TLS certificates for internal services done right TLS certificates for internal services done right<br>2026-07-09 Jakub Kołodziejczak dnstlsacmeletsencryptnetbirdnginxacme.sh<br>Title is a bit clickbait-y — YMMV, but let me explain why I think “this is the<br>way”. Let’s start with a simple example — we have a server which hosts bunch<br>of HTTP services. Some of those services are external, others internal. In<br>order to reach the internal ones you need to be connected to the VPN.
For the sake of simplicity let’s consider we have two choices:
We use top-level domain restricted by ICANN for private<br>use<br>— e.g. .internal.
We use a public apex domain that we own — e.g. tuxnet.dev
Grafana would be our example internal app. Let’s assume that it’s reachable on<br>10.0.1.10 internal IP address and our VPN has DNS resolver features.
What’s wrong with .internal then?
We could simply create a DNS record of type “A” that resolves to 10.0.1.10<br>internal IP address — e.g. grafana.tuxnet.internal. But then if we don’t<br>want it to be a plain text HTTP service we would need to create a self-signed<br>certificate.
The good part is that there are plenty tutorials that show you how to do this<br>(e.g. this<br>one).<br>The ugly part is that suddenly every HTTP client should be configured to trust<br>this self-signed certificate. Alternatively we could just tell our users to<br>ignore the TLS certificate errors ¯\_(ツ)_/¯.
How to do it “the right way”
Meet the “split-horizon DNS” configuration. For public DNS resolvers our<br>grafana.tuxnet.dev domain resolves to a public IP and for clients connected<br>to the VPN this domain resolves to an internal IP.
The good part is that since it resolves to a public IP we can use some public<br>CA like Let’s Encrypt or ZeroSSL. The ugly part is that we still need some WAF<br>rejecting traffic that does not originate from the VPN.
Considering pros and cons of both solutions I think it’s much easier to set up<br>a WAF in one place (on our server) than to install self-signed certificate on<br>every machine that joins our internal network (… or advising our users to<br>suppress the TLS errors).
”Talk is cheap show me the code”
We have theory now, time to get our hands dirty. We will need:
A VPN with DNS resolver features — I choose NetBird.
ACME client for issuing a certificate — I choose acme.sh.
A reverse proxy with WAF features in front of our grafana — I choose nginx.
If you’ve read my other blog posts you probably noticed that I am a fan of<br>NetBird (sorry Tailscale). Thanks to Custom<br>Zones feature, NetBird does<br>all the heavy lifting required for “split-horizon DNS” for us. By using user<br>groups or peer<br>groups we can<br>selectively apply Custom Zones so that server uses public DNS resolver for<br>grafana.tuxnet.dev.
Why exclude server from that custom zone? It’s not required unless we want to<br>use http-01 challange. Using<br>other methods is also possible but for the sake of this blog post I choose<br>http-01. Ok, let’s get the certificate now with:
acme.sh --issue -d grafana.tuxnet.dev --server letsencrypt --standalone<br>acme.sh is quite flexible and has a lot of<br>modes.<br>The cool part about the standalone mode (enabled with --standalone flag) is<br>that our nginx doesn’t have to listen on port 80 at all. This port becomes<br>“active” only when acme.sh gets the certificate.
Ok now we can put nginx into action. This is our config:
upstream grafana {<br>server localhost:3000;
map $http_upgrade $connection_upgrade {<br>default upgrade;<br>'' close;
server {<br>listen our-server.netbird.cloud:443 ssl;<br>server_name grafana.tuxnet.dev;<br>http2 on;
ssl_certificate /etc/ssl/certs/grafana.tuxnet.dev.crt;<br>ssl_certificate_key /etc/ssl/private/grafana.tuxnet.dev.key;
access_log /var/log/nginx/grafana.tuxnet.dev.access.log main;<br>error_log /var/log/nginx/grafana.tuxnet.dev.error.log warn;
location / {<br>proxy_pass http://grafana;<br>proxy_set_header Host $host;
# Proxy Grafana Live WebSocket connections.<br>location /api/live/ {<br>proxy_pass http://grafana;<br>proxy_http_version 1.1;<br>proxy_set_header Upgrade $http_upgrade;<br>proxy_set_header Connection $connection_upgrade;<br>proxy_set_header Host $host;<br>There is one key setting in this configuration worth explaining — listen our-server.netbird.cloud:443 ssl;. We are binding to the VPN network interface<br>of our server. Instead of our-server.netbird.cloud this could be a VPN IP<br>address. In practice this will reject any traffic to grafana.tuxnet.dev that<br>originates from public internet — this is our Web Access Firewall.
Security is about layers (just like onions and ogres). Our first layer is<br>split-horizon DNS but if for whatever reason it fails or is cleverly bypassed<br>we have a 2nd layer - WAF - that should hold the line.
Last but not least — certificate auto renewal. acme.sh has a out-of-the box<br>--cron flag. Now we need a daily cron job that will call
acme.sh --cron<br>acme.sh automatically chooses which certificates should be renewed. All we<br>need to do is to make sure that cron job copies new certificates into the<br>location defined in...