Internal DNS¶
freckle.services, freckle.systems, freckle.media, freckle.network and
the two site reverse zones are served whole by BIND9 inside the clusters.
Internal-only names in them only resolve from inside the network; public DNS
at Cloudflare never learns them.
The three freckle.* zones are split-horizon: Cloudflare keeps serving the
public view, BIND serves a superset internally, and the gateways forward the
whole zones to BIND. Anything published publicly by something other than
our two clusters' external-dns (tofu-managed records, mail records, another
cluster, hand edits) has to be mirrored in tofu or it vanishes from the
inside view — internal-dns-records.tf in the private repo is that list.
Zones¶
| Zone | Purpose | Written by |
|---|---|---|
freckle.services / freckle.systems / freckle.media / freckle.tools |
whole zones, superset of public: every record either cluster publishes plus the mirrored non-cluster ones | external-dns (both), tofu |
freckle.network |
network gear, OOB, anything not a Kubernetes Service; <host>.den1 and <host>.btx1 |
tofu |
140.10.in-addr.arpa |
den1 reverse (10.140.0.0/16) | tofu |
189.10.in-addr.arpa |
btx1 reverse (10.189.0.0/16) | tofu |
Every zone is dynamic: the skeleton (SOA + NS) is written once by the primary's init container and every record after that arrives as an RFC2136 update. Never edit a zone file on the PVC by hand.
Components¶
external-dns (fairy) ──RFC2136──┐
external-dns (atlantis) ─RFC2136┤
tofu (hashicorp/dns) ───RFC2136─┤
▼
bind9-primary (atlantis, hidden, 10.140.250.54)
│ AXFR + NOTIFY (TSIG freckle-dns-xfr)
┌───────────┴───────────┐
▼ ▼
bind9-secondary (fairy) bind9-secondary (atlantis)
10.189.250.53 10.140.250.53
▲ ▲
UDM forward-domain den1 gateway forward-domain
(internal zones only) (internal zones only)
- bind9-primary —
kubernetes/apps/networking/bind9/primary, one replica on atlantis with aceph-blockPVC for the zone journals. Never handed to clients; it takes updates with thefreckle-dns-updateTSIG key and NOTIFYs both secondaries. Losing it freezes updates, never resolution. - bind9-secondary —
kubernetes/apps/networking/bind9/secondary, two replicas per cluster behind a pinned internal LoadBalancer IP. Authoritative for the hosted zones, plus just enough recursion (forward-only, DNS-over-TLS to Cloudflare, internal clients only) to complete CNAME chains that leave our zones — the UDM does not chase those, and stub resolvers cannot; zones on emptyDir because a PVC would make DNS depend on Ceph during a cold start, and a fresh pod simply re-pulls from the primary. SOA expiry is 14 days so a secondary keeps serving long after the primary goes away. - external-dns-internal —
kubernetes/apps/networking/external-dns/internal, one per cluster,rfc2136provider, the threefreckle.*zones. No gateway label filter: it mirrors every route on every gateway (tailnet, tunnel, internal), so the internal zone is a superset of the public one. SeparatetxtOwnerId(<cluster>-internal) from the Cloudflare instances. - Static records —
tofu/platform/internal-dns-records.tfin the private repo: one host map that generates thefreckle.networkA/AAAA records and the matching PTRs, internal twins of every tofu-managed Cloudflare record (cluster API names, tunnel gateways), and the hand-kept mirrors (mail records, the R2-backedk8s-oidc/k8s-schemasnames, another cluster's names).
Both TSIG keys and the ESO grants for the dns namespace come from
tofu/platform/internal-dns.tf (private repo) via GSM.
Who resolves what¶
Site gateways are the only resolvers clients see. Each forwards just the seven hosted zones to its site's secondary and resolves everything else itself, so the gateway's own DNS features (filtering, logging, encrypted upstream) stay in the loop. BIND's only upstream traffic is DoT to Cloudflare for CNAME targets outside our zones.
| Client | Resolver | Internal zones go to |
|---|---|---|
| fairy LAN + Talos nodes | UDM (DHCP; nodes use the compute-VLAN gateway 10.189.3.1) | UniFi Forward Domain DNS policies → 10.189.250.53 (network/btx01/udm-dns in the private repo, Integration API) |
| atlantis Talos nodes | the secondaries directly: 10.140.250.53 then 10.189.250.53 | authoritative locally; the rest forwarded over DoT |
| den1 devices | nothing yet — border1 runs no resolver; the UniFi gateway on order will get the same Forward Domain policies as fairy | — |
| Tailnet | MagicDNS split DNS for the internal domains | both secondaries |
| Pods | kube-dns → node resolvers | same path as the node |
Guest and IoT VLANs can resolve the names; the firewall is what denies them access.
Failure modes¶
- Whole cluster reboots. Nodes resolve via the gateway, whose default upstream is unaffected, so kubelet and image pulls never wait on anything internal. That cluster's own internal names are unavailable until its secondary is back (seconds after the CNI), which is roughly when the services behind them return anyway.
- Primary down. Secondaries serve their copies for up to 14 days; external-dns and tofu updates queue until it returns.
- Tunnel down. Each site's secondary keeps answering from its last transfer. The fairy secondary cannot refresh until the tunnel returns.
Operating¶
# Is a zone loaded and current on a secondary?
dig +short SOA fairy.freckle.services @10.189.250.53
dig +short SOA fairy.freckle.services @10.140.250.53 # serials should match
# What did external-dns write? (TSIG-authenticated AXFR from the primary)
kubectl -n dns exec deploy/bind9-primary -- \
dig +short AXFR fairy.freckle.services @127.0.0.1 -p 5353 -k /secrets/tsig-keys.conf
# Primary / secondary logs show every update, transfer and NOTIFY
kubectl -n dns logs deploy/bind9-primary
Adding a static host: edit internal_hosts in internal-dns-records.tf,
tofu apply from a machine on the tailnet (the provider talks RFC2136 to
the primary at 10.140.250.54).