It's always DNS

There's a famous haiku among system administrators:

It's not DNS. There's no way it's DNS. It was DNS.

It's funny because it's true. Some of the biggest outages of recent years, at cloud providers and large websites, came down to DNS problems. And in daily work, a huge share of "the app can't connect" mysteries end with someone saying "oh, it's resolving to the old address".

Why does DNS cause so much trouble, and what can we do about it?

Why DNS is everywhere

Almost every network connection starts with a DNS lookup. Your browser, your app connecting to its database, a pod calling another service, a CI job pulling an image. If DNS is slow, everything is slow. If it's wrong, everything goes to the wrong place. If it's down, nothing works, even if every server is healthy.

And DNS is cached at many levels: in applications, in operating systems, in local resolvers, at your ISP. Which means problems can be inconsistent, delayed and very hard to reproduce.

Records and TTLs

The records you'll work with most:

• A / AAAA: name to IPv4 / IPv6 address.
• CNAME: alias to another name. (Can't be used at the zone apex like example.com; providers offer ALIAS or ANAME records for that.)
• TXT: arbitrary text, used for domain verification, SPF, DKIM and ACME challenges.
• MX: mail servers.
• SRV: service discovery with ports.
• NS / SOA: which servers are authoritative for a zone.

TTL (time to live) tells resolvers how long to cache an answer. It's the most important lever you have:

• Low TTL (60 to 300 seconds): changes take effect quickly. Good for records you might need to change in an emergency, like failover endpoints.
• High TTL (hours): less load on DNS, more resilience if your DNS provider has problems. Good for stable records.

A classic mistake: planning a migration, changing the record, and discovering the old TTL was 24 hours. Lower the TTL a day or two before a planned change, then raise it again afterwards.

DNS changes don't happen when you make them. They happen when the last cache in the world forgets the old answer.

DNS inside Kubernetes

Kubernetes has its own DNS, usually CoreDNS. Every service gets a name like orders.shop.svc.cluster.local.

A few things that trip people up:

• Short names work only in the same namespace. orders works from the shop namespace; from elsewhere, use orders.shop or the full name.
• The ndots:5 setting. By default, pods treat names with fewer than five dots as relative and try several search domains first. A lookup for api.example.com can generate several failed queries before the real one. For external names, a trailing dot (api.example.com.) or tuning ndots in the pod's dnsConfig reduces that.
• CoreDNS can become a bottleneck in busy clusters. Scale it, and consider NodeLocal DNSCache, which runs a cache on every node.
• Conntrack and UDP issues on some setups cause occasional slow lookups (classic five-second delays). NodeLocal DNSCache helps with those too.

ExternalDNS

Managing public DNS records by hand for every service is error-prone. ExternalDNS watches Kubernetes resources (Services, Ingresses, Gateway API routes) and creates DNS records automatically in Route 53, Cloud DNS, Azure DNS, Cloudflare and many others.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
  namespace: shop
spec:
  parentRefs:
  - name: web
    namespace: infra
  hostnames: ["shop.example.com"]   # ExternalDNS creates this record
  rules:
  - backendRefs:
    - name: shop-api
      port: 8080

Configure it with a TXT registry so it only touches records it owns, and a domain filter so it can't modify zones it shouldn't.

DNS-based failover

DNS can route traffic away from failures:

• Health-checked failover records (Route 53, Cloudflare, Azure Traffic Manager, and others): if the primary endpoint fails health checks, DNS starts returning the secondary.
• Latency-based or geo routing: send users to the nearest region.
• Weighted records: gradually shift traffic between old and new environments during migrations.

Remember the TTL: failover can only be as fast as caches expire. And some clients cache DNS much longer than the TTL says (older Java versions were famous for this).

Outages I've seen (and you probably will too)

• Expired domain or a lapsed registrar payment.
• Wrong TTL during migration: half the users on the old system for a day.
• Deleted or overwritten records by a tool that thought it owned them.
• Internal resolvers down, so every service fails at once, while the servers themselves are healthy.
• Split-horizon confusion: the name resolves differently inside and outside the network, and someone tested from the wrong side.
• DNSSEC misconfiguration making a domain unresolvable for validating resolvers.

My debugging routine

dig shop.example.com +short                 # what does my resolver say?
dig @1.1.1.1 shop.example.com +short        # what does a public resolver say?
dig shop.example.com +trace                 # follow it from the root
dig shop.example.com NS                     # who is authoritative?
kubectl run -it --rm dnstest --image=busybox:1.36 -- nslookup orders.shop   # from inside the cluster

Compare answers from different places. If they differ, it's caching or split-horizon. If the authoritative answer is wrong, it's the record itself.

So, how do you make DNS break less?

Manage records as code or with ExternalDNS, plan TTLs deliberately, monitor your domains and certificates, cache DNS close to workloads, and always test from where the client actually is.

My take: DNS is simple until it isn't. Treat it like critical infrastructure, because it's the first thing every request touches.

— Desly