Kubernetes
What happens if DNS stops resolving inside the cluster?
Short answer
Almost everything breaks, in a way that looks like several unrelated failures at once — because service discovery is a dependency of nearly every call, and DNS caching means it breaks unevenly.
Starting state
A working cluster where CoreDNS becomes unavailable, is overloaded, or starts returning errors. Nothing else has changed.
The sequence
Colour marks which component acts. Steps sharing a timestamp happen concurrently — nothing here waits for anything else unless it says so.
- your app
- kubelet
- network
- you
- t+0your app
Nothing happens. Any connection already established keeps working, because it resolved its name before the failure and is holding an open socket.
What you would see Long-lived clients — database pools, gRPC channels, message consumers — appear completely healthy.
- before the failurekubelet
Every pod's `/etc/resolv.conf` was written by the kubelet at start-up, pointing at the cluster DNS service and listing several search domains. That file does not change when DNS breaks, so every container keeps confidently asking a resolver that is no longer answering.
- as caches expireyour app
New lookups start failing. Because caches expire at different times in different processes, services begin failing in an order that has nothing to do with the dependency graph.
What you would see The confusing signature: errors appearing across unrelated services over several minutes, in no sensible pattern.
- on each failed lookupnetwork
The resolver works through the search domains in `/etc/resolv.conf`. A single failed lookup becomes several queries, each waiting for a timeout, so a name that used to resolve in a millisecond now takes seconds to fail.
What you would see Latency rising sharply *before* errors appear. Requests are not erroring yet; they are waiting on resolution.
- as retries pile upyour app
Retry logic makes it worse. Every retry is another lookup, so the load on an already-struggling resolver increases exactly when it needs relief.
What you would see Query volume at the DNS layer climbing while success rate falls — the classic self-reinforcing failure.
- throughoutyou
Your retry and timeout configuration decides how bad this gets. Aggressive retries with no backoff turn a resolver under strain into a resolver being attacked by your own fleet; long timeouts turn a fast failure into exhausted connection pools.
- at the edgesyour app
Health endpoints that do not resolve anything keep returning healthy, so pods stay in Service endpoints and keep receiving traffic they cannot serve.
What you would see Everything reports healthy while nothing works — one of the most disorienting states to walk into.
What you actually control
Separated from what happens regardless, because conflating the two is how people end up tuning the wrong thing.
- Where health checks look
- A readiness probe that never resolves a name cannot detect this, and a liveness probe that *does* resolve one will restart every pod in the cluster during a DNS outage. Neither is obviously right, which is worth deciding deliberately rather than by accident.
- Connection reuse
- Long-lived pooled connections are the reason parts of the system survive. They are also why the failure is uneven and hard to read.
- `ndots` and search domains
- The default cluster config appends several search domains, so an external name can take five queries to resolve. Under DNS pressure this multiplies your query volume; setting `ndots` appropriately for fully-qualified names is a genuine mitigation.
- A node-local DNS cache
- Reduces both latency and query volume against the central resolver, and keeps working briefly when the central one does not. This is the structural fix rather than a workaround.
Where this goes wrong
The investigation starts from the application errors
Hours lost. Five services are failing with five different error messages — connection timeouts, pool exhaustion, upstream errors — and none of them says "DNS". The common cause is only visible if you think to look at resolution.
Liveness probes resolve a hostname
Every pod fails its liveness probe and restarts, simultaneously, during the outage. A degradation becomes a total outage, caused by the health checking rather than by the fault.
Retries have no backoff
The retry storm keeps the resolver saturated after the original trigger has passed, so recovery does not happen when the cause is fixed.
The part worth remembering
DNS is worth thinking about differently from other dependencies, because it is a dependency of the act of *finding* dependencies. When it fails, the failure does not appear at DNS — it appears everywhere else, wearing whatever error message each caller happens to produce.
The uneven timing is the diagnostic signature. A cause that takes out several unrelated services over several minutes, in no order that matches your architecture, is almost always something shared and cached: DNS, certificates, a token, a config source.
The practical habit is to check resolution early when a failure is broad and incoherent, rather than late after every application-level theory has been eliminated. It costs one command and it is the difference between twenty minutes and an afternoon.
What comes next
Did this get you to an answer?
No text box on purpose — please do not paste production logs anywhere