Kubernetes · what happens if

What happens if DNS stops resolving inside the cluster?

Short answer

Almost everything breaks, in a way that looks like several unrelated failures at once — because service discovery is a dependency of nearly every call, and DNS caching means it breaks unevenly.

First published 4 Sept 2026

On this page · 8 sections

Starting state

A working cluster where CoreDNS becomes unavailable, is overloaded, or starts returning errors. Nothing else has changed.

Words used on this page

CoreDNS: The cluster DNS server that kubeadm installs and the Kubernetes DNS guides assume. Its pods carry the label k8s-app=kube-dns and its Service is called kube-dns, for compatibility with the older kube-dns.
resolv.conf: The file in each container that tells the C library which DNS server to ask, which search domains to try and with which options. The kubelet sets it up for every pod.
search domain: A suffix the resolver adds to a name before asking. In a pod in namespace test, data is tried as data.test.svc.cluster.local.
ndots: How many dots a name needs before the resolver first tries it as written. Kubernetes sets 5 for pods that use cluster DNS; the C library's own default is 1.
dnsPolicy: A pod's DNS mode. ClusterFirst, used when nothing is set, sends lookups to cluster DNS; Default uses the node's settings instead.
TTL: Time to live: how many seconds an answer may be cached before it must be asked for again.
NodeLocal DNSCache: An optional DaemonSet that runs a DNS cache on every node. Lookups it cannot answer from cache still go to CoreDNS.
liveness probe: A kubelet check that restarts the container when it fails. A failing readiness probe, by contrast, only takes the pod out of its Services.
retry storm: Clients retrying failed requests so quickly that the retries themselves keep the struggling service overloaded.

The sequence

Colour marks which component acts. Steps sharing a timestamp happen concurrently — nothing here waits for anything else unless it says so.

your app
kubelet
network
you

t+0your app
Nothing happens. Any connection already established keeps working, because it resolved its name before the failure and is holding an open socket.
What you would see Long-lived clients — database pools, gRPC channels, message consumers — appear completely healthy.
before the failurekubelet
Every pod using the default ClusterFirst DNS policy got an /etc/resolv.conf from the kubelet when it started, pointing at the cluster DNS service and listing several search domains. That file does not change when DNS breaks, so every container keeps confidently asking a resolver that is no longer answering.
as caches expireyour app
New lookups start failing. Cluster names are served with short TTLs, 5 seconds by default in CoreDNS and 30 in kubeadm's configuration, so cached answers run out within seconds. What spreads the failure out is when each process next needs a lookup, for a new connection or a reconnect, and any cache inside the application, so services begin failing in an order that has nothing to do with the dependency graph.
What you would see The confusing signature: errors appearing across unrelated services over several minutes, in no sensible pattern.
on each failed lookupnetwork
The resolver works through the search domains in /etc/resolv.conf. A query that gets no answer waits out the resolver's timeout, 5 seconds by default in glibc, and is sent twice before the resolver gives up, and a name with fewer than five dots can be tried with several search domains. So a name that used to resolve in a millisecond now takes seconds to fail.
What you would see Latency rising sharply before errors appear. Requests are not erroring yet; they are waiting on resolution.
as retries pile upyour app
Retry logic makes it worse. Every retry is another lookup, so the load on an already-struggling resolver increases exactly when it needs relief.
What you would see Query volume at the DNS layer climbing while success rate falls: a self-reinforcing failure, of the kind Google's SRE book describes for retries.
throughoutyou
Your retry and timeout configuration decides how bad this gets. Aggressive retries with no backoff turn a resolver under strain into a resolver being attacked by your own fleet; long timeouts turn a fast failure into exhausted connection pools.
at the edgesyour app
Health endpoints that do not resolve anything keep returning healthy, so pods stay in Service endpoints and keep receiving traffic they cannot serve.
What you would see Everything reports healthy while nothing works — one of the most disorienting states to walk into.

What you actually control

Separated from what happens regardless, because conflating the two is how people end up tuning the wrong thing.

Where health checks look: A readiness probe that never resolves a name cannot detect this, and a liveness probe that does resolve one will restart every pod that runs it during a DNS outage. Kubernetes' own guidance is to keep liveness probes to the app itself and to check dependencies, if at all, in the readiness probe. Even that is a trade-off: a readiness check that needs DNS takes those pods out of their Services during the outage, so decide it deliberately.
Connection reuse: Long-lived pooled connections are the reason parts of the system survive. They are also why the failure is uneven and hard to read.
ndots and search domains: The default cluster config lists three cluster search domains plus any the node adds, with ndots:5, so an external name with fewer than five dots, such as api.example.com, is first tried with every search domain appended. Each try is a separate query, and asking for both IPv4 and IPv6 addresses doubles them. Under DNS pressure this multiplies your query volume; setting ndots appropriately for fully-qualified names is a genuine mitigation.
A node-local DNS cache: Reduces both latency and query volume against the central resolver, and keeps answering for names it has already cached while those answers are still valid. It does not replace a healthy CoreDNS: lookups it has not cached still go to CoreDNS, and the cache on each node is one more component that can fail.

Try it yourself

Before you start

kubectl access that can run kubectl exec in a test namespace and read pods, Services and EndpointSlices in kube-system, where CoreDNS runs.
A pod with DNS tools to test from. The Kubernetes DNS debugging guide creates one called dnsutils with kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml; your application images may have no nslookup at all.
Your cluster's DNS configuration: the CoreDNS Corefile (kubectl -n kube-system get configmap coredns -o yaml) and whether NodeLocal DNSCache is installed. Cache lifetimes depend on it: CoreDNS gives cluster names a 5-second TTL unless configured otherwise, and kubeadm 1.37.1's Corefile sets 30.
Which resolver your images use. The search, ndots and timeout behaviour on this page is glibc's, as its resolv.conf(5) manual describes; the Kubernetes guide warns that images on Alpine 3.17 or earlier can fail on DNS answers over 512 bytes.
A test cluster if you want to watch the failure happen, for example by scaling CoreDNS to zero there. In production, observe; do not reproduce.

Checking DNS from inside a pod

Quoted verbatim from the Kubernetes documentation page Debugging DNS Resolution; not reproduced, because no cluster was available. The guide runs these checks from its test pod dnsutils in the default namespace, and each output is its example for that step: the expected resolv.conf, a failing lookup, and healthy CoreDNS pods and endpoints. The ... lines are in the documentation's own output.

Command

kubectl exec -ti dnsutils -- cat /etc/resolv.conf
kubectl exec -i -t dnsutils -- nslookup kubernetes.default
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
kubectl get endpointslice -l kubernetes.io/service-name=kube-dns --namespace=kube-system

Output · as shown in Kubernetes — Debugging DNS Resolution

$ kubectl exec -ti dnsutils -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internal
nameserver 10.0.0.10
options ndots:5

$ kubectl exec -i -t dnsutils -- nslookup kubernetes.default
Server:    10.0.0.10
Address 1: 10.0.0.10

nslookup: can't resolve 'kubernetes.default'

$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                       READY     STATUS    RESTARTS   AGE
...
coredns-7b96bf9f76-5hsxb   1/1       Running   0           1h
coredns-7b96bf9f76-mvmmt   1/1       Running   0           1h
...

$ kubectl get endpointslice -l kubernetes.io/service-name=kube-dns --namespace=kube-system
NAME             ADDRESSTYPE   PORTS   ENDPOINTS                  AGE
coredns-zxoja    IPv4          53      10.180.3.17,10.180.3.17    1h

How to read it

resolv.conf is what the kubelet set up for this pod: the cluster DNS Service's address as the only nameserver, three cluster search domains followed by the node's own (google.internal and c.gce_project_id.internal here; the guide notes these vary by provider), and ndots:5.
With ndots:5, any name with fewer than five dots, such as kubernetes.default or api.example.com, is first tried with search domains appended. In this file that is up to five extra names per lookup, which is where the extra queries on this page come from.
nslookup: can't resolve 'kubernetes.default' after Server: 10.0.0.10 means the pod asked the right address and got no usable answer. The guide says errors like this point to a problem with the CoreDNS add-on or its Services.
1/1 Running only says the CoreDNS containers are up. The EndpointSlice line says the kube-dns Service has addresses to send queries to; if it lists none, the guide sends you to debugging Services.
If pods and endpoints look healthy and lookups still fail, the same guide has you read CoreDNS's logs (kubectl logs --namespace=kube-system -l k8s-app=kube-dns) and, if needed, turn on its log plugin to see whether queries arrive at all.

Where this goes wrong

The investigation starts from the application errors
Hours lost. Five services are failing with five different error messages — connection timeouts, pool exhaustion, upstream errors — and none of them says "DNS". The common cause is only visible if you think to look at resolution.
Liveness probes resolve a hostname
Every pod fails its liveness probe and restarts, simultaneously, during the outage. A degradation becomes a total outage, caused by the health checking rather than by the fault.
Retries have no backoff
The retry storm keeps the resolver saturated after the original trigger has passed, so recovery does not happen when the cause is fixed.

If it goes wrong

nslookup: can't resolve 'kubernetes.default' from a test pod: Work through the guide's checks: are the CoreDNS pods running (kubectl get pods -n kube-system -l k8s-app=kube-dns), does the kube-dns Service have endpoints (kubectl get endpointslice -n kube-system -l kubernetes.io/service-name=kube-dns), and what do CoreDNS's logs say.
CoreDNS logs show SERVFAIL for cluster names, like the guide's sample "A IN serverproxy.contoso.net.cluster.local. udp 52 false 512" SERVFAIL qr,aa,rd: CoreDNS may lack permission to read Services and EndpointSlices. Compare kubectl describe clusterrole system:coredns -n kube-system with the guide's expected rules (list and watch on endpoints, namespaces, pods, services and endpointslices) and add only what is missing.
kubernetes.default resolves, but a short Service name from another namespace does not: A name without a namespace is looked up only in the pod's own namespace. Use <service>.<namespace>, or the full <service>.<namespace>.svc.cluster.local.
With NodeLocal DNSCache installed, lookups fail briefly and repeatedly on one node: Check whether that node's node-local-dns pod is being OOMKilled. The documentation warns that a killed cache leaves its packet filtering rules behind, so queries go to an unhealthy pod until it restarts. Measure its peak memory and set the limit above it.
A pod with hostNetwork: true cannot resolve Service names: With hostNetwork, the default ClusterFirst policy falls back to the node's DNS settings. Set dnsPolicy: ClusterFirstWithHostNet on that pod.

The part worth remembering

DNS is worth thinking about differently from other dependencies, because it is a dependency of the act of finding dependencies. When it fails, the failure does not appear at DNS — it appears everywhere else, wearing whatever error message each caller happens to produce.

The uneven timing is the diagnostic signature. A cause that takes out several unrelated services over several minutes, in no order that matches your architecture, points to something shared that every affected service depends on, such as DNS, certificates, a token or a config source.

The practical habit is to check resolution early when a failure is broad and incoherent, rather than late after every application-level theory has been eliminated. It costs one command and it is the difference between twenty minutes and an afternoon.

Security considerations

Lowering ndots to cut queries changes where short names go first. With ndots:1, a name with a dot such as data.prod is tried as written before the search list, and ClusterFirst forwards names outside the cluster domain to the upstream resolver, so internal Service names are sent outside the cluster. The resolv.conf manual itself points to ndots as a way to avoid man-in-the-middle attacks and unnecessary traffic. Use full names ending in a dot for external dependencies instead.
The search list that turns data.prod into data.prod.svc.cluster.local also tries example.com as example.com.svc.cluster.local, which is a Service named example in a namespace named com. Whoever can create namespaces and Services can answer for such names, so restrict namespace creation, and use a trailing dot (example.com.) for external names, which the resolver looks up with no further processing.
Turning on CoreDNS's log plugin to debug records every query name and the client address that asked, which shows what each workload talks to. Remove it when you are done, and treat the logs as sensitive while they exist.
When you fix SERVFAIL by editing the system:coredns ClusterRole, add only the list and watch rules the guide lists. CoreDNS can read Services, EndpointSlices, pods and namespaces across the whole cluster already; widening the role widens what a compromised CoreDNS can read or change.

Sources

The behaviour this page describes, as documented by the projects themselves.

Terms used here (1)plain definitions

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Service: A Kubernetes abstraction that provides stable networking for a set of Pods. Services assign a fixed IP address and DNS name to a logical group of Pods, enabling reliable communication even as individual Pods are created and destroyed.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it