Skip to main content
DevOps Insightsguides & answers

Kubernetes · troubleshooting

DNS lookup fails inside a pod

A pod cannot turn a name into an address, so connections by Service or host name fail. The cluster's DNS has a short, fixed chain — CoreDNS pods, the kube-dns Service, its endpoints, and the pod's own resolv.conf — and each link has a one-line check. Most failures are one namespace assumption or one unready pod, not a broken network.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 9 sections

Before you start

  • A pod in the affected namespace to test from, with nslookup in its image, or permission to create the dnsutils test pod from Debugging DNS Resolution, which uses the image registry.k8s.io/e2e-test-images/agnhost:2.39.
  • Permission to kubectl exec into pods in that namespace.
  • Read access in kube-system to pods, pod logs, services and endpointslices, where CoreDNS runs.
  • Your cluster domain. It is cluster.local unless the kubelet's --cluster-domain flag says otherwise; use yours wherever this page says cluster.local.
  • Cluster-admin rights, or the platform owner's agreement, before you edit the coredns ConfigMap or the system:coredns ClusterRole. Both affect name lookups for every pod in the cluster.

Run this first

step 1 of 5
kubectl exec -i -t <pod> -- nslookup kubernetes.default

Look for An address. nslookup: can't resolve means the chain is broken somewhere below

The documentation's own first test. It uses a name that always exists, so a failure is about DNS rather than about your service.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Does nslookup kubernetes.default work from inside the pod?

kubectl exec -i -t <pod> -- nslookup kubernetes.default

Cause space

7 of 7 still possible

  • The CoreDNS pods are not running, or not readyCommon
  • The kube-dns Service has no endpointsCommon
  • The name is being resolved from the wrong namespaceCommon
  • The pod's resolv.conf is not what the cluster expectsOccasional
  • CoreDNS cannot read the API objects it answers fromRare
  • Short external names go through every search domain firstCommon
  • Queries never reach CoreDNSOccasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    kubectl exec -i -t <pod> -- nslookup kubernetes.default

    The documentation's own first test. It uses a name that always exists, so a failure is about DNS rather than about your service.

    Look for An address. nslookup: can't resolve means the chain is broken somewhere below

  2. Step 2

    kubectl exec -ti <pod> -- cat /etc/resolv.conf

    Establishes which resolver the pod is actually using before anything about the cluster is investigated.

    Look for nameserver, a search list starting with the pod's namespace, and options ndots:5

  3. Step 3

    kubectl get pods --namespace=kube-system -l k8s-app=kube-dns

    Separates "nothing is serving DNS" from "DNS is serving and the answer is wrong", which need different fixes.

    Look for Running and 1/1 Ready

  4. Step 4

    kubectl get endpointslice -l kubernetes.io/service-name=kube-dns --namespace=kube-system

    A Service with a ClusterIP and no endpoints looks healthy in kubectl get svc and answers nothing.

    Look for Addresses on port 53

  5. Step 5

    kubectl logs --namespace=kube-system -l k8s-app=kube-dns --tail=50

    Distinguishes a configuration error, a permission error (SERVFAIL) and a name that genuinely does not exist.

    Look for Corefile errors, SERVFAIL lines, or nothing at all — which means the queries are not arriving

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

The CoreDNS pods are not running, or not ready

Common

Nothing answers. CoreDNS is an ordinary Deployment in kube-system and fails for ordinary reasons — evicted under memory pressure, unschedulable after a node drain, or crash-looping on a bad Corefile after somebody edited the ConfigMap. Every lookup in the cluster then times out at once, which makes this look like a network outage rather than one Deployment being down.

Confirm

kubectl get pods --namespace=kube-system -l k8s-app=kube-dns

Pods that are Running and 1/1 Ready. A pod in any other state, including 0/1 Ready, gets no queries; if no pod is Ready, nothing is serving

Fix

  • Read the CoreDNS logs before restarting anything — a Corefile syntax error is named there and survives a restart.
  • If the pods are Pending, this is a scheduling problem and the pod's events explain it.
  • Restart the Deployment only once you know why it stopped; a restart that fixes it without explanation will recur.
kubectl logs --namespace=kube-system -l k8s-app=kube-dns --tail=50

A healthy start logs the CoreDNS version and a plugin/reload line. Errors appear here, not in the pod status.

The kube-dns Service has no endpoints

Common

The Service object exists and has a ClusterIP, so kubectl get svc looks healthy, but nothing is behind it. Queries go to the Service IP and nothing answers them. In iptables mode, kube-proxy installs a REJECT rule for a Service with no endpoints, so the client is refused rather than left waiting. This is the same failure mode as any other endpoint-less Service: the selector matches no ready pod. It follows from the previous cause but is worth checking separately, because the Service can also be present with the pods healthy if a label was changed.

Confirm

kubectl get endpointslice -l kubernetes.io/service-name=kube-dns --namespace=kube-system

At least one slice with addresses on port 53. ENDPOINTS showing <unset>, or addresses whose conditions.ready is false in -o yaml, is the fault

Fix

  • Confirm the CoreDNS pods are Ready: an unready pod stays in the EndpointSlice with ready: false and gets no Service traffic. Check with kubectl get endpointslice -n kube-system -l kubernetes.io/service-name=kube-dns -o yaml.
  • Check that the Service's selector still matches the pods' labels. A hand-edited Deployment is the usual way these diverge.

The name is being resolved from the wrong namespace

Common

A bare myservice is completed from the search list, which starts with the calling pod's own namespace. Moving a workload to another namespace, or calling a service that lives elsewhere, silently changes what that name means. Nothing is misconfigured; the name is being asked in a context where it does not exist.

Confirm

kubectl exec -i -t <pod> -- nslookup <service>.<namespace>.svc.cluster.local

The fully qualified name resolving while the short name does not. That difference is the whole answer

Fix

  • Use a namespace-qualified name such as <service>.<namespace>, or the full <service>.<namespace>.svc.cluster.local, for anything crossing a namespace boundary. cluster.local is only the default cluster domain.
  • Keep short names for same-namespace calls, where they are correct and readable.
  • If the service genuinely should be local, check it exists in this namespace at all rather than assuming DNS is at fault.

The pod's resolv.conf is not what the cluster expects

Occasional

A pod can be given a different resolver entirely — dnsPolicy: Default inherits the node's configuration rather than the cluster's, hostNetwork: true changes it too, and dnsPolicy: None with a dnsConfig replaces it outright (under the other policies a dnsConfig is merged into the generated settings). The pod then asks a nameserver that knows nothing about cluster.local, so in-cluster names fail while external ones work perfectly. That split is the signature.

Confirm

kubectl exec -ti <pod> -- cat /etc/resolv.conf

A cluster nameserver, a search list beginning with <namespace>.svc.cluster.local, and options ndots:5

Fix

  • Unless there is a reason, leave dnsPolicy at its default so the pod uses cluster DNS.
  • A pod on the host network needs dnsPolicy: ClusterFirstWithHostNet to keep cluster resolution.
  • Where a custom dnsConfig is required, keep the cluster search entries rather than replacing them.

CoreDNS cannot read the API objects it answers from

Rare

CoreDNS answers service queries by listing Services and EndpointSlices through the API. If its ClusterRole has been narrowed — commonly after a hand-written RBAC change or an older chart — it can start and serve while failing the queries that need those objects. The documentation gives the signature: a SERVFAIL in the CoreDNS log rather than a clean NXDOMAIN.

Confirm

kubectl describe clusterrole system:coredns

list and watch on services and endpointslices. Missing either produces SERVFAIL on service names

Fix

  • Restore the permissions on services and endpointslices in the CoreDNS ClusterRole.
  • Distinguish this from a genuine NXDOMAIN first: SERVFAIL means the server failed to answer, not that the name does not exist.

Short external names go through every search domain first

Common

With the default ndots:5, a name with fewer than five dots is tried against each search domain before it is sent as written, so every lookup of an external host costs several failed queries first. Nothing is broken; it is slow.

Confirm

kubectl exec <pod> -- cat /etc/resolv.conf

options ndots:5 and several search domains: an external name like api.example.com is tried with each of them before it goes out as written

Fix

  • Add a trailing dot to hot external names (api.example.com.) so they are sent as written.
  • Or lower ndots for that pod with dnsConfig.options, accepting that partly qualified in-cluster names with at least that many dots, such as myservice.myns.svc with ndots:2, are then tried as written first and only reach the search list after that query fails.

Queries never reach CoreDNS

Occasional

When the pod points at cluster DNS, CoreDNS is Ready with endpoints and its log shows no errors, the next question is whether the query arrives at all. A NetworkPolicy that does not allow DNS, or a kube-proxy problem on the node, drops it before CoreDNS sees it.

Confirm

kubectl -n kube-system edit configmap coredns   # add the `log` plugin, then query again and watch:
kubectl -n kube-system logs -l k8s-app=kube-dns -f

Whether your query appears in the log at all. Nothing logged means it is dropped on the way: check the NetworkPolicies that select the pod and CoreDNS, for UDP and TCP port 53

Fix

  • Allow egress to kube-dns on UDP and TCP 53 in any NetworkPolicy that selects the pod.
  • Remove the log plugin again afterwards; it logs every query.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Inside a cluster, pods find each other by name, and a DNS server called CoreDNS turns those names into addresses. When a lookup fails, one link in a short chain is broken: the CoreDNS pods, the Service in front of them, the pod's own DNS settings, or a name asked from the wrong namespace. You test the links in order, from the pod outwards.

A pod that cannot resolve even kubernetes.default

Output quoted from the Kubernetes task page Debugging DNS Resolution, run in its dnsutils test pod in the default namespace while cluster DNS is failing. No cluster was run for this page.

Command

kubectl exec -i -t dnsutils -- nslookup kubernetes.default

Output · as shown in Kubernetes — Debugging DNS Resolution

Server:    10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'kubernetes.default'

How to read it

  • Server: 10.0.0.10 is the nameserver from the pod's /etc/resolv.conf, and the name on the next line shows it is the kube-dns Service. The address on your cluster is likely different.
  • So the pod is asking cluster DNS, which makes a wrong dnsPolicy unlikely. The question becomes why CoreDNS cannot answer: check its pods, the Service's endpoints and the CoreDNS logs next.
  • kubernetes.default always exists: it is the Service for the Kubernetes API in the default namespace. When even this fails, DNS itself is broken, not your Service.
  • The short name only works through the search list, where the svc.cluster.local entry turns it into kubernetes.default.svc.cluster.local.
  • The docs show this output with and without the name on the Address 1 line and treat both as a problem with CoreDNS or its Service. The exact layout depends on the nslookup in your image.

Words used on this page

CoreDNS
The cluster's DNS server, running as pods in kube-system. It answers Service and pod names from the Kubernetes API and forwards other names to upstream servers.
kube-dns
The name of the Service in front of CoreDNS, and the value of its pods' k8s-app label. The name is kept from the older kube-dns server for backward compatibility.
/etc/resolv.conf
The file in each container that tells its resolver which nameserver to ask, which search domains to try and which options to use. The kubelet writes it for every pod.
Search list
The search line in resolv.conf. A short name is tried with each suffix in turn, starting with <namespace>.svc.cluster.local, so myservice means the Service in your own namespace.
ndots
A resolv.conf option. A name with fewer dots than this is tried with each search suffix before it is tried as written. Kubernetes sets 5; the resolver's own default is 1.
Trailing dot
A name that ends in ., such as api.example.com.. The glibc resolver sends it as written and skips the search list.
dnsPolicy
A pod setting that picks its DNS configuration. ClusterFirst, the default, uses cluster DNS; Default copies the node's settings; None takes everything from dnsConfig.
NXDOMAIN and SERVFAIL
Two DNS answers. NXDOMAIN says the name does not exist. SERVFAIL says the server could not produce an answer, so the name was never really checked.
What is actually happening

Every pod is handed a nameserver and a search list in /etc/resolv.conf, pointing at a cluster DNS Service. A lookup inside a pod walks that chain: the resolver asks the Service IP, kube-proxy sends it to a CoreDNS pod, and CoreDNS answers from the Kubernetes API or forwards upstream. A failure is one of those links, and the links are testable in order.

The first thing worth knowing is that the Service is called kube-dns even when the software behind it is CoreDNS. The Kubernetes documentation states this plainly — the name is kept for backward compatibility with the original kube-dns. People grep for a coredns Service, find nothing, and conclude DNS is not installed.

The second is that resolution depends on the namespace of the asking pod. A short name like myservice is completed using the search list, which begins with the pod's own namespace. The same short name in a different namespace resolves to a different place, or to nothing. The documentation flags this next to its own example for exactly this reason.

The third is options ndots:5, which Kubernetes sets deliberately and which changes how every name is tried. resolv.conf(5) defines ndots:n as the threshold for how many dots a name needs before the resolver tries it as an absolute name first; the system default is 1. At 5, a name with fewer than five dots goes through every search-list entry before it is tried as written. Short in-cluster names are the reason it exists, and external names pay for it: api.example.com has two dots, so it is tried as api.example.com.<namespace>.svc.cluster.local, then with each remaining search-list entry (svc.cluster.local, cluster.local, plus any search domains inherited from the node) before the real query.

That last one is why DNS is often blamed for slowness rather than failure. The man page is explicit that walking a search list "may be slow and will generate a lot of network traffic if the servers for the listed domains are not local, and that queries will time out if no server is available". Nothing is broken; each lookup is just making one extra lookup per search domain before the real one: three for the cluster's own suffixes, plus any the pod inherits from the node.

How to tell this is your problem
WhereWhat you see
Application logsno such host, Name or service not known, or a connection attempt to a name that never became an address
kubectl exec -i -t <pod> -- nslookup kubernetes.defaultnslookup: can't resolve 'kubernetes.default'
kubectl get pods -n kube-system -l k8s-app=kube-dnsCoreDNS pods not Running, or Running but not 1/1 Ready
kubectl get endpointslice -l kubernetes.io/service-name=kube-dns -n kube-systemNo endpoints listed — the Service resolves to nothing

How to know it is actually fixed

  • nslookup kubernetes.default succeeds from a pod in the affected namespace, not only from kube-system.
  • The application's own name resolves, both short and fully qualified, from the namespace it actually runs in.
  • CoreDNS pods are 1/1 Ready and the kube-dns EndpointSlice lists addresses on port 53.
  • Where ndots was the issue, an external name resolves without the delay — compare timings before and after rather than assuming.

Stopping it happening again

  • Use the fully qualified <service>.<namespace>.svc.cluster.local for cross-namespace calls. It costs a few characters and removes the entire class of namespace-dependent name bugs.
  • For external hostnames that are called often, a trailing dot makes the name absolute and skips the search list, which is where the ndots:5 cost comes from.
  • Alert on CoreDNS readiness rather than on application errors. Every workload fails at once when DNS does, so the application alerts arrive as noise and the cause is one Deployment.
  • Treat the CoreDNS ConfigMap as production configuration — a Corefile edit takes a minute or two to propagate and a syntax error does not reach pods that are already running, because the reload plugin keeps the old configuration and logs an error, but every CoreDNS pod that starts afterwards reads the broken file.

Security considerations

  • The CoreDNS log plugin records every query with the client pod's IP and the name it asked for. Turn it on only while you debug, remove it afterwards, and treat those logs as sensitive.
  • With ndots:5, an external name is first tried under every search domain, including any the pod inherits from the node. The resolv.conf man page warns that search lists can expose lookups to man-in-the-middle attacks, so use a trailing dot for sensitive external hosts and keep TLS certificate checks on.
  • When you repair CoreDNS permissions, restore only list and watch on the resources the docs name: endpoints, namespaces, pods, services and endpointslices. A broader grant, such as cluster-admin, gives a network-facing pod far more than it needs.
  • Allow DNS in a NetworkPolicy with a narrow egress rule to the CoreDNS pods on UDP and TCP port 53. A broad egress allow, or deleting a deny policy, removes the protection the policy gave.
  • Pods with dnsPolicy: Default, or with hostNetwork: true and no ClusterFirstWithHostNet, use the node's resolver. Rules you set in the Corefile, such as stub domains or upstream servers, do not apply to them.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next