Skip to main content
DevOps Insightsguides & answers

Kubernetes · troubleshooting

Readiness probe failed

The kubelet's readiness check on a container failed, so the pod is marked not ready and Services stop sending it traffic. The container is not restarted. The event's text after Readiness probe failed: says how it failed: a status code, connection refused, or a timeout. Each points somewhere different, and the commonest causes are an app listening only on localhost, the wrong port or path, and a 1-second timeout.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 9 sections

Before you start

  • kubectl access to the pod's namespace, including events and EndpointSlices.
  • The pod's manifest, to read the probe's port, path, scheme and timeout.
  • A shell or wget in the container, or permission to use kubectl debug to add one.

Run this first

step 1 of 4
kubectl describe pod <pod> -n <namespace>

Look for The text after Readiness probe failed:, and the Readiness line: kubectl 1.37 prints successThreshold and failureThreshold, older versions #success and #failure

Shows the probe's configuration and the Unhealthy events with their messages.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

What follows Readiness probe failed: in the event?

kubectl describe pod <pod>, at the bottom

Cause space

6 of 6 still possible

  • The application listens only on 127.0.0.1, not on the pod's IPCommon
  • The probe points at the wrong port or pathCommon
  • The application answers, but not within timeoutSecondsCommon
  • The application reports itself not ready because something it needs is downOccasional
  • The application is still startingOccasional
  • The probe's scheme does not match the portRare

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    kubectl describe pod <pod> -n <namespace>

    Shows the probe's configuration and the Unhealthy events with their messages.

    Look for The text after Readiness probe failed:, and the Readiness line: kubectl 1.37 prints successThreshold and failureThreshold, older versions #success and #failure

  2. Step 2

    kubectl get events -n <namespace> --field-selector involvedObject.name=<pod>,reason=Unhealthy

    Just the probe failures, with how often they happen.

    Look for statuscode, connection refused, or Client.Timeout exceeded

  3. Step 3

    POD_IP=$(kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.podIP}')
kubectl exec <pod> -n <namespace> -- wget -T 1 -qO- http://$POD_IP:<port>/<path>

    Makes the request the kubelet makes: to the pod IP, with a 1-second limit. Testing localhost or using port-forward checks something else.

    Look for The same failure as the event. For an image without wget, use kubectl debug -it <pod> --image=busybox:1.28 --target=<container>

  4. Step 4

    kubectl get endpointslices -n <namespace> -l kubernetes.io/service-name=<service> -o yaml

    Shows whether the Service considers the pod ready. The ENDPOINTS column of the default view lists every address, ready or not.

    Look for conditions.ready: false on the pod's address

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

The application listens only on 127.0.0.1, not on the pod's IP

Common

The kubelet probes the pod's IP. An application bound to 127.0.0.1 or ::1 refuses that connection, so the event says connection refused. It is easy to miss, because kubectl exec … curl localhost and kubectl port-forward both connect to localhost inside the pod and succeed.

Confirm

POD_IP=$(kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.podIP}')
kubectl exec <pod> -n <namespace> -- wget -qO- http://$POD_IP:<port>/<path>

Connection refused on the pod IP while the same request to localhost works

Fix

  • Make the application listen on 0.0.0.0, or on the pod IP, for the port the probe uses. Many frameworks default to localhost in development mode.

The probe points at the wrong port or path

Common

The probe's port must be the container's port, not the Service's. A wrong port gives connection refused; a wrong path gives HTTP probe failed with statuscode: 404. A renamed health endpoint or a changed container port leaves the probe behind.

Confirm

kubectl describe pod <pod> -n <namespace> | grep -E 'Port|Readiness'

The Readiness line's port and path against the container's ports and the application's real health route

Fix

  • Set the probe's port to the containerPort, ideally by name, so the two cannot drift.
  • Set path to the route the application actually serves, and check it returns a code from 200 to 399.

The application answers, but not within timeoutSeconds

Common

The default timeout is 1 second. A readiness handler that queries a database or calls other services, a busy process, or a CPU-throttled container can take longer, and the event says Client.Timeout exceeded or, for exec probes, command timed out.

Confirm

kubectl exec <pod> -n <namespace> -- sh -c 'time wget -qO- http://<pod-ip>:<port>/<path>'

Response times near or above timeoutSeconds, especially under normal load

Fix

  • Make the readiness handler cheap: report the process's own state, and avoid slow calls in it.
  • Set timeoutSeconds from measured response times with margin, rather than the 1-second default.

The application reports itself not ready because something it needs is down

Occasional

The handler checks a database or another service, finds it unavailable and returns 503. The probe is doing its job. The trade-off is real: kubernetes.io suggests checking required back-ends, while the AWS EKS best practices guide warns that a shared dependency then takes every replica out at once.

Confirm

kubectl exec <pod> -n <namespace> -- wget -S -qO- http://<pod-ip>:<port>/<path>

A 503 whose body, or the application log at the same time, names a dependency

Fix

  • Fix the dependency.
  • Decide deliberately what readiness should include. If every replica depends on the same database, consider leaving it out of readiness and handling its failure in the application, so pods are not all removed at once.

The application is still starting

Occasional

Until the first success the pod counts as not ready, and every probe before then records an event. That is expected noise during startup. If the application takes longer than initialDelaySeconds + failureThreshold × periodSeconds, the documentation recommends a startup probe.

Confirm

kubectl get pod <pod> -n <namespace> -w

READY changing to 1/1 after a while, with the events stopping then

Fix

  • Add a startup probe sized to the real start time, so readiness and liveness checks wait for it.
  • If the pod is restarted while starting, the cause is a liveness probe, not readiness; readiness never restarts a container.
startupProbe:
  httpGet:
    path: /ready
    port: http
  periodSeconds: 10
  failureThreshold: 30

Gives the application up to 300 seconds to start before the other probes begin.

The probe's scheme does not match the port

Rare

scheme defaults to HTTP. Probing a TLS port with HTTP usually gets a 400 or a dropped connection, depending on the server; probing a plain HTTP port with HTTPS fails in Go's client with http: server gave HTTP response to HTTPS client. Both messages are derived from the source code, not observed here.

Confirm

kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].readinessProbe.httpGet}'

scheme against whether the port serves TLS

Fix

  • Set scheme: HTTPS for a TLS port, or probe a plain HTTP health port.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Kubernetes asks each container, every few seconds, whether it is ready for traffic, usually by requesting a health URL. If the answer is wrong or late, the pod is marked not ready and stops receiving requests, but it keeps running. The warning tells you what the check got back: an error code, no connection at all, or no reply in time. That one detail tells you where to look.

Words used on this page

Readiness probe
A check the kubelet runs on a container for its whole life. Failing it marks the pod not ready and takes it out of Service traffic. It never restarts the container; that is what a liveness probe does.
Pod IP
The address the kubelet probes, unless the probe sets a host. An application listening only on 127.0.0.1 cannot be reached there.
timeoutSeconds
How long the kubelet waits for one probe to answer. The default is 1 second.
failureThreshold and periodSeconds
How many failures in a row mark the pod not ready, 3 by default, and how often the probe runs, every 10 seconds by default.
Startup probe
A probe that holds off readiness and liveness checks until it succeeds once, for applications that start slowly.
EndpointSlice
The list of addresses behind a Service. A not-ready pod stays in it, marked ready: false, and receives no traffic.
What is actually happening

The kubelet runs the readiness probe on the container throughout its life. Each failure records a Warning event with reason Unhealthy and the message Readiness probe failed: followed by the probe's output. Events fire on every failed probe, before the threshold, which is why they show counts such as x3 over 20s.

After failureThreshold failures in a row, 3 by default, the kubelet sets the pod's Ready condition to False with reason ContainersNotReady. kubectl shows READY 0/1 while STATUS stays Running. The container keeps running and the kubelet keeps probing.

Services stop sending the pod traffic. The documentation says its IP is removed from the Service's EndpointSlices; in fact the endpoint stays, marked ready: false, and kubectl get endpointslices still lists the address in its ENDPOINTS column. Read the endpoint's conditions to see whether it is ready.

The text after the colon is the diagnosis. HTTP probe failed with statuscode: 503 means the application answered and said no. Get "http://…": dial tcp …: connect: connection refused means nothing was listening on that address and port. Client.Timeout exceeded while awaiting headers means it did not answer within timeoutSeconds. The words before that bracket vary with the Go version the kubelet was built with, so match on the bracket.

A pod is born not ready: the readiness result counts as failure until the first probe succeeds. A rolling update waits for new pods to become ready, so a probe that never passes stalls the rollout rather than breaking the old pods.

How to tell this is your problem
WhereWhat you see
kubectl describe pod, EventsWarning Unhealthy … kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Event message, nothing listeningReadiness probe failed: Get "http://10.244.0.15:8080/ready": dial tcp 10.244.0.15:8080: connect: connection refused
Event message, too slowReadiness probe failed: Get "http://…": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Event message, exec probeThe command's own output, or command timed out: "…" timed out after 1s
kubectl get podsREADY 0/1, STATUS Running
Pod Ready conditionstatus False, reason ContainersNotReady

How to know it is actually fixed

  • kubectl get pod shows READY 1/1 and no new Unhealthy events appear over several probe periods.
  • The pod's address in the Service's EndpointSlice has conditions.ready: true.
  • The probe still fails when the application genuinely cannot serve, for example with its port closed, so it has not been made to pass unconditionally.

Stopping it happening again

  • Reference the container port by name in probes, so changing the port cannot leave the probe behind.
  • Give readiness its own lightweight endpoint, and set timeoutSeconds and failureThreshold from measured behaviour.
  • Use a startup probe for slow starters instead of a long initialDelaySeconds.

Security considerations

  • Keep the readiness endpoint cheap and free of sensitive detail. The kubelet does not put the response body in the event, but anyone who can reach the pod can call the endpoint, so it should not return configuration, versions of dependencies or stack traces.
  • Do not fix a failing probe by pointing it at a path that always returns 200. The pod then receives traffic it cannot serve, and the failure moves to your users.
  • If the probe uses scheme: HTTPS, the kubelet skips certificate verification. That is fine for a health check, but do not read a passing probe as proof that the certificate is valid.
  • Standard NetworkPolicy cannot block the kubelet's probes, because traffic from the pod's own node is always allowed. A node firewall or a CNI-specific policy can, so check those before loosening a NetworkPolicy.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next