Kubernetes · what happens if

What happens if a Kubernetes node suddenly disappears?

Short answer

More slowly than people expect. The control plane cannot distinguish a dead node from an unreachable one, so it waits: about 50 seconds (40 before Kubernetes 1.32) before the node's pods are marked not ready and taken out of Service endpoints, and about five more minutes before they are evicted. A Deployment then replaces them elsewhere; a StatefulSet's pods wait until the node returns or someone acts. Your callers feel the first part before Kubernetes notices anything.

First published 4 Sept 2026

On this page · 8 sections

Starting state

A three-node cluster. One node loses power, or its network, or its kubelet crashes. From the control plane's point of view these are indistinguishable: heartbeats simply stop arriving.

Words used on this page

kubelet: The agent on each node that starts the node's containers, runs their probes and sends the node's heartbeats to the control plane.
Lease: A small object in the kube-node-lease namespace, one per node. The kubelet renews it every 10 seconds by default, and the renew time is the heartbeat the control plane watches.
node controller: Part of kube-controller-manager. It notices when heartbeats stop, sets the node's Ready condition to Unknown, marks its pods not ready and taints the node.
Ready condition: A node's main health flag: True, False when the kubelet reports a problem, or Unknown when nothing has been heard for the grace period. kubectl get nodes shows anything other than True as NotReady.
taint: A mark on a node that repels pods. With the NoExecute effect it also evicts pods already running there, unless they tolerate it.
tolerationSeconds: How long a pod may stay on a node with a NoExecute taint it tolerates. Kubernetes adds 300 seconds for the unreachable and not-ready taints to pods that do not set their own.
eviction: Here, the control plane deleting a pod from a failed node so its controller can replace it. On an unreachable node the pod is then shown as Terminating, because no kubelet confirms that it stopped.
EndpointSlice: The list of pod addresses behind a Service. Each address has conditions that follow the pod's Ready condition, and kube-proxy routes Service traffic from them.
force detach: Detaching a volume from a node that never confirmed it stopped using it. Kubernetes does this when a pod's deletion has been pending for six minutes and the node is unhealthy.

The sequence

Colour marks which component acts. Steps sharing a timestamp happen concurrently — nothing here waits for anything else unless it says so.

kubelet
your app
network
control plane

t+03 things at once
- kubelet
  The kubelet stops renewing its Lease — because the process is gone, the machine is gone, or it can no longer reach the API server. From outside, those three are indistinguishable.
- your app
  If the machine or its network is gone, callers notice before the cluster does. Requests to pods on that node stop being answered, and each caller starts waiting on its own timeout. If only the kubelet crashed, the containers keep running and keep answering.
  What you would see Rising latency in the callers, up to about a minute before any Kubernetes object changes state.
- network
  Nothing removes the endpoints yet. Nothing has been detected yet: the absence of a heartbeat is not immediately meaningful, because heartbeats are missed all the time. Until the next step, Services keep sending new requests to the pods on that node.
~t+50s (default)control plane
The node controller has not seen a heartbeat for node-monitor-grace-period (50 seconds by default since Kubernetes 1.32, 40 seconds before that) and marks the node's Ready condition as Unknown.
What you would see kubectl get nodes shows NotReady.
on that transitioncontrol plane
Two things happen. The node controller marks every pod on the node as not ready, so their endpoints stop being ready and Services stop routing new requests to them. And the node gets the node.kubernetes.io/unreachable taint. Pods tolerate it for five minutes by default, so nothing is evicted yet.
What you would see The pods' Ready condition turns False (kubectl get pod <name> -o yaml, under status.conditions), and their endpoints show ready: false in kubectl get endpointslice -o yaml.
meanwhilenetwork
Requests that were already in flight to that node are still stuck. Nothing is answering to refuse them, so each one waits out its caller's timeout.
What you would see Timeouts rather than connection refusals, concentrated in the first minute.
~t+5m50s (default)control plane
The toleration expires. Pods on the node are evicted, and their controllers create replacements elsewhere. DaemonSet pods are the exception: they tolerate this taint with no time limit, so it never evicts them.
after replacementcontrol plane
For a Deployment, new pods schedule onto healthy nodes and capacity recovers. A StatefulSet can wait much longer: its replacement must reuse the old pod's name, and the old pod stays Terminating because the kubelet that would confirm its deletion is gone. It stays that way until the kubelet returns, the Node object is deleted (by you, or by the cloud node controller once the VM no longer exists), or someone force-deletes the pod. Its volumes are a second block: Kubernetes force-detaches them only once the pod's deletion has been pending for six minutes and the node is still unhealthy, and not at all if disable-force-detach-on-timeout is set.

What you actually control

Separated from what happens regardless, because conflating the two is how people end up tuning the wrong thing.

Caller-side health checks: The fastest lever, because it does not wait for the node controller at all. Outlier detection in a client library or service mesh stops sending to an endpoint that has stopped answering within seconds. Readiness probes cannot do this: the kubelet that runs them was on the node that disappeared.
Client timeouts and retries: What determines the blast radius until the node's pods are marked not ready, and after that for any caller still holding connections to them. A caller with a 30-second timeout and no retry budget will queue up requests and exhaust its own connection pool.
Topology spread constraints: Decides how much you lose. Three replicas on three nodes lose a third of capacity; three replicas that all landed on one node lose everything.
tolerationSeconds on the node.kubernetes.io/unreachable taint: Shortens the eviction wait. A node that stops heartbeating is Unknown, which applies unreachable, not not-ready, so that is the toleration to change. Tempting, and worth being careful with: the wait exists so that a brief network partition does not trigger a cluster-wide reschedule.

Try it yourself

Before you start

kubectl access that can read cluster-wide objects: nodes, pods in every namespace, events and EndpointSlices. Check with kubectl auth can-i list nodes; a role limited to one namespace cannot see nodes at all.
Your Kubernetes version (kubectl version). The timings on this page are the defaults for 1.32 and later; before 1.32 the node controller waited 40 seconds, not 50.
Whether you can see or change the control plane's settings. --node-monitor-grace-period belongs to kube-controller-manager, and the 300-second default tolerations to kube-apiserver; on a managed cluster your provider decides whether you can change them.
For stateful workloads, which PersistentVolumeClaims are ReadWriteOnce (kubectl get pvc -A). Those volumes cannot follow their pod to another node until they are detached from the lost one.
A test cluster where you can stop a worker's kubelet or cut its network if you want to watch this happen. Do not rehearse it on a node that runs the only replica of anything.

Reading a node that stopped heartbeating

Quoted verbatim from the example "debugging a down/unreachable node" on the Kubernetes documentation page Troubleshooting Clusters; not reproduced, because no cluster was available. That cluster ran Kubernetes v1.23.3, older than 1.32, so its node controller waited 40 seconds rather than 50. Lines of ... mark parts of the describe output left out here, and the times are as the documentation prints them.

Command

kubectl get nodes
kubectl describe node kube-worker-1

Output · as shown in Kubernetes — Troubleshooting Clusters

$ kubectl get nodes
NAME                     STATUS       ROLES     AGE     VERSION
kube-worker-1            NotReady     <none>    1h      v1.23.3
kubernetes-node-bols     Ready        <none>    1h      v1.23.3
kubernetes-node-st6x     Ready        <none>    1h      v1.23.3
kubernetes-node-unaj     Ready        <none>    1h      v1.23.3

$ kubectl describe node kube-worker-1
...
Taints:             node.kubernetes.io/unreachable:NoExecute
                    node.kubernetes.io/unreachable:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  kube-worker-1
  AcquireTime:     <unset>
  RenewTime:       Thu, 17 Feb 2022 17:13:09 -0500
Conditions:
  Type                 Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message
  ----                 ------    -----------------                 ------------------                ------              -------
  ...
  Ready                Unknown   Thu, 17 Feb 2022 17:12:40 -0500   Thu, 17 Feb 2022 17:13:52 -0500   NodeStatusUnknown   Kubelet stopped posting node status.
...

How to read it

kubectl get nodes prints NotReady for any Ready condition that is not True. That covers False, where the kubelet reported a problem, and Unknown, where nothing was heard at all. describe tells you which.
Unknown with reason NodeStatusUnknown and the message Kubelet stopped posting node status. was written by the node controller, not by the kubelet. It is what the control plane records when heartbeats stop arriving.
The Lease was last renewed at 17:13:09 and the Ready condition changed at 17:13:52, 43 seconds later. That fits this cluster's 40-second grace period plus the controller's 5-second check interval. On Kubernetes 1.32 or later, expect about 50 seconds.
LastHeartbeatTime (17:12:40) is older than the Lease renewal because the kubelet posts node status far less often than it renews its Lease. The control plane counts whichever it heard from last.
Both unreachable taints are there. NoSchedule keeps new pods off the node. NoExecute starts the clock on each pod's toleration, 300 seconds by default, after which the pod is evicted.

Where this goes wrong

All replicas were on the failed node
Complete outage for that service, for the full detection and rescheduling window. Replica count without spread is a number, not redundancy.
Callers have long timeouts and no circuit breaker
The failure propagates. Each caller holds connections open waiting on a node that is gone, exhausts its pool, and starts failing requests that had nothing to do with the dead node.
The workload is a StatefulSet with a ReadWriteOnce volume
The replacement cannot be created while the old pod is stuck Terminating, and its volume is still attached to a node that is unreachable rather than confirmed dead. Kubernetes force-detaches it once the pod's deletion has been pending for six minutes and the node is still unhealthy, which is safe only if the node really is gone. If you know it is, force-delete the pod or apply the node.kubernetes.io/out-of-service taint. That is the judgement call somebody has to make.

If it goes wrong

The lost node's pods stay Terminating long after the five minutes: Expected while the node is unreachable: no kubelet is left to confirm they stopped. A Deployment has already created replacements without waiting for them. If the machine is confirmed dead, delete the Node object (kubectl delete node <node>), which the Kubernetes documentation recommends over force-deleting pods. If it might come back, wait: its kubelet finishes the deletions when it reconnects.
A StatefulSet's replacement pod is stuck with event FailedAttachVolume and Waiting for detach for volume "<volume>" Volume is already exclusively attached to one node, waiting on detach before it can be attached to another node (before 1.37: Multi-Attach error for volume "<volume>" Volume is already exclusively attached to one node and can't be attached to another): The volume is still attached to the lost node. Kubernetes force-detaches it once the old pod's deletion has been pending for six minutes and the node is still unhealthy, unless disable-force-detach-on-timeout is set. If you have confirmed the node is shut down, kubectl taint nodes <node> node.kubernetes.io/out-of-service:NoExecute detaches it at once. Remove the taint yourself once the node has recovered.
Several nodes are NotReady at once and their pods are not evicted after five minutes: That is deliberate. When at least 55% of a zone's nodes (and at least 3), but not all of them, are unhealthy, the node controller slows evictions, and stops them in clusters of 50 nodes or fewer. When no node in the cluster is healthy, it assumes the control plane has lost its connection to the nodes and evicts nothing. Look for the shared cause, such as the network or the control plane, rather than waiting for rescheduling.
Pods were evicted and replaced, but the old containers are still serving on the original node: The kubelet crashed or was partitioned, not the machine. Stopping the kubelet does not stop its pods, and pods marked for deletion may keep running until the kubelet reconnects. Restart the kubelet or restore the network; it then completes the deletions. Until then, each replaced pod runs twice.

The part worth remembering

The uncomfortable truth underneath this is that the control plane cannot tell a dead node from an unreachable one. Nothing can — that is a fundamental property of distributed systems, not a gap in Kubernetes.

So it has to choose between acting quickly and being wrong sometimes, or waiting and being slow. The defaults choose waiting, because evicting every pod on a node during a two-second network blip would be far more damaging than a slower recovery.

Which means node failure recovery is not something to tune away. The fast path is to not depend on the control plane noticing: spread your replicas, and give your callers sensible timeouts and their own health checks. Those work in seconds, because they never needed to know whether the node was dead.

Security considerations

Force-deleting a StatefulSet pod, or tainting a node out-of-service, tells Kubernetes the old pod will never run again. If the node was only cut off, the old pod may still be running and writing, and its replacement gives you two members with one identity, which the documentation warns can mean split brain and data loss. Confirm the machine is powered off first.
The six-minute forced detach assumes an unhealthy node is a dead one. If it is still running, a workload using the force-detached volume can corrupt its data, a violation of the CSI specification by the documentation's own account. Where that risk is unacceptable, set disable-force-detach-on-timeout and recover through the non-graceful node shutdown procedure instead.
Shortening tolerationSeconds on the unreachable taint turns short network blips into evictions. Pods on a partitioned node may keep running while their replacements start, so anything that must not run twice, such as a job without idempotency or a leader without fencing, can run twice.
A PodDisruptionBudget does not protect against this: involuntary disruptions cannot be prevented by PDBs, though they count against the budget. While the lost pods are counted, kubectl drain on another node can block, so finish recovering before you start maintenance.

Sources

The behaviour this page describes, as documented by the projects themselves.

Terms used here (1)plain definitions

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Node: A physical or virtual machine in a Kubernetes cluster that runs workloads. Each node runs the kubelet agent and a container runtime, and usually kube-proxy (optional when the network plugin implements Services itself). Nodes are managed by the control plane and can be added or removed to scale the cluster.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it