Kubernetes
What happens if a Kubernetes node suddenly disappears?
Short answer
Much more slowly than people expect. The control plane cannot distinguish a dead node from an unreachable one, so it waits — and by default your pods keep receiving traffic for most of that wait.
Starting state
A three-node cluster. One node loses power, or its network, or its kubelet crashes. From the control plane's point of view these are indistinguishable: heartbeats simply stop arriving.
The sequence
Colour marks which component acts. Steps sharing a timestamp happen concurrently — nothing here waits for anything else unless it says so.
- kubelet
- your app
- network
- control plane
- t+0kubelet
The kubelet stops renewing its Lease — because the process is gone, the machine is gone, or it can no longer reach the API server. From outside, those three are indistinguishable.
- t+0your app
Callers begin to notice, before the cluster does. Requests to pods on that node stop being answered, and each caller starts waiting on its own timeout.
What you would see Rising latency in the *callers*, minutes before any Kubernetes object changes state.
- t+0network
Nothing removes the endpoints yet. Nothing has been detected yet — the absence of a heartbeat is not immediately meaningful, because heartbeats are missed all the time.
- ~t+40s (default)control plane
The node controller has not seen a heartbeat for `node-monitor-grace-period` and marks the node's Ready condition as `Unknown`.
What you would see `kubectl get nodes` shows `NotReady`.
- on that transitioncontrol plane
Taints are applied to the node. Pods with matching tolerations — which most pods have by default — begin a five-minute eviction timer rather than being evicted immediately.
- meanwhilenetwork
This is the part that surprises people. Until the pods are marked for deletion, their addresses are still in the Service endpoints. Traffic keeps being sent to a node that is not there.
What you would see Timeouts rather than connection refusals — nothing is answering to say no, so callers wait for their own timeout to expire.
- ~t+5m (default)control plane
The eviction timer expires. Pods on the node are marked for deletion, removed from endpoints, and their controllers create replacements elsewhere.
- after replacementcontrol plane
For a Deployment, new pods schedule onto healthy nodes and traffic recovers. For a StatefulSet with attached storage, the replacement may block until the volume can be detached from a node nobody can reach — which is why stateful recovery is slower and sometimes needs a human.
What you actually control
Separated from what happens regardless, because conflating the two is how people end up tuning the wrong thing.
- Readiness probes
- The most effective lever, and it works because it does not involve the node controller at all. A caller-side readiness signal fails fast when the pod stops answering, so traffic drains in seconds rather than minutes.
- Client timeouts and retries
- What actually determines the blast radius during those five minutes. A caller with a 30-second timeout and no retry budget will queue up requests and exhaust its own connection pool.
- Topology spread constraints
- Decides how much you lose. Three replicas on three nodes lose a third of capacity; three replicas that all landed on one node lose everything.
- `tolerationSeconds` on the not-ready taint
- Shortens the eviction wait. Tempting, and worth being careful with: the wait exists so that a brief network partition does not trigger a cluster-wide reschedule.
Where this goes wrong
All replicas were on the failed node
Complete outage for that service, for the full detection and rescheduling window. Replica count without spread is a number, not redundancy.
Callers have long timeouts and no circuit breaker
The failure propagates. Each caller holds connections open waiting on a node that is gone, exhausts its pool, and starts failing requests that had nothing to do with the dead node.
The workload is a StatefulSet with a ReadWriteOnce volume
The replacement pod cannot start until the volume is detached, and the volume cannot be safely detached from a node that is unreachable rather than confirmed dead. This is where automation stops and somebody has to make a judgement call.
The part worth remembering
The uncomfortable truth underneath this is that the control plane cannot tell a dead node from an unreachable one. Nothing can — that is a fundamental property of distributed systems, not a gap in Kubernetes.
So it has to choose between acting quickly and being wrong sometimes, or waiting and being slow. The defaults choose waiting, because evicting every pod on a node during a two-second network blip would be far more damaging than a slower recovery.
Which means node failure recovery is not something to tune away. The fast path is to not depend on the control plane noticing: spread your replicas, keep readiness probes honest, and give your callers sensible timeouts. Those work in seconds, because they never needed to know whether the node was dead.
What comes next
Did this get you to an answer?
No text box on purpose — please do not paste production logs anywhere