Skip to main content
DevOps Insightsguides & answers

Kubernetes

What happens if a Pod is deleted while it is serving traffic?

Short answer

Two things start at once and neither waits for the other: the pod begins shutting down, and its address begins being removed from Services. Requests that arrive in the gap between those get sent to a container that is already stopping.

Starting state

A Deployment with three replicas behind a Service. One pod is deleted — by a rollout, a node drain, an eviction or `kubectl delete`. The mechanism does not matter; what follows is the same.

The sequence

Colour marks which component acts. Steps sharing a timestamp happen concurrently — nothing here waits for anything else unless it says so.

  • control plane
  • kubelet
  • network
  • your app
  1. t+0control plane

    The pod's `deletionTimestamp` is set. It is not removed from the API yet — it is marked as going away.

    What you would see `kubectl get pod` shows status `Terminating`.

  2. t+0control plane

    In parallel, the endpoint controller notices and starts removing the pod's address from the Service's endpoints.

    What you would see `kubectl get endpointslices` stops listing the address — eventually.

  3. t+0kubelet

    Also in parallel, the kubelet runs any `preStop` hook and then sends SIGTERM to PID 1 in each container.

  4. next few hundred msnetwork

    Every node's kube-proxy (or CNI equivalent) has to be told about the endpoint change and update its own rules. This is where the gap lives: the pod is already receiving SIGTERM while some nodes still consider it a valid destination.

    What you would see A small burst of connection errors or resets at exactly the moment of a rollout — often blamed on the load balancer.

  5. t+0 to grace periodyour app

    Your application decides what SIGTERM means. If it stops accepting new connections and finishes in-flight requests, this is graceful. If it exits immediately, in-flight requests are dropped. If it ignores the signal, nothing happens yet.

  6. t+30s (default)kubelet

    The grace period expires and any container still running receives SIGKILL. There is no negotiation and no cleanup.

    What you would see Exit code 137, and a pod that took the full grace period to disappear.

  7. after terminationcontrol plane

    The pod object is removed. If it was managed by a ReplicaSet, the controller has already created a replacement to restore the desired replica count.

What you actually control

Separated from what happens regardless, because conflating the two is how people end up tuning the wrong thing.

`terminationGracePeriodSeconds`
How long SIGTERM has before SIGKILL. Raise it if your application legitimately needs longer to drain; lowering it makes deploys faster and drops more requests.
A `preStop` hook with a short sleep
The standard mitigation for the endpoint-propagation gap. Sleeping a couple of seconds before SIGTERM gives the network time to stop routing to the pod, so the requests that would have landed in the gap go elsewhere instead. It feels like a hack because it is one; it is also the accepted answer.
Handling SIGTERM in your application
The part nothing else can do for you. Stop accepting new work, finish what you have, close connections, exit. Without this, none of the above helps.
A PodDisruptionBudget
Does not change any of this sequence — it changes how many pods may be in it at once during voluntary disruptions like a node drain.

Where this goes wrong

  • The application ignores SIGTERM

    Every pod takes the full grace period to die and is then killed. Deploys are slow and every rollout drops in-flight requests. Usually caused by shell-form ENTRYPOINT making `/bin/sh` PID 1.

  • There is no preStop delay and traffic is high

    A reliable trickle of errors on every deploy, proportional to request rate. Hard to attribute because each individual error looks like a transient network fault.

  • The grace period is shorter than the work in flight

    Long-running requests are cut off mid-response. For a background worker, a job can be lost silently unless it is idempotent and re-queued.

The part worth remembering

The generalisable point is that shutdown is not a single event with a single owner. Four different components act, they act concurrently, and none of them coordinates with the others. Kubernetes does not stop traffic and then signal the pod; it does both at once and lets the timing work itself out.

Once you see that, a whole class of "transient errors during deploys" stops being mysterious. It is not a bug in the load balancer and not flakiness — it is a race that exists by design, and the mitigations are all about giving the slower participant a head start.

What comes next

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere