Skip to main content
DevOps Insightsguides & answers

Kubernetes · troubleshooting

Pod stuck in Terminating

Deletion of the pod was accepted but not completed, so it shows Terminating and stays there. The pod carries a deletionTimestamp and is waiting on something — a grace period that has not expired, a process ignoring SIGTERM, a finalizer nobody is clearing, or a node that stopped answering. Force-deleting ends the symptom without ending the cause, and the docs are explicit that the container may keep running.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 9 sections

Before you start

  • kubectl read access to the pod and its events, and cluster-wide read on nodes so you can check the node the pod ran on.
  • Permission to kubectl exec into the pod, and cat in its image, if you want to see what runs as PID 1.
  • A way to read the kubelet's logs on that node, through SSH or your provider's console. Unmount and container-runtime errors show there, not in the pod's status.
  • Before any force delete or finalizer removal: the workload type (StatefulSet pods need extra care), the controller that owns each finalizer, and, if the node is unreachable, confirmation that it is really shut down.
  • RBAC to act: a force delete uses the delete verb on pods and removing a finalizer uses patch. Keep both with people who can make the checks above.

Run this first

step 1 of 4
kubectl get pod <name> -o jsonpath='{.metadata.deletionTimestamp}{"  grace="}{.spec.terminationGracePeriodSeconds}{"  finalizers="}{.metadata.finalizers}{"\n"}'

Look for Elapsed time well past the grace period, or a non-empty finalizer list

Three of the five causes are decided by this one line: how long it has been waiting, how long it is allowed to wait, and whether anything is holding it.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

How long past its grace period has it been Terminating?

kubectl get pod <name> -o jsonpath='{.metadata.deletionTimestamp}{" "}{.spec.terminationGracePeriodSeconds}'

Cause space

5 of 5 still possible

  • It has not actually been long enoughCommon
  • The main process never receives or never handles SIGTERMCommon
  • A finalizer is still on the objectCommon
  • A preStop hook is outlasting the grace periodOccasional
  • The node stopped answeringOccasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    kubectl get pod <name> -o jsonpath='{.metadata.deletionTimestamp}{"  grace="}{.spec.terminationGracePeriodSeconds}{"  finalizers="}{.metadata.finalizers}{"\n"}'

    Three of the five causes are decided by this one line: how long it has been waiting, how long it is allowed to wait, and whether anything is holding it.

    Look for Elapsed time well past the grace period, or a non-empty finalizer list

  2. Step 2

    kubectl get pod <name> -o jsonpath='{.spec.nodeName}{"\n"}' | xargs kubectl get node

    Separates "the pod will not stop" from "nobody can tell us whether it stopped". These need opposite responses.

    Look for NotReady, or a node that no longer exists

  3. Step 3

    kubectl describe pod <name> | sed -n '/Events/,$p'

    Shows failing preStop hooks and unmount errors, which otherwise leave no trace in the pod's status.

    Look for FailedPreStopHook, or repeated unmount or detach failures

  4. Step 4

    kubectl get pod <name> -o jsonpath='{range .spec.containers[*]}{.name}{" preStop="}{.lifecycle.preStop}{"\n"}{end}'

    A hook longer than the grace period guarantees a forced kill every time, which is worth knowing before changing anything else.

    Look for Any preStop, and whether its duration fits inside terminationGracePeriodSeconds

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

It has not actually been long enough

Common

The grace period defaults to 30 seconds and a workload may legitimately set it much higher to drain connections. Within that window Terminating is the correct state. The documentation also notes that if the kubelet or the runtime's management service restarts mid-shutdown, the cluster retries from the start including the full original grace period — so the clock can legitimately begin again.

Confirm

kubectl get pod <name> -o jsonpath='{.metadata.deletionTimestamp}{"  grace="}{.spec.terminationGracePeriodSeconds}{"\n"}'

deletionTimestamp is already the deadline: the time of the delete plus the grace period. If it is still in the future, nothing is wrong yet. If the pod was deleted with --grace-period, the period in use is .metadata.deletionGracePeriodSeconds, not the spec value

Fix

  • Wait out the grace period before treating this as a fault.
  • If the grace period is long on purpose, that is a deliberate drain window and shortening it to make deletes feel faster will cut connections mid-flight.

The main process never receives or never handles SIGTERM

Common

The signal goes to PID 1 in the container. If PID 1 is a shell wrapping the real process, the shell receives SIGTERM and the application underneath never does, so nothing shuts down until SIGKILL arrives at the end of the grace period. The pod then takes exactly the grace period to die, every time — a very recognisable signature.

Confirm

kubectl exec <name> -c <container> -- cat /proc/1/cmdline | tr '\0' ' '; echo

What actually runs as PID 1, including the image's ENTRYPOINT and CMD defaults. /bin/sh -c ... means the shell is PID 1 rather than your process. (Needs cat in the image. If the pod sets shareProcessNamespace: true, PID 1 is the pod's /pause process, so this check does not apply.)

Fix

  • Use the exec form so the application becomes PID 1 and receives signals directly.
  • Where a wrapper is genuinely needed, use an init that forwards signals, or have the shell exec the final process so it replaces the shell.
  • Handle SIGTERM in the application: stop accepting new work, finish what is in flight, exit.

A finalizer is still on the object

Common

metadata.finalizers is a list of keys that must be cleared by their owning controllers before the object is removed. While the list is non-empty the object stays terminating no matter what the container is doing. If the controller that owns a key has been uninstalled, or is crash-looping, nobody will ever clear it and the wait is unbounded.

Confirm

kubectl get pod <name> -o jsonpath='{.metadata.finalizers}{"\n"}'

Any non-empty list. The key names which controller is being waited on

Fix

  • Find the controller that owns the key and work out why it is not completing — that is the actual fix, and it usually affects more than this one object.
  • Removing a finalizer by hand skips the cleanup it existed to guarantee, which can leave external resources — cloud disks, load balancer rules, DNS records — orphaned and billable. Do it knowingly, not reflexively.
kubectl patch pod <name> -p '{"metadata":{"finalizers":null}}' --type=merge

Last resort. The Kubernetes docs say to remove a finalizer only when you understand its purpose and its cleanup has been done another way, for example by deleting the dependent resource yourself.

A preStop hook is outlasting the grace period

Occasional

If a container defines a preStop hook and the grace period is not zero, the kubelet runs that hook before signalling the container. A hook that sleeps or waits on something slow eats the window the application needed. The documentation notes the kubelet grants only a small one-off extension of 2 seconds if the hook is still running when the period expires — so a hook longer than the grace period requires raising terminationGracePeriodSeconds to match.

Confirm

kubectl get pod <name> -o jsonpath='{.spec.containers[*].lifecycle.preStop}{"\n"}'

A hook, and how long it could plausibly take compared with terminationGracePeriodSeconds

Fix

  • Raise terminationGracePeriodSeconds so it comfortably exceeds the hook plus the application's own shutdown.
  • Or shorten the hook. A fixed sleep used to wait for endpoint removal is a common one worth re-examining.
  • If the hook exists to order shutdown against a sidecar, native sidecar containers now handle that ordering: the kubelet delays SIGTERM to sidecars until the last main container has fully terminated, and stops them in reverse order of definition.

The node stopped answering

Occasional

Deletion completes when the kubelet confirms the containers are gone. If the node is NotReady — powered off, partitioned, or the kubelet is dead — no confirmation is coming and the pod stays Terminating indefinitely. Nothing is wrong with the pod spec; the reporting path is broken. This is also the one case where the pod may genuinely still be running somewhere you cannot see.

Confirm

kubectl get pod <name> -o jsonpath='{.spec.nodeName}{"\n"}' | xargs kubectl get node

STATUS NotReady, or the node missing entirely

Fix

  • Deal with the node. If it is coming back, the deletion completes on its own when the kubelet reconnects.
  • If the node is genuinely gone, removing the Node object lets the control plane clean up pods bound to it.
  • Be careful with StatefulSets here: a replacement pod with the same identity must not start while the original may still be writing. Kubernetes publishes a separate procedure for force-deleting StatefulSet pods for exactly this reason.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Deleting a pod is a request, not an instant removal. Kubernetes marks the pod as Terminating, gives its containers a set time to shut down cleanly, and removes it once the node confirms they have stopped. A pod that stays Terminating is waiting on something: a slow shutdown, a cleanup step called a finalizer, or a node that has stopped answering.

A pod that is Terminating on purpose

Output quoted from the Kubernetes tutorial Explore Termination Behavior for Pods And Their Endpoints. A one-replica nginx Deployment sets terminationGracePeriodSeconds: 120 and a preStop hook that sleeps for 180 seconds, and the command runs just after kubectl delete pod on its pod. No cluster was run for this page.

Command

kubectl get pods

Output · as shown in Kubernetes — Explore Termination Behavior for Pods And Their Endpoints

NAME                                READY   STATUS        RESTARTS      AGE
nginx-deployment-7768647bf9-b4b9s   1/1     Terminating   0             4m1s
nginx-deployment-7768647bf9-rkxlw   1/1     Running       0             8s

How to read it

  • STATUS Terminating with READY 1/1 means the delete is recorded and the container is still running, here inside its preStop sleep. It is within its 120-second grace period, so nothing is stuck yet.
  • AGE counts from when the pod was created, not from the delete, so 4m1s tells you nothing about how long it has been Terminating. Read .metadata.deletionTimestamp instead: it is the deadline, the delete time plus the grace period.
  • The replacement pod is already Running after 8s, because a ReplicaSet stops counting a pod as soon as it starts shutting down. A StatefulSet replacement needs the same name, so it waits until the old pod is gone, which is why force-deleting StatefulSet pods has its own procedure.
  • The hook sleeps 180 seconds but the grace period is 120. When the period ends with the hook still running, the kubelet grants one 2-second extension and then kills the container, so the hook never finishes.
  • In the same tutorial, the EndpointSlice keeps this pod's address with ready: false, serving: true and terminating: true, so it gets no new regular traffic while it shuts down.

Words used on this page

Terminating
What kubectl get pods shows in STATUS once a delete has been requested and the pod is not gone yet. It is a display value, not one of the pod's phases.
deletionTimestamp
Set in .metadata.deletionTimestamp when you delete the pod. For a graceful delete it is the deadline: the time of the delete plus the grace period. Once it is in the past, the pod is overdue.
Grace period
How long the kubelet waits after asking the containers to stop before it kills them. It comes from terminationGracePeriodSeconds, 30 by default, unless kubectl delete --grace-period overrides it.
SIGTERM and SIGKILL
SIGTERM asks a process to exit and can be handled. SIGKILL ends it at once and cannot be caught. Kubernetes sends SIGTERM, or the image's STOPSIGNAL, first and SIGKILL when the grace period ends.
PID 1
The first process in a container. The stop signal goes to it, so if PID 1 is a shell that does not pass signals on, your application never hears it.
preStop hook
A command or sleep the kubelet runs in a container before sending the stop signal. It uses up the same grace period.
Finalizer
A key in metadata.finalizers that keeps an object until a controller finishes some cleanup and removes the key. While any key remains, the object cannot be deleted.
Force delete
kubectl delete pod <name> --grace-period=0 --force. The API server removes the pod without waiting for the kubelet to confirm the containers stopped, unless finalizers still hold it.
What is actually happening

kubectl delete does not delete a pod. It records an intent: the API server stamps .metadata.deletionTimestamp, tracks a grace period, and the pod shows as Terminating while the kubelet does the actual shutdown. The design aim, in the documentation's words, is to let processes terminate gracefully rather than being stopped abruptly with a KILL signal.

The normal sequence is that the kubelet asks the runtime to stop each container by sending SIGTERM — or whatever STOPSIGNAL the image defines, which containerd and CRI-O default to SIGTERM — waits out the grace period, then sends SIGKILL to whatever is still running and removes the pod from the API server. terminationGracePeriodSeconds defaults to 30.

So a pod that has been Terminating for ten seconds is not stuck, it is working. Stuck means it has outlived its grace period by a clear margin, and that only happens for a small set of reasons: a finalizer still on the object, a kubelet that cannot finish tearing the pod down (volume or container-runtime errors), or a kubelet that cannot report back because its node is unreachable. A slow preStop hook only stretches shutdown to the grace period plus 2 seconds.

Finalizers are the reason that surprises people, because they are not about the container at all. A finalizer is a key in metadata.finalizers telling Kubernetes to wait until a controller has done some cleanup. The API marks the object for deletion and returns 202 Accepted, and the object stays in a terminating state until that list is empty. If the controller that owns the key is gone, nothing will ever empty it.

The tempting fix is --force --grace-period=0, and it is worth knowing exactly what it does before reaching for it: the API server stops waiting for the kubelet to confirm anything and removes the object immediately, unless finalizers are still listed: those keep the object in place even through a force delete. The documentation's own caution is blunt — immediate deletion does not wait for confirmation that the running resource has terminated, and the resource may continue to run on the cluster indefinitely. You get a clean kubectl get pods and possibly a container still holding a volume or a lock.

How to tell this is your problem
WhereWhat you see
kubectl get podsSTATUS Terminating long after the delete. AGE counts from the pod's creation, not from the delete, so read .metadata.deletionTimestamp to see whether the grace period has run out
kubectl get pod <name> -o jsonpath='{.metadata.deletionTimestamp}'A timestamp well in the past
kubectl get pod <name> -o jsonpath='{.metadata.finalizers}'A non-empty list means the wait is on a controller, not on the container
kubectl get nodeThe pod's node showing NotReady: the kubelet cannot confirm anything, and the pod's own STATUS may read Unknown instead of Terminating

How to know it is actually fixed

  • kubectl get pod <name> returns NotFound, rather than the pod merely disappearing from a filtered list.
  • If a controller owns the pod, the replacement reaches Running — a StatefulSet replacement in particular should start only after the original is genuinely gone.
  • Deleting a comparable pod now completes within its grace period instead of taking the full window every time.
  • Where a volume was involved, it detaches and the next pod mounts it without a FailedAttachVolume event (worded Multi-Attach error up to Kubernetes v1.36 and Waiting for detach from v1.37).

Stopping it happening again

  • Handle SIGTERM in the application and make sure it reaches PID 1. Most pods that always take the full grace period are failing this, quietly, long before anyone calls it a problem.
  • Set terminationGracePeriodSeconds from the workload's real drain time rather than leaving the 30-second default to decide it.
  • Treat --force --grace-period=0 as an incident action, not a habit. It removes the API object without confirming the container stopped, and the documentation warns the resource may continue to run indefinitely.
  • Before uninstalling an operator or CRD, check for finalizers it owns. A removed controller leaves keys nobody can clear, and the objects holding them become undeletable.
  • Prefer native sidecar containers over preStop hooks written to order shutdown — the kubelet handles that ordering itself, stopping sidecars in reverse order after the last main container exits.

Security considerations

  • On an unreachable node, --force --grace-period=0 frees the pod's name without stopping its containers. For a StatefulSet the controller can then start a second pod with the same identity while the first may still be writing, which the docs warn can cause split brain and data loss. Confirm the node is powered off, or delete the Node object once you know it is gone.
  • Remove a finalizer only after its cleanup has been done another way. The Finalizers page says to do it only when you understand the finalizer's purpose; skipping it can leave resources such as cloud disks or load balancers behind with nothing tracking them.
  • Add the node.kubernetes.io/out-of-service taint only after you have confirmed the node is shut down or powered off, not restarting. It force-deletes the node's pods and detaches their volumes at once, and the docs warn that straying from the procedure can corrupt data.
  • When a pod on an unhealthy node has not finished deleting for 6 minutes, Kubernetes force-detaches its volumes by default. If the workload is in fact still running there, the docs warn the volume may be corrupted, so power off or isolate a partitioned node rather than waiting it out.
  • Do not shorten the grace period just to make deletes feel faster. It cuts in-flight work, and for StatefulSet pods the docs call a grace period of 0 unsafe and strongly discouraged.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next