Skip to main content
DevOps Insightsguides & answers

Kubernetes · troubleshooting

CrashLoopBackOff

A container starts, exits, and Kubernetes restarts it — repeatedly. The loop itself is not the fault; it is the symptom of whatever makes the container exit.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

  • kubectl configured for the cluster and the pod's namespace. Kubernetes supports a kubectl within one minor version, older or newer, of the API server.
  • Permission to read pods, events and the pods/log subresource in that namespace, which kubectl logs --previous needs. The built-in view role grants all three.
  • The manifest or workload (Deployment, StatefulSet, Job) that owns the pod, so you can compare its command, environment, mounted ConfigMaps and Secrets, probes and memory limit with what the container did.
  • For a container with no shell, rights to use kubectl debug: access to the pod's ephemeralcontainers subresource, or permission to create pods for a --copy-to copy.
  • Rights to change the owning workload, because a fix goes into its pod template and rolls out new pods; editing the failing pod itself does not stick.

Run this first

step 1 of 3
kubectl describe pods <pod>

Look for Last State reason and exit code; probe failure events; missing-object events

The Kubernetes documentation names this as the primary debugging command. Events at the bottom usually name the failure outright, and Last State gives you the exit code.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

In kubectl describe pod, what does Last State say the container terminated with?

Look for the Last State block. The reason and exit code narrow this faster than anything else.

Cause space

5 of 5 still possible

  • The application exits on startupCommon
  • The container is being OOMKilledCommon
  • The liveness probe is failingCommon
  • A referenced ConfigMap, Secret or volume does not existOccasional
  • The image entrypoint or command is wrongOccasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    kubectl describe pods <pod>

    The Kubernetes documentation names this as the primary debugging command. Events at the bottom usually name the failure outright, and Last State gives you the exit code.

    Look for Last State reason and exit code; probe failure events; missing-object events

  2. Step 2

    kubectl logs <pod> --previous

    The container may already be running its next attempt, which has not failed yet. --previous shows the run that actually failed, if there is one.

    Look for The application's own error on the way down

  3. Step 3

    kubectl get events --sort-by=.lastTimestamp -n <namespace>

    Widens the view when the pod's own events are not enough — scheduling, image and node problems appear here.

    Look for Anything about this pod in the minutes before the first restart

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

The application exits on startup

Common

The most common case, and the least glamorous: the process itself fails. A missing environment variable, an unreachable database at boot, a config file that is not where the code expects, an unhandled exception during initialisation. The container is doing exactly what it was told; the program is wrong.

Confirm

kubectl logs <pod> --previous

A stack trace or fatal error from the run that just died. --previous is the important part: once the container has restarted and is running again, plain kubectl logs shows the new run, which may not have failed yet, while --previous always shows the last run that terminated.

Fix

  • Read the actual error. In most cases the container is telling you precisely what is missing.
  • Check that every environment variable and mounted file the process reads at startup exists in this namespace.
  • If it depends on another service being reachable at boot, either make the dependency optional with retries, or use an init container so the failure is reported honestly instead of as a crash loop.

The container is being OOMKilled

Common

The container exceeded its memory limit and the kernel terminated it. This presents as a crash loop because the restart succeeds, memory climbs again, and it is killed again. Reason OOMKilled is the signature. Exit code 137 on its own only says the process was killed with SIGKILL, which also happens when a container ignores the stop signal until its grace period runs out.

Confirm

kubectl describe pod <pod> | grep -A3 'Last State'

Reason: OOMKilled, and Exit Code: 137

Fix

  • Raise the memory limit if the workload genuinely needs more than it was given.
  • If usage grows without bound, this is a leak and raising the limit only lengthens the interval between kills.

The liveness probe is failing

Common

A liveness probe that fails causes the kubelet to kill and restart the container — by design. If the probe is stricter than the application's real startup time, Kubernetes kills a healthy process before it ever finishes booting, forever. This one is cruel because nothing is actually broken.

Confirm

kubectl describe pod <pod>

Events containing "Liveness probe failed". Compare initialDelaySeconds and the probe path against how long the app really takes to serve traffic.

Fix

  • Use a startup probe for slow-booting applications so the liveness probe only begins once the app is up.
  • Point the probe at an endpoint that means "this process is alive", not one that checks downstream dependencies — otherwise a database blip restarts your app.

A referenced ConfigMap, Secret or volume does not exist

Occasional

The pod references something absent from the namespace. A required ConfigMap or Secret that is missing stops the container from starting at all (CreateContainerConfigError or ContainerCreating, with an event naming the object), which is not a crash loop. It becomes one when the reference is optional, or a key the application reads is absent, and the application exits on startup.

Confirm

kubectl describe pod <pod>

Events mentioning a ConfigMap, Secret or volume that cannot be found

Fix

  • Create the missing object, or correct the name in the pod spec.
  • Confirm you are looking in the right namespace — this is very often a namespace mismatch rather than a missing object.

The image entrypoint or command is wrong

Occasional

A command that does not exist in the image, or one that runs to completion and exits 0. Kubernetes restarts a container that exits successfully too, if the restart policy says so — a one-shot script in a Deployment will crash-loop while behaving perfectly.

Confirm

kubectl logs <pod> --previous

No output at all with exit code 0 (the command finished). If the command itself is missing from the image, the error comes from the container runtime, not your program: look in kubectl describe pod for a Warning event with reason Failed whose message begins Error: and quotes the runtime's explanation.

Fix

  • Verify the command exists inside the image: docker run --rm --entrypoint sh <image> -c 'which <cmd>'.
  • If the workload is genuinely meant to run once and exit, it is a Job, not a Deployment.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Kubernetes keeps a container running by starting it again whenever it stops. If the container keeps stopping soon after it starts, Kubernetes waits a little longer before each new start and shows CrashLoopBackOff while it waits. The loop is only the symptom: the fault is whatever makes the container exit, and its exit code and last logs usually say what that is.

A container that exits as soon as it starts

Documented output, not reproduced: there was no cluster to run it on. The Debug Running Pods page on kubernetes.io starts a pod whose only command is false, which exits with status 1 straight away, using kubectl run --image=busybox:1.28 myapp -- false, and then shows this part of kubectl describe pod myapp. The ... line is in the original.

Command

kubectl describe pod myapp

Output · as shown in Kubernetes — Debug Running Pods (Copying a Pod while changing its command)

Containers:
  myapp:
    Image:         busybox
    ...
    Args:
      false
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1

How to read it

  • State: Waiting with Reason: CrashLoopBackOff means the container is not running at this moment. The kubelet is sitting out the back-off before the next start.
  • Last State: Terminated describes the run that just ended. Reason: Error with Exit Code: 1 says the process exited by itself with a failure, so look at the program and its configuration, not at memory or probes.
  • Other values point elsewhere: Reason: OOMKilled (usually with exit code 137) points at the memory limit, and exit code 0 means the command finished successfully and the restart policy started it again.
  • Next, run kubectl logs myapp --previous to read that run's output. For false it is empty, which is why the exit code and the command matter as much as the logs.
  • Further down, the Events list a BackOff warning. Older docs show its message as Back-off restarting failed container; the v1.37.1 kubelet adds the container and pod names after it.

Words used on this page

Back-off
The growing wait between restarts. By default the kubelet starts at 10 seconds and doubles it after each exit (10s, 20s, 40s and so on) up to 300 seconds, and resets it once a container has run for 10 minutes without trouble. Two feature gates change this when a cluster admin enables them: ReduceDefaultCrashLoopBackOffDecay (start at 1 second, cap at 60) and KubeletCrashLoopBackOffMax (a per-node cap set with crashLoopBackOff.maxContainerRestartPeriod).
restartPolicy
A pod setting with the values Always (the default), OnFailure and Never. Deployments allow only Always, so their containers are restarted even after a clean exit with code 0.
Waiting reason
A short word the kubelet sets on a container that is not running, such as CrashLoopBackOff or ImagePullBackOff. kubectl's STATUS column shows it; it is not a pod phase.
Last State
The block in kubectl describe pod about the container's previous run: its reason (such as Error or OOMKilled), exit code, and start and finish times. It holds only the most recent termination.
Exit code
The number the container's main process ended with. 0 means success. 137 is 128 + 9: the process was ended by signal 9, SIGKILL, which is what both the kernel's out-of-memory killer and a forced stop after the grace period send.
Liveness probe
A periodic health check the kubelet runs against a container. After failureThreshold failures in a row (3 by default) the kubelet kills the container and the restart policy applies, so a probe that is too strict causes a crash loop by itself.
Startup probe
A probe that holds off liveness and readiness checks until it succeeds once, giving a slow application time to boot. With failureThreshold: 30 and periodSeconds: 10 the application gets up to 300 seconds.
kubectl logs --previous
Prints the logs of the container's previous instance, if one exists. Use it to read the run that crashed rather than the one that has just started.
What is actually happening

Kubernetes restarted your container, it exited again, and the kubelet is now waiting before trying once more. Each failure increases that wait exponentially, so the pod spends most of its time deliberately doing nothing. The Kubernetes documentation gives the progression as 10s, 20s, 40s and so on, capped at five minutes by default — that cap is why a broken pod appears to stop changing.

It is worth being precise about one thing, because it changes where you look: CrashLoopBackOff is a value kubectl shows in the STATUS column, not a pod phase. The Kubernetes documentation explicitly warns against confusing the two — phase is part of the API, STATUS is a display field assembled for human intuition. There is no CrashLoopBackOff phase or pod condition: it is the reason on the container's waiting state (status.containerStatuses[*].state.waiting.reason), so you are reading a summary of container state.

So the useful question is never "how do I fix CrashLoopBackOff". It is "why did the container exit", and the answer is almost always in the previous container's logs or in the pod's events.

How to tell this is your problem
WhereWhat you see
kubectl get podsSTATUS shows CrashLoopBackOff and RESTARTS climbs
kubectl describe podEvents include "Back-off restarting failed container"
Last StateTerminated with a non-zero exit code, or with reason OOMKilled

How to know it is actually fixed

  • RESTARTS stops incrementing. Check twice, more than five minutes apart. The wait between restarts can reach 300 seconds, so a broken pod can look calm for several minutes.
  • kubectl get pod shows Running with all containers ready, not just Running.
  • kubectl logs <pod> (without --previous) shows the application's normal startup output.

Stopping it happening again

  • Use a startup probe wherever boot time is variable, so liveness never races initialisation.
  • Set memory requests and limits from observed usage, not from a round number.
  • Make startup dependencies retry rather than exit, so a transient outage does not become a crash loop.
  • Keep liveness probes checking only the process itself; check dependencies with readiness.

Security considerations

  • Applications that fail at startup often print their configuration on the way down, such as connection strings with passwords or API tokens in a stack trace. Read kubectl logs --previous output before pasting it into a ticket or chat, and redact it.
  • The built-in view role lets people read pods/log but not Secrets. Anything your application logs is therefore readable by people who are not allowed to see the Secrets themselves, so keep credentials out of startup errors.
  • kubectl debug --copy-to creates a second pod from the same spec, so it runs with the same service account and mounts the same Secrets. Delete the copy when you finish, as the Debug Running Pods page tells you to.
  • The entry's docker run --entrypoint sh <image> check runs the image on your own machine. Do that only with images you built or trust; otherwise debug a copy inside the cluster, where the cluster's own restrictions apply.
  • If a missing Secret caused the loop, create it from a file or your secret manager rather than typing values on the command line. The Kubernetes docs warn that secrets typed on the command line may be stored in your shell history.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Terms used here (1)

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Kubernetes
An open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications across clusters of machines. It works declaratively: you describe the state you want, and its controllers continuously work to move the cluster toward it.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next