Skip to main content
DevOps Insightsguides & answers

Kubernetes · troubleshooting

ImagePullBackOff

The kubelet cannot retrieve the container image, and is now backing off between attempts. Almost always the name, the credentials, or the network — in that order of likelihood.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

  • kubectl configured for the cluster and the pod's namespace, within one minor version of the API server, which is the supported skew.
  • Permission to read pods and events in the namespace. Checking pull credentials also needs read access to Secrets there, which the built-in view role deliberately leaves out.
  • The full image reference the pod uses, and a way to see the registry yourself (a login or its web console) to confirm the repository and tag exist.
  • For network problems, shell access to a node or help from someone who has it: what matters is whether the node can reach the registry, not your laptop.
  • For a private image, registry credentials with pull-only scope to put in an image pull Secret, or a kubelet credential provider set up by the cluster admin.

Run this first

step 1 of 3
kubectl describe pods <pod>

Look for The exact text after "Failed to pull image"

The kubelet's Warning event (Failed to pull image ...) carries the container runtime's error, which usually includes the registry's reply. It separates not-found, authentication, network and rate-limit failures; telling a wrong name from a missing registry host also needs the image reference.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

In kubectl describe pod, what does the error after "Failed to pull image" actually say?

Quote the text after the image name exactly. It separates not-found, authentication, network and rate-limit failures; for not-found, the next question checks for a missing registry host.

Cause space

5 of 5 still possible

  • The image name or tag is wrongCommon
  • Missing or wrong registry credentialsCommon
  • The registry is implied rather than statedOccasional
  • The node cannot reach the registryOccasional
  • Registry rate limitingRare

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    kubectl describe pods <pod>

    The kubelet's Warning event (Failed to pull image ...) carries the container runtime's error, which usually includes the registry's reply. It separates not-found, authentication, network and rate-limit failures; telling a wrong name from a missing registry host also needs the image reference.

    Look for The exact text after "Failed to pull image"

  2. Step 2

    kubectl get pod <pod> -o jsonpath='{.spec.containers[*].image}'

    Shows the reference as the cluster sees it, which is not always what the manifest appears to say after templating.

    Look for A missing registry host, a wrong tag, or a typo

  3. Step 3

    kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets}'

    Confirms whether credentials are attached at all before you go looking at the registry.

    Look for Empty output for a private image, or a Secret absent from this namespace

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

The image name or tag is wrong

Common

A typo, a tag that was never pushed, or a tag deleted from the registry. The registry answers correctly — it simply has nothing by that name. "manifest unknown" and "not found" both point here.

Confirm

kubectl describe pod <pod> | grep -A5 Events

"manifest unknown", "manifest for ... not found", or "repository does not exist"

Fix

  • Read the image reference back character by character, including the tag.
  • Confirm the tag actually exists in the registry rather than assuming CI pushed it.
  • Avoid :latest — it makes this failure mode ambiguous and non-reproducible.
docker manifest inspect <image>:<tag>

Run from a machine with registry access. Succeeds if the reference resolves.

Missing or wrong registry credentials

Common

The image is private and the kubelet has no usable credentials for it. The imagePullSecrets field is absent, references a Secret that does not exist in this namespace, or holds credentials that have expired. Pull secrets are namespaced, which is the detail that catches people — a Secret that works in one namespace does nothing in another.

Confirm

kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets}'

Empty output for a private image, or a Secret name that kubectl get secret cannot find in this namespace

Fix

  • Create a docker-registry Secret in the same namespace as the pod and reference it from imagePullSecrets.
  • Check the credentials have not expired — many registry tokens are short-lived.
  • For cloud registries, prefer the provider's workload identity over long-lived pull secrets.
kubectl create secret docker-registry regcred --docker-server=<registry> --docker-username=<user> --docker-password=<token> -n <namespace>

The registry is implied rather than stated

Occasional

An image reference with no registry host defaults to Docker Hub. If the image actually lives in a private or cloud registry, the kubelet dutifully asks the wrong place and is told the image does not exist. The error looks like a missing image; the cause is a missing hostname.

Confirm

kubectl get pod <pod> -o jsonpath='{.spec.containers[*].image}'

A reference with no registry host, when the image is not on Docker Hub

Fix

  • Fully qualify the reference, including the registry host.
  • Prefer digests over tags where reproducibility matters.

The node cannot reach the registry

Occasional

DNS cannot resolve the registry, a firewall or egress policy blocks it, or a required proxy is not configured on the node. Distinguished from the other causes by its error shape: a timeout or connection failure rather than a clear registry response.

Confirm

kubectl describe pod <pod> | grep -i -E 'timeout|no such host|connection refused|i/o timeout'

A network-level error rather than an authentication or not-found response

Fix

  • Test connectivity from the node itself, not from your laptop — the node's network is what matters.
  • Check egress rules, NAT and any proxy configuration the container runtime needs.
  • If DNS is the issue, confirm the node can resolve the registry hostname.

Registry rate limiting

Rare

Public registries limit anonymous pulls. A busy cluster pulling unauthenticated can exhaust the allowance, and pulls that previously worked start failing — which makes this look like an intermittent network problem.

Confirm

kubectl describe pod <pod> | grep -i -E 'rate limit|toomanyrequests'

"toomanyrequests" or an explicit rate-limit message

Fix

  • Authenticate pulls even for public images. On Docker Hub, unauthenticated pulls are limited to 100 per 6 hours per IPv4 address or IPv6 /64 subnet, shared by every node behind that address; a Personal account gets 200, and paid plans are unlimited.
  • Mirror or cache images you depend on rather than pulling from upstream on every node.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Before a container can start, the node it is scheduled on has to download the container's image from a registry. If that download fails, Kubernetes waits and tries again, waiting longer each time, and shows ImagePullBackOff while it waits. The usual causes are a wrong image name or tag, missing or wrong credentials for a private registry, or a node that cannot reach the registry.

A tag typo during a rollout

Documented output, not reproduced: there was no cluster to run it on. On the Deployments page on kubernetes.io, the image of a three-replica Deployment is changed with a typo, nginx:1.161 instead of nginx:1.16.1, using kubectl set image deployment/nginx-deployment nginx=nginx:1.161, and then the pods are listed.

Command

kubectl get pods

Output · as shown in Kubernetes — Deployments (Rolling Back a Deployment)

NAME                                READY     STATUS             RESTARTS   AGE
nginx-deployment-1564180365-70iae   1/1       Running            0          25s
nginx-deployment-1564180365-jbqqo   1/1       Running            0          25s
nginx-deployment-1564180365-hysrc   1/1       Running            0          25s
nginx-deployment-3066724191-08mng   0/1       ImagePullBackOff   0          6s

How to read it

  • Only the pod from the new ReplicaSet (3066724191) is stuck. The three pods from the old ReplicaSet keep serving, because the Deployment controller stops a bad rollout instead of replacing every pod; how far it gets depends on maxUnavailable, 25% by default.
  • READY 0/1 with RESTARTS 0 means the container never started. There is nothing to restart and no logs to read, which is how you tell this apart from CrashLoopBackOff, where RESTARTS climbs.
  • The table does not say why. Run kubectl describe pod nginx-deployment-3066724191-08mng: in the v1.37.1 kubelet the failure is a Warning event with reason Failed whose message starts Failed to pull image "nginx:1.161": and then carries the container runtime's error.
  • To recover, the same page rolls back with kubectl rollout undo deployment/nginx-deployment; setting the correct tag works too. Either way the stuck pod is replaced.

Words used on this page

Image reference
The image value in the pod spec, such as registry.example/team/app:1.4. Without a registry host, Kubernetes assumes the Docker public registry (unless the container runtime is configured otherwise); without a tag, it assumes latest.
Tag and digest
A tag such as 1.16.1 is a label the registry can move to another image. A digest such as sha256:... is a hash of the content and never changes, so pinning one makes every node run the same image.
ErrImagePull
The container's waiting reason right after a pull attempt fails. The kubelet records the failure and starts backing off.
ImagePullBackOff
The waiting reason while the kubelet holds off before the next pull attempt. The delay starts at 10 seconds and grows to a compiled-in limit of 300 seconds.
imagePullSecrets
A list in the pod spec naming Secrets of type kubernetes.io/dockerconfigjson (or the older kubernetes.io/dockercfg) that hold registry credentials. The Secrets must be in the same namespace as the pod.
imagePullPolicy
When the kubelet pulls. IfNotPresent skips the pull if the node already has the image and is the default for a tag other than latest or a digest; Always asks the registry every time; Never never pulls.
Kubelet credential provider
A plugin the kubelet runs to fetch short-lived registry credentials, for example from a cloud provider, instead of storing them in Secrets.
What is actually happening

Before a container can run, the kubelet has to fetch its image. Pulling an image means resolving the name to a registry, reaching that registry over the network, authenticating if it is private, and finding the requested tag. ImagePullBackOff means one of those four failed and the kubelet is waiting before retrying.

You will often see ErrImagePull first and ImagePullBackOff a moment later. They are the same problem at different stages: the first is the failed attempt, the second is the wait that follows.

The Kubernetes documentation treats this as a case of a pod stuck Waiting — scheduled onto a node, but unable to start. That framing is useful, because it tells you scheduling already succeeded and the fault lies entirely in image retrieval.

How to tell this is your problem
WhereWhat you see
kubectl get podsSTATUS shows ImagePullBackOff or ErrImagePull, 0 restarts
kubectl describe podEvents with "Failed to pull image" and a registry error
The error text"manifest unknown", "unauthorized", "not found", or a timeout

How to know it is actually fixed

  • kubectl describe pod shows a "Successfully pulled image" event.
  • The pod moves past Waiting into Running.
  • Make sure the event says Successfully pulled image, not already present on machine. With IfNotPresent, a node that already has the image does not pull at all, so the fix is proven only once a node without the image pulls it.

Stopping it happening again

  • Pin images by digest, or at least by an immutable tag. Never :latest in production.
  • Keep pull secrets in every namespace that needs them, and rotate before expiry.
  • Authenticate even public pulls, so rate limits do not surprise you during an incident.
  • Have CI verify the image is pullable before the manifest referencing it is deployed.

Security considerations

  • kubectl create secret docker-registry ... --docker-password=<token> puts the token in your shell history, and the Kubernetes docs warn it may be visible to other users of the machine while kubectl runs. Prefer --from-file with an existing Docker config file, or a credential provider.
  • An image pull Secret is only base64-encoded: anyone who can read Secrets in the namespace can decode the registry password, as the Pull an Image from a Private Registry page shows step by step. Use a pull-only token and keep get and list on Secrets narrow.
  • Once a private image is on a node, other pods on that node may be able to start it without credentials. The AlwaysPullImages admission controller, or the kubelet's image pull credential verification (KubeletEnsureSecretPulledImages, beta and on by default since v1.35), makes each pod prove access.
  • Fixing a "not found" by trying similar names is risky: a name without a registry host resolves to Docker's public registry, so a typo can match an image someone else published. Use the full registry host and pin a digest.
  • Prefer short-lived credentials from a kubelet credential provider, including service account tokens for image pulls (beta since v1.34), over long-lived passwords stored in Secrets.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Terms used here (2)

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Container Registry
A service for storing and distributing container images. A registry holds many repositories, each a collection of related images. Registries like Docker Hub, GitHub Container Registry, and Amazon ECR allow teams to push, pull, and version their images as part of the CI/CD pipeline.
Namespace
A mechanism in Kubernetes for isolating groups of resources within a single cluster. Namespaces allow multiple teams or environments to coexist on the same cluster while providing logical boundaries for access control and resource quotas.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next