Skip to main content
DevOps Insightsguides & answers

Kubernetes · troubleshooting

Pod stuck in Pending

The pod exists in the API but no node has accepted it. Nothing is running yet, so logs are empty — the answer is always in the scheduler's own explanation.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

  • kubectl configured for the cluster and the pod's namespace, within one minor version of the API server, which is the supported skew.
  • Permission to read pods, events and PersistentVolumeClaims in the namespace (the built-in view role grants these), and read access to nodes, which are cluster-scoped and need a ClusterRole.
  • The pod's resource requests, nodeSelector, affinity, tolerations and volumes, from its manifest or the owning workload's pod template.
  • For the fix, someone who can add nodes or resize node pools, or who manages the StorageClass and its provisioner, depending on what the scheduler reports.

Run this first

step 1 of 3
kubectl describe pods <pod>

Look for FailedScheduling, and the count in "0/N nodes are available" with the reason for each

The scheduler writes its refusal here, with a count of nodes for each reason. This single command usually contains the whole answer.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

What reason does the FailedScheduling event give?

In kubectl describe pod, look for the line beginning "0/N nodes are available".

Cause space

5 of 5 still possible

  • No node has enough free CPU or memoryCommon
  • Node selectors, affinity or taints exclude every nodeCommon
  • A PersistentVolumeClaim is unboundOccasional
  • No scheduler is handling the podRare
  • There are no schedulable nodesRare

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    kubectl describe pods <pod>

    The scheduler writes its refusal here, with a count of nodes for each reason. This single command usually contains the whole answer.

    Look for FailedScheduling, and the count in "0/N nodes are available" with the reason for each

  2. Step 2

    kubectl get nodes

    Rules out the case where nothing is schedulable at all, which would make the rest of the investigation moot.

    Look for Ready status, and SchedulingDisabled on any cordoned node

  3. Step 3

    kubectl describe nodes | grep -A5 'Allocated resources'

    Shows committed requests rather than live usage — the numbers the scheduler actually reasons about.

    Look for CPU or memory requests already near 100% of allocatable

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

No node has enough free CPU or memory

Common

The pod's requests — not its usage — exceed what any single node has unallocated. This catches people because a cluster can look busy at 40% actual utilisation while being fully committed on requests. The scheduler reasons about requests, not about what is really being used.

Confirm

kubectl describe pod <pod> | grep -A10 Events

"Insufficient cpu" or "Insufficient memory" in the FailedScheduling message

Fix

  • Lower the request if it was set higher than the workload genuinely needs.
  • Add capacity, or free it by right-sizing over-requested pods elsewhere.
  • Check the request is not accidentally enormous — a units mistake such as 1000 instead of 1000m asks for a thousand CPUs.
kubectl describe nodes | grep -A5 'Allocated resources'

Shows committed requests per node, which is what the scheduler actually weighs.

Node selectors, affinity or taints exclude every node

Common

The pod asks for nodes with a particular label, or the only suitable nodes carry a taint the pod does not tolerate. Capacity is irrelevant here — the scheduler has excluded those nodes from consideration entirely.

Confirm

kubectl describe pod <pod> | grep -i -E 'didn.t match|untolerated taint|node.selector|affinity'

"node(s) didn't match Pod's node affinity/selector" or "had untolerated taint"

Fix

  • Compare the pod's nodeSelector and affinity against real node labels: kubectl get nodes --show-labels.
  • Add the matching toleration, or remove the constraint if it was inherited from a copied manifest.
  • Control-plane nodes are tainted deliberately, and so are nodes with problems: the control plane adds taints such as node.kubernetes.io/not-ready, node.kubernetes.io/unreachable and node.kubernetes.io/memory-pressure from node conditions. If the taint starts with node.kubernetes.io/, fix the node instead of adding a toleration.

A PersistentVolumeClaim is unbound

Occasional

The pod mounts a PVC that has not been satisfied. No matching PersistentVolume exists, no StorageClass can provision one, or the volume is zone-bound to a zone with no suitable node. The pod waits on storage, not on compute.

Confirm

kubectl get pvc -n <namespace>

A PVC in Pending rather than Bound, when its StorageClass uses the default Immediate binding mode. With WaitForFirstConsumer, a PVC stays Pending until a pod that uses it is scheduled, so there a Pending PVC can be a result of the pod's problem rather than its cause.

Fix

  • Check the StorageClass named by the PVC exists and can provision dynamically.
  • For zonal volumes, confirm a node exists in the volume's zone.
  • Read the PVC's own events — they usually name the provisioning failure directly.

No scheduler is handling the pod

Rare

A pending pod waits for a scheduler to place it. If the pod names a schedulerName that is not running, or the default kube-scheduler is down, nothing records a FailedScheduling event and the pod simply waits.

Confirm

kubectl get pod <pod> -o jsonpath='{.spec.schedulerName}'; echo
kubectl get pods -n kube-system -l component=kube-scheduler

A scheduler name other than default-scheduler with nothing running to serve it, or no running kube-scheduler (the component=kube-scheduler label is the one kubeadm sets on its scheduler pod, so other installers may need a different selector; on a managed control plane it is not visible, so check the provider's status page)

Fix

  • Point the pod at a running scheduler, usually by removing a custom schedulerName.
  • Restore the scheduler if it is down.

There are no schedulable nodes

Rare

Every node is NotReady, cordoned, or the cluster genuinely has none. Obvious once seen, and worth ruling out early because it makes every other investigation pointless.

Confirm

kubectl get nodes

All nodes NotReady or SchedulingDisabled, or an empty list

Fix

  • Recover or uncordon the nodes: kubectl uncordon <node>.
  • If a cluster autoscaler should have added capacity, check its logs — it may be at its own maximum.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

When you create a pod, a part of Kubernetes called the scheduler picks a machine, a node, for it to run on. Pending means no node has been picked yet, or the pod is still waiting for something such as its storage. Nothing has started, so there are no logs to read; the scheduler's own message in the pod's events says what is missing.

Not enough CPU on any node

Documented output, not reproduced: there was no cluster to run it on. The Resource Management for Pods and Containers page on kubernetes.io shows the events of a pending pod named frontend in a 42-node cluster.

Command

kubectl describe pod frontend | grep -A 9999999999 Events

Output · as shown in Kubernetes — Resource Management for Pods and Containers (My Pods are pending with event message FailedScheduling)

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  23s   default-scheduler  0/42 nodes available: insufficient cpu

How to read it

  • FailedScheduling from default-scheduler means the scheduler tried and every node failed its checks. The grep only trims the output to the Events section.
  • 0/42 nodes available: insufficient cpu says none of the 42 nodes had enough unreserved CPU for the pod's request. The scheduler compares the request with each node's allocatable CPU minus what other pods have already requested, not with live usage.
  • The docs' sample is shortened. In the v1.37.1 scheduler the message begins 0/42 nodes are available: and then lists each reason with the number of nodes it applied to; the CPU reason is worded Insufficient cpu.
  • Next, run kubectl describe nodes and compare Allocatable with Allocated resources. In the same page's node example, 1800m of allocatable CPU with 680m already requested leaves room for a pod requesting at most 1.120 CPUs.
  • The scheduler records a new event each time it fails to place the pod, so the message keeps up as nodes are added or freed.

Words used on this page

Scheduler
The control-plane component that picks a node for each new pod. The default one is called default-scheduler in pod specs and events. A pod is scheduled only once in its lifetime.
FailedScheduling
The Warning event the scheduler records each time it fails to place a pod. Its message starts 0/N nodes are available: and lists each reason with the number of nodes it ruled out.
Requests
resources.requests for CPU and memory. The scheduler places a pod only on a node whose allocatable capacity, minus the requests of pods already there, covers them. Live usage does not count.
Allocatable
What a node offers to pods: its capacity minus what is reserved for system daemons. kubectl describe node shows it next to Capacity, with the committed requests under Allocated resources.
Taint and toleration
A taint on a node keeps pods away unless they carry a matching toleration. Kubernetes also adds taints by itself for node problems, such as node.kubernetes.io/not-ready and node.kubernetes.io/memory-pressure.
nodeSelector and node affinity
Pod rules that limit, by node label, which nodes the pod may run on. If no node carries the labels, no node qualifies.
PersistentVolumeClaim (PVC)
A pod's request for storage. With the default Immediate binding mode it is bound as soon as it is created; with WaitForFirstConsumer it stays Pending until a pod that uses it is scheduled.
Cordon
Marking a node unschedulable, for example with kubectl cordon. kubectl get nodes then shows SchedulingDisabled; pods already there keep running, new ones go elsewhere.
What is actually happening

A bare Pending in kubectl's STATUS column means the pod has been created but not yet placed on a node. (The Pending phase also covers scheduled pods still pulling images, which kubectl shows as ContainerCreating or ImagePullBackOff.) The Kubernetes documentation is direct about this: a Pending pod is one that cannot be scheduled. There is no container, so there are no logs to read and no exit code to interpret. Looking for either is the most common wasted first move. And if pods are missing entirely rather than Pending, look elsewhere: a ResourceQuota rejects them at creation, which shows as FailedCreate ... exceeded quota on the ReplicaSet (kubectl describe rs <name>).

What you do have is better. The scheduler records why it declined in the pod's events, as a FailedScheduling message that starts "0/3 nodes are available:" and then lists each reason with the number of nodes it ruled out. That message is the diagnosis; everything else is confirming it.

One nuance worth knowing: a pod can also sit Pending while waiting on a volume rather than a node. If the events mention a PersistentVolumeClaim, the scheduler is not the thing blocking you.

How to tell this is your problem
WhereWhat you see
kubectl get podsSTATUS Pending, 0/1 READY, and it does not change
kubectl describe podFailedScheduling events, typically "0/N nodes are available"
kubectl logsNo output at all — there is no container yet

How to know it is actually fixed

  • kubectl get pod shows the pod has left Pending and reached Running or ContainerCreating.
  • kubectl describe pod shows a Scheduled event naming the node it landed on.
  • If a PVC was involved, it now reports Bound rather than Pending.

Stopping it happening again

  • Set requests from measurement so the scheduler has an accurate picture and the cluster is not falsely full.
  • Watch committed requests, not just utilisation — a cluster can be fully committed while looking idle.
  • Keep node labels and pod selectors under review; copied manifests carry constraints that no longer apply.
  • Confirm StorageClasses can actually provision in the zones your nodes occupy.

Security considerations

  • Do not tolerate the control-plane taint just to get a pod scheduled. kubeadm keeps ordinary pods off control-plane nodes for security reasons, and removing or tolerating node-role.kubernetes.io/control-plane undoes that.
  • Setting nodeName to force a pod onto a node makes the scheduler ignore the pod and overrules nodeSelector and affinity. Use node affinity or nodeSelector instead, so placement rules still apply.
  • A pod with a high priority class can make the scheduler preempt (evict) lower-priority pods to fit. In shared clusters, limit who can use high-priority classes with a ResourceQuota scoped to PriorityClass.
  • Do not delete or loosen a namespace ResourceQuota to unblock a rollout. The docs note that anyone with full write access to a namespace can remove its quota, so keep that access narrow.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Terms used here (1)

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Pod
The smallest deployable unit in Kubernetes, consisting of one or more containers that share networking and storage. Pods are ephemeral by design — they can be created, destroyed, and replaced at any time by higher-level controllers like Deployments.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next