Skip to main content
DevOps Insightsguides & answers

Kubernetes · troubleshooting

Pod Evicted

The kubelet stopped the pod to protect its node, or because the pod went over its own ephemeral-storage limit. The pod is left behind with phase Failed, reason Evicted, and a message that names the resource. Read that message first: memory, node disk and the pod's own limit each have a different fix.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 9 sections

Before you start

  • kubectl access to the namespace, and permission to describe nodes, since most causes are visible only on the node.
  • For disk causes, shell access to the node or a way to run commands on it, such as kubectl debug node/<node> -it --image=busybox.
  • The workload's manifest, to compare its resource requests and limits with what it uses.

Run this first

step 1 of 4
kubectl get pods -A --field-selector=status.phase=Failed

Look for Which pods, which namespaces, and whether they share a node

Lists failed pods, evicted ones included. status.reason is not a supported field selector, so Evicted cannot be filtered on the server.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

What does the pod's status.message start with?

kubectl get pod <pod> -o jsonpath='{.status.message}'

Cause space

5 of 5 still possible

  • The node ran low on memoryCommon
  • The node ran low on disk space or inodesCommon
  • The pod went over its own ephemeral-storage limitOccasional
  • The node refused the pod because it was already under pressureRare
  • The node ran low on process IDsRare

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    kubectl get pods -A --field-selector=status.phase=Failed

    Lists failed pods, evicted ones included. status.reason is not a supported field selector, so Evicted cannot be filtered on the server.

    Look for Which pods, which namespaces, and whether they share a node

  2. Step 2

    kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.reason}{"\n"}{.status.message}{"\n"}{.spec.nodeName}{"\n"}'

    The message names the resource and, for memory, which containers used more than they requested.

    Look for Evicted, then low on resource: memory, ephemeral-storage, a limit, or The node had condition

  3. Step 3

    kubectl describe node <node>

    Shows the node's pressure conditions and how much of it is promised to pods.

    Look for Conditions with MemoryPressure, DiskPressure or PIDPressure, and Allocated resources

  4. Step 4

    kubectl get events -A --field-selector reason=Evicted

    Lists recent evictions across the cluster. Events are kept for one hour by default, so this only covers the last hour.

    Look for Several pods evicted from one node at about the same time, which points at the node rather than the pod

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

The node ran low on memory

Common

memory.available fell below the threshold, 100Mi by default on Linux. Pods using more memory than they requested are evicted first, so a pod with a low or missing memory request goes early even when another pod caused the pressure.

Confirm

kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.message}{"\n"}'

The node was low on resource: memory, and the per-container lines showing usage against request

Fix

  • Set memory requests close to real usage, so the scheduler does not overcommit the node and the pod is not first in line.
  • For pods that must not be evicted for others, set memory requests equal to limits.
  • Reserve memory for the operating system and the kubelet with system-reserved and kube-reserved, so pods cannot squeeze them.
  • Add nodes or larger nodes if requests honestly exceed capacity.

The node ran low on disk space or inodes

Common

nodefs or imagefs fell below its threshold: by default 10% free on nodefs, 15% on imagefs, or 5% of inodes. Images, container logs, writable layers and emptyDir volumes all share that disk.

Confirm

kubectl describe node <node> | sed -n '/Conditions:/,/Addresses:/p'
df -h; df -i

DiskPressure True, and on the node a filesystem near full or out of inodes. The pod's message says The node was low on resource: ephemeral-storage

Fix

  • Find what grew: images, logs, or a pod's emptyDir. The kubelet's image garbage collection already runs from 85% disk use, so persistent pressure usually means something else is growing.
  • Give pods ephemeral-storage requests and limits, and emptyDir volumes a sizeLimit, so one pod cannot fill the node.
  • Keep container log rotation on: the kubelet defaults are containerLogMaxSize 10Mi and containerLogMaxFiles 5. Logs written by the system outside the kubelet are not rotated by Kubernetes.
  • Use larger node disks if the steady state really needs them.
crictl images
sudo crictl rmi --prune

On the node. --prune removes images no container uses; pinned images are kept. The kubelet's own image garbage collection does this under pressure, so this is a stopgap.

The pod went over its own ephemeral-storage limit

Occasional

The node was fine. A container's writable layer and logs exceeded its ephemeral-storage limit, the containers together exceeded the pod's total, or an emptyDir exceeded its sizeLimit, and the kubelet evicted only this pod.

Confirm

kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.message}{"\n"}'

exceeded its local ephemeral storage limit, Pod ephemeral local storage usage exceeds the total limit of containers, or Usage of EmptyDir volume … exceeds the limit

Fix

  • Find what the container writes locally: files under its own filesystem, a temporary directory, or very verbose logs.
  • Write large data to a volume meant for it, or raise the limit to a measured figure.

The node refused the pod because it was already under pressure

Rare

The kubelet rejects new pods while it has a pressure condition: under memory pressure only BestEffort pods that do not tolerate it, under any other condition all non-critical pods. These pods are also marked Evicted, but they never ran there.

Confirm

kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.message}{"\n"}'

A message starting The node had condition:

Fix

  • Fix the node's pressure, using the memory or disk cause above.
  • Check how the pod reached a node the scheduler normally avoids, for example a nodeName set in the spec or a toleration for the pressure taint.

The node ran low on process IDs

Rare

pid.available crossed its threshold, usually because one workload forks without limit. The kubelet ranks pods for PID pressure by priority, then by process count, since PIDs have no requests.

Confirm

kubectl describe node <node> | grep -i pidpressure

PIDPressure True

Fix

  • Find the pod with the most processes on that node and fix its process or thread leak.
  • Set a per-pod PID limit in the kubelet configuration, podPidsLimit, so one pod cannot exhaust the node.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Every node keeps a safety margin of free memory and disk. When the margin runs out, the kubelet, the agent on the node, picks pods to stop so the node itself survives, and marks them Evicted. It also stops a pod that writes more to local disk than its limit allows. The stopped pod stays listed with a message saying which resource ran short, and that message tells you what to fix.

Words used on this page

Node-pressure eviction
The kubelet stopping pods because a node resource crossed an eviction threshold. It does not respect PodDisruptionBudgets, and with the default hard thresholds it gives no grace period.
Eviction threshold
The free-resource level that triggers eviction. Linux defaults are memory.available<100Mi, nodefs.available<10%, imagefs.available<15%, and nodefs.inodesFree<5% and imagefs.inodesFree<5%.
nodefs and imagefs
The node's main filesystem, which holds the kubelet's directory, and the filesystem where the container runtime keeps images and writable layers. They may be the same disk.
Ephemeral storage
Local disk a pod uses that disappears with it: container writable layers, container logs and emptyDir volumes. Pods can request it and be limited on it like CPU and memory.
status.reason
A pod-level field. Evicted pods have reason Evicted and phase Failed. kubectl's STATUS column can show a container's reason instead, so read this field directly.
What is actually happening

Node-pressure eviction is the kubelet protecting the node. When a signal such as memory.available or nodefs.available crosses its threshold, the kubelet sets the chosen pod's phase to Failed, reason Evicted, and terminates it. It does not respect PodDisruptionBudgets or the pod's terminationGracePeriodSeconds, and with the default hard thresholds it allows no grace period at all.

The kubelet ranks pods first by whether their usage exceeds their requests, then by priority, then by how far usage exceeds requests. Pods using less than they requested go last. The Kubernetes documentation notes that the kubelet does not use QoS class to choose, although the outcome often matches it.

The message carries the evidence. Since Kubernetes 1.26 it reads The node was low on resource: memory, then Threshold quantity and available, then one line per container using more than its request, such as Container app was using …, request is …, has larger consumption of memory. Older posts quote which exceeds its request of, the format before 1.26.

Three things look similar and are not this. A container killed by the kernel's out-of-memory killer is OOMKilled, a container state, and can be restarted. A pod removed by kubectl drain, preemption or a NoExecute taint is deleted, not left as Evicted. Only the kubelet's own eviction leaves a Failed pod with reason Evicted.

Evicted pods are not cleaned up quickly. They stay until someone deletes them, or until the cluster holds more than 12,500 terminated pods, kube-controller-manager's default garbage-collection threshold, which deletes Evicted pods first. A Deployment or StatefulSet has already created a replacement.

How to tell this is your problem
WhereWhat you see
kubectl get pod -o jsonpath='{.status.reason}'Evicted
status.message, memoryThe node was low on resource: memory. Threshold quantity: …, available: …. Container … was using …, request is …, has larger consumption of memory.
status.message, the pod's own limitContainer … exceeded its local ephemeral storage limit "…".
status.message, refused at admissionThe node had condition: …
kubectl describe node, ConditionsMemoryPressure, DiskPressure or PIDPressure True
kubectl get pods STATUS columnEvicted, or a container reason such as Error, for pods with phase Failed

How to know it is actually fixed

  • kubectl describe node shows MemoryPressure, DiskPressure and PIDPressure False.
  • The workload's replacement pods stay Running, and no new pods with reason Evicted appear over a full cycle of the workload's normal load.
  • Once you have saved what you need, delete the old evicted pods. kubectl delete pods -n <namespace> --field-selector=status.phase=Failed deletes every failed pod there, including failed Job pods and their logs, so list them first.

Stopping it happening again

  • Set memory requests from measured usage, and ephemeral-storage requests and limits, on every workload.
  • Alert on node conditions and on free disk and memory before they reach the eviction thresholds, not on the evictions themselves.
  • If you change eviction thresholds, set mergeDefaultEvictionSettings: true. The documentation warns that setting one evictionHard value otherwise sets the others to zero.

Security considerations

  • A pod that writes without bound to its filesystem or logs can push a node into DiskPressure and get other teams' pods evicted. Give every workload ephemeral-storage limits, and cap each namespace's total with a ResourceQuota on requests.ephemeral-storage and limits.ephemeral-storage.
  • Pod priority decides who is evicted first among pods using more than they requested. Restrict which namespaces may use high PriorityClasses, with a ResourceQuota scopeSelector on PriorityClass, since anyone who can use them can shift evictions onto other workloads.
  • Before deleting evicted pods, save their status.message. The pod object is the lasting evidence; the eviction events expire after an hour by default.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next