Skip to main content
DevOps Insightsguides & answers

Kubernetes · troubleshooting

OOMKilled (exit code 137)

The kernel killed your container because memory ran out: usually the container went over its own limit, sometimes the whole node ran short. Reason OOMKilled is the signature, and the question is whether the limit is too low, the application is leaking, a spike hit, or the node was overcommitted.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

  • kubectl configured for the cluster and the pod's namespace, within one minor version of the API server, which is the supported skew.
  • Permission to read pods, events and pods/log in the namespace (the built-in view role grants these), and read access to nodes, which are cluster-scoped and need a ClusterRole, to check node memory.
  • A source of resource metrics for kubectl top, such as metrics-server. Without one kubectl top pod returns no data, and new pods can show none for the first few minutes.
  • Rights to change the workload that owns the pod (Deployment, StatefulSet, Job). New requests and limits go into its pod template and roll out new pods.
  • Knowledge of any ResourceQuota in the namespace: a quota on limits.memory caps the sum of memory limits, so raising a limit can fail until the quota is raised.

Run this first

step 1 of 3
kubectl describe pod <pod>

Look for Last State: Terminated · Reason: OOMKilled · Exit Code: 137

Confirms it really was an OOM kill rather than another non-zero exit. The kernel does the killing; the container runtime records the reason OOMKilled and the kubelet reports it with the exit code.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Sample kubectl top pod a few times over several minutes. What does memory do?

One reading cannot show a trend. Take at least three under normal traffic.

Cause space

4 of 4 still possible

  • The limit is below what the workload actually needsCommon
  • The application leaks memoryCommon
  • A workload spike, not a steady-state problemOccasional
  • The node ran out of memory, not the containerRare

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    kubectl describe pod <pod>

    Confirms it really was an OOM kill rather than another non-zero exit. The kernel does the killing; the container runtime records the reason OOMKilled and the kubelet reports it with the exit code.

    Look for Last State: Terminated · Reason: OOMKilled · Exit Code: 137

  2. Step 2

    kubectl top pod <pod> --containers

    Separates "limit too low" from "leaking". Sample several times over a few minutes rather than once — one reading cannot show a trend.

    Look for Steady usage near the limit, versus usage climbing monotonically

  3. Step 3

    kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources}'

    Shows what the limit and request actually are, which is often not what the author believes.

    Look for A limit far below observed usage, or a request with no limit at all

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

The limit is below what the workload actually needs

Common

The most common case. The limit was set from a guess, a copied manifest or a tidy round number, and the application's real steady-state usage is higher. Nothing is broken; the ceiling is simply in the wrong place.

Confirm

kubectl top pod <pod> --containers

Usage sitting close to the limit even when the app is behaving. If steady-state usage is already near the ceiling, the limit is the problem.

Fix

  • Observe usage over a normal cycle, including any startup spike, then set the limit above the observed peak with genuine headroom.
  • Set the request to typical usage and the limit above peak. Requests drive scheduling; limits are the kill line.
kubectl set resources deployment <name> --limits=memory=512Mi --requests=memory=256Mi

Illustrative numbers only — use your own measurements. This triggers a rollout.

The application leaks memory

Common

Usage climbs steadily from start until the limit is hit, then the cycle repeats with almost mechanical regularity. The give-away is the rhythm: kills at consistent intervals under consistent load. Raising the limit here only lengthens the interval — it does not fix anything.

Confirm

kubectl top pod <pod> --containers

Sample repeatedly over several minutes. Monotonic growth under steady traffic is a leak; usage that rises and falls with load is not.

Fix

  • Treat it as an application bug, not a configuration one. A higher limit buys time to investigate, nothing more.
  • Check for unbounded caches, accumulating connections, or a runtime heap allowed to grow past the container limit.
  • For JVM and Node workloads, make sure the runtime knows its own ceiling — a heap sized independently of the container limit can grow past it.

A workload spike, not a steady-state problem

Occasional

Usage is fine until a large request, a bulk import or a batch job briefly needs far more. The container is killed at the peak and looks healthy immediately afterwards, which makes this the hardest variant to catch — by the time you look, everything is normal.

Confirm

kubectl describe pod <pod> | grep -B2 -A5 'Last State'

The Started and Finished times of the most recent kill, and Restart Count. Last State holds only the latest termination, so note the time of each kill as it happens to tell a one-off or kills at busy times from a steady cadence, then correlate those times with what the system was doing.

Fix

  • Size the limit for the peak the workload must survive, not the average.
  • If the spike comes from batch work, isolate it into a Job with its own limits instead of sizing the whole service around it.

The node ran out of memory, not the container

Rare

Distinct from a container hitting its own limit: the node ran out of memory before the kubelet could evict anything, and the kernel's OOM killer chose this container, going first for BestEffort pods and those far above their requests. Pods the kubelet evicts show as Evicted, not OOMKilled. The remedy is capacity or scheduling, not the container's limit.

Confirm

kubectl describe node <node>

A Warning event with reason SystemOOM ("System OOM encountered") around the time of the kill. MemoryPressure can still read False, because the kubelet samples memory at intervals and the kernel can act first. Eviction events mean the kubelet evicted pods, which then show Evicted rather than OOMKilled.

Fix

  • Set requests accurately so the scheduler stops overcommitting the node.
  • Add capacity, or move the workload to a node that can hold it.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

You can give each container a memory limit. If the container uses more than its limit, or the machine it runs on runs out of memory, the Linux kernel stops it instantly and Kubernetes reports OOMKilled, short for out-of-memory killed, usually with exit code 137. Your job is to find out whether the limit is too low, the program keeps growing, or something else used up the memory.

A container killed at its memory limit

Documented output, not reproduced: there was no cluster to run it on. The Resource Management for Pods and Containers page on kubernetes.io runs kubectl describe pod on a pod whose container has a 50Mi memory limit. Shown here is the Containers section of that output, quoted verbatim; the lines above and below it are left out.

Command

kubectl describe pod simmemleak-hra99

Output · as shown in Kubernetes — Resource Management for Pods and Containers (My container is terminated)

Containers:
  simmemleak:
    Image:  saadali/simmemleak:latest
    Limits:
      cpu:          100m
      memory:       50Mi
    State:          Running
      Started:      Tue, 07 Jul 2019 12:54:41 -0700
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Fri, 07 Jul 2019 12:54:30 -0700
      Finished:     Fri, 07 Jul 2019 12:54:33 -0700
    Ready:          False
    Restart Count:  5

How to read it

  • Limits: memory: 50Mi is the ceiling the kernel enforces. Compare it with what the application actually uses at its peak, not with what the manifest author expected.
  • Last State: Terminated with Reason: OOMKilled and Exit Code: 137 is the run that was killed. The reason is what proves it was memory; exit code 137 alone only says the process got SIGKILL.
  • State: Running with Ready: False shows the container has already been started again: the kubelet restarts an OOM-killed container according to its restart policy.
  • Restart Count: 5 means five terminations so far, but Last State keeps only the latest. To tell a steady rhythm (a leak) from kills at busy times (a spike), note the time of each restart as it happens.
  • Right after a kill, kubectl get pod can show OOMKilled in the STATUS column, then Running again after the restart, as the Assign Memory Resources page shows.

Words used on this page

Memory limit
resources.limits.memory, the most memory a container may use. The kernel enforces it reactively: a container can briefly go over, but if it does and the kernel detects memory pressure, the container can be killed.
Memory request
resources.requests.memory, the amount the scheduler reserves for the container on a node. It decides where the pod can run; it is not the line at which the container's own limit kills it.
OOM killer
The part of the Linux kernel that ends a process when memory runs out, either inside one container's limit or across the whole node.
Exit code 137
128 + 9: the process was ended by signal 9, SIGKILL. A program cannot catch that signal, so it gets no chance to log or clean up. An OOM kill produces 137, and so does a forced stop after a grace period, so read the code together with the reason.
Last State
The block in kubectl describe pod about the container's previous run. It holds only the most recent termination: reason, exit code, and start and finish times.
QoS class
Guaranteed, Burstable or BestEffort, set from a pod's requests and limits. When the whole node runs out of memory, the kernel is most likely to kill containers in BestEffort pods and those using far more memory than they requested.
Node-pressure eviction
The kubelet stopping pods on purpose when a node runs low on memory. Evicted pods are marked Failed with reason Evicted, unlike an OOM-killed container, which the kubelet can restart in place.
What is actually happening

A container's memory limit is enforced by the kernel, not by Kubernetes politely asking. When usage goes over the limit and the kernel cannot reclaim enough memory, the OOM killer sends SIGKILL — no graceful shutdown, no chance to flush anything. Exit code 137 is 128 + 9, meaning killed by SIGKILL.

This matters for interpretation: the container did not choose to exit and had no opportunity to explain itself. Its logs will usually just stop mid-sentence. Looking for a crash message wastes time — the evidence is in the pod's Last State, not the application output.

There is one distinction worth holding onto. Memory requests affect scheduling; memory limits are what get you killed. A pod can be scheduled comfortably and still be OOMKilled seconds later if the limit is below what the workload actually needs.

How to tell this is your problem
WhereWhat you see
kubectl describe podLast State: Terminated, Reason: OOMKilled, Exit Code: 137
kubectl get podsOften CrashLoopBackOff, because the restart succeeds and it is killed again
Container logsOutput that simply stops, with no error — SIGKILL leaves no message

How to know it is actually fixed

  • No new OOMKilled entries in Last State after a full load cycle, including whatever peak originally triggered it.
  • kubectl top pod shows steady-state usage with real headroom below the limit, not brushing against it.
  • If you suspected a leak: usage plateaus instead of climbing. If it still climbs, the limit change only postponed the kill.

Stopping it happening again

  • Set requests and limits from measurement. A round number is a guess wearing a suit.
  • Tell language runtimes about the container limit so their heaps stay inside it.
  • Alert on memory usage approaching the limit, so you learn about this before the kernel decides for you.
  • Keep batch work in Jobs with their own limits rather than inflating a long-running service.

Security considerations

  • Removing the memory limit to stop the kills moves the risk to everyone else on the node: a container with no limit can use all of the node's memory and push the kernel into killing other workloads.
  • Heap dumps and core dumps taken to chase a leak contain whatever was in memory, including tokens, passwords and customer data. Store and share them like secrets, and never attach them to a public issue.
  • Give critical pods the Guaranteed QoS class, with requests equal to limits for CPU and memory in every container. If the node runs out of memory, the kubelet's oom_score_adj of -997 for them, against 1000 for BestEffort, makes the kernel pick them last.
  • If a ResourceQuota blocks a higher limit, raise the quota through its owner rather than moving the workload to a namespace without one. Anyone with full write access to a namespace can delete its quota, so keep that access narrow.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next