Skip to main content
DevOps Insightsguides & answers

Kubernetes · troubleshooting

OOMKilled (exit code 137)

The kernel terminated your container for exceeding its memory limit. Exit code 137 is the signature, and the only question that matters is whether the limit is too low or the application is leaking.

Run this first

step 1 of 3
kubectl describe pod <pod>

Look for Last State: Terminated · Reason: OOMKilled · Exit Code: 137

Confirms it really was an OOM kill rather than another non-zero exit. Reason and exit code come from the kernel.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Sample kubectl top pod a few times over several minutes. What does memory do?

One reading cannot show a trend. Take at least three under normal traffic.

Cause space

4 of 4 still possible

  • The limit is below what the workload actually needsCommon
  • The application leaks memoryCommon
  • A workload spike, not a steady-state problemOccasional
  • The node ran out of memory, not the containerRare

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    kubectl describe pod <pod>

    Confirms it really was an OOM kill rather than another non-zero exit. Reason and exit code come from the kernel.

    Look for Last State: Terminated · Reason: OOMKilled · Exit Code: 137

  2. Step 2

    kubectl top pod <pod> --containers

    Separates "limit too low" from "leaking". Sample several times over a few minutes rather than once — one reading cannot show a trend.

    Look for Steady usage near the limit, versus usage climbing monotonically

  3. Step 3

    kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources}'

    Shows what the limit and request actually are, which is often not what the author believes.

    Look for A limit far below observed usage, or a request with no limit at all

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

The limit is below what the workload actually needs

Common

The most common case. The limit was set from a guess, a copied manifest or a tidy round number, and the application's real steady-state usage is higher. Nothing is broken; the ceiling is simply in the wrong place.

Confirm

kubectl top pod <pod> --containers

Usage sitting close to the limit even when the app is behaving. If steady-state usage is already near the ceiling, the limit is the problem.

Fix

  • Observe usage over a normal cycle, including any startup spike, then set the limit above the observed peak with genuine headroom.
  • Set the request to typical usage and the limit above peak. Requests drive scheduling; limits are the kill line.
kubectl set resources deployment <name> --limits=memory=512Mi --requests=memory=256Mi

Illustrative numbers only — use your own measurements. This triggers a rollout.

The application leaks memory

Common

Usage climbs steadily from start until the limit is hit, then the cycle repeats with almost mechanical regularity. The give-away is the rhythm: kills at consistent intervals under consistent load. Raising the limit here only lengthens the interval — it does not fix anything.

Confirm

kubectl top pod <pod> --containers

Sample repeatedly over several minutes. Monotonic growth under steady traffic is a leak; usage that rises and falls with load is not.

Fix

  • Treat it as an application bug, not a configuration one. A higher limit buys time to investigate, nothing more.
  • Check for unbounded caches, accumulating connections, or a runtime heap allowed to grow past the container limit.
  • For JVM and Node workloads, make sure the runtime knows its own ceiling — a heap sized independently of the container limit will exceed it.

A workload spike, not a steady-state problem

Occasional

Usage is fine until a large request, a bulk import or a batch job briefly needs far more. The container is killed at the peak and looks healthy immediately afterwards, which makes this the hardest variant to catch — by the time you look, everything is normal.

Confirm

kubectl describe pod <pod> | grep -B2 -A5 'Last State'

A single kill, or kills clustered at particular times, rather than a steady cadence. Correlate the timestamp with what the system was doing.

Fix

  • Size the limit for the peak the workload must survive, not the average.
  • If the spike comes from batch work, isolate it into a Job with its own limits instead of sizing the whole service around it.

The node ran out of memory, not the container

Rare

Distinct from a container hitting its own limit: the node itself came under memory pressure and the kubelet evicted pods to protect it. The remedy is capacity or scheduling, not the container's limit.

Confirm

kubectl describe node <node> | grep -i pressure

MemoryPressure True, and eviction events referencing your pod

Fix

  • Set requests accurately so the scheduler stops overcommitting the node.
  • Add capacity, or move the workload to a node that can hold it.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

What is actually happening

A container's memory limit is enforced by the kernel, not by Kubernetes politely asking. When usage crosses the limit the OOM killer terminates the process immediately — no graceful shutdown, no chance to flush anything. Exit code 137 is 128 + 9, meaning killed by SIGKILL.

This matters for interpretation: the container did not choose to exit and had no opportunity to explain itself. Its logs will usually just stop mid-sentence. Looking for a crash message wastes time — the evidence is in the pod's Last State, not the application output.

There is one distinction worth holding onto. Memory *requests* affect scheduling; memory *limits* are what get you killed. A pod can be scheduled comfortably and still be OOMKilled seconds later if the limit is below what the workload actually needs.

How to tell this is your problem
WhereWhat you see
kubectl describe podLast State: Terminated, Reason: OOMKilled, Exit Code: 137
kubectl get podsOften CrashLoopBackOff, because the restart succeeds and it is killed again
Container logsOutput that simply stops, with no error — SIGKILL leaves no message

How to know it is actually fixed

  • No new OOMKilled entries in Last State after a full load cycle, including whatever peak originally triggered it.
  • kubectl top pod shows steady-state usage with real headroom below the limit, not brushing against it.
  • If you suspected a leak: usage plateaus instead of climbing. If it still climbs, the limit change only postponed the kill.

Stopping it happening again

  • Set requests and limits from measurement. A round number is a guess wearing a suit.
  • Tell language runtimes about the container limit so their heaps stay inside it.
  • Alert on memory usage approaching the limit, so you learn about this before the kernel decides for you.
  • Keep batch work in Jobs with their own limits rather than inflating a long-running service.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Related

CrashLoopBackOffPod stuck in PendingKubernetesObservability
OOMKilled (exit code 137) — causes, diagnosis and fix | DevOps Insights