Skip to main content
DevOps Insightsguides & answers

Kubernetes

What happens if a container exceeds its memory limit?

Short answer

The kernel kills it immediately. There is no warning, no signal your application can catch, and no opportunity to shut down cleanly — memory is not something that can be throttled.

Starting state

A container with `limits.memory: 512Mi`, running normally, whose working set is growing — a leak, a large request, or an unbounded cache.

The sequence

Colour marks which component acts. Steps sharing a timestamp happen concurrently — nothing here waits for anything else unless it says so.

  • your app
  • kernel
  • kubelet
  1. before anything visibleyour app

    Your process allocates memory and does not release it — a cache with no ceiling, a query loading an unbounded result set, or a runtime whose heap is sized against the node rather than the container. Nothing is wrong yet.

    What you would see Normal operation. This is the phase where the problem is cheap to find and nobody is looking.

  2. as usage growskernel

    The cgroup's memory usage approaches its limit. The kernel starts reclaiming what it can: page cache, buffers, anything droppable.

    What you would see Nothing in your application logs. If you are collecting container metrics you would see usage flatten near the ceiling while performance degrades.

  3. at the limitkernel

    Reclaim can no longer satisfy the next allocation. The cgroup OOM killer selects a process in that cgroup and sends SIGKILL.

    What you would see An OOM entry in the node's kernel log. Your application's last log line is whatever it happened to be doing — there is no shutdown message, because SIGKILL cannot be handled.

  4. immediately afterkubelet

    The kubelet observes the container exited with code 137 and records the termination reason as `OOMKilled`.

    What you would see `kubectl describe pod` shows `Last State: Terminated`, `Reason: OOMKilled`, `Exit Code: 137`.

  5. per restart policykubelet

    With the default `restartPolicy: Always`, the container is restarted. Its memory usage begins from zero.

  6. after restartyour app

    Your process starts again with an empty heap and begins the same growth. If the cause is a leak rather than an under-sized limit, the sequence from here is identical — which is why the interval between kills is the useful signal.

  7. if it recurskubelet

    Repeated kills produce an exponentially increasing restart delay, and the pod's status becomes `CrashLoopBackOff` — which is a symptom of the OOM, not a separate problem.

    What you would see Restart count climbing, with the gaps between restarts getting longer.

What you actually control

Separated from what happens regardless, because conflating the two is how people end up tuning the wrong thing.

`limits.memory`
The kill line. Set it above your observed peak with headroom. Raising it is the right fix for a container that was simply under-provisioned, and the wrong fix for a leak — it buys time and nothing else.
`requests.memory`
Does not affect the kill at all. It affects scheduling and QoS class, which determines eviction order under *node* pressure — a different failure with a different signature.
Application-level bounds
The only real fix for the common causes: a cache with no maximum size, a query that loads an unbounded result set, a runtime heap larger than the container limit.
The runtime's own heap setting
Worth checking explicitly. A JVM or Node process that has not been told the container's limit will happily size its heap against the *node's* memory and get killed long before it thinks it is under pressure.

Where this goes wrong

  • No limit is set at all

    The container grows until the *node* is under pressure, and then the kubelet starts evicting pods — possibly other people's. A memory problem in one workload becomes an incident in an unrelated one.

  • The limit is raised without finding the cause

    The interval between kills gets longer, which reads as a fix. A leak reaches any ceiling eventually, so this converts a frequent, obvious failure into a rare, confusing one.

  • The kill lands on a worker mid-job

    SIGKILL means no cleanup, so any job that was not idempotent and re-queued is simply lost. This is the case where the silence matters most.

The part worth remembering

The asymmetry between CPU and memory is the thing to internalise. Exceeding a CPU limit throttles you — unpleasant, survivable, and visible as latency. Exceeding a memory limit kills you, because there is no way to give a process less memory than it just asked for.

That is why the advice on the two differs so much, and why a memory limit deserves careful sizing while a CPU limit is often better left off entirely.

It also explains why OOM kills are so quiet. Every other failure mode gives your application a chance to say something. This one does not, so the only evidence is in the container's termination reason and the node's kernel log — neither of which is where anyone looks first.

What comes next

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere