Skip to main content
DevOps Insightsguides & answers

Kubernetes · commonly confused

Requests vs limits

Short answer

Requests are what the scheduler reserves for you. Limits are the ceiling the kernel enforces. For memory, the limit is a hard ceiling: a container the kernel cannot reclaim back under it is killed. For CPU it only slows you down.

First published

On this page · 10 sections

In simple terms

A request is a promise made at scheduling time: “this container needs at least this much, so only place it on a node that has it spare.” It shapes where the pod goes, and it is what the cluster subtracts from a node's available capacity.

A limit is a boundary enforced while the container runs. It says “whatever happens, do not use more than this.”

The part that surprises people is that the two resources behave completely differently at the boundary. CPU is compressible — exceed your share and you simply get scheduled less often. Memory is not. Once the kernel cannot reclaim enough memory to keep the container under its limit, its remaining option is to kill a process.

Words used on this page

request
The amount of CPU or memory the scheduler sets aside for a container on a node. The container can use more when the node has it spare.
limit
The most CPU or memory a container may use. The kernel enforces it on the node: CPU by throttling, memory by killing.
millicpu (m)
A thousandth of a CPU core. 100m and 0.1 mean the same amount.
Mi and M
Mebibytes (1,048,576 bytes) and megabytes (1,000,000 bytes). Case matters: a lowercase m means millibytes, so 400m of memory is 0.4 bytes.
allocatable
What a node offers to Pods: its capacity minus what is reserved for the system. The scheduler fits requests into this, not into free memory.
QoS class
Guaranteed, Burstable or BestEffort, worked out from a Pod's requests and limits. It affects which containers the kernel kills first when a node runs out of memory.
cgroup
The Linux kernel feature that groups a container's processes and caps their CPU and memory. The container runtime sets it up from your requests and limits.
CPU throttling
The kernel pausing a container that has used up its CPU time for the current period. The container slows down; nothing is killed.
OOMKilled
The reason Kubernetes reports when the kernel's out-of-memory (OOM) killer ended a container.
eviction
The kubelet stopping whole Pods to free memory or disk when a node runs short. Evicted Pods are not restarted on that node.

What actually happens

Requests are consumed by the scheduler when it filters nodes, and they determine the pod's QoS class. Every container having CPU and memory requests equal to its limits gives Guaranteed; any request or limit short of that gives Burstable; nothing set gives BestEffort. Under node pressure the kubelet evicts first the pods using more than they requested, ordered by priority, and a BestEffort pod is always in that group.

A CPU limit is implemented as CFS quota — a cap on how much CPU time the cgroup receives per period. Exceeding it means throttling, which shows up as latency rather than as an error.

A memory limit is the cgroup memory ceiling. When usage reaches it and the kernel cannot reclaim enough, the OOM killer runs against processes in that cgroup. On cgroup v2 nodes the kubelet by default has them killed together, so the container terminates with exit code 137. Unless an administrator has turned on Memory QoS throttling in the kubelet, nothing slows the container down first, and the kill is a SIGKILL, which gives no chance to shut down.

A starting shape for a service

resources:
  requests:
    memory: "256Mi"   # near typical observed usage
    cpu: "100m"       # near typical observed usage
  limits:
    memory: "512Mi"   # above observed peak — this is the kill line
    # No CPU limit. See the note.

The omission is deliberate. A CPU limit throttles exactly when the container is busiest, which presents as latency nobody can explain rather than as throttling. Unless you need hard multi-tenant isolation, a CPU request alone allocates fairly and lets a service use spare capacity during a spike.

Try it yourself

Before you start

  • A Kubernetes cluster you can safely break things in, and kubectl configured for it. The Kubernetes task page used below asks for nodes with at least 300 MiB of memory each, and has you create a namespace for the exercise: kubectl create namespace mem-example.
  • Permission to create Pods in that namespace (kubectl auth can-i create pods -n mem-example prints yes). Reading node capacity with kubectl describe nodes needs read access to nodes, which are cluster-wide: only a ClusterRole can grant it.
  • The Metrics API, for kubectl top. kubectl get apiservices should list v1beta1.metrics.k8s.io; on minikube, minikube addons enable metrics-server turns it on.
  • The namespace's own rules. kubectl get resourcequota,limitrange -n <namespace> shows them: a ResourceQuota can reject Pods that leave out requests or limits, and a LimitRange can add defaults you did not write.
  • The example below is quoted from the Kubernetes v1.37 documentation, not run here. Behaviour on this page is for Linux nodes using cgroup v2, which the kubelet requires by default since v1.35.

One Pod killed by its memory limit, one never scheduled because of its request

Quoted from the Kubernetes task page Assign Memory Resources to Containers and Pods; not reproduced here, because no cluster was available. In the namespace mem-example, memory-demo-2 requests 50 MiB of memory with a 100 MiB limit and runs stress to allocate 250 MiB. memory-demo-3 requests 1000 GiB, more than any node has. The page shows only an excerpt of the --output=yaml output, and its timestamps date from 2017.

Command

kubectl get pod memory-demo-2 --namespace=mem-example
kubectl get pod memory-demo-2 --output=yaml --namespace=mem-example
kubectl get pod memory-demo-3 --namespace=mem-example

Output · as shown in Kubernetes: Assign Memory Resources to Containers and Pods

$ kubectl get pod memory-demo-2 --namespace=mem-example
NAME            READY     STATUS      RESTARTS   AGE
memory-demo-2   0/1       OOMKilled   1          24s

$ kubectl get pod memory-demo-2 --output=yaml --namespace=mem-example
lastState:
   terminated:
     containerID: 65183c1877aaec2e8427bc95609cc52677a454b56fcb24340dbd22917c23b10f
     exitCode: 137
     finishedAt: 2017-06-20T20:52:19Z
     reason: OOMKilled
     startedAt: null

$ kubectl get pod memory-demo-3 --namespace=mem-example
NAME            READY     STATUS    RESTARTS   AGE
memory-demo-3   0/1       Pending   0          25s

How to read it

  • OOMKilled with RESTARTS 1: memory-demo-2 went past its 100 MiB limit and was killed. Its 50 MiB request played no part. A request is a reservation, not a ceiling.
  • exitCode: 137 is 128 plus 9, the number of SIGKILL, a signal a process cannot catch, so the program got no chance to clean up. reason: OOMKilled is what says the out-of-memory killer sent it; a 137 on its own only says something sent SIGKILL.
  • The kubelet restarts the container under the Pod's restart policy, and stress asks for 250 MiB again every time. The page shows it cycling between OOMKilled and Running while RESTARTS climbs. It will keep doing so until the limit is raised or the program asks for less.
  • memory-demo-3 is Pending with RESTARTS 0: it never started, so no limit was ever tested. Its request is bigger than any node can offer, and the scheduler fits Pods to nodes by their requests, so it waits indefinitely. kubectl describe pod memory-demo-3 --namespace=mem-example gives the reason in a FailedScheduling event.

The mistake this causes

Setting requests equal to peak usage, to be safe.

Why people do it It feels conservative. In practice requests are a reservation, so inflating them makes the cluster look full while nodes sit mostly idle — and you pay for the difference.

What you see Pods stuck in Pending with 0/5 nodes are available: 5 Insufficient memory., on a cluster whose actual memory utilisation is 30%. The scheduler is refusing to place pods against reservations, not against usage.

If it goes wrong

A FailedScheduling event such as 0/3 nodes are available: 3 Insufficient memory.
No node has enough unreserved allocatable memory for the Pod's requests, whatever the nodes are actually using. kubectl describe nodes shows Allocatable and Allocated resources for each node. Bring requests down to measured usage, or add capacity. The number before Insufficient counts the nodes that failed for that reason.
kubectl apply is refused with failed quota: <quota>: must specify limits.cpu for: <container>
The namespace has a ResourceQuota that counts CPU limits, so every container must declare one, even if you meant to leave CPU unlimited. Add limits.cpu, or ask the namespace owner whether the quota should count requests only. kubectl describe resourcequota -n <namespace> shows what it counts.
Pod "<name>" is invalid: spec.containers[0].resources.requests: Invalid value: "<request>": must be less than or equal to cpu limit, although you set no CPU limit
A LimitRange in the namespace added its default CPU limit to your container, and your request is above it. Set a CPU limit yourself, at or above the request, or ask for the LimitRange default to be raised. kubectl describe limitrange -n <namespace> shows the defaults.
Latency rises under load, there are no errors, and kubectl top pod <pod> --containers shows CPU at the limit
The container is being throttled. If you collect cAdvisor metrics, compare container_cpu_cfs_throttled_periods_total with container_cpu_cfs_periods_total; cAdvisor only exports these for containers that have a CPU limit. Raise or remove the CPU limit and keep the request.
kubectl top pod prints error: Metrics API not available
Nothing in the cluster serves the resource metrics API. Install metrics-server or another provider (on minikube, minikube addons enable metrics-server), then check that kubectl get apiservices lists v1beta1.metrics.k8s.io.

How it shows up in production

The opposite mistake is more dangerous. With no memory limit, a container with a slow leak grows until the node runs short. The kubelet then starts evicting Pods, beginning with those using more than they requested, and if memory runs out faster than it can react, the kernel's OOM killer picks the victim. Neither is sure to pick the leaking one, so a memory problem in one workload can become an outage in an unrelated one on the same node.

Both mistakes come from the same root: sizing from intuition rather than measurement. The numbers are available from kubectl top pod --containers, and they only mean anything if you sample across a full load cycle including start-up.

How to tell which one you are hitting

Pod stuck in Pending with an insufficient-resource message?
That is requests, not limits — the scheduler cannot find a node with enough unreserved capacity. Compare your requests against real usage before adding nodes.
Container terminated with exit code 137?
Exit code 137 means the container was killed with SIGKILL. If kubectl describe pod shows Last State: Terminated with Reason: OOMKilled under it, the kernel's OOM killer did it, enforcing either the container's memory limit or, when the whole node ran out of memory, the node's.
Latency that rises under load with no errors?
Suspect CPU throttling. Check the container_cpu_cfs_throttled_seconds_total metric if you have it — throttling is invisible in application logs by design.

Security considerations

  • A container with no memory limit has no upper bound on memory, and that includes files written to a memory-backed emptyDir. The Kubernetes docs warn this can use all of a node's memory and become a denial of service for every Pod on it. Set a memory limit on anything that shares nodes with other workloads.
  • Pods with no requests or limits are BestEffort, and the kubelet gives their containers an oom_score_adj of 1000, the highest. When a node runs out of memory they are the kernel's first choice to kill, so give requests to anything you rely on, such as a log shipper or security agent.
  • A quota only limits a team that cannot edit it. The built-in admin role, granted in a namespace with a RoleBinding, can change most resources there but has no write access to ResourceQuota, according to the Kubernetes RBAC reference. Binding cluster-admin in the namespace, or a custom role that includes resourcequotas, gives that away, so keep quota objects writable by cluster administrators only.
  • Leaving out a CPU limit lets a container use every idle core on its node. That is the point, but it also means a runaway or compromised container, such as one mining cryptocurrency, runs at full speed without failing anything, and no throttling metric is exported for it. Alert on CPU use well above the request instead.

In an interview

How this gets asked in an interview

Usually phrased as “What is the difference between a resource request and a limit?”

What a strong answer contains Scheduling versus enforcement, then the asymmetry between the two resources: CPU is compressible so a limit throttles, memory is not so a limit kills. Bringing up QoS classes and eviction order shows you have thought about what happens under node pressure.

The follow-up “Should you set a CPU limit?” is a genuinely open question, and having a reasoned position — either way — is worth more than repeating a default.

Sources

The behaviour this page describes, as documented by the projects themselves.

Terms used here (3)

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Auto Scaling
The ability to automatically adjust the number of running compute resources based on real-time demand. Auto scaling ensures applications can handle traffic spikes without over-provisioning during quiet periods, optimizing both performance and cost.
Helm
A package manager for Kubernetes whose packages, called 'charts', are collections of files — chiefly templated Kubernetes manifests and their default values — used to define, install, and upgrade applications. Helm simplifies complex deployments by bundling related Kubernetes resources into a single, versioned package.
HPA
Horizontal Pod Autoscaler — a Kubernetes resource that automatically scales the number of Pod replicas based on observed CPU utilization, memory usage, or custom metrics. HPA enables applications to handle variable workloads without manual intervention.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it