Kubernetes · commonly confused
Requests vs limits
Short answer
First published
On this page · 10 sections
In simple terms
A request is a promise made at scheduling time: “this container needs at least this much, so only place it on a node that has it spare.” It shapes where the pod goes, and it is what the cluster subtracts from a node's available capacity.
A limit is a boundary enforced while the container runs. It says “whatever happens, do not use more than this.”
The part that surprises people is that the two resources behave completely differently at the boundary. CPU is compressible — exceed your share and you simply get scheduled less often. Memory is not. Once the kernel cannot reclaim enough memory to keep the container under its limit, its remaining option is to kill a process.
Words used on this page
- request
- The amount of CPU or memory the scheduler sets aside for a container on a node. The container can use more when the node has it spare.
- limit
- The most CPU or memory a container may use. The kernel enforces it on the node: CPU by throttling, memory by killing.
- millicpu (m)
- A thousandth of a CPU core.
100mand0.1mean the same amount. - Mi and M
- Mebibytes (1,048,576 bytes) and megabytes (1,000,000 bytes). Case matters: a lowercase
mmeans millibytes, so400mof memory is 0.4 bytes. - allocatable
- What a node offers to Pods: its capacity minus what is reserved for the system. The scheduler fits requests into this, not into free memory.
- QoS class
Guaranteed,BurstableorBestEffort, worked out from a Pod's requests and limits. It affects which containers the kernel kills first when a node runs out of memory.- cgroup
- The Linux kernel feature that groups a container's processes and caps their CPU and memory. The container runtime sets it up from your requests and limits.
- CPU throttling
- The kernel pausing a container that has used up its CPU time for the current period. The container slows down; nothing is killed.
- OOMKilled
- The reason Kubernetes reports when the kernel's out-of-memory (OOM) killer ended a container.
- eviction
- The kubelet stopping whole Pods to free memory or disk when a node runs short. Evicted Pods are not restarted on that node.
What actually happens
Requests are consumed by the scheduler when it filters nodes, and they determine the pod's QoS class. Every container having CPU and memory requests equal to its limits gives Guaranteed; any request or limit short of that gives Burstable; nothing set gives BestEffort. Under node pressure the kubelet evicts first the pods using more than they requested, ordered by priority, and a BestEffort pod is always in that group.
A CPU limit is implemented as CFS quota — a cap on how much CPU time the cgroup receives per period. Exceeding it means throttling, which shows up as latency rather than as an error.
A memory limit is the cgroup memory ceiling. When usage reaches it and the kernel cannot reclaim enough, the OOM killer runs against processes in that cgroup. On cgroup v2 nodes the kubelet by default has them killed together, so the container terminates with exit code 137. Unless an administrator has turned on Memory QoS throttling in the kubelet, nothing slows the container down first, and the kill is a SIGKILL, which gives no chance to shut down.
A starting shape for a service
resources:
requests:
memory: "256Mi" # near typical observed usage
cpu: "100m" # near typical observed usage
limits:
memory: "512Mi" # above observed peak — this is the kill line
# No CPU limit. See the note.The omission is deliberate. A CPU limit throttles exactly when the container is busiest, which presents as latency nobody can explain rather than as throttling. Unless you need hard multi-tenant isolation, a CPU request alone allocates fairly and lets a service use spare capacity during a spike.
Try it yourself
Before you start
- A Kubernetes cluster you can safely break things in, and
kubectlconfigured for it. The Kubernetes task page used below asks for nodes with at least 300 MiB of memory each, and has you create a namespace for the exercise:kubectl create namespace mem-example. - Permission to create Pods in that namespace (
kubectl auth can-i create pods -n mem-exampleprintsyes). Reading node capacity withkubectl describe nodesneeds read access to nodes, which are cluster-wide: only a ClusterRole can grant it. - The Metrics API, for
kubectl top.kubectl get apiservicesshould listv1beta1.metrics.k8s.io; on minikube,minikube addons enable metrics-serverturns it on. - The namespace's own rules.
kubectl get resourcequota,limitrange -n <namespace>shows them: a ResourceQuota can reject Pods that leave out requests or limits, and a LimitRange can add defaults you did not write. - The example below is quoted from the Kubernetes v1.37 documentation, not run here. Behaviour on this page is for Linux nodes using cgroup v2, which the kubelet requires by default since v1.35.
One Pod killed by its memory limit, one never scheduled because of its request
Quoted from the Kubernetes task page Assign Memory Resources to Containers and Pods; not reproduced here, because no cluster was available. In the namespace mem-example, memory-demo-2 requests 50 MiB of memory with a 100 MiB limit and runs stress to allocate 250 MiB. memory-demo-3 requests 1000 GiB, more than any node has. The page shows only an excerpt of the --output=yaml output, and its timestamps date from 2017.
Command
kubectl get pod memory-demo-2 --namespace=mem-example kubectl get pod memory-demo-2 --output=yaml --namespace=mem-example kubectl get pod memory-demo-3 --namespace=mem-example
Output · as shown in Kubernetes: Assign Memory Resources to Containers and Pods
$ kubectl get pod memory-demo-2 --namespace=mem-example
NAME READY STATUS RESTARTS AGE
memory-demo-2 0/1 OOMKilled 1 24s
$ kubectl get pod memory-demo-2 --output=yaml --namespace=mem-example
lastState:
terminated:
containerID: 65183c1877aaec2e8427bc95609cc52677a454b56fcb24340dbd22917c23b10f
exitCode: 137
finishedAt: 2017-06-20T20:52:19Z
reason: OOMKilled
startedAt: null
$ kubectl get pod memory-demo-3 --namespace=mem-example
NAME READY STATUS RESTARTS AGE
memory-demo-3 0/1 Pending 0 25sHow to read it
OOMKilledwithRESTARTS 1:memory-demo-2went past its 100 MiB limit and was killed. Its 50 MiB request played no part. A request is a reservation, not a ceiling.exitCode: 137is 128 plus 9, the number ofSIGKILL, a signal a process cannot catch, so the program got no chance to clean up.reason: OOMKilledis what says the out-of-memory killer sent it; a 137 on its own only says something sentSIGKILL.- The kubelet restarts the container under the Pod's restart policy, and
stressasks for 250 MiB again every time. The page shows it cycling betweenOOMKilledandRunningwhileRESTARTSclimbs. It will keep doing so until the limit is raised or the program asks for less. memory-demo-3isPendingwithRESTARTS 0: it never started, so no limit was ever tested. Its request is bigger than any node can offer, and the scheduler fits Pods to nodes by their requests, so it waits indefinitely.kubectl describe pod memory-demo-3 --namespace=mem-examplegives the reason in aFailedSchedulingevent.
The mistake this causes
Setting requests equal to peak usage, to be safe.
Why people do it It feels conservative. In practice requests are a reservation, so inflating them makes the cluster look full while nodes sit mostly idle — and you pay for the difference.
What you see Pods stuck in Pending with 0/5 nodes are available: 5 Insufficient memory., on a cluster whose actual memory utilisation is 30%. The scheduler is refusing to place pods against reservations, not against usage.
If it goes wrong
- A
FailedSchedulingevent such as0/3 nodes are available: 3 Insufficient memory. - No node has enough unreserved allocatable memory for the Pod's requests, whatever the nodes are actually using.
kubectl describe nodesshowsAllocatableandAllocated resourcesfor each node. Bring requests down to measured usage, or add capacity. The number beforeInsufficientcounts the nodes that failed for that reason. kubectl applyis refused withfailed quota: <quota>: must specify limits.cpu for: <container>- The namespace has a ResourceQuota that counts CPU limits, so every container must declare one, even if you meant to leave CPU unlimited. Add
limits.cpu, or ask the namespace owner whether the quota should count requests only.kubectl describe resourcequota -n <namespace>shows what it counts. Pod "<name>" is invalid: spec.containers[0].resources.requests: Invalid value: "<request>": must be less than or equal to cpu limit, although you set no CPU limit- A LimitRange in the namespace added its default CPU limit to your container, and your request is above it. Set a CPU limit yourself, at or above the request, or ask for the LimitRange default to be raised.
kubectl describe limitrange -n <namespace>shows the defaults. - Latency rises under load, there are no errors, and
kubectl top pod <pod> --containersshows CPU at the limit - The container is being throttled. If you collect cAdvisor metrics, compare
container_cpu_cfs_throttled_periods_totalwithcontainer_cpu_cfs_periods_total; cAdvisor only exports these for containers that have a CPU limit. Raise or remove the CPU limit and keep the request. kubectl top podprintserror: Metrics API not available- Nothing in the cluster serves the resource metrics API. Install metrics-server or another provider (on minikube,
minikube addons enable metrics-server), then check thatkubectl get apiserviceslistsv1beta1.metrics.k8s.io.
How it shows up in production
The opposite mistake is more dangerous. With no memory limit, a container with a slow leak grows until the node runs short. The kubelet then starts evicting Pods, beginning with those using more than they requested, and if memory runs out faster than it can react, the kernel's OOM killer picks the victim. Neither is sure to pick the leaking one, so a memory problem in one workload can become an outage in an unrelated one on the same node.
Both mistakes come from the same root: sizing from intuition rather than measurement. The numbers are available from kubectl top pod --containers, and they only mean anything if you sample across a full load cycle including start-up.
How to tell which one you are hitting
- Pod stuck in Pending with an insufficient-resource message?
- That is requests, not limits — the scheduler cannot find a node with enough unreserved capacity. Compare your requests against real usage before adding nodes.
- Container terminated with exit code 137?
- Exit code 137 means the container was killed with
SIGKILL. Ifkubectl describe podshowsLast State: TerminatedwithReason: OOMKilledunder it, the kernel's OOM killer did it, enforcing either the container's memory limit or, when the whole node ran out of memory, the node's. - Latency that rises under load with no errors?
- Suspect CPU throttling. Check the
container_cpu_cfs_throttled_seconds_totalmetric if you have it — throttling is invisible in application logs by design.
Security considerations
- A container with no memory limit has no upper bound on memory, and that includes files written to a memory-backed
emptyDir. The Kubernetes docs warn this can use all of a node's memory and become a denial of service for every Pod on it. Set a memory limit on anything that shares nodes with other workloads. - Pods with no requests or limits are
BestEffort, and the kubelet gives their containers anoom_score_adjof 1000, the highest. When a node runs out of memory they are the kernel's first choice to kill, so give requests to anything you rely on, such as a log shipper or security agent. - A quota only limits a team that cannot edit it. The built-in
adminrole, granted in a namespace with a RoleBinding, can change most resources there but has no write access to ResourceQuota, according to the Kubernetes RBAC reference. Bindingcluster-adminin the namespace, or a custom role that includesresourcequotas, gives that away, so keep quota objects writable by cluster administrators only. - Leaving out a CPU limit lets a container use every idle core on its node. That is the point, but it also means a runaway or compromised container, such as one mining cryptocurrency, runs at full speed without failing anything, and no throttling metric is exported for it. Alert on CPU use well above the request instead.
In an interview
How this gets asked in an interviewpreparation
Usually phrased as “What is the difference between a resource request and a limit?”
What a strong answer contains Scheduling versus enforcement, then the asymmetry between the two resources: CPU is compressible so a limit throttles, memory is not so a limit kills. Bringing up QoS classes and eviction order shows you have thought about what happens under node pressure.
The follow-up “Should you set a CPU limit?” is a genuinely open question, and having a reasoned position — either way — is worth more than repeating a default.
Sources
The behaviour this page describes, as documented by the projects themselves.
- Kubernetes: Node-pressure Eviction
- Kubernetes: Resource Quotas
- Kubernetes: Limit Ranges
- Kubernetes: Kubelet Configuration (v1beta1)
- Linux Kernel: Control Group v2
- Kubernetes source v1.37.1: scheduler FitError message
- cAdvisor source lib/v0.60.5: CPU throttling metrics
- signal(7) manual page
- Kubernetes: Using RBAC Authorization (user-facing roles)
- Kubernetes — Resource Management for Pods and Containers
- Kubernetes — Pod Quality of Service Classes
- Linux Kernel — CFS Bandwidth Control
- Kubernetes: Assign Memory Resources to Containers and Pods
Terms used here (3)plain definitions
Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.
- Auto Scaling
- The ability to automatically adjust the number of running compute resources based on real-time demand. Auto scaling ensures applications can handle traffic spikes without over-provisioning during quiet periods, optimizing both performance and cost.
- Helm
- A package manager for Kubernetes whose packages, called 'charts', are collections of files — chiefly templated Kubernetes manifests and their default values — used to define, install, and upgrade applications. Helm simplifies complex deployments by bundling related Kubernetes resources into a single, versioned package.
- HPA
- Horizontal Pod Autoscaler — a Kubernetes resource that automatically scales the number of Pod replicas based on observed CPU utilization, memory usage, or custom metrics. HPA enables applications to handle variable workloads without manual intervention.
Did this get you to an answer?
No text box on purpose — please do not paste production logs anywhere
Something wrong or out of date? Report it
Where to go next
- When it goes wrongPersistentVolumeClaim stays Pending
- When it goes wrongThe connection to the server localhost:8080 was refused
- When it goes wrongerror: You must be logged in to the server (Unauthorized)
- What comes nextOOMKilled — causes and diagnosis
- What comes nextPod stuck in Pending
- What comes nextSize these from real numbers