Why Your Cluster Is Full at 10% Utilisation

Your monitoring says the cluster is bored. CPU sits around 10%, memory is comfortable, the graphs are flat. Then a deploy fails to schedule:

0/6 nodes are available: 6 Insufficient cpu.

Both things are true at once, and the reason is a detail about Kubernetes that is easy to miss and expensive to miss: the scheduler does not look at usage. It looks at requests.

What the scheduler is actually doing

When a pod is created, the scheduler looks for a node with enough unreserved capacity. Unreserved means allocatable capacity minus the sum of the CPU and memory requests of every pod already placed there.

It does not consult what those pods are currently using. It could not sensibly do so — a pod using 50m right now might need 800m in ten seconds, and a scheduler that packed against live usage would place work onto nodes that are about to need it back.

So requests is a reservation. Once a pod is scheduled, that capacity is committed to it whether it uses any of it or not.

This produces the situation above exactly. A cluster can be 95% committed and 10% used. Both numbers are correct; they measure different things.

Seeing your own numbers

Three commands, and the gap between the first two is the whole story.

What is committed:

kubectl describe node <node-name>

Look for the Allocated resources section near the bottom:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource   Requests      Limits
  cpu        3400m (85%)   6000m (150%)
  memory     7800Mi (67%)  12Gi (103%)

That 85% is why nothing else will schedule.

What is actually used:

kubectl top nodes

This needs metrics-server installed. If it returns an error, that absence is itself worth noting — you have been sizing workloads without usage data.

Which pods hold the reservations:

kubectl get pods --all-namespaces -o custom-columns=\
NS:.metadata.namespace,\
NAME:.metadata.name,\
CPU_REQ:.spec.containers[*].resources.requests.cpu,\
MEM_REQ:.spec.containers[*].resources.requests.memory \
  --sort-by=.metadata.namespace

Compare that against kubectl top pods --all-namespaces. The workloads where requested and used differ by an order of magnitude are where your capacity went.

A worked example

Numbers invented for the arithmetic, but the shape is the common one.

A node offers 4 CPU allocatable. Six services run on it, each requesting 600m:

committed:  6 × 600m = 3600m  (90% of allocatable)
actual use: 6 ×  70m =  420m  (10% of allocatable)

The seventh service requests 600m. There is 400m unreserved, so it does not fit — and it does not fit despite 3.5 CPU sitting idle at that exact moment.

You can respond to this in two ways. Add a node, which costs money and does nothing about the 90% gap. Or find out why six services reserve 600m each to use 70m.

Why the requests are too high

This is rarely carelessness. The reasons are all rational in isolation.

They were a guess, and guesses round up. Nobody knew what the service would need, so somebody picked a number with headroom. That was correct at the time. It stayed.

Being throttled is visible and waste is not. Set CPU requests too low and the service is slow under load, somebody notices, and it is your fault. Set them too high and a number in a billing console goes up that nobody attributes to you. The incentives are not symmetric, and engineers respond to that sensibly.

They live in a Helm chart. A value set once in values.yaml at the start of a project gets copied into every later environment and is never revisited, because nothing ever prompts anyone to revisit it.

The feedback loop does not exist. The developer who sets the request does not see the bill. The person who sees the bill cannot tell which request caused which line of it. Without a path between those two facts, nothing corrects.

That last one is the actual root of it, and it is organisational rather than technical. Tooling helps, but a team that never sees the cost of its own requests will drift upward again after any one-off cleanup.

What to do about it

Measure before changing anything. Look at actual usage over a period that includes your real peak — a full week at minimum, and include whatever your monthly or quarterly spike is. Right-sizing against a quiet Tuesday is how you cause an incident.

Set CPU requests near observed p95, not near peak. CPU is compressible: exceeding your request means you get throttled, which is degradation rather than death. Some headroom is right; three times headroom is a reservation nobody is using.

Set memory requests near observed peak, with real headroom. Memory is not compressible. Exceeding a memory limit gets the container killed — see OOMKilled for what that looks like when it happens. Be more generous here than with CPU, and be deliberate about it rather than generous everywhere out of habit.

Understand requests and limits as separate decisions. They are two different mechanisms and setting them equal, or setting only one, is the most common configuration mistake in this area. requests vs limits covers what each actually controls.

Change it in small steps and watch. Reduce a request, wait a full cycle of real traffic, look at throttling and eviction. Then do the next one. A cluster-wide right-sizing applied in one afternoon is an outage looking for a date.

The resource calculator will do the arithmetic for a workload if you have the usage numbers.

The trade-off, stated honestly

Lower requests pack more work onto fewer nodes and cost less. They also reduce the buffer between normal operation and trouble.

What you are actually buying with generous requests is isolation: the guarantee that your service's capacity is there regardless of what its neighbours do. Tighten requests and you give some of that up in exchange for money. That is a real trade, not free efficiency, and how much of it to make depends on the workload.

Which means the sensible answer is not "make everything tight".

When not to do this

Some workloads should keep generous requests, and treating right-sizing as a universal good is how a cost exercise becomes an incident.

Latency-critical paths. If p99 matters commercially, throttling is a product problem and not just an engineering one. Leave headroom.

Spiky workloads. Something that idles at 5% and needs 3 CPU for ninety seconds every hour is correctly reserving for the spike. Its average utilisation is a misleading number.

Stateful workloads. Databases, queues and anything holding state suffer disproportionately from eviction. The saving is rarely worth the risk.

Anything you cannot observe properly. If you do not have reliable usage data for a service, you do not have the input this exercise requires. Get the metrics first.

During an incident. Obvious, and it still happens. Do not right-size while something is on fire.

Common mistakes

Right-sizing against average usage. The average includes every quiet hour and hides every peak. Use a high percentile.

Lowering limits along with requests. These do different jobs. Cutting the limit is what turns a throttling problem into an OOMKill.

Doing it once. Workloads change. A single cleanup drifts back within a couple of quarters unless something makes the cost visible to the people setting the numbers.

Confusing this with idle nodes. Over-reserved workloads and over-provisioned node pools are two different problems with different fixes. Fixing the first without touching the second leaves you with a cluster that is now genuinely empty and still costs the same — the node pool has to shrink for any of it to reach a bill.

Assuming Pending always means this. It often does not. A pod can also sit Pending waiting on a volume, or because of taints, affinity or topology constraints. Pod stuck in Pending covers the other causes and how to tell them apart — the scheduler records its reason per node, and that message is the diagnosis.

The short version

The scheduler commits against requests, so a cluster can be full and idle simultaneously and both dashboards are telling the truth. Find the workloads where requested and used differ by an order of magnitude, measure them across a real peak, lower CPU requests toward p95 and memory toward peak with headroom, and move in small steps.

Then make the number visible to the people who set it, because otherwise you will be doing this again next year.