Question 1

A deployment says 3 replicas but only 2 pods are running. How do you investigate?

Accepted Answer

Start by finding out where the third one is stuck, because the answer branches completely. kubectl get pods tells you whether it does not exist, is Pending, or is crash-looping. Pending means scheduling — kubectl describe pod and read the events for insufficient resources, taints or unbound volumes. Crash-looping means the container starts and exits, so logs and last-state. Not existing at all points at the ReplicaSet or a quota.

Question 2

It's Pending. describe says insufficient memory, but the cluster is at 30% utilisation. Explain that.

Accepted Answer

Requests, not usage. The scheduler places pods against reserved capacity, so a cluster whose pods request far more than they use will refuse to schedule while looking idle. The fix is sizing requests from observed usage rather than adding nodes — and being able to say that inflated requests are the common cause is the point.

Question 3

Different pod: restart count is climbing but the logs show a clean startup with no error. What now?

Accepted Answer

Suspect the probes before the application. A liveness probe whose initialDelaySeconds is shorter than real start-up time kills the container mid-boot, which produces exactly this signature: a clean, truncated log and no error, because nothing went wrong. Check describe for Liveness probe failed events, and note that a startupProbe is the correct tool for a slow start.

Question 4

Now every pod in the cluster restarted at roughly the same time. What does that tell you?

Accepted Answer

Simultaneity rules out pod-level faults — independent failures do not synchronise. So look for something shared: a liveness probe that checks a dependency which briefly went down, a node-level event, DNS, an expiring certificate or token. The liveness-checking-a-dependency case is worth naming, because it turns a partial degradation into a total outage.

Question 5

A rollout reported success. Users are seeing errors. Where do you look?

Accepted Answer

Say plainly what the success signal asserts: pods started and passed their readiness probe. Nothing more. So either the probe does not check anything meaningful, or the failure is in a path the probe never touches — config read on first request, an unrun migration, a downstream contract change. The diagnostic tell is timing: an error rate that starts at the rollout and rises as it progresses is proportional to how much traffic the new pods serve.

Question 6

How would you stop this class of problem from recurring?

Accepted Answer

Two threads, and the honest answer includes both. Make readiness probes do real work, since every automated deployment decision inherits their definition of health. And gate rollouts on something beyond readiness — progressive delivery watching real error rate — because that is what converts “pods started” into “nothing got worse”. Acknowledging that plain Deployments cannot do the second thing is better than implying they can.

Kubernetes: debugging under pressure

A deployment says 3 replicas but only 2 pods are running. How do you investigate?

Sources

Where to go next