Kubernetes
What happens if a deployment reports success but users still see errors?
Short answer
It means every check that ran was satisfied — and none of them checked the thing that is broken. A successful rollout proves that pods started and passed their probes, which is a much weaker claim than "the application works".
Starting state
A rolling update completes. `kubectl rollout status` returns success, all replicas are Ready, and the error rate at the edge has gone up.
The sequence
Colour marks which component acts. Steps sharing a timestamp happen concurrently — nothing here waits for anything else unless it says so.
- control plane
- kubelet
- network
- your app
- during the rolloutcontrol plane
New pods are created and old ones removed according to the update strategy. The Deployment's only criterion for progress is that new pods become Ready.
- per new podkubelet
Each new pod's readiness probe passes. Whatever that probe checks is now the entire definition of "healthy" that the rollout used.
What you would see This is the crux: if the probe returns 200 from a handler that does nothing, then a completely broken application is indistinguishable from a working one.
- as pods become readynetwork
Each new pod is added to the Service endpoints and starts receiving real traffic — traffic that exercises code paths the probe never touched.
- at rollout completioncontrol plane
The desired replica count is met by Ready pods, so the rollout is reported successful. Nothing in this signal has any knowledge of your error rate.
What you would see `rollout status` says success. It is not lying; it is answering a narrower question than you asked.
- at the edgeyour app
Users encounter whatever is actually wrong: a missing config value, a migration that has not run, a downstream call that now fails, a serialisation change the caller does not understand.
What you would see Error rate up, and — the useful tell — the increase starts at the rollout and rises as the rollout progresses, in proportion to how much traffic the new pods are serving.
What you actually control
Separated from what happens regardless, because conflating the two is how people end up tuning the wrong thing.
- What the readiness probe actually checks
- The highest-leverage decision here. A probe that verifies the application can do a representative unit of work makes the rollout signal mean something. A probe that returns a static 200 makes it meaningless.
- Whether the rollout watches anything beyond readiness
- Plain Deployments cannot. Progressive delivery — canary, or a rollout gated on real metrics — is what turns "pods started" into "error rate did not rise", and it is the actual answer to this class of problem.
- `minReadySeconds`
- Requires a pod to stay Ready for a period before counting. Catches the failure mode of a pod that becomes ready and then immediately breaks, which is common with a lazily-initialised dependency.
- Whether you can roll back quickly
- The thing to establish before you need it. `kubectl rollout undo` is fast, but only if the change was backwards-compatible — a migration that rewrote data is not something a rollout undo reverses.
Where this goes wrong
The rollout is trusted as a health signal
The deploy is marked done, everybody moves on, and the errors are discovered by users or by an alert twenty minutes later — by which point the deploy is no longer the obvious suspect.
The change included a database migration
Rolling back the code does not roll back the schema. If the old code cannot read the new schema, you are now unable to go either forwards or backwards, which is a considerably worse position than the original bug.
Config comes from somewhere the probe does not exercise
The classic version: a missing environment variable that is only read on the first request to one endpoint. Everything is Ready, and one route returns 500.
The part worth remembering
The general lesson is about what a green signal actually asserts. "Rollout successful" means the pods reached a state your probe defined as ready. It says nothing about correctness, and it cannot, because nothing in the rollout has any idea what your application is for.
So the value of that signal is exactly the quality of your readiness probe. This is the strongest practical argument for probes that do real work: not the probe itself, but that it is the definition of health every automated deployment decision depends on.
The corollary is worth stating too. If the error rate rises at the moment of a rollout and climbs as the rollout progresses, that shape is itself strong evidence — you do not need correlation tooling to read it, and it is the first thing to check when a deploy and an incident share a timestamp.
What comes next
Did this get you to an answer?
No text box on purpose — please do not paste production logs anywhere