Kubernetes · what happens if

What happens if a deployment reports success but users still see errors?

Short answer

It means every check that ran was satisfied — and none of them checked the thing that is broken. A successful rollout proves that pods started and passed their probes, which is a much weaker claim than "the application works".

First published 4 Sept 2026

On this page · 8 sections

Starting state

A rolling update completes. kubectl rollout status returns success, all replicas are Ready, and the error rate at the edge has gone up.

Words used on this page

rolling update: The Deployment's default strategy: replace old pods with new ones a few at a time, within maxSurge and maxUnavailable (25% each by default).
readiness probe: A check the kubelet runs against a container. While it fails, the pod is not Ready and its Services send it no traffic. A container without one counts as ready once it starts.
available: Ready for at least minReadySeconds without a container crashing. With the default of 0, available simply means Ready. Deployments count available pods.
progressDeadlineSeconds: How long a rollout may go without progress before the Deployment reports ProgressDeadlineExceeded. Defaults to 600 seconds. Reporting is all Kubernetes does about it.
revision: A numbered version of a Deployment's Pod template. A new one is created only when .spec.template changes, and kubectl rollout undo returns to the previous one.
rollout status: kubectl rollout status waits for a rollout to finish. It exits 0 once every replica is updated and available, and non-zero if the progress deadline is exceeded.
canary: Running a new release beside the old one, for part of the real traffic, before rolling it out fully. The Kubernetes docs show it with two Deployments told apart by a track label.

The sequence

Colour marks which component acts. Steps sharing a timestamp happen concurrently — nothing here waits for anything else unless it says so.

control plane
kubelet
network
your app

during the rolloutcontrol plane
New pods are created and old ones removed according to the update strategy. The only health signal the Deployment uses is each new pod's Ready condition: the rollout moves on as new pods become available, which means Ready for at least minReadySeconds (0 unless you set it).
per new podkubelet
Each new pod's readiness probe passes. A container with no readiness probe counts as ready as soon as it starts. Whatever that probe checks is now the entire definition of "healthy" that the rollout used.
What you would see This is the crux: if the probe returns 200 from a handler that does nothing, then a completely broken application is indistinguishable from a working one.
as pods become readynetwork
Each new pod is added to the Service endpoints and starts receiving real traffic — traffic that exercises code paths the probe never touched.
at rollout completioncontrol plane
Every replica runs the new Pod template, all of them are available, and no old replica is left, so the rollout is reported successful. Nothing in this signal has any knowledge of your error rate.
What you would see rollout status says success. It is not lying; it is answering a narrower question than you asked.
at the edgeyour app
Users encounter whatever is actually wrong: a missing config value, a migration that has not run, a downstream call that now fails, a serialisation change the caller does not understand.
What you would see Error rate up, and — the useful tell — the increase starts at the rollout and rises as the rollout progresses, in proportion to how much traffic the new pods are serving.

What you actually control

Separated from what happens regardless, because conflating the two is how people end up tuning the wrong thing.

What the readiness probe actually checks: The highest-leverage decision here. A probe that verifies the application can do a representative unit of work makes the rollout signal mean something. A probe that returns a static 200 makes it meaningless.
Whether the rollout watches anything beyond readiness: Plain Deployments cannot. Progressive delivery — canary, or a rollout gated on real metrics — is what turns "pods started" into "error rate did not rise", and it is the actual answer to this class of problem.
minReadySeconds: Requires a pod to stay Ready for a period before counting. Catches a pod that becomes ready and then fails its readiness probe or crashes within that period. It cannot catch a fault the probe never exercises, such as a dependency that is only set up on the first real request.
Whether you can roll back quickly: The thing to establish before you need it. kubectl rollout undo is fast, but only if the change was backwards-compatible — a migration that rewrote data is not something a rollout undo reverses.

Try it yourself

Before you start

kubectl access to the Deployment's namespace that can read Deployments, ReplicaSets, pods and events. Rolling back also needs patch on the Deployment, because kubectl rollout undo patches its Pod template.
The Deployment's settings, from kubectl get deployment <name> -o yaml: its strategy, minReadySeconds, progressDeadlineSeconds and each container's readinessProbe. A container with no readiness probe counts as ready as soon as it starts.
A measure of health that does not come from Kubernetes: error rate and latency at your load balancer, ingress or application metrics, ideally split by version or by pod, so you can see whether the new pods are the ones failing.
What else the release changed. kubectl rollout history deployment/<name> lists Pod template revisions, but a ConfigMap, a Secret or a database migration is not part of them, and a rollback will not touch it.
A non-production namespace where you have rehearsed kubectl rollout undo before you need it.

What rollout status reports, and what it does not

Quoted verbatim from the Kubernetes documentation page Deployments (sections "Complete Deployment" and "Failed Deployment"); not reproduced, because no cluster was available. The first run is a rollout that completed and the second one that passed its progress deadline. The documentation prints the second error with the name nginx, and its waiting lines are worded differently from kubectl 1.37.1's.

Command

kubectl rollout status deployment/nginx-deployment
echo $?
kubectl rollout status deployment/nginx-deployment
echo $?

Output · as shown in Kubernetes — Deployments

$ kubectl rollout status deployment/nginx-deployment
Waiting for rollout to finish: 2 of 3 updated replicas are available...
deployment "nginx-deployment" successfully rolled out

$ echo $?
0

$ kubectl rollout status deployment/nginx-deployment
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
error: deployment "nginx" exceeded its progress deadline

$ echo $?
1

How to read it

Both runs count replicas and nothing else: how many run the new Pod template, and how many of those are available, meaning Ready for at least minReadySeconds. No line mentions requests, errors or latency, because the command never looks at them.
successfully rolled out with exit code 0 means every replica runs the new template, all of them are available, and no old replica is left. In kubectl 1.37.1's source, apart from checking that the controller has seen the latest change, those three checks are the whole test. A pipeline step that waits on this command learns that and nothing more.
Exit code 1 with exceeded its progress deadline comes from the Deployment's Progressing condition turning False with reason ProgressDeadlineExceeded, after progressDeadlineSeconds (600 by default) without progress. Kubernetes only reports it; it does not roll back for you.
Current kubectl names the Deployment in its waiting lines, for example Waiting for deployment "nginx-deployment" rollout to finish: 2 of 3 updated replicas are available..., so your output will not match the documentation word for word.

Where this goes wrong

The rollout is trusted as a health signal
The deploy is marked done, everybody moves on, and the errors are discovered by users or by an alert twenty minutes later — by which point the deploy is no longer the obvious suspect.
The change included a database migration
Rolling back the code does not roll back the schema. If the old code cannot read the new schema, you are now unable to go either forwards or backwards, which is a considerably worse position than the original bug.
Config comes from somewhere the probe does not exercise
For example, a missing environment variable that is only read on the first request to one endpoint. Everything is Ready, and one route returns 500.

If it goes wrong

error: deployment "<name>" exceeded its progress deadline, and kubectl rollout status exits 1: The rollout made no progress for progressDeadlineSeconds. Kubernetes reports this and does nothing else. Read the conditions in kubectl describe deployment <name> and the new pods' events (readiness probe failures, image pulls, quota), then fix forward or run kubectl rollout undo deployment/<name>.
New pods show Unhealthy events with Readiness probe failed: HTTP probe failed with statuscode: 500 while the rollout waits: The probe is doing its job: those pods get no Service traffic and the rollout does not complete. With the default maxUnavailable of 25%, most old pods keep serving. Read the failing handler's logs, then roll back or fix forward.
kubectl rollout status reports success within seconds, though the application takes much longer to warm up: The containers probably have no readiness probe, so each counted as ready as soon as it started. Add one that exercises a real request path. If failures tend to appear shortly after start, set minReadySeconds as well.
kubectl rollout undo finished, but the errors continue: A rollback restores only the Pod template. A changed ConfigMap or Secret, a database migration or a change in another service stays as it is. Check what else shipped with the release and undo that separately, if it can be undone.

The part worth remembering

The general lesson is about what a green signal actually asserts. "Rollout successful" means the pods reached a state your probe defined as ready. It says nothing about correctness, and it cannot, because nothing in the rollout has any idea what your application is for.

So the value of that signal is exactly the quality of your readiness probe. This is the strongest practical argument for probes that do real work: not the probe itself, but that it is the definition of health every automated deployment decision depends on.

The corollary is worth stating too. If the error rate rises at the moment of a rollout and climbs as the rollout progresses, that shape is itself strong evidence — you do not need correlation tooling to read it, and it is the first thing to check when a deploy and an incident share a timestamp.

Security considerations

A rollback runs the previous image again, including any vulnerability the release fixed. Before undoing a release that carried a security fix, decide whether the errors or the vulnerability are worse, and record why.
kubectl rollout undo needs patch on the Deployment, and anyone who can patch it can also change its image or command. Give that permission to the people and pipelines that deploy, not to everyone who can read the namespace.
Config and secrets live outside the revision history, so after a rollback the old code runs against the new ConfigMap or Secret. Give each release's config a new name referenced from the Pod template, so that a rollback carries the matching config with it.
A readiness probe that does real work runs every periodSeconds (10 by default) in every pod, for the pod's whole life. Keep it cheap and bounded: the documentation warns that badly implemented readiness probes can starve the container of resources, and a probe that hits your database multiplies that load by your replica count.

Sources

The behaviour this page describes, as documented by the projects themselves.

Terms used here (3)plain definitions

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Blue-Green Deployment: A release strategy that runs two identical production environments called 'blue' and 'green.' Traffic is switched from one to the other once the new version is verified, enabling instant rollback by redirecting traffic back to the previous environment.
Canary Release: A deployment technique where a new version is rolled out to a small subset of users before being promoted to the entire fleet. This limits the blast radius of bugs and allows teams to monitor real-world performance before full rollout.
Zero-Downtime Deployment: A deployment strategy where new versions are released without any service interruption to end users. Techniques include rolling updates, blue-green deployments, and canary releases, each of which can keep the service available during the transition, provided new instances receive traffic only once ready and old ones finish in-flight requests before exiting.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it