Kubernetes · why it works that way

Why does Kubernetes use desired state instead of commands?

Short answer

Because over an unreliable network you cannot know whether a command arrived. Declaring the intended end state makes that uncertainty harmless — the same declaration can be applied any number of times and the result is identical.

First published 4 Sept 2026

On this page · 7 sections

The constraint behind it

Every instruction to a distributed system crosses a network that can drop, delay or duplicate it. When a request times out, the sender genuinely does not know whether it succeeded, failed, or is still in flight.

For a command like "create one more replica", that ambiguity is unrecoverable. Retrying might create a second one. Not retrying might leave you short. There is no way to tell which, because the information you need is on the other side of the failure.

Declaring "there should be three replicas" removes the problem entirely. Send it once or a hundred times; the outcome is three replicas. The uncertainty still exists, but it no longer matters.

Words used on this page

desired state: What you have declared should exist, written in an object's spec, such as three replicas of a given image.
current state: What actually exists right now. Controllers report it back in the object's status.
controller: A control loop that watches the cluster and makes or requests changes to move the current state towards the desired state.
declarative configuration: Keeping each object's full configuration in files and running kubectl apply, so the files rather than a history of commands say what should exist.
idempotent: Safe to repeat: doing it twice leaves the same result as doing it once. Applying an unchanged file again makes kubectl apply print unchanged.
Pod template: The part of a Deployment's spec that describes the Pods to create. Changing it, for example the image, starts a rollout.
ReplicaSet: The object a Deployment creates for each version of its Pod template. It creates and deletes Pods to keep the declared number running.
rollout: A Deployment replacing Pods from the old template with Pods from the new one, a few at a time.
event: A short record of something a component did or could not do, shown at the end of kubectl describe and by kubectl get events.
progress deadline: .spec.progressDeadlineSeconds, 600 by default: how long a rollout may make no progress before the Deployment reports that it has failed.

If it were the other way

A command-based cluster would need every instruction to be exactly-once, which over a network means sequence numbers, acknowledgements, deduplication and a durable log of what has been applied — effectively a distributed transaction for every scaling operation.

And drift would have nowhere to go. If a pod dies, no command was issued, so nothing happens. Recovery would require something to notice the discrepancy and issue a corrective command — which is a reconciliation loop, arrived at reluctantly and with worse guarantees.

What it buys and costs you

What it buys you

Idempotency for free. kubectl apply twice does what it did once, and GitOps relies on that: a git repository is a declaration, replayed continuously.
Self-healing as a side effect rather than a feature. The controller does not distinguish "a pod died" from "you asked for one more"; both are just a gap between desired and actual.
Auditability. The declaration is the intent, so reviewing a diff tells you the end state you are asking for, not a list of steps to attempt. It does not tell you that the end state will be reached.

What it costs you

Everything is eventually consistent. kubectl apply returns when the declaration is accepted, not when it is true, which is why rollout status exists as a separate thing to wait on.
Failures are asynchronous and appear somewhere else. A manifest is accepted, and the reason it cannot be satisfied shows up later and elsewhere: in a Pod's status, such as ImagePullBackOff, or, when no Pod could be created at all, in a Deployment condition such as ReplicaFailure.
Imperative operations are awkward. "Restart this deployment" is not a declaration, which is why the idiom is to patch an annotation and let the diff trigger a rollout — a workaround that reveals the model's edge.

Where it bites in practice

When something silently does not happen. A manifest applies cleanly, and nothing runs. There is no error, because accepting a declaration and satisfying it are different steps — and the reason lives in kubectl describe, in events, rather than in the response to the command you ran.

Try it yourself

Before you start

A Kubernetes cluster you are free to break, not production, and kubectl configured to talk to it.
The Deployments page's example Deployment, created first with kubectl apply -f https://k8s.io/examples/controllers/nginx-deployment.yaml: three replicas of nginx, named nginx-deployment.
Permission in that namespace to update Deployments and to read Deployments, ReplicaSets, Pods and events, so that kubectl describe can show you the reasons.
A second terminal, or Ctrl-C ready: kubectl rollout status keeps watching until the rollout finishes or its progress deadline passes.

An accepted change that cannot be satisfied

Quoted from the Kubernetes Deployments page, section Rolling Back a Deployment; not reproduced here, because no cluster was available. The Deployment nginx-deployment was running three replicas when the image was changed with a typo: nginx:1.161 instead of nginx:1.16.1. The page shows each output as "similar to this"; Pod names and ages will differ in your cluster.

Command

kubectl set image deployment/nginx-deployment nginx=nginx:1.161
kubectl rollout status deployment/nginx-deployment
kubectl get pods

Output · as shown in Kubernetes — Deployments: Rolling Back a Deployment

$ kubectl set image deployment/nginx-deployment nginx=nginx:1.161
deployment.apps/nginx-deployment image updated

$ kubectl rollout status deployment/nginx-deployment
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...

$ kubectl get pods
NAME                                READY     STATUS             RESTARTS   AGE
nginx-deployment-1564180365-70iae   1/1       Running            0          25s
nginx-deployment-1564180365-jbqqo   1/1       Running            0          25s
nginx-deployment-1564180365-hysrc   1/1       Running            0          25s
nginx-deployment-3066724191-08mng   0/1       ImagePullBackOff   0          6s

How to read it

image updated means the API server accepted the new Pod template, the desired state. It does not mean the image exists: the mistyped tag was accepted exactly as a correct one would have been. kubectl set image is an imperative command, but what it changes is still the declaration.
kubectl rollout status is the separate wait the page describes. It stays on 1 out of 3 new replicas have been updated...; the documentation says to press Ctrl-C. Left alone, it exits non-zero once the progress deadline (600 seconds by default) has passed.
The reason appears only in the Pod list: the new Pod, nginx-deployment-3066724191-08mng, is in ImagePullBackOff. Nothing in the response to the command said so.
The three old Pods are still Running. The Deployment controller stopped scaling up the new ReplicaSet because of maxUnavailable, 25% by default, so the declaration is unsatisfied but the old version keeps serving. The documented fix is to roll back with kubectl rollout undo deployment/nginx-deployment, or to correct the tag and apply again.

If it goes wrong

kubectl rollout status stays on Waiting for rollout to finish: 1 out of 3 new replicas have been updated...: The change was accepted but cannot be satisfied. Run kubectl get pods and kubectl describe pod <new-pod> to find the reason, such as ImagePullBackOff for a mistyped image. Correct the declaration and apply it again, or go back with kubectl rollout undo deployment/<name>.
error: deployment "nginx" exceeded its progress deadline, and kubectl rollout status exits with status 1: The rollout made no progress for .spec.progressDeadlineSeconds (600 by default), and the Deployment's Progressing condition is now False with reason ProgressDeadlineExceeded. The controller keeps retrying, so fix the cause shown by kubectl describe deployment <name> rather than waiting.
Fewer Pods than replicas, and kubectl describe deployment shows ReplicaFailure True FailedCreate: No Pod was created, so there is no Pod to describe. Read the condition's message with kubectl get deployment <name> -o yaml; in the documentation's example it is Error creating: pods "nginx-deployment-4262182780-" is forbidden: exceeded quota. Scale something down or raise the namespace's quota, and the controller finishes the rollout on its own.
kubectl apply warns that a resource is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply: The object was made with kubectl create or kubectl replace, and mixing those with kubectl apply is not supported. kubectl adds the annotation itself this time; from then on manage the object only with kubectl apply, or create such objects with kubectl create --save-config.

Security considerations

Whoever can change the declaration controls what runs. Kubernetes' RBAC guidance warns that permission to create workloads implicitly grants access to the namespace's Secrets, ConfigMaps and PersistentVolumes, and to the API access of any service account in it. Treat write access to Deployments, and to any repository that is applied to the cluster automatically, as that powerful.
Reconciliation undoes manual fixes. kubectl apply sets every field in the file on the live object, so a hotfix made with kubectl edit, such as removing a vulnerable sidecar, is reverted by the next apply of the old file. Make the fix in the declaration.
An accepted change is not a deployed change. In the example above the old Pods kept serving while the new image failed; if the change had been a security patch, the unpatched version would still be live. Make deploy jobs wait on kubectl rollout status, which exits non-zero once the progress deadline passes.
Deleting a Pod does not stop a workload: its ReplicaSet creates a replacement from the same template to restore the declared count. To stop a compromised workload, change the declaration, for example scale it to 0 replicas.

Sources

The behaviour this page describes, as documented by the projects themselves.

Terms used here (3)plain definitions

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Deployment: A Kubernetes resource that manages a set of identical Pods and provides declarative updates, rolling updates, and rollback to an earlier revision on request; it does not roll back a failed rollout by itself. Deployments are the standard way to run stateless applications on Kubernetes.
GitOps: An operational framework where the desired state of applications and infrastructure is declared in a versioned store, usually Git, which serves as the single source of truth. Changes are made by committing to it (typically through pull requests), and software agents automatically pull that state and continuously reconcile the running system toward it.
Rolling Update: A deployment strategy that incrementally replaces instances of the old version with the new version in batches (a Kubernetes Deployment by default allows 25% extra and 25% unavailable Pods at a time). Rolling updates avoid downtime by keeping a minimum number of healthy instances available throughout, provided new instances only receive traffic once ready and old ones shut down gracefully.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it