Kubernetes · why it works that way

Why are Pods ephemeral?

Short answer

Because a system that can replace a workload cannot also promise it will persist. Ephemerality is not a limitation Kubernetes imposes — it is the price of automatic recovery, and you cannot have one without the other.

First published 4 Sept 2026

On this page · 7 sections

The constraint behind it

Kubernetes' central promise is that it keeps your declared state true. You say three replicas; it maintains three replicas, through node failures, evictions, rescheduling and rollouts.

Honouring that promise requires the freedom to destroy and recreate. A node dies, so a pod must be recreated elsewhere. A rollout happens, so old pods must go. A node is drained or a pod evicted, so it is replaced somewhere else. A pod never moves: it is scheduled once, and anything that looks like a move is a new pod.

Once a system has that freedom, it cannot simultaneously guarantee that any particular pod survives. The two properties are in direct conflict, and Kubernetes resolves it in favour of the workload rather than the instance.

Words used on this page

Pod: The smallest thing Kubernetes runs: one or more containers that share a network address and volumes, placed together on one node.
container restart: The kubelet starting a new container inside the same Pod, on the same node, after the old one exited. The Pod keeps its name and UID; the container's own filesystem starts clean.
UID: The unique ID each Pod gets. A replacement Pod can reuse the old name but always has a new UID, and anything tied to the Pod's lifetime, such as an emptyDir volume, is created anew with it.
controller: A control loop that keeps a resource such as a Deployment or ReplicaSet at its declared state, for example by creating a replacement Pod when one goes away.
restartPolicy: The Pod field that says whether the kubelet restarts containers that exit: Always (the default), OnFailure or Never.
emptyDir: A volume created empty when a Pod is assigned to a node. It survives container restarts and is deleted when the Pod is removed from the node.
PersistentVolumeClaim: A request for storage that exists separately from any Pod, so its data outlives the Pods that use it.
StatefulSet: A workload resource that gives each of its Pods a stable name and its own persistent storage, for applications that need them.
eviction: Ending a Pod to free a node, either because the node is short of a resource or because someone is draining it. The Pod is not moved anywhere; a controller may create a new one.

If it were the other way

Imagine pods were durable — a pod, once created, was expected to persist with its identity and local state intact.

Node failure now becomes a human problem, because nothing can safely recreate the pod elsewhere without knowing what was on its disk. Rollouts require in-place upgrades, which means the old and new versions share a filesystem and a lifecycle. Rescheduling for efficiency becomes impossible, because moving a pod would lose data.

That system exists. It is a fleet of long-lived virtual machines, and it works — with configuration management, snapshots, and a person deciding what to do when one fails. Kubernetes is what you get when you trade that durability for automatic recovery.

What it buys and costs you

What it buys you

Self-healing that needs no human decision, because there is never a question of whether a replacement is safe to create.
Rollouts that are just create-and-destroy, which is why a deploy and a recovery use the same machinery.
Genuine portability: a workload that assumes nothing about its host can be scheduled anywhere with capacity.
Horizontal scaling as a number rather than a procedure.

What it costs you

Anything stateful becomes explicit work. Volumes, StatefulSets, PersistentVolumeClaims and their failure modes all exist to reintroduce, carefully, the durability that was traded away.
A container's own filesystem and its emptyDir volumes are scratch space that goes away with the container or the Pod. Software that assumed its local files would persist needs adapting, or a PersistentVolume.
Debugging is harder. The pod that exhibited the problem may not exist by the time you look, which is why centralised logging stops being optional.

Where it bites in practice

File uploads are a clear example. An application writes to local disk, works perfectly in testing, and starts losing files in production. In Kubernetes even a container restart begins with a clean filesystem; an emptyDir volume survives restarts, but not the pod being replaced, which is what every deploy does. Nothing errors, because from the new container's perspective the directory was always empty.

Try it yourself

Before you start

A Kubernetes cluster you are free to experiment in, not production, and kubectl configured to talk to it. The task page this example comes from asks you to check with kubectl version, and suggests minikube or an online playground if you have no cluster.
Permission to create Pods, open a shell in them with kubectl exec, and delete them, in a namespace of your own.
Network access for the cluster to pull the public redis image, and for your machine to fetch the example manifest from k8s.io/examples.
Two terminals: one to keep kubectl get pod redis --watch running, one for the shell inside the container.
For your own application, the path it writes files to, so you can check whether that path is under a volume's mountPath or only in the container's own filesystem.

A file in an emptyDir volume survives a container restart

Quoted from the Kubernetes task page Configure a Pod to Use a Volume for Storage, not reproduced here: no cluster was available. The Pod, redis, was created with kubectl apply -f https://k8s.io/examples/pods/storage/redis.yaml and runs one container with an emptyDir volume mounted at /data/redis. The watch runs in one terminal and the shell in another; its last two lines appear after the kill. The page's steps that install procps and run ps aux are left out here: ps listed redis-server as PID 1, which is the <pid> to kill.

Command

kubectl get pod redis --watch
kubectl exec -it redis -- /bin/bash
cd /data/redis/
echo Hello > test-file
kill <pid>
kubectl exec -it redis -- /bin/bash
cd /data/redis/
ls

Output · as shown in Kubernetes — Configure a Pod to Use a Volume for Storage

$ kubectl get pod redis --watch
NAME      READY     STATUS     RESTARTS   AGE
redis     1/1       Running    0          13s
redis     0/1       Completed  0         6m
redis     1/1       Running    1         6m

$ kubectl exec -it redis -- /bin/bash
root@redis:/data# cd /data/redis/
root@redis:/data/redis# echo Hello > test-file
root@redis:/data/redis# kill <pid>

$ kubectl exec -it redis -- /bin/bash
root@redis:/data/redis# cd /data/redis/
root@redis:/data/redis# ls
test-file

How to read it

Killing the Redis process ended the container, not the Pod. The watch shows the same Pod name throughout, with RESTARTS going from 0 to 1: the kubelet started a new container in the same Pod, on the same node, because the Pod's restartPolicy is Always.
AGE is the Pod's age, not the container's, so it carries on at 6m across the restart. A replacement Pod would start again from a few seconds.
test-file is still in /data/redis in the new container because that directory is the emptyDir volume, which belongs to the Pod. A file written anywhere else in the container would be gone: the page says a container's filesystem changes are lost when it terminates and restarts.
The page ends with kubectl delete pod redis. That is the step that loses test-file. When a Pod is removed from a node for any reason its emptyDir data is deleted permanently, and a replacement Pod, even one with the same name, gets a new UID and a new, empty volume.

If it goes wrong

Files your application wrote are gone, and RESTARTS in kubectl get pods has gone up: The files were in the container's own filesystem, which starts clean on every restart. Mount a volume at the path the application writes to: emptyDir if the data only has to outlive the container, a PersistentVolumeClaim if it has to outlive the Pod. Check with kubectl exec that the path really is under the volume's mountPath.
Files in an emptyDir volume disappear after a rollout or a node drain: That is the documented behaviour: the volume is deleted with the Pod, and a rollout or drain replaces Pods rather than moving them. Store the data on a PersistentVolumeClaim, or in a StatefulSet with volumeClaimTemplates if each replica needs its own.
kubectl logs shows nothing from before the crash you are investigating: Add --previous to read the logs of the last terminated container. By default the kubelet keeps only that one, and an evicted Pod's logs are removed along with its containers, so for anything older you need logs shipped off the node to a central store.
A Pod that uses a local PersistentVolume cannot run after its node fails: A local volume is tied to one node: while that node is unhealthy the volume is inaccessible and the Pod cannot run. Bring the node back or restore the data elsewhere. For data that has to survive losing a node, use network-attached storage, which the Kubernetes documentation prefers for critical data.

Security considerations

Do not reach for hostPath to make data outlive the Pod. Kubernetes' documentation warns that access to the host filesystem can expose privileged credentials, such as the kubelet's, or the container runtime socket, which can be used for container escape. It recommends a local PersistentVolume instead.
A container restart is not a clean-up. It resets the container's own filesystem, but anything written into a volume, emptyDir included, is still there in the restarted container. Deleting the Pod discards its emptyDir; data on a PersistentVolumeClaim survives even that and has to be checked separately.
Evidence goes with the Pod. By default the kubelet keeps the logs of one terminated container, and an evicted Pod's logs are removed with it. If you may need logs for an incident review later, ship them off the node as they are written.
Persistent data outlives the workload on purpose. Deleting or scaling down a StatefulSet does not delete its volumes, so secrets or personal data written there stay until someone deletes the PersistentVolumeClaims.

Sources

The behaviour this page describes, as documented by the projects themselves.

Terms used here (1)plain definitions

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Pod: The smallest deployable unit in Kubernetes, consisting of one or more containers that share networking and storage. Pods are ephemeral by design — they can be created, destroyed, and replaced at any time by higher-level controllers like Deployments.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it