CI/CD · why it works that way

Why do CI/CD systems insist on immutable artifacts?

Short answer

So that testing something means anything. If the artifact can change between the test and the deploy, then passing tests tells you about a thing that no longer exists.

Reproduced on shasum 6.02, 12 Oct 2026. Where the text says “reproduced”, the behaviour was run on that version, not taken from documentation.

First published 4 Sept 2026

On this page · 7 sections

The constraint behind it

A pipeline's value rests on a chain of inference: this artifact passed these tests, therefore deploying it is reasonably safe. Every link has to hold for the conclusion to.

A mutable reference breaks the chain silently. If myapp:latest is tested and then myapp:latest is deployed, those are two lookups of a name, not two uses of a thing. Anything pushed in between changes what runs while every check stays green.

The failure is invisible where people look. The repository is unchanged, the pipeline is green, and production is running something nobody tested.

Words used on this page

artifact: The file or image a build produces and later stages test and deploy: a tarball, a package, a container image.
digest: A hash of an artifact's exact bytes, such as sha256:2e86... Change one byte and the digest changes.
tag: A human-friendly name, such as latest or v1.42.0, that a registry points at one image. It can be pointed somewhere else.
mutable tag: A tag the registry lets you move by pushing a different image under the same name.
tag immutability: A registry setting that refuses a push reusing an existing tag. In Amazon ECR it is set per repository and can list exceptions.
pinning: Referring to an artifact by digest, such as image@sha256:..., so the reference can only ever mean those bytes.
promotion: Moving the same artifact from one environment to the next, for example staging to production, without rebuilding it.
rollback: Deploying the previous artifact again. It only returns to a known state if that artifact still exists unchanged.
lifecycle policy: Registry rules that delete or archive old artifacts automatically, by age or count.

If it were the other way

With mutable artifacts, rollback stops meaning anything. "Deploy the previous version" resolves a tag that may now point somewhere else, so you cannot reliably return to a known state.

Reproducing an incident becomes guesswork, because "we were running v2.1" no longer identifies specific bytes.

And promotion between environments loses its guarantee: what staging validated and what production received are two resolutions of the same name, which may differ.

What it buys and costs you

What it buys you

A rollback that actually returns to a known state, because the previous artifact still exists and is still itself.
Meaningful promotion. The thing that passed staging is bit-for-bit the thing production runs.
Answerable questions after an incident: exactly what was running, and what was in it.

What it costs you

Storage and lifecycle work. Every build is a distinct artifact, so retention becomes a policy rather than an afterthought.
Discipline in version handling. Tags have to be generated rather than reused, which is more machinery than latest.
It does not come for free from tagging alone. Whether a tag can be moved is a registry setting, not a property of the tag: some registries can lock tags, but nothing in the reference tells a consumer so. Only a digest is bound to content.

Where it bites in practice

In the gap between an immutable tag and a verified one. Teams stop using latest, adopt version tags, and reasonably believe the problem is solved — but unless the registry locks that particular tag, a tag is still a pointer that can be moved. Only a digest, or a signature, makes immutability something you can check rather than something you agreed to.

Try it yourself

Before you start

A shell with shasum. It ships with macOS and, on Debian-based Linux, with the perl package. Everything on this page was reproduced with shasum 6.02.
A scratch folder to stand in for an artifact store. Do not try this against a shared bucket or registry: overwriting a real artifact is the problem this page is about.
For container images, the digest your registry reports. docker push and docker pull print it as Digest: sha256:....
Read access to your registry's tag settings. In Amazon ECR that is the repository's imageTagMutability and any tag exclusion filters.
A place where a later pipeline stage can read the digest the build recorded, such as a build output or the deployment manifest, and that the artifact store cannot overwrite.

Catching an artifact that changed under the same name

Reproduced on shasum 6.02 (Perl Digest::SHA) on macOS, with the time zone set to UTC, in a scratch folder. store/ stands in for an artifact store that allows overwrites, such as a bucket. store/myapp-latest.tar.gz and build-2/myapp.tar.gz are two builds of version 1.4.0 that differ in one setting. Your digests will differ, because tar records file times.

Command

shasum -a 256 store/myapp-latest.tar.gz | tee tested.sha256
cp build-2/myapp.tar.gz store/myapp-latest.tar.gz
ls store
shasum -a 256 -c tested.sha256
echo $?
shasum -a 256 store/myapp-latest.tar.gz

Output

$ shasum -a 256 store/myapp-latest.tar.gz | tee tested.sha256
daf2d86ed98080f4500681245c369b9fce025bfec5ffe13da22cb7561d8d170a  store/myapp-latest.tar.gz

$ cp build-2/myapp.tar.gz store/myapp-latest.tar.gz

$ ls store
myapp-latest.tar.gz

$ shasum -a 256 -c tested.sha256
store/myapp-latest.tar.gz: FAILED
shasum: WARNING: 1 computed checksum did NOT match

$ echo $?
1

$ shasum -a 256 store/myapp-latest.tar.gz
1f0f708bfa882483e887f302c7803ab552f328c3555e9023781ed14beb41f91e  store/myapp-latest.tar.gz

How to read it

At test time the pipeline records the SHA-256 digest of the exact bytes it tested, daf2d86e..., in tested.sha256. The file name alone says nothing about content.
A second build is copied over the same name, and ls shows nothing different. Anything that deploys myapp-latest.tar.gz from here ships bytes that were never tested.
The check compares the bytes on disk with the recorded digest. It prints FAILED and WARNING: 1 computed checksum did NOT match, and exits with 1. A deploy step that runs this check first and stops on a non-zero exit refuses the untested file.
The new file has a different digest, 1f0f708b.... Container images work the same way: Docker prints an image's digest after a push or pull, and docker pull ubuntu@sha256:<digest> fetches exactly that image, where ubuntu:24.04 can move to a newer one.

If it goes wrong

store/myapp-latest.tar.gz: FAILED followed by shasum: WARNING: 1 computed checksum did NOT match: The artifact changed after its digest was recorded. Do not deploy it. Find out what wrote to that name, deploy the artifact that matches the recorded digest, and make the store refuse overwrites or give every build a unique name.
Amazon ECR rejects a push with ImageTagAlreadyExistsException: The repository has tag immutability on, and the tag already exists. That is the protection working. Push under a new, unique tag such as the commit ID. If one tag really has to move, ECR can list it as an exclusion, but every consumer of that tag then loses the guarantee.
Using default tag: latest in docker pull output, when you meant a specific version: The image reference had no tag, and Docker and Kubernetes both fill in latest. Give an explicit version tag, or better, the digest.
Pods started from the same Deployment run different code under the same tag: The tag was moved at the registry, and nodes that already had the old image kept using it. Kubernetes' documentation describes exactly this mix of versions. Reference the image by digest, image@sha256:..., so every node runs the same bytes.

Security considerations

A tag is only as trustworthy as everyone who can push to it. Anyone with push access to a mutable tag, including an attacker with stolen registry credentials, changes what every tag-based deploy runs next. Kubernetes' documentation says pinning by digest means "a change at the registry cannot lead to that mix of versions".
Record the digest from the build or push output, not by looking the tag up later. Docker prints the digest when pushing so you can pin what you just pushed. Resolving the tag at deploy time repeats the lookup you were trying to avoid.
Keep the recorded digest out of reach of whoever can write the artifact. A checksum file stored beside the artifact can be overwritten along with it. The digest belongs in the pipeline's own output or the deployment manifest.
Immutability in ECR is a repository setting with exceptions. A repository set to IMMUTABLE_WITH_EXCLUSION still lets the excluded tags move, and nothing in an image reference shows which tags those are. Check the setting, and manage it as code.
Lifecycle policies can delete what a rollback needs. ECR expires images within 24 hours of them meeting a policy's criteria, and recommends the lifecycle policy preview before applying a policy. Make sure the rules keep every digest a running or rollback deployment points to.

Sources

The behaviour this page describes, as documented by the projects themselves.

Terms used here (2)plain definitions

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Blue-Green Deployment: A release strategy that runs two identical production environments called 'blue' and 'green.' Traffic is switched from one to the other once the new version is verified, enabling instant rollback by redirecting traffic back to the previous environment.
Immutable Infrastructure: An approach where servers or containers are never modified after deployment. Instead of patching running systems, you build a new image with the changes and replace the old instances entirely, eliminating configuration drift and improving reliability.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it