Skip to main content
DevOps Insightsguides & answers

Docker · troubleshooting

no space left on device

Docker has filled the filesystem, and almost always with things nothing is using: build cache, dangling images, stopped containers and orphaned volumes. Deleting containers does not reclaim their volumes, which is why the space does not come back.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

  • A terminal on the Docker host as root or a docker group member (root-equivalent), with sudo to run du inside Docker's data directory.
  • The location of Docker's data directory from docker info --format '{{.DockerRootDir}}', which is /var/lib/docker by default on Linux.
  • A list of the volumes holding data you must keep, and a backup of any you are unsure about, before you run any prune command.
  • Permission and a maintenance window to restart the Docker daemon if you change daemon.json. Docker applies new log settings only after a restart, and only to containers created afterwards.
  • On Docker Desktop, access to Settings > Docker Engine, which is where daemon.json is edited.

Run this first

step 1 of 3
docker system df

Look for The largest RECLAIMABLE value. That is where the space is, and it is usually build cache or volumes.

One command that attributes the usage across images, containers, volumes and build cache, with a reclaimable figure for each.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Is the full filesystem the one holding Docker's data root?

Check with df -h $(docker info --format '{{.DockerRootDir}}').

Cause space

5 of 5 still possible

  • Build cache has grown without limitCommon
  • Dangling images from repeated rebuildsCommon
  • Volumes left behind by removed containersCommon
  • Container logs have grown unboundedOccasional
  • The full filesystem is not Docker'sRare

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    docker system df

    One command that attributes the usage across images, containers, volumes and build cache, with a reclaimable figure for each.

    Look for The largest RECLAIMABLE value. That is where the space is, and it is usually build cache or volumes.

  2. Step 2

    df -h $(docker info --format '{{.DockerRootDir}}')

    Confirms the full filesystem is Docker's before you delete anything.

    Look for Use% at or near 100 on the filesystem holding Docker's data root.

  3. Step 3

    docker system df -v | head -40

    The per-object breakdown, which names the specific volumes and images consuming the space.

    Look for Individual volumes or images large enough to explain the shortfall.

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

Build cache has grown without limit

Common

BuildKit keeps cache for every build stage, and its default garbage collection is generous: unused cache can stay for up to 60 days, within a size limit that can be more than a small disk can spare. On a machine that builds several times a day this is usually the single largest consumer, and it is entirely reclaimable.

Confirm

docker system df

The Build Cache row. A large TOTAL with an equally large RECLAIMABLE means this is your answer.

Fix

  • Reclaim it with docker builder prune. Add -a to remove all unused build cache, not just dangling entries.
  • Set a ceiling so it does not recur, rather than pruning by hand when builds start failing.
docker builder prune -a

# Then cap it, in daemon.json:
# { "builder": { "gc": { "defaultKeepStorage": "20GB", "enabled": true } } }

The cap is the actual fix. Pruning is the symptom relief. 20GB is already Docker Desktop's default, so on a small disk choose a lower figure.

Dangling images from repeated rebuilds

Common

Rebuilding a tag moves the tag to the new image and leaves the old one untagged. Those <none> images keep every layer they own, and there is no limit on how many accumulate.

Confirm

docker images -f dangling=true --format 'table {{.ID}}	{{.Size}}' | head -20

A long list of untagged images. Each one is holding layers nothing references.

Fix

  • docker image prune removes dangling images. docker image prune -a also removes tagged images no container uses, which is more aggressive than most people intend.
  • On CI hosts, prune as a scheduled job rather than waiting for a build to fail.
docker image prune

Volumes left behind by removed containers

Common

This is the one that surprises people. Removing a container does not remove its named volumes, because a volume exists precisely so that data survives the container. So the space stays used, and docker ps -a shows nothing to explain it.

Confirm

docker volume ls -qf dangling=true | wc -l && docker system df -v | head -30

A non-zero count of dangling volumes, and their sizes in the verbose output. Database volumes are often the largest single objects on the host.

Fix

  • Read the list before deleting anything. docker volume prune removes data permanently, and a dangling volume can still be the only copy of something — a development database whose container was recreated, for example.
  • Once you have confirmed what they are, docker volume prune --all reclaims them (plain docker volume prune removes only anonymous volumes since Docker 23), or remove specific ones with docker volume rm <name>.
# Look first.
docker volume ls -f dangling=true

# Then, if you are sure (--all includes named volumes):
docker volume prune --all

This is the one destructive step on the page. There is no undo, and the data may not exist anywhere else.

Container logs have grown unbounded

Occasional

With the default json-file logging driver and no rotation configured, a container's log file grows for as long as the container runs. A chatty application over several months can produce tens of gigabytes in a single file.

Confirm

sudo sh -c 'du -sh /var/lib/docker/containers/*/*-json.log' 2>/dev/null | sort -h | tail -10

Individual log files measured in gigabytes.

Fix

  • Configure log rotation in the daemon config, which applies to newly created containers.
  • Do not truncate or edit the log file by hand. Docker's documentation says these files are meant for the daemon only and that other tools can interfere with its logging. Set rotation, then recreate the container so it picks the setting up.
# daemon.json
{ "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } }

Applies to containers created after the daemon restarts, not to existing ones.

The full filesystem is not Docker's

Rare

Worth ruling out before pruning anything. If Docker's data root is on its own partition, a full root filesystem produces the same error while Docker's own usage is fine — and pruning will free nothing.

Confirm

df -h $(docker info --format '{{.DockerRootDir}}') && df -h /

Which of the two is actually full. If Docker's data root has space and root does not, pruning images is not the fix.

Fix

  • Find the real consumer on the full filesystem with du -sh /* | sort -h before touching Docker.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Docker keeps old images, build leftovers, stopped containers, log files and volumes until you tell it to delete them. Over time they fill the disk, and the next write fails with this error. The job is to find which of those is using the space, then free it without deleting data someone still needs.

Reading docker system df, from Docker's documentation

Documented, not reproduced: Docker is not installed on this machine. The output is the example on Docker's docker system df reference page. It was captured on an older release, so it has no Build Cache row.

Command

docker system df

Output · as shown in Docker — docker system df

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              5                   2                   16.43 MB            11.63 MB (70%)
Containers          2                   0                   212 B               212 B (100%)
Local Volumes       2                   1                   36 B                0 B (0%)

How to read it

  • RECLAIMABLE is the part nothing uses. For Images it counts the space unique to images no container uses: 11.63 MB of 16.43 MB here, which docker image prune -a could free.
  • Containers: 2 containers, ACTIVE 0, so neither is running. Their writable layers, 212 B in total, are all reclaimable by removing them. The verbose output on the same page shows both as Exited (0).
  • Local Volumes: one of the two volumes is unused, yet 0 B is reclaimable. The verbose output shows why: the unused one, my-named-vol, has LINKS 0 and is empty. LINKS counts the containers using a volume, and a volume with 0 links is what docker volume prune --all would delete.
  • Current Docker CLI source prints a fourth row, Build Cache. On a machine that builds images, check that row before anything else.
  • Container sizes here are writable layers only. Docker Engine's source does not add container log files to this table, so if df -h shows far more used than these rows explain, look at the log files next.

Words used on this page

Docker root directory
Where Docker stores images, containers, volumes and build cache: /var/lib/docker by default on Linux. docker info shows it as Docker Root Dir.
Build cache
Layers and files BuildKit keeps from earlier builds so later builds run faster. Garbage collection trims it by age and size; docker builder prune removes it on demand.
Dangling image
An image with no tag that no container uses, usually the old version left behind when you rebuild a tag. docker image prune removes these.
Writable layer
The layer a container writes its changes to. A stopped container keeps it, and Docker's documentation notes it still takes up disk space until the container is removed.
Named and anonymous volumes
Volumes keep data outside a container's writable layer. Named volumes have a name you chose; anonymous ones get a random ID. Since Docker Engine 23, docker volume prune removes only unused anonymous volumes unless you add --all.
RECLAIMABLE
The docker system df column for space held by things nothing is using: stopped containers, images with no container, volumes no container references.
json-file and local logging drivers
json-file is the default and does no log rotation unless you set max-size. Docker's documentation recommends the local driver to prevent disk exhaustion, because it rotates logs by default.
What is actually happening

Docker accumulates by design. Every build leaves cache, every rebuilt image leaves the previous layers dangling, every stopped container keeps its writable layer, and every named volume outlives the container that created it.

None of that is a leak — it is what makes rebuilds fast and data durable. It just has no automatic ceiling, so on a machine that builds regularly it grows until the disk is full.

The important asymmetry is volumes. Removing a container removes its writable layer but deliberately does not remove named volumes, because the whole point of a volume is to survive the container. So docker rm frees less than people expect, and the missing space is usually in volumes nothing references any more.

It is also worth checking that the full filesystem is actually Docker's. On a host with a separate partition for Docker's data root, the error can come from a different filesystem entirely.

How to tell this is your problem
WhereWhat you see
docker build / docker runwrite /var/lib/docker/...: no space left on device
docker system dfLarge RECLAIMABLE figures against Build Cache, Images or Local Volumes.
df -hThe filesystem holding Docker's data root at or near 100%.

How to know it is actually fixed

  • docker system df shows the reclaimable figure substantially reduced.
  • df -h shows the filesystem back below its threshold.
  • The build or run that failed now completes.

Stopping it happening again

  • Cap the build cache in the daemon config. This is the single highest-value setting on a build host, and it removes the most common cause entirely.
  • Configure log rotation before you need it. The default is unbounded, which is fine for a laptop and not for anything long-lived.
  • Prune on a schedule rather than in response to a failure — a weekly docker system prune on a build host keeps cache, dangling images and stopped containers in check (volumes and logs need their own handling).
  • Treat volume pruning as a separate, deliberate act with the list read first. It deletes the most, but it is not the only loss: docker system prune also removes every stopped container and anything written inside it.

Security considerations

  • docker volume prune --all and docker volume rm delete data permanently. Docker's documentation says volumes are never removed automatically "because to do so could destroy data". List the candidates with docker volume ls -f dangling=true and back up anything you cannot recreate first.
  • docker system prune removes every stopped container, and with it anything written inside those containers. Check docker ps -a for containers you meant to restart or inspect before you run it.
  • Be careful with -f in scheduled prune jobs: it skips the confirmation prompt that warns what will be removed. Never schedule docker volume prune --all -f, and protect volumes you keep with a label and --filter "label!=keep", as Docker's pruning guide shows.
  • Do not edit or truncate files under Docker's containers directory by hand. Docker's json-file documentation says these files are for the daemon only and that other tools can interfere with its logging. Set rotation and recreate the container instead.
  • Container logs hold whatever the application printed, which can include tokens or personal data. Setting max-size and max-file also limits how much of that history stays on disk.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Terms used here (1)

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Volume
A directory accessible to containers in a Pod, used for persisting data beyond the container's lifecycle. Kubernetes supports many volume types including hostPath, NFS, and cloud provider disks, each with different durability and performance characteristics.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it