Skip to main content
DevOps Insightsguides & answers

Docker · troubleshooting

net/http: timeout awaiting response headers

Docker 29.7+ and containerd 2.3+ give a registry 30 seconds to start answering, so pushing or pulling a large layer from a slow registry fails. Registries that finish storing or scanning a layer before they answer (Nexus, Harbor and similar) take longer for multi-gigabyte layers. The limit is not configurable yet: push from the classic image store, or make the registry answer faster.

First published

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

  • The failing command's full output.
  • docker version and docker info from the machine or dind service that pushes, or containerd --version on the Kubernetes node that pulls.
  • For the Docker workaround: permission to change the Docker daemon configuration or the dind service's arguments.

Run this first

step 1 of 2
docker info --format '{{.DriverStatus}}'

Look for [[driver-type io.containerd.snapshotter.v1]]

The 30-second limit applies to the containerd image store, not to the classic one.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Where does the error appear?

Cause space

2 of 2 still possible

  • Docker pushes through the containerd image store to a slow registryCommon
  • A Kubernetes node on containerd 2.3+ pulls from a slow registryOccasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

  1. Step 1

    docker info --format '{{.DriverStatus}}'

    The 30-second limit applies to the containerd image store, not to the classic one.

    Look for [[driver-type io.containerd.snapshotter.v1]]

  2. Step 2

    docker image history --human=false --format '{{.Size}}\t{{.CreatedBy}}' REGISTRY/IMAGE:TAG | sort -n | tail -5

    The failing layer is usually the largest; its size tells you how long the registry needs to finalise it.

    Look for Sizes in bytes; layers of several gigabytes (10 digits or more) are the likely ones

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

Docker pushes through the containerd image store to a slow registry

Common

Docker 29.7 or later with the containerd image store (the default on fresh installs and in docker:dind) waits only 30 seconds for the registry's answer to the final upload request of each layer.

Confirm

docker version --format '{{.Server.Version}}'
docker info --format '{{.DriverStatus}}'

29.7 or later and driver-type io.containerd.snapshotter.v1

Fix

  • Push from the classic image store: set "features": {"containerd-snapshotter": false} in /etc/docker/daemon.json and restart Docker, or start a dind service with --feature=containerd-snapshotter=false. Images in the other store are hidden until you switch back, so rebuild or pull what the job needs.
  • Make layers smaller, so each one is finalised within 30 seconds: split large files across layers, and keep data out of images.
  • Ask the registry owner whether blobs can be finalised or scanned asynchronously, so the upload is answered before the work finishes.
{
  "features": {
    "containerd-snapshotter": false
  }
}

In /etc/docker/daemon.json, then restart Docker. Hides images stored in the containerd store until switched back.

A Kubernetes node on containerd 2.3+ pulls from a slow registry

Occasional

The same transport is used for image pulls. A registry that delays its answer for more than 30 seconds, for example while a scanner checks a large layer on first pull, makes the pull fail, and because the scan restarts with each attempt, retries do not help.

Confirm

kubectl get nodes -o wide
kubectl describe pod POD -n NAMESPACE | grep -A3 'Failed'

containerd://2.3 or later in CONTAINER-RUNTIME, and timeout awaiting response headers in the pull error

Fix

  • Have the registry pre-scan or cache the image, so pulls are answered at once; many registries can scan on push instead of on first pull.
  • Pull through a registry mirror or cache that answers quickly.
  • Follow containerd PR #13636 for a per-registry response_header_timeout in hosts.toml.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Uploading a big image layer has two parts: sending the bytes, and the registry confirming it has stored them. Some registries copy or scan the whole layer before confirming, which for a multi-gigabyte layer takes minutes. Newer containerd, which Docker uses for its default image store, stops waiting after 30 seconds and calls the push failed, even though the registry finishes fine a little later. Older Docker storage waits as long as it takes.

Words used on this page

Response headers
The first part of an HTTP answer. The timeout counts from the end of the request until the registry starts answering, not the upload itself.
containerd image store
Docker's image storage based on containerd snapshotters; the default on fresh installs of Docker Engine 29.
Classic image store
The older graph-driver storage (overlay2). Docker's own push code, which has no 30-second header limit.
Blob commit
The final request of a layer upload, after which the registry stores the layer and answers 201 Created.
What is actually happening

containerd added ResponseHeaderTimeout: 30 * time.Second to its default registry HTTP transport (PR #13164, first released in containerd 2.3.0), so that a stalled registry connection fails fast instead of hanging for minutes. It applies to every request, including the final upload request of a layer and image pulls.

Docker Engine 29.7.0 moved from containerd 2.2.5 to 2.3.3; 29.9.2 uses 2.4.2, which still has the 30-second limit. Pushes and pulls through the containerd image store use it; the classic store's own code does not.

Registries that finalise a blob before answering, by copying it into their blob store or scanning it, take time proportional to the layer's size. Reports in moby#53700 describe Nexus answering 201 after about 100 seconds for a 5 GB layer, after Docker had already given up, and Harbor failing on a 3.8 GB layer every retry.

A per-registry override in hosts.toml is proposed (containerd PR #13636) but was not merged on 21 Oct 2026; hosts.toml supports only dial_timeout today.

Kubernetes nodes running containerd 2.3 or later can hit the same limit while pulling from such a registry, which shows up as an image pull error rather than a push error (containerd#13682).

How to tell this is your problem
WhereWhat you see
docker pushfailed commit on ref "layer-sha256:…": … net/http: timeout awaiting response headers
Push progressA large layer reaches 100%, waits about 30 seconds, then fails; retries fail on the same layer.
Registry request logThe final PUT for the layer completes with 201 after the client has disconnected.
Kubernetes eventsFailed to pull image … timeout awaiting response headers, from a node on containerd 2.3+

How to know it is actually fixed

  • The push completes and the registry lists the tag.
  • Back on the containerd image store (if you switched), the same push succeeds once the registry answers within 30 seconds or a configurable timeout ships.

Stopping it happening again

  • Keep image layers small; very large layers are slow everywhere, not just here.
  • Pin the Docker or dind version in CI, and test a push of your largest image before moving to a new minor version.
  • Watch containerd PR #13636 and the Docker release notes for a configurable timeout, then return to the default image store.

Security considerations

  • Do not turn off the registry's malware scanning to beat the timeout without the registry owner's agreement; scanning delays are one of the documented triggers, and the scan is there for a reason.
  • Switching Docker's image store hides, but does not delete, the images and containers in the other store. Check what a CI host relies on before changing it.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Where to go next