Prometheus · troubleshooting

context deadline exceeded

The scrape took longer than scrape_timeout, so Prometheus gave up and marked the target down. The default window is 10 seconds. The usual cause is an exporter doing real work per scrape or returning far more series than anyone intended — and raising the timeout alone will not load once it is greater than the scrape interval.

First published 6 Oct 2026

New to this? Read what is actually happening first, then come back to the steps.

On this page · 9 sections

Before you start

Access to the Prometheus web UI or its HTTP API, to read each target's health, last error and scrape settings. The examples here use port 9090.
A shell on the Prometheus host with curl, so a hand-run fetch of the target's /metrics URL takes the same network path Prometheus does.
The job's scrape settings (scheme, path, proxy and credentials) from prometheus.yml or the targets API, so your fetch matches what Prometheus sends.
promtool from the same Prometheus release, to check a changed configuration before you load it.
Permission to edit the configuration and reload it: send SIGHUP to the process, or POST to /-/reload when --web.enable-lifecycle is set.

Run this first

step 1 of 4

curl -o /dev/null -s -w 'time_total=%{time_total}s size=%{size_download}B\n' http://<target>:<port>/metrics

Look for A total near the scrape_timeout, and a response size large enough to explain it

Measures the same thing Prometheus measures, from outside Prometheus. It separates a slow target from a slow path in one step.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Timed by hand with curl, from the Prometheus host and again from next to the target, how long does /metrics take?

curl -o /dev/null -s -w '%{time_total}\n' http://<target>:<port>/metrics

Cause space

5 of 5 still possible

The exporter does real work on every scrapeCommon
The target returns far more series than it shouldCommon
The path between Prometheus and the target is slowOccasional
The target cannot get enough CPU to answer in timeOccasional
The difference is in how Prometheus fetches itRare

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Or diagnose it manually

In this order. The first command usually contains the whole answer.

Step 1
```
curl -o /dev/null -s -w 'time_total=%{time_total}s size=%{size_download}B\n' http://<target>:<port>/metrics
```
Measures the same thing Prometheus measures, from outside Prometheus. It separates a slow target from a slow path in one step.
Look for A total near the scrape_timeout, and a response size large enough to explain it
Step 2
```
curl -sG --data-urlencode 'query=scrape_duration_seconds{job="<job>"}' --data-urlencode 'start=<3 days ago, RFC 3339>' --data-urlencode 'end=<now, RFC 3339>' --data-urlencode 'step=1h' http://<prometheus>:9090/api/v1/query_range
```
A range query shows whether durations jumped suddenly or crept toward the timeout over days, and whether one instance or the whole job is affected.
Look for Durations close to the timeout, and whether peers in the same job are unaffected
Step 3
```
curl -sG --data-urlencode 'query=topk(10, scrape_samples_scraped)' http://<prometheus>:9090/api/v1/query
```
Ranks targets by how many samples they expose. Cardinality problems stand out immediately against their peers.
Look for One instance far above comparable ones running the same software
Step 4
```
curl -s http://<target>:<port>/metrics | awk '!/^#/ && NF {print $1}' | cut -d'{' -f1 | sort | uniq -c | sort -rn | head -20
```
Names the specific metric responsible when the sample count is the problem.
Look for A single metric name with a disproportionate series count

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

The exporter does real work on every scrape

Common

Many exporters compute at scrape time rather than in the background — querying a database, walking a filesystem, calling a cloud API, running SNMP against a device. As the thing being measured grows, so does the work, and the scrape crosses 10 seconds. Timing the endpoint by hand is decisive, because it measures exactly what Prometheus measures.

Confirm

curl -o /dev/null -s -w 'time_total=%{time_total}s size=%{size_download}B\n' http://<target>:<port>/metrics

time_total at or above the configured scrape_timeout. Run it a few times — the first may be warm-cache fast

Fix

Raise scrape_timeout for that job — but note it cannot exceed scrape_interval, so a timeout above the job's interval (1 minute by default) means raising the interval too.
Better, make the exporter stop doing the work inline: cache results and refresh them on a background schedule, so a scrape reads a value rather than computing one.
Give slow exporters their own scrape config with a longer interval and timeout, instead of lifting the global values for every target.

promtool check config /etc/prometheus/prometheus.yml

Catches the timeout-greater-than-interval rejection before a reload fails.

The target returns far more series than it should

Common

A label carrying something unbounded — a request ID, a user ID, a full URL path, a timestamp — multiplies series without limit. The response grows until generating and transferring it no longer fits the window. The tell is that one instance's sample count dwarfs its peers running the same code, and the body is large enough to be slow on its own.

Confirm

curl -s http://<target>:<port>/metrics | awk '!/^#/ && NF {print $1}' | cut -d'{' -f1 | sort | uniq -c | sort -rn | head -20

One metric name with thousands of series. That metric's labels are where the unbounded value is

Fix

Remove the unbounded label at the source. That is the only fix that reduces the work rather than moving it.
Where the source cannot be changed quickly, drop the offending series with metric_relabel_configs — but the target still generates and sends them, so this protects storage rather than the scrape.
Consider sample_limit as a guard so one runaway target fails loudly instead of degrading the whole server. It defaults to 0, meaning no limit, and exceeding it fails the entire scrape by design.

The path between Prometheus and the target is slow

Occasional

The deadline covers the whole request, including DNS resolution, connection setup and TLS handshake. A target behind a proxy, across a region, or resolving through a struggling DNS server can blow the window while the exporter itself is fast. The giveaway is that scraping the same endpoint from close by is quick.

Confirm

curl -o /dev/null -s -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' http://<target>:<port>/metrics

curl's timers are cumulative, so subtract consecutive values and see which gap is large: connect minus namelookup, then for https:// appconnect minus connect and starttransfer minus appconnect. For an http:// URL tls stays 0, so the wait before the first byte is starttransfer minus connect. A large DNS or connect share points at the path; a large wait before the first byte points at the exporter

Fix

Fix whichever phase the breakdown names — DNS resolution, routing, or a TLS handshake renegotiating per scrape.
Scrape from a Prometheus closer to the target and federate or remote-write upward, rather than scraping long-haul on a short deadline.

The target cannot get enough CPU to answer in time

Occasional

An application throttled by a CPU limit, or paused by garbage collection, serves /metrics slowly even when the handler is trivial. Scrape failures then correlate with load on the target rather than with anything about the exporter. This one is easy to misread, because the monitoring fails exactly when the system is busiest and most worth watching.

Confirm

curl -o /dev/null -s -w 'total=%{time_total}\n' http://<target>:<port>/metrics   # run during peak and again when idle

A large gap between the two. Fast when idle, slow under load, points at contention rather than at the handler

Fix

Raise the CPU limit, or reduce throttling, for the target workload.
Where the target is a sidecar exporter, give it its own small reservation so it is not starved by the main container.
Treat this as a finding about the workload, not only about monitoring — if scrapes cannot be served under load, user requests are likely suffering too.

The difference is in how Prometheus fetches it

Rare

If the endpoint is fast when you fetch it yourself, the difference is in how Prometheus fetches it: the exact URL, scheme, proxy or credentials the job uses, which a hand-run curl did not reproduce, or a Prometheus server too loaded to finish its scrapes in time.

Confirm

curl -s http://<prometheus>:9090/api/v1/targets | grep -o '"scrapeUrl":"[^"]*"'

The exact URL Prometheus scrapes for this target. Fetch that URL from the Prometheus host, with the job's proxy and credentials, and time it

Fix

Reproduce the scrape exactly before tuning anything: same URL, scheme, proxy and credentials as the job.
If it is fast even then, look at the server: scrape_duration_seconds rising across every job points at Prometheus, not at one target.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Prometheus collects metrics by fetching a page from each target on a schedule, and it gives every fetch a time limit. This error means the target had not finished answering when the limit ran out, so Prometheus threw that fetch away and marked the target as down. You can allow more time, but never more than the gap between fetches, so the real job is finding out why the target is slow.

Where the timeout shows up: the targets API

Documented output, not reproduced: Prometheus is not installed on the machine used for this page. This is an excerpt of the example response for GET /api/v1/targets in the Prometheus HTTP API documentation, showing one healthy target; the full example also lists discovered labels and dropped targets. The notes below say what changes for a target that times out, using the documentation and the Prometheus v3.15.0 source.

Command

curl http://localhost:9090/api/v1/targets

Output · as shown in Prometheus HTTP API: Targets (example response)

        "labels": {
          "instance": "127.0.0.1:9090",
          "job": "prometheus"
        },
        "scrapePool": "prometheus",
        "scrapeUrl": "http://127.0.0.1:9090/metrics",
        "globalUrl": "http://example-prometheus:9090/metrics",
        "lastError": "",
        "lastScrape": "2017-01-17T15:07:44.723715405+01:00",
        "lastScrapeDuration": 0.050688943,
        "health": "up",
        "scrapeInterval": "1m",
        "scrapeTimeout": "10s"

How to read it

health is up here. After a failed scrape, a timeout included, it becomes down, matching the up series for the target dropping to 0.
lastError is empty because the last scrape worked. Go's HTTP client wraps request errors with the method and URL, so a timeout before the target starts answering reads Get "<scrapeUrl>": context deadline exceeded.
lastScrapeDuration is in seconds: 0.05 here. Compare it with scrapeTimeout; a value close to the timeout means one slower scrape will fail.
scrapeInterval and scrapeTimeout are the values this target really uses, after defaults and any relabeling are applied: here the defaults, 1 minute and 10 seconds. Check them here after a configuration change rather than trusting the file.
scrapeUrl is the exact URL Prometheus fetches. Time that URL with curl from the Prometheus host to see what Prometheus sees.

Words used on this page

Scrape: One fetch of a target's metrics page, usually /metrics, by Prometheus.
scrape_interval: How often each target is scraped. The default is 1 minute; it can be set globally and overridden per scrape job.
scrape_timeout: How long one scrape may run before it is cancelled. The default is 10 seconds and it cannot be greater than the interval: a job that sets a larger value fails to load, and one that inherits a larger global value gets its own interval as the timeout instead.
up: A series Prometheus writes for every target on every scrape: 1 if the target was reached and the scrape worked, 0 if the scrape failed.
scrape_duration_seconds: How long each scrape took, recorded for every target. Compare it with the timeout to see how close a target is to failing.
Context deadline: Go's name for a time limit attached to a piece of work. context deadline exceeded is the exact text of Go's error when that limit passes, so it says only that time ran out, not why.
Stale marker: What Prometheus records when a series stops appearing. A failed scrape counts as an empty one, so the target's series are marked stale and queries stop returning them.
Cardinality: How many distinct series a metric produces: one for each combination of label values. A label with unbounded values, such as a user ID, multiplies it.

What is actually happeningadvanced

Prometheus gives every scrape a deadline. If the target has not finished responding by then, the request is cancelled and the scrape is recorded as failed. The message itself comes from Go's context package and only means a deadline passed. When the deadline passes before the target answers, Go's HTTP client puts the method and URL in front of it, so the full error reads Get "<scrape URL>": context deadline exceeded.

The deadline is scrape_timeout, which defaults to 10 seconds, and the companion setting is scrape_interval, which defaults to 1 minute. Both can be set globally and overridden per scrape config. The documentation states one constraint plainly: scrape_timeout cannot be greater than the scrape interval. That single line explains why the obvious fix sometimes refuses to load.

A failed scrape is not a gap in one metric. The whole scrape is discarded, so every series from that target is missing for that interval, and up for the target goes to 0. A dashboard built on those series shows a hole, and alerts written on them can resolve for lack of data, while alerts on up == 0 or absent() fire — which is why a slow exporter often looks like an outage in the thing being monitored rather than a problem with the monitoring.

Scrape duration is itself a metric. Prometheus records scrape_duration_seconds for every scrape, alongside scrape_samples_scraped for how many samples the target exposed and scrape_series_added for roughly how many are new. A target that fails intermittently almost always shows a duration creeping toward the timeout for a while first, which makes this one of the more predictable failures to catch early.

Worth separating from the timeout: Prometheus can also fail a scrape deliberately on limits. sample_limit and label_limit both default to 0, meaning no limit, and when either is exceeded the documentation says the entire scrape is treated as failed. That is a different error from a deadline, so read the target's reported error rather than assuming.

How to tell this is your problemsignals

Where	What you see
Prometheus UI, Status > Target health (Status > Targets in 2.x)	The target in red with Last Error: context deadline exceeded
Query: up{job="<job>"}	0 for the affected instances, 1 for the healthy ones
Query: scrape_duration_seconds{job="<job>"}	A value at or just under the configured timeout, often climbing over days
Query: scrape_samples_scraped{job="<job>"}	A sample count far larger than comparable targets in the same job

How to know it is actually fixed

The target shows healthy in Status > Target health (Status > Targets in 2.x), with no Last Error.
up for the instance is 1 and stays there across several intervals, not just immediately after a reload.
scrape_duration_seconds sits well below scrape_timeout with real headroom, rather than just inside it.
If the timeout or interval was changed, promtool check config passes and the reload is reflected in the target's configured values.

Stopping it happening again

Alert on scrape_duration_seconds / scrape_timeout_seconds approaching 1 rather than on up == 0. The ratio warns while there is time to act; up == 0 is the aftermath. The scrape_timeout_seconds series is recorded only when extra_scrape_metrics: true is set in the global or scrape config; the older --enable-feature=extra-scrape-metrics flag is deprecated.
Watch scrape_samples_scraped per job over time. Cardinality growth is gradual and invisible until the day it is not.
Keep unbounded values out of labels — IDs, paths, timestamps, anything per-request. This is the single biggest cause of both slow scrapes and expensive storage.
Give genuinely slow exporters their own scrape config rather than raising global values, so one slow target does not loosen the deadline for everything.
Remember the constraint when planning: scrape_timeout may not exceed scrape_interval, so the two have to be changed together.

Security considerations

The targets page and /api/v1/targets list every target's URL, labels and last error. Prometheus's security model presumes that untrusted users can reach its HTTP endpoint, so redact internal hostnames and addresses before you paste this output into a public issue.
When you repeat a scrape by hand with the job's bearer token or password, keep it off the command line, where shell history and process listings can show it. curl's -H @file reads the headers from a file instead.
Do not fix a slow TLS handshake with insecure_skip_verify: true. It disables validation of the target's certificate, so anything that can intercept the connection can feed Prometheus false metrics or collect the credentials the job sends.
Labels that blow up cardinality, such as user IDs or full request paths, also copy that data into Prometheus, and its security model treats metrics as not secret. Removing them fixes the slow scrape and stops the exposure.
Keep scrape credentials in the fields the configuration documentation marks as secret, such as a password or credentials file, not in params or labels. Prometheus keeps secret fields out of its logs and HTTP API but makes no such promise for other fields.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it

Before you start

Run this first

Work out which cause you have

Or diagnose it manually

Every cause, and how to fix it

The exporter does real work on every scrape

The target returns far more series than it should

The path between Prometheus and the target is slow

The target cannot get enough CPU to answer in time

The difference is in how Prometheus fetches it

Understanding it properly

In plain terms

Where the timeout shows up: the targets API

Words used on this page

How to know it is actually fixed

Stopping it happening again

Security considerations

Sources

Where to go next