Skip to main content
DevOps Insightsguides & answers

HTTP · commonly confused

502 Bad Gateway vs 504 Gateway Timeout

Short answer

A 502 means the attempt ended — refused, or the upstream closed the connection before its response header was complete. A 504 means the attempt never ended, and the proxy stopped waiting. One is a failure; the other is a decision, and they need opposite fixes.

Reproduced on curl 8.7.1 and Python 3.9.6, . Where the text says “reproduced”, the behaviour was run on that version, not taken from documentation.

First published

On this page · 10 sections

In simple terms

Both are a proxy telling you it could not give you the upstream's answer, which is why they get conflated. The difference is what happened to the attempt.

With a 502, something concluded badly. The connection was refused, or it opened and then closed without a complete response. There is an event you can point at.

With a 504, nothing concluded at all. The proxy gave up waiting, and — this is the part people miss — the upstream is quite possibly still working on it. Your request may well complete after you have been told it failed.

That distinction decides the fix. For a 502, the job is finding what broke. A 504 means deciding whether the upstream is too slow or the timeout is too short, which are different problems with opposite remedies.

Words used on this page

gateway
A server that takes the client's request, passes it on to another server and relays the answer. RFC 9110 also calls it a reverse proxy; nginx with proxy_pass is one.
upstream
The server the proxy passes requests to. nginx can group several in an upstream block and pick one per request.
error log
nginx's own record of failures, separate from the access log. The access log has the 502 or 504; the error log has the reason.
errno
The operating system's error number, which nginx prints in brackets with its text, as in (111: Connection refused). 110 and 111 are Linux's numbers; macOS uses 60 and 61 for the same errors.
TCP handshake
The exchange of packets that opens a connection. while connecting to upstream in nginx's log means the failure happened here, before any request was sent.
accept queue
Connections waiting for the server program to pick them up. When it is full, new connection attempts may be refused or ignored, depending on the system.
proxy_read_timeout
How long nginx waits between two reads from the upstream before giving up, 60 seconds by default. It is not a limit on the whole response.
idempotent
A request that has the same effect whether it runs once or several times. RFC 9110 lists PUT, DELETE and the safe methods such as GET; POST is not, so repeating it can do the work twice.

What actually happens

A 502 in nginx comes with a specific error-log line, and the lines map almost one-to-one onto causes: connect() failed (111: Connection refused), or connect() to <address> failed when the connect() call itself fails, means nothing was listening; upstream prematurely closed connection means the upstream closed the connection before sending a complete response header; upstream sent too big header is a buffer limit; no live upstreams means the whole pool was ejected.

A 504 comes from a timeout expiring, and nginx names the stage: while connecting to upstream is the TCP handshake never completing, while reading response header is the upstream not answering in time. The default proxy_read_timeout is 60 seconds, so a 504 from nginx at its defaults comes after a round 60 seconds of waiting.

The stage matters more than the status. A connect-stage 504 means the handshake never completed: packets dropped somewhere on the path, or an upstream so overloaded that its accept queue is full and new connections are ignored. A refusal would have produced a fast 502 instead. A read-stage 504 is about latency or queueing.

One consequence worth internalising: a 504 is not safe to retry blindly. The request may have been fully processed, so retrying a non-idempotent operation can duplicate it. A 502 from a refused connection is safe to retry; a 502 from a mid-response close is not, for the same reason.

Telling them apart from the proxy log alone

# 502 — the attempt concluded badly
connect() failed (111: Connection refused) while connecting to upstream
upstream prematurely closed connection while reading response header from upstream

# 504 — the attempt never concluded
upstream timed out (110: Connection timed out) while reading response header from upstream
upstream timed out (110: Connection timed out) while connecting to upstream
#                                                    └── network path, not latency

Both statuses look identical to the user and to most dashboards. The proxy's error log is where the distinction lives.

Try it yourself

Before you start

  • Read access to the proxy's error log, and its path. For nginx, the error_log directive sets it (default logs/error.log at level error), and nginx -T prints the whole configuration, with sudo if the files are not readable by you.
  • Read access to the upstream's own log, with both clocks in sync, so you can line up a request's timestamps on each side.
  • A shell on the proxy host with curl, so you can ask the upstream directly from where the proxy sits. The example on this page was reproduced with curl 8.7.1.
  • The timeouts along the chain: nginx's proxy_connect_timeout, proxy_send_timeout and proxy_read_timeout (60 seconds each by default), plus whatever the load balancer, CDN and application server in front of or behind it use.

Asking three broken upstreams directly

Reproduced on curl 8.7.1 and Python 3.9.6 on macOS, with the time zone set to UTC. nginx is not installed here, so curl sits where the proxy would and asks the upstreams directly, which is also a useful first step on a real proxy host. Port 18621 has nothing listening. A Python script written for this page listens on 18622, reads the request and closes the connection without replying, and on 18623 replies 200 OK after 8 seconds and logs that it did. --connect-timeout 3 and --max-time 5 stand in for nginx's proxy_connect_timeout and proxy_read_timeout. The log was read a few seconds after the last curl. The nginx messages named below are quoted from nginx's source at release-1.30.5, not reproduced. 127.0.0.1 is the loopback address; nothing in the output identifies this machine.

Command

curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18621/
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18622/
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18623/
cat upstream.log

Output

$ curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18621/
curl: (7) Failed to connect to 127.0.0.1 port 18621 after 0 ms: Couldn't connect to server
000 0.000206s

$ curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18622/
curl: (52) Empty reply from server
000 0.000363s

$ curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18623/
curl: (28) Operation timed out after 5005 milliseconds with 0 bytes received
000 5.005721s

$ cat upstream.log
12:26:48 GET / HTTP/1.1 -> 200 sent after 8.0s

How to read it

  • Nothing listening on 18621: Failed to connect ... after 0 ms. In nginx's place this is a refusal, logged on Linux as connect() failed (111: Connection refused) while connecting to upstream, or connect() to <address> failed (111: Connection refused) while connecting to upstream when the connect() call itself fails, and the client gets a 502. Search the log for both forms. Nothing reached the upstream, so a retry is safe.
  • 18622 took the request and closed without a byte: (52) Empty reply from server, also immediate. nginx logs upstream prematurely closed connection while reading response header from upstream and answers 502. This time the upstream did receive the request, so whether a retry is safe depends on what it did with it.
  • 18623 took the request and said nothing, so curl stopped waiting at exactly its limit: Operation timed out after 5005 milliseconds with 0 bytes received. Behind nginx the limit is proxy_read_timeout, the log says upstream timed out (110: Connection timed out) while reading response header from upstream, and the client gets 504.
  • upstream.log shows the same request answered 200 sent after 8.0s: it succeeded three seconds after the client was told it failed. This is the 504 that looks healthy from the upstream's side, and the reason not to retry a 504 blindly.
  • The times are the quickest tell: 0.0002 and 0.0004 seconds for the two failures, 5.0 seconds, the exact limit, for the timeout. 000 is curl saying no HTTP status arrived at all.

The mistake this causes

Raising the proxy timeout because a 504 appeared.

Why people do it It works, immediately and visibly, which is exactly the problem. The 504 goes away and the underlying latency does not.

What you see The error rate drops and the p95 latency climbs. Users no longer see an error page; they wait thirty seconds instead. Worse, the timeout was the only thing reporting the regression, so raising it removed the signal along with the symptom — and the next investigation starts from a much weaker position.

If it goes wrong

connect() to unix:<path> failed (11: Resource temporarily unavailable) while connecting to upstream
The upstream listens on a Unix socket and its accept queue is full. nginx's source notes that Linux returns EAGAIN instead of ECONNREFUSED for Unix sockets when the listen queue is full. The upstream is overloaded, not missing: look at its worker count and backlog, not at nginx. The client still gets a 502.
upstream timed out (110: Connection timed out) while reading response header from upstream, 60 seconds after the request
That is proxy_read_timeout at its default. Before changing it, find the request in the upstream's log: still running means the upstream is slow, and finished after the 504 means something behind nginx waits longer than nginx does. Raise the timeout only in the location that genuinely needs longer, not for the whole server.
no live upstreams while connecting to upstream
Every server in the group is marked unavailable. With the defaults, max_fails=1 and fail_timeout=10s, one failed attempt takes a server out for 10 seconds. The error lines just before this one say why each server failed, and that is what to fix. A group with a single server is never marked unavailable.
502 or 504 on POST requests while GET requests to the same upstream group succeed
nginx passes a failed GET to the next server (proxy_next_upstream error timeout by default), but since 1.9.13 it does not pass on a POST, LOCK or PATCH that was already sent. One bad server is hidden from reads and exposed by writes. Find it by logging $upstream_addr and $upstream_status in the access log.
upstream prematurely closed connection while reading upstream, and clients get cut-off responses rather than a 502
The upstream died after nginx had already passed its status line and headers to the client, so it was too late to send a 502. nginx's documentation says so: if a failure happens in the middle of transferring a response, fixing this is impossible. Look for the upstream crashing or restarting mid-request.

How it shows up in production

A hard version to diagnose is a 504 with no errors in the upstream's logs and a fast response when tested directly. Both facts seem to rule out the upstream, so the investigation goes to the network.

One cause that fits both facts is timeout ordering. Some inner hop has a longer timeout than the proxy, so the proxy gives up first while the chain is still working. The request completes eventually, logs a success, and nobody is listening by then — which is why the upstream logs look healthy and the client saw a failure.

The tell is comparing timestamps rather than reading either log alone: a request logged as completing successfully after the client's 504 is the whole diagnosis.

How to tell which one you are hitting

How quickly did the error arrive?
Immediately, or at a consistent round number of seconds? An instant failure is a 502: a refusal, or an upstream that closed the connection without answering. A consistent delay, such as exactly 60 seconds at nginx's defaults, is a timeout expiring, so a 504.
What stage does the proxy log name?
while connecting means the request never got past the TCP handshake: refused, because nothing was listening, for a 502; timed out, on the path or at a full accept queue, for a 504. while reading response header points at the upstream's behaviour. The stage narrows this faster than anything else.
Did the upstream log the request at all?
No log at all means it never arrived: a refused connection (502) or a connect timeout (504). Logged and completed after the client gave up means a timeout ordering problem. Logged and still running means genuine slowness.
Is it safe to retry?
A refused connection, yes — nothing was processed. A 504 or a mid-response close, not blindly: the work may have completed, so a non-idempotent retry can duplicate it.

Security considerations

  • nginx retries for you. By default proxy_next_upstream error timeout passes a failed request to the next server in the group, so a GET that timed out on one server can run again on another while the first is still working on it. Since 1.9.13 nginx does not do this for POST, LOCK or PATCH once the request was sent, unless you add non_idempotent; adding it to make errors go away can repeat a payment or an order.
  • Raise timeouts in the location that needs it, not server-wide. A longer proxy_read_timeout everywhere means every stuck request holds its client and upstream connections for longer, so fewer stuck or deliberately slow requests are needed to use up the upstream's capacity.
  • nginx's error log lines end with the client address, the full request line including any query string, and the upstream address (client:, request:, upstream:). Treat excerpts as personal data, and remove tokens and internal addresses before pasting them into a ticket or chat.
  • 502 and 504 pages are nginx error pages, and with the default server_tokens on they show the nginx version, as does the Server header. server_tokens off; removes it, so an outage page does not tell every visitor which release you run.

In an interview

How this gets asked in an interview

Usually phrased as “What is the difference between a 502 and a 504?”

What a strong answer contains Frame it as concluded-badly versus never-concluded rather than reciting the status definitions, then give the consequence: a 502 means find what broke, a 504 means decide whether the upstream is too slow or the timeout is too short. Mentioning that a 504 is unsafe to retry blindly, because the work may have completed, is the point that shows operational experience.

The follow-up “You get a 504 and the upstream logs look fine — what now?” The answer they want involves timeout ordering across the chain, not raising the proxy timeout.

Sources

The behaviour this page describes, as documented by the projects themselves.

Terms used here (1)

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Load Balancer
A device or software component that distributes incoming network traffic across multiple servers to ensure no single server becomes overwhelmed. Load balancers improve availability, reliability, and responsiveness of applications.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it