HTTP · commonly confused
502 Bad Gateway vs 504 Gateway Timeout
Short answer
Reproduced on curl 8.7.1 and Python 3.9.6, . Where the text says “reproduced”, the behaviour was run on that version, not taken from documentation.
First published
On this page · 10 sections
In simple terms
Both are a proxy telling you it could not give you the upstream's answer, which is why they get conflated. The difference is what happened to the attempt.
With a 502, something concluded badly. The connection was refused, or it opened and then closed without a complete response. There is an event you can point at.
With a 504, nothing concluded at all. The proxy gave up waiting, and — this is the part people miss — the upstream is quite possibly still working on it. Your request may well complete after you have been told it failed.
That distinction decides the fix. For a 502, the job is finding what broke. A 504 means deciding whether the upstream is too slow or the timeout is too short, which are different problems with opposite remedies.
Words used on this page
- gateway
- A server that takes the client's request, passes it on to another server and relays the answer. RFC 9110 also calls it a reverse proxy; nginx with
proxy_passis one. - upstream
- The server the proxy passes requests to. nginx can group several in an
upstreamblock and pick one per request. - error log
- nginx's own record of failures, separate from the access log. The access log has the 502 or 504; the error log has the reason.
- errno
- The operating system's error number, which nginx prints in brackets with its text, as in
(111: Connection refused). 110 and 111 are Linux's numbers; macOS uses 60 and 61 for the same errors. - TCP handshake
- The exchange of packets that opens a connection.
while connecting to upstreamin nginx's log means the failure happened here, before any request was sent. - accept queue
- Connections waiting for the server program to pick them up. When it is full, new connection attempts may be refused or ignored, depending on the system.
- proxy_read_timeout
- How long nginx waits between two reads from the upstream before giving up, 60 seconds by default. It is not a limit on the whole response.
- idempotent
- A request that has the same effect whether it runs once or several times. RFC 9110 lists PUT, DELETE and the safe methods such as GET; POST is not, so repeating it can do the work twice.
What actually happens
A 502 in nginx comes with a specific error-log line, and the lines map almost one-to-one onto causes: connect() failed (111: Connection refused), or connect() to <address> failed when the connect() call itself fails, means nothing was listening; upstream prematurely closed connection means the upstream closed the connection before sending a complete response header; upstream sent too big header is a buffer limit; no live upstreams means the whole pool was ejected.
A 504 comes from a timeout expiring, and nginx names the stage: while connecting to upstream is the TCP handshake never completing, while reading response header is the upstream not answering in time. The default proxy_read_timeout is 60 seconds, so a 504 from nginx at its defaults comes after a round 60 seconds of waiting.
The stage matters more than the status. A connect-stage 504 means the handshake never completed: packets dropped somewhere on the path, or an upstream so overloaded that its accept queue is full and new connections are ignored. A refusal would have produced a fast 502 instead. A read-stage 504 is about latency or queueing.
One consequence worth internalising: a 504 is not safe to retry blindly. The request may have been fully processed, so retrying a non-idempotent operation can duplicate it. A 502 from a refused connection is safe to retry; a 502 from a mid-response close is not, for the same reason.
Telling them apart from the proxy log alone
# 502 — the attempt concluded badly connect() failed (111: Connection refused) while connecting to upstream upstream prematurely closed connection while reading response header from upstream # 504 — the attempt never concluded upstream timed out (110: Connection timed out) while reading response header from upstream upstream timed out (110: Connection timed out) while connecting to upstream # └── network path, not latency
Both statuses look identical to the user and to most dashboards. The proxy's error log is where the distinction lives.
Try it yourself
Before you start
- Read access to the proxy's error log, and its path. For nginx, the
error_logdirective sets it (defaultlogs/error.logat levelerror), andnginx -Tprints the whole configuration, with sudo if the files are not readable by you. - Read access to the upstream's own log, with both clocks in sync, so you can line up a request's timestamps on each side.
- A shell on the proxy host with
curl, so you can ask the upstream directly from where the proxy sits. The example on this page was reproduced with curl 8.7.1. - The timeouts along the chain: nginx's
proxy_connect_timeout,proxy_send_timeoutandproxy_read_timeout(60 seconds each by default), plus whatever the load balancer, CDN and application server in front of or behind it use.
Asking three broken upstreams directly
Reproduced on curl 8.7.1 and Python 3.9.6 on macOS, with the time zone set to UTC. nginx is not installed here, so curl sits where the proxy would and asks the upstreams directly, which is also a useful first step on a real proxy host. Port 18621 has nothing listening. A Python script written for this page listens on 18622, reads the request and closes the connection without replying, and on 18623 replies 200 OK after 8 seconds and logs that it did. --connect-timeout 3 and --max-time 5 stand in for nginx's proxy_connect_timeout and proxy_read_timeout. The log was read a few seconds after the last curl. The nginx messages named below are quoted from nginx's source at release-1.30.5, not reproduced. 127.0.0.1 is the loopback address; nothing in the output identifies this machine.
Command
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18621/
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18622/
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18623/
cat upstream.logOutput
$ curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18621/
curl: (7) Failed to connect to 127.0.0.1 port 18621 after 0 ms: Couldn't connect to server
000 0.000206s
$ curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18622/
curl: (52) Empty reply from server
000 0.000363s
$ curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --connect-timeout 3 --max-time 5 http://127.0.0.1:18623/
curl: (28) Operation timed out after 5005 milliseconds with 0 bytes received
000 5.005721s
$ cat upstream.log
12:26:48 GET / HTTP/1.1 -> 200 sent after 8.0sHow to read it
- Nothing listening on 18621:
Failed to connect ... after 0 ms. In nginx's place this is a refusal, logged on Linux asconnect() failed (111: Connection refused) while connecting to upstream, orconnect() to <address> failed (111: Connection refused) while connecting to upstreamwhen theconnect()call itself fails, and the client gets a 502. Search the log for both forms. Nothing reached the upstream, so a retry is safe. - 18622 took the request and closed without a byte:
(52) Empty reply from server, also immediate. nginx logsupstream prematurely closed connection while reading response header from upstreamand answers 502. This time the upstream did receive the request, so whether a retry is safe depends on what it did with it. - 18623 took the request and said nothing, so curl stopped waiting at exactly its limit:
Operation timed out after 5005 milliseconds with 0 bytes received. Behind nginx the limit isproxy_read_timeout, the log saysupstream timed out (110: Connection timed out) while reading response header from upstream, and the client gets 504. upstream.logshows the same request answered200 sent after 8.0s: it succeeded three seconds after the client was told it failed. This is the 504 that looks healthy from the upstream's side, and the reason not to retry a 504 blindly.- The times are the quickest tell: 0.0002 and 0.0004 seconds for the two failures, 5.0 seconds, the exact limit, for the timeout.
000is curl saying no HTTP status arrived at all.
The mistake this causes
Raising the proxy timeout because a 504 appeared.
Why people do it It works, immediately and visibly, which is exactly the problem. The 504 goes away and the underlying latency does not.
What you see The error rate drops and the p95 latency climbs. Users no longer see an error page; they wait thirty seconds instead. Worse, the timeout was the only thing reporting the regression, so raising it removed the signal along with the symptom — and the next investigation starts from a much weaker position.
If it goes wrong
connect() to unix:<path> failed (11: Resource temporarily unavailable) while connecting to upstream- The upstream listens on a Unix socket and its accept queue is full. nginx's source notes that Linux returns EAGAIN instead of ECONNREFUSED for Unix sockets when the listen queue is full. The upstream is overloaded, not missing: look at its worker count and backlog, not at nginx. The client still gets a 502.
upstream timed out (110: Connection timed out) while reading response header from upstream, 60 seconds after the request- That is
proxy_read_timeoutat its default. Before changing it, find the request in the upstream's log: still running means the upstream is slow, and finished after the 504 means something behind nginx waits longer than nginx does. Raise the timeout only in thelocationthat genuinely needs longer, not for the whole server. no live upstreams while connecting to upstream- Every server in the group is marked unavailable. With the defaults,
max_fails=1andfail_timeout=10s, one failed attempt takes a server out for 10 seconds. The error lines just before this one say why each server failed, and that is what to fix. A group with a single server is never marked unavailable. - 502 or 504 on POST requests while GET requests to the same upstream group succeed
- nginx passes a failed GET to the next server (
proxy_next_upstream error timeoutby default), but since 1.9.13 it does not pass on a POST, LOCK or PATCH that was already sent. One bad server is hidden from reads and exposed by writes. Find it by logging$upstream_addrand$upstream_statusin the access log. upstream prematurely closed connection while reading upstream, and clients get cut-off responses rather than a 502- The upstream died after nginx had already passed its status line and headers to the client, so it was too late to send a 502. nginx's documentation says so: if a failure happens in the middle of transferring a response, fixing this is impossible. Look for the upstream crashing or restarting mid-request.
How it shows up in production
A hard version to diagnose is a 504 with no errors in the upstream's logs and a fast response when tested directly. Both facts seem to rule out the upstream, so the investigation goes to the network.
One cause that fits both facts is timeout ordering. Some inner hop has a longer timeout than the proxy, so the proxy gives up first while the chain is still working. The request completes eventually, logs a success, and nobody is listening by then — which is why the upstream logs look healthy and the client saw a failure.
The tell is comparing timestamps rather than reading either log alone: a request logged as completing successfully after the client's 504 is the whole diagnosis.
How to tell which one you are hitting
- How quickly did the error arrive?
- Immediately, or at a consistent round number of seconds? An instant failure is a 502: a refusal, or an upstream that closed the connection without answering. A consistent delay, such as exactly 60 seconds at nginx's defaults, is a timeout expiring, so a 504.
- What stage does the proxy log name?
while connectingmeans the request never got past the TCP handshake: refused, because nothing was listening, for a 502; timed out, on the path or at a full accept queue, for a 504.while reading response headerpoints at the upstream's behaviour. The stage narrows this faster than anything else.- Did the upstream log the request at all?
- No log at all means it never arrived: a refused connection (502) or a connect timeout (504). Logged and completed after the client gave up means a timeout ordering problem. Logged and still running means genuine slowness.
- Is it safe to retry?
- A refused connection, yes — nothing was processed. A 504 or a mid-response close, not blindly: the work may have completed, so a non-idempotent retry can duplicate it.
Security considerations
- nginx retries for you. By default
proxy_next_upstream error timeoutpasses a failed request to the next server in the group, so a GET that timed out on one server can run again on another while the first is still working on it. Since 1.9.13 nginx does not do this for POST, LOCK or PATCH once the request was sent, unless you addnon_idempotent; adding it to make errors go away can repeat a payment or an order. - Raise timeouts in the
locationthat needs it, not server-wide. A longerproxy_read_timeouteverywhere means every stuck request holds its client and upstream connections for longer, so fewer stuck or deliberately slow requests are needed to use up the upstream's capacity. - nginx's error log lines end with the client address, the full request line including any query string, and the upstream address (
client:,request:,upstream:). Treat excerpts as personal data, and remove tokens and internal addresses before pasting them into a ticket or chat. - 502 and 504 pages are nginx error pages, and with the default
server_tokens onthey show the nginx version, as does theServerheader.server_tokens off;removes it, so an outage page does not tell every visitor which release you run.
In an interview
How this gets asked in an interviewpreparation
Usually phrased as “What is the difference between a 502 and a 504?”
What a strong answer contains Frame it as concluded-badly versus never-concluded rather than reciting the status definitions, then give the consequence: a 502 means find what broke, a 504 means decide whether the upstream is too slow or the timeout is too short. Mentioning that a 504 is unsafe to retry blindly, because the work may have completed, is the point that shows operational experience.
The follow-up “You get a 504 and the upstream logs look fine — what now?” The answer they want involves timeout ordering across the chain, not raising the proxy timeout.
Sources
The behaviour this page describes, as documented by the projects themselves.
- nginx source: ngx_http_upstream.c at release-1.30.5
- nginx source: ngx_event_connect.c at release-1.30.5
- nginx: Core functionality (error_log)
- nginx: server_tokens
- connect(2) manual page
- listen(2) manual page
- nginx — Module ngx_http_proxy_module
- nginx — Module ngx_http_upstream_module
- RFC 9110 — HTTP Semantics
Terms used here (1)plain definitions
Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.
- Load Balancer
- A device or software component that distributes incoming network traffic across multiple servers to ensure no single server becomes overwhelmed. Load balancers improve availability, reliability, and responsiveness of applications.
Did this get you to an answer?
No text box on purpose — please do not paste production logs anywhere
Something wrong or out of date? Report it