HTTP · troubleshooting

502 Bad Gateway

The proxy could not get a usable response from the thing behind it. The proxy is reporting a fact about somebody else — so the answer is almost never in the proxy's configuration, and almost always in whether the upstream was reachable, alive, and speaking HTTP.

Run this first

step 1 of 3

sudo tail -50 /var/log/nginx/error.log

Look for `connect() failed (111)` = nothing listening · `prematurely closed` = upstream died mid-request · `too big header` = buffer · `no live upstreams` = whole pool ejected.

This is the step that decides everything else. The proxy already knows what went wrong and has written it down, and the distinct messages map almost one-to-one onto the causes below.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

What does the proxy's error log say?

The distinct messages map almost directly onto distinct causes, which is why this is first.

Cause space

6 of 6 still possible

Nothing is listening at the address the proxy is usingCommon
The upstream accepted the connection and then diedCommon
The proxy is configured for the wrong addressCommon
The upstream is speaking a different protocol than the proxy expectsOccasional
The upstream's response headers exceed the proxy's bufferOccasional
Every upstream in the pool has been marked unavailableOccasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Or diagnose it manually

In this order. The first command usually contains the whole answer.

Step 1
```
sudo tail -50 /var/log/nginx/error.log
```
This is the step that decides everything else. The proxy already knows what went wrong and has written it down, and the distinct messages map almost one-to-one onto the causes below.
Look for `connect() failed (111)` = nothing listening · `prematurely closed` = upstream died mid-request · `too big header` = buffer · `no live upstreams` = whole pool ejected.
Step 2
```
curl -sv http://<upstream-address>:<port>/ -o /dev/null
```
Run from wherever the proxy runs. This separates "the proxy cannot reach it" from "the application is broken", which are different problems with the same status code.
Look for A refusal means reachability. A valid response means the proxy config or protocol is the fault.
Step 3
```
sudo nginx -T 2>/dev/null | grep -B2 -A6 proxy_pass | head -40
```
Prints the configuration actually in effect, which is often not the file you last edited.
Look for The upstream address and scheme, checked against where the application really listens.

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

Nothing is listening at the address the proxy is using

Common

The upstream process is not running, is still starting, or is listening somewhere other than where the proxy is looking. The proxy's connection is actively refused, which is a fast failure rather than a hang — the 502 arrives immediately.

Confirm

sudo tail -50 /var/log/nginx/error.log | grep -i 'connect() failed'

`(111: Connection refused)`. That is the kernel saying nothing holds the port, which is unambiguous — the address in the message is where the proxy tried.

Fix

Confirm the upstream is running and listening on exactly the address in the proxy config, with `ss -tlnp | grep <port>`.
Check the bind address, not just the port. A process on `127.0.0.1:3000` is unreachable from a proxy in a different container or on another host, even though the port is correct.
If the upstream is a container, check the proxy is using the container name and container port rather than a published host port.

ss -tlnp | grep ':3000'   # is anything listening, and on what address?
curl -sv http://127.0.0.1:3000/ -o /dev/null   # from the proxy's own host

Run the curl from wherever the proxy runs. Reachability from your laptop proves nothing about reachability from the proxy.

The upstream accepted the connection and then died

Common

The connection succeeded, so this is not a reachability problem. The application then crashed, was killed, or closed the connection before sending a complete response — most often an unhandled exception in a worker, or the process being OOM-killed while handling the request.

Confirm

sudo tail -50 /var/log/nginx/error.log | grep -i 'prematurely closed\|upstream sent'

`upstream prematurely closed connection while reading response header`. This is the signature — the proxy got a socket and then got nothing.

Fix

Look at the upstream's own logs at the timestamp of the 502. A crash usually leaves a stack trace, and this cause is the one where the application has the most to say.
Check whether the process was killed rather than crashed. Exit code 137 or an OOM entry means the request pushed it over a memory limit.
If it is intermittent and correlates with load, suspect worker exhaustion or a per-request memory spike rather than a logic bug.

The proxy is configured for the wrong address

Common

A hostname that no longer resolves, a service name that does not exist on the proxy's network, or a stale address after a deploy. Common the first time a proxy and an application are containerised together, because `localhost` in the proxy config means the proxy's container.

Confirm

sudo nginx -T 2>/dev/null | grep -A3 'proxy_pass\|upstream ' | head -30

The actual resolved configuration, not the file you think is loaded. `nginx -T` prints what is in effect, which is frequently not what you last edited.

Fix

Resolve the upstream name from inside the proxy: `docker exec proxy getent hosts api`. A name that does not resolve is a different fault from a connection refused.
In compose, the upstream is the service name and the container port. Neither `localhost` nor the published host port is correct from inside the proxy container.

The upstream is speaking a different protocol than the proxy expects

Occasional

The connection works and bytes come back, but they are not what the proxy is parsing. Usually `proxy_pass http://` against a TLS listener, or an HTTP/1.1 proxy in front of a gRPC or HTTP/2-only service.

Confirm

curl -sv http://<upstream-host>:<port>/ -o /dev/null 2>&1 | head -20

A TLS handshake error, or a response that is not recognisable HTTP. If plain HTTP fails but `https://` works, the scheme in `proxy_pass` is the fault.

Fix

Match the scheme to the listener: `proxy_pass https://` for a TLS upstream, and `grpc_pass` for gRPC.
For an HTTP/2-only upstream, either enable the appropriate module or put an HTTP/1.1-capable listener in front of it.

The upstream's response headers exceed the proxy's buffer

Occasional

The proxy allocates a fixed buffer for response headers. An unusually large set — a long chain of cookies, a large `Set-Cookie`, a verbose auth token — overflows it, and the proxy gives up with a 502 rather than truncating.

Confirm

sudo tail -50 /var/log/nginx/error.log | grep -i 'too big header\|upstream sent too big'

`upstream sent too big header while reading response header from upstream`.

Fix

Raise `proxy_buffer_size` and `proxy_buffers` to accommodate the real header size.
Then ask why the headers are that large. A multi-kilobyte cookie is usually a design problem that will cause other trouble.

proxy_buffer_size 16k;
proxy_buffers 4 16k;

Treat the raise as a stopgap and the header size as the actual finding.

Every upstream in the pool has been marked unavailable

Occasional

With multiple upstreams, the proxy takes failing ones out of rotation. Once all of them are out, every request 502s — including requests that would now succeed, because a passively-marked upstream is not retried until its window expires.

Confirm

sudo tail -50 /var/log/nginx/error.log | grep -i 'no live upstreams'

`no live upstreams while connecting to upstream`. That is the whole pool, not one member.

Fix

Fix the underlying upstream failure — the pool being empty is a consequence rather than the cause.
Review `max_fails` and `fail_timeout`. Aggressive values will eject the entire pool during a brief wobble and then refuse to try again for the timeout period.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

What is actually happeningmechanism

A 502 is a proxy telling you it tried and failed. That framing matters, because it means the proxy is working: it accepted your request, attempted to forward it, and is honestly reporting that the attempt did not produce a valid response.

So there are three places the fault can be, and they need different investigations. The **proxy** may be pointing at the wrong address. The **network path** may be refusing or resolving incorrectly. Or the **upstream application** may have accepted the connection and then died, closed early, or returned something that is not valid HTTP.

The proxy's own error log distinguishes these almost immediately, and it is the single most under-used artefact in this whole class of problem. `connect() failed (111: Connection refused)` and `upstream prematurely closed connection` are two completely different faults that produce the identical status code for the user.

It is worth being precise about one thing: a 502 is not a timeout. If the upstream took too long you get a 504. A 502 means the attempt ended in a way other than waiting.

How to tell this is your problemsignals

Where	What you see
Browser / curl	`HTTP/1.1 502 Bad Gateway`, often with the proxy's own branded error page.
nginx error log	`connect() failed (111: Connection refused) while connecting to upstream`, or `upstream prematurely closed connection while reading response header`.
Upstream application logs	Either nothing at all (the request never arrived) or a crash at the moment of the request.

How to know it is actually fixed

A request through the proxy returns the application's own response rather than the proxy's error page.
The proxy error log stops adding entries for the affected upstream.
The 502 does not return under load, which distinguishes a fixed cause from one that only appears when workers are saturated.

Stopping it happening again

Alert on the proxy's upstream error rate rather than only on the application's own error rate. A 502 is the one failure the application may never see, because the request did not reach it.
Give the upstream a readiness signal the proxy respects, so a starting or draining instance is taken out of rotation rather than producing 502s.
Keep the proxy and the upstream's bind address in sync deliberately — this is the mismatch that recurs on every environment change.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

504 Gateway Timeout Connection refused between containers Linux

Run this first

Work out which cause you have

Check it with a tool

Or diagnose it manually

Every cause, and how to fix it

Nothing is listening at the address the proxy is using

The upstream accepted the connection and then died

The proxy is configured for the wrong address

The upstream is speaking a different protocol than the proxy expects

The upstream's response headers exceed the proxy's buffer

Every upstream in the pool has been marked unavailable

Understanding it properly

How to know it is actually fixed

Stopping it happening again

Sources

Related