504 Gateway Timeout

Confirm

sudo grep 'upstream timed out' /var/log/nginx/error.log | tail -20

Timeouts clustered on one endpoint rather than spread across the application. A single slow route is a strong signal for this cause.

Fix

• Make it asynchronous. Accept the request, return 202 with a job identifier, and let the client poll or receive a callback. This is the real fix and it removes the class of problem.
• If it must stay synchronous, raise `proxy_read_timeout` for that route only rather than globally — a global raise removes the protection everywhere else.

location /api/export {
  proxy_pass http://app;
  proxy_read_timeout 300s;   # this route only
}

Scope it to the location. Raising the global timeout hides genuine slowness across the whole application.

Confirm

sudo awk '{print $NF}' /var/log/nginx/access.log | tail -1000 | sort -n | tail -20

Whether request durations have shifted upward generally, or only for a few endpoints. A broad shift points at a shared dependency.

Fix

• Do not raise the timeout. It is the only thing currently telling you about the regression, and raising it converts a visible failure into a slow site.
• Find what changed — a deploy, a data volume threshold crossed, a dependency's own latency. The timing of onset is usually the strongest clue.

Confirm

ss -tn state established '( dport = :3000 or sport = :3000 )' | wc -l

A connection count at or near the upstream's configured worker or pool limit, with low CPU. Saturation without CPU load is the fingerprint.

Fix

• Find what the workers are blocked on. Exhaustion is nearly always a downstream wait — a slow database call, an external API, a lock — rather than genuine compute.
• Raising the worker count without finding the block moves the queue rather than draining it, and can make the dependency worse.

Confirm

sudo nginx -T 2>/dev/null | grep -E 'proxy_(read|connect|send)_timeout'

The proxy's timeouts compared against the application's own client timeouts for its dependencies. The outer should exceed the inner, not the reverse.

Fix

• Order timeouts deliberately from the outside in, so the innermost gives up first and can return a meaningful error instead of being abandoned.
• Give the application a client timeout shorter than the proxy's read timeout, so it fails and reports rather than being cut off mid-flight.

Confirm

sudo grep 'upstream timed out' /var/log/nginx/error.log | grep -o 'while [a-z ]*' | sort | uniq -c

`while connecting to upstream` rather than `while reading response header`. The stage named in the message is the diagnosis.

Fix

• Treat this as a network path problem, not a performance one. Check security groups, firewall rules and routing between the proxy and the upstream.
• A silent drop rather than a refusal is the signature of a packet filter, which is why the failure is slow rather than immediate.