NginxHigh
Fix: Nginx 504 Gateway Timeout
Users see '504 Gateway Timeout' after a long wait
!Symptoms
- Users see '504 Gateway Timeout' after a long wait
- Nginx error log shows 'upstream timed out'
- Long-running API requests fail consistently
- Only affects certain endpoints, not the whole site
?Root Causes
- Upstream application taking too long to respond (slow queries, heavy processing)
- Nginx proxy_read_timeout too short for the operation
- Database or external API the upstream depends on is slow
- Deadlock or resource contention in the upstream application
- Network latency between Nginx and the upstream server
#Diagnosis Steps
- 1Check Nginx error log for timeout messages: `tail -f /var/log/nginx/error.log`
- 2Identify which endpoint is slow from access logs: `awk '$9 == 504' access.log`
- 3Test the upstream directly to measure response time: `time curl http://upstream:port/endpoint`
- 4Check upstream application logs for slow operations
- 5Monitor database query times for the affected endpoints
>Fix
- 1Increase Nginx timeout values: proxy_read_timeout, proxy_connect_timeout, proxy_send_timeout
- 2Optimize the slow upstream endpoint (database indexes, caching, async processing)
- 3Move long-running tasks to a background job queue (Celery, Sidekiq, Bull)
- 4Add request buffering: `proxy_buffering on` with appropriate buffer sizes
- 5Implement pagination or streaming for large data responses
*Prevention
- Set appropriate timeouts at every layer (Nginx, app, database)
- Implement request timeouts in the application code itself
- Use async processing for operations that take more than a few seconds
- Monitor response time percentiles (p95, p99) and alert on degradation
- Load test endpoints with realistic payloads before release
Related Error Messages
504 Gateway Timeoutupstream timed out (110: Connection timed out)upstream timed out (110: Operation timed out)proxy_read_timeout