DevOpsil
Sign In

On this page

HAProxy
85%
Needs Review

Fix HAProxy '503 Service Unavailable' Backend Down

Riku TanakaRiku Tanaka4 min read

The Error: "503 Service Unavailable"

Users hit your application and see:

HTTP/1.1 503 Service Unavailable
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

HAProxy is running, the frontend is accepting connections, but every backend server is marked as down.

Root Cause

HAProxy returns 503 when no backend servers are available to handle a request. This means all servers in the backend have either:

  • Failed health checks and are marked DOWN
  • Reached their maxconn limit and are refusing new connections
  • Been put into maintenance mode (MAINT or DRAIN state)
  • Never been reachable due to wrong IP, port, or network configuration

Step-by-Step Fix

1. Check Backend Server Status

Use the HAProxy stats socket or stats page:

# Via stats socket
echo "show stat" | sudo socat stdio /var/run/haproxy/admin.sock | \
  awk -F',' '{print $1,$2,$18,$19}' | column -t

Or via the stats URL (if enabled): http://haproxy-host:8404/stats

Look for the status column. Healthy servers show UP. Problem servers show DOWN, MAINT, or NOLB.

2. Test Backend Server Connectivity

From the HAProxy server, manually test each backend:

# Test HTTP health check endpoint
curl -v http://backend-server:8080/health

# Test TCP connectivity
nc -zv backend-server 8080

# Check if the port is listening on the backend
ssh backend-server 'ss -tlnp | grep 8080'

3. Check HAProxy Logs

# Recent logs
sudo journalctl -u haproxy --since "10 minutes ago" --no-pager

# Look for health check failures
sudo journalctl -u haproxy | grep -i "DOWN" | tail -20

You'll see messages like:

Server mybackend/server1 is DOWN, reason: Layer4 connection problem, info: "Connection refused"
Server mybackend/server2 is DOWN, reason: Layer7 wrong status, info: "HTTP status 500"

4. Fix Health Check Configuration

If the health check path returns a non-2xx status, HAProxy marks the server as down. Update your config:

backend myapp
    option httpchk GET /health
    http-check expect status 200

    server app1 10.0.1.10:8080 check inter 5s fall 3 rise 2
    server app2 10.0.1.11:8080 check inter 5s fall 3 rise 2

Key parameters:

  • inter 5s -- check every 5 seconds
  • fall 3 -- mark DOWN after 3 consecutive failures
  • rise 2 -- mark UP after 2 consecutive successes

If your app uses a different health endpoint or returns 204:

    option httpchk GET /api/status
    http-check expect status 200-299

5. Fix Connection Limits

If servers are up but overloaded:

echo "show stat" | sudo socat stdio /var/run/haproxy/admin.sock | \
  awk -F',' '{print $1,$2,$5,$6,$7}' | column -t

Check if scur (current sessions) equals slim (session limit). If so, increase the limit:

backend myapp
    server app1 10.0.1.10:8080 check maxconn 500
    server app2 10.0.1.11:8080 check maxconn 500

6. Force a Server Back Up

If you've fixed the underlying issue and want to force HAProxy to re-check:

# Via stats socket
echo "set server mybackend/app1 state ready" | \
  sudo socat stdio /var/run/haproxy/admin.sock

# Enable a server that was in maintenance
echo "set server mybackend/app1 state enabled" | \
  sudo socat stdio /var/run/haproxy/admin.sock

7. Reload HAProxy

After config changes:

# Graceful reload (no dropped connections)
sudo systemctl reload haproxy

# Verify config syntax first
sudo haproxy -c -f /etc/haproxy/haproxy.cfg

8. Verify the Fix

# Check all backends are UP
echo "show stat" | sudo socat stdio /var/run/haproxy/admin.sock | \
  grep -E "^myapp" | awk -F',' '{print $2, $18}'

# Test from a client
curl -I http://your-site.com
# Should return HTTP/1.1 200 OK

Prevention Tips

  • Use multiple backend servers. A single backend means any failure causes a 503.
  • Set reasonable fall values. fall 3 means 3 failures before marking down -- this prevents flapping on a single slow response.
  • Configure a maintenance page using errorfile 503 /etc/haproxy/errors/503.http so users see a helpful message instead of a bare 503.
  • Monitor backend health via the stats page or export metrics to Prometheus with haproxy_exporter.
  • Use connection queuing (timeout queue 30s) to queue requests during brief backend unavailability instead of immediately returning 503.
Advertise here
Share:
haproxy503backendtroubleshooting

Was this article helpful?

Riku Tanaka
Riku Tanaka

SRE & Observability Engineer

If it's not measured, it doesn't exist. SLO-driven, metrics-obsessed, and the person who gets paged at 3 AM so you don't have to. Observability isn't optional.

Twitter/XLinkedIn

Related Articles

Discussion

View all