HAProxy Health Checks and Automatic Failover

A load balancer that sends traffic to dead backends is worse than no load balancer at all — it takes healthy requests and throws them into the void. HAProxy's health check system is sophisticated enough to detect failures at the TCP, HTTP, and application layer, then automatically fail over before users notice anything is wrong.

Health Check Fundamentals

HAProxy makes periodic health check connections to each backend server. Based on the results, it marks servers as UP, DOWN, or in transition states (NOLB — no load balancing, MAINT — maintenance).

The check cycle:

1. HAProxy connects to the backend on the configured interval
2. Sends the check request (TCP, HTTP, or custom)
3. Evaluates the response
4. Updates the server's health state
5. Routes or withholds traffic accordingly

TCP Health Checks (Layer 4)

The simplest check — just verify the port is open and accepting connections:

backend app_servers
    balance roundrobin

    server app1 10.0.1.10:8080 check
    server app2 10.0.1.11:8080 check
    server app3 10.0.1.12:8080 check

The bare check keyword enables TCP health checks. HAProxy opens a connection, and if it succeeds, the server is UP. Quick and cheap, but it won't catch application-level failures — a process could be listening on port 8080 but stuck in a deadlock.

Configure check timing:

backend app_servers
    # Check interval: 5s, timeout: 3s
    # Rise: 2 consecutive successes to mark UP
    # Fall: 3 consecutive failures to mark DOWN
    timeout check 3s

    server app1 10.0.1.10:8080 check inter 5s rise 2 fall 3
    server app2 10.0.1.11:8080 check inter 5s rise 2 fall 3

Parameter	Default	Meaning
inter	2000ms	Check interval when server is UP
fastinter	same as inter	Check interval after state change
downinter	same as inter	Check interval when server is DOWN
rise	2	Successes needed to mark UP
fall	3	Failures needed to mark DOWN
timeout check	none	Max time to wait for check response

Use downinter to check dead servers less frequently:

server app1 10.0.1.10:8080 check inter 5s downinter 30s fastinter 2s rise 2 fall 3

HTTP Health Checks (Layer 7)

HTTP checks let HAProxy send a real HTTP request and validate the response code:

backend app_servers
    balance roundrobin

    # Check using GET /healthz, expect 200
    option httpchk GET /healthz HTTP/1.1\r\nHost:\ app.example.com

    http-check expect status 200

    server app1 10.0.1.10:8080 check inter 5s
    server app2 10.0.1.11:8080 check inter 5s

Validating Response Content

Check the response body, not just the status code — your app might return 200 with {"status":"degraded"}:

backend app_servers
    option httpchk GET /healthz

    # Expect the body to contain "ok" or "healthy"
    http-check expect rstring (ok|healthy)

    server app1 10.0.1.10:8080 check
    server app2 10.0.1.11:8080 check

Advanced HTTP Check Configuration (HAProxy 2.2+)

The new http-check syntax gives fine-grained control:

backend app_servers
    # Define the check request
    http-check connect
    http-check send meth GET  uri /healthz  ver HTTP/1.1 \
                  hdr Host app.example.com \
                  hdr Accept application/json \
                  hdr X-Health-Check-Secret mysecrettoken

    # Accept 200 or 204
    http-check expect status 200:204

    # Or match JSON body
    http-check expect rstring "\"status\":\"up\""

    server app1 10.0.1.10:8080 check inter 10s
    server app2 10.0.1.11:8080 check inter 10s

Database and Redis Health Checks

HAProxy can send custom TCP payloads for non-HTTP services:

# MySQL health check — send a ping, expect a valid response
backend mysql_servers
    option mysql-check user haproxy_check

    server db1 10.0.2.10:3306 check
    server db2 10.0.2.11:3306 check backup

# PostgreSQL
backend postgres_servers
    option pgsql-check user haproxy

    server db1 10.0.2.10:5432 check
    server db2 10.0.2.11:5432 check backup

# Redis — send PING, expect +PONG
backend redis_servers
    option tcp-check
    tcp-check connect
    tcp-check send PING\r\n
    tcp-check expect string +PONG

    server redis1 10.0.3.10:6379 check
    server redis2 10.0.3.11:6379 check

Backup Servers and Failover

HAProxy supports explicit backup servers that only receive traffic when all primary servers are down:

backend app_servers
    balance roundrobin

    # Primary servers
    server app1 10.0.1.10:8080 check
    server app2 10.0.1.11:8080 check

    # Backup — used only when both app1 and app2 are down
    server app-dr 10.0.99.10:8080 check backup

Maintenance Mode

Take a server out of rotation gracefully — in-flight requests complete, no new requests are sent:

# Via stats socket — set server to maintenance mode
echo "set server app_servers/app1 state maint" | \
  socat stdio /run/haproxy/admin.sock

# Bring it back
echo "set server app_servers/app1 state ready" | \
  socat stdio /run/haproxy/admin.sock

# Drain a server (finish current sessions, accept no new ones)
echo "set server app_servers/app1 state drain" | \
  socat stdio /run/haproxy/admin.sock

This pattern is essential for zero-downtime deployments: drain the server, deploy, verify, return to ready.

Monitoring Health Check Status

Stats Page

Enable HAProxy's built-in web stats:

frontend stats
    bind *:8404
    stats enable
    stats uri /haproxy-stats
    stats refresh 10s
    stats auth admin:strongpassword

    # Restrict to internal IPs
    acl internal src 10.0.0.0/8
    http-request deny unless internal

Visit http://your-server:8404/haproxy-stats to see a live dashboard with server states, check results, and traffic counters.

Stats Socket Queries

# Show all server states
echo "show servers state" | socat stdio /run/haproxy/admin.sock

# Show just the backend you care about
echo "show servers state app_servers" | socat stdio /run/haproxy/admin.sock

# Output columns: backend_name server_name status check_status ...

# Show health check details
echo "show health" | socat stdio /run/haproxy/admin.sock

# Live connection counts
echo "show stat" | socat stdio /run/haproxy/admin.sock | cut -d',' -f1,2,5,6,18

Log Health Check Events

global
    log /dev/log local0

defaults
    log global

    # Log health check state changes
    option log-health-checks

This logs messages like:

haproxy[1234]: Health check for server app_servers/app1 succeeded, reason: Layer7 check passed, code: 200, info: "OK", check duration: 3ms. Server app_servers/app1 is UP.
haproxy[1234]: Health check for server app_servers/app2 failed, reason: Layer7 timeout, check duration: 3001ms. Server app_servers/app2 is DOWN (1 active and 0 backup servers left).

Parse these with your log aggregator (Loki, Elasticsearch) to alert on state changes.

External Health Check Scripts (Agent Checks)

For complex health logic that HAProxy can't express natively, use agent checks. HAProxy connects to a separate agent port on each backend; the agent returns a simple string indicating the desired state:

backend app_servers
    balance roundrobin

    # Regular health check + agent check on port 9001
    server app1 10.0.1.10:8080 check agent-check agent-port 9001 agent-inter 10s
    server app2 10.0.1.11:8080 check agent-check agent-port 9001 agent-inter 10s

A minimal agent in Python running on each backend:

#!/usr/bin/env python3
# /usr/local/bin/haproxy-agent.py
import socketserver
import subprocess
import os

def get_health_response():
    # Check disk space
    disk = os.statvfs('/')
    disk_pct = 100 * (1 - disk.f_bavail / disk.f_blocks)
    if disk_pct > 90:
        return b"drain\n"  # Drain: finish sessions, take no new ones

    # Check load average
    load = os.getloadavg()[0]
    cpu_count = os.cpu_count()
    if load > cpu_count * 2:
        return b"50%\n"  # Take half normal traffic

    # Check if the app process is running
    result = subprocess.run(['systemctl', 'is-active', 'myapp'], capture_output=True)
    if result.returncode != 0:
        return b"down\n"

    return b"up 100%\n"

class AgentHandler(socketserver.BaseRequestHandler):
    def handle(self):
        response = get_health_response()
        self.request.sendall(response)

if __name__ == '__main__':
    with socketserver.TCPServer(("0.0.0.0", 9001), AgentHandler) as server:
        server.serve_forever()

Agent response strings:

Response	Effect
up	Mark server UP, restore full weight
down	Mark server DOWN
drain	Drain: finish sessions, no new traffic
maint	Maintenance mode
50%	Set server weight to 50% of configured
ready	Clear maintenance/drain, restore to UP

Run the agent as a systemd service on each backend server. This gives you full programmatic control over HAProxy's load balancing decisions based on application-specific conditions.

Health Check Tuning Reference

backend app_servers
    # HTTP check with full options
    option httpchk GET /healthz HTTP/1.1\r\nHost:\ app.example.com
    http-check expect status 200
    timeout check 3s

    # Faster detection when a server just came back up
    server app1 10.0.1.10:8080 \
        check \
        inter 5s \
        fastinter 1s \
        downinter 15s \
        rise 2 \
        fall 3 \
        weight 100

The key insight: fastinter (1s) means you detect recovery in 2 seconds (rise 2), while downinter (15s) reduces check load on already-dead servers. Normal operation uses the standard inter interval.