Keepalived Health Check Scripts: Beyond Basic VRRP

Basic Keepalived VRRP failover only detects one failure mode: the entire node going down. But what about when the node is up but Nginx has crashed? Or when HAProxy is running but can't reach any backends? track_script lets you define custom health checks that trigger failover based on any condition you can script.

Why track_script Matters

Without track_script, Keepalived stays MASTER as long as the node is alive. The VIP doesn't move even if:

• Nginx or HAProxy crashes (the actual load balancer is dead)
• All backend servers are down (the LB is up but serving 502s)
• A critical dependency (database, Redis) becomes unavailable
• Disk is full and the LB can't write logs
• Certificate has expired

With track_script, you define scripts that run periodically. If a script returns a non-zero exit code, Keepalived reduces that node's VRRP priority. If priority drops below the BACKUP node's priority, the VIP moves.

Basic track_script Setup

# /etc/keepalived/keepalived.conf

global_defs {
    router_id LB_PRIMARY
    script_user keepalived_script
    enable_script_security
}

# Define the check script
vrrp_script check_nginx {
    script "/etc/keepalived/scripts/check_nginx.sh"
    interval 2          # Run every 2 seconds
    timeout 1           # Kill script after 1 second
    rise 2              # 2 successes to recover
    fall 2              # 2 failures to trigger
    weight -20          # Subtract 20 from priority on failure
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass strongpassword
    }

    virtual_ipaddress {
        10.0.0.100/24 dev eth0
    }

    # Link the check to this VRRP instance
    track_script {
        check_nginx
    }
}

With weight -20 and a primary priority of 100: when the script fails, priority drops to 80. If the backup has priority 90, it wins the election and claims the VIP.

Creating a Dedicated Script User

For security, don't run health check scripts as root:

# Create a user with minimal privileges
useradd -r -s /bin/false keepalived_script

# Scripts need to run as this user — make them executable
chown root:keepalived_script /etc/keepalived/scripts/
chmod 750 /etc/keepalived/scripts/

Some checks need elevated privileges (e.g., checking listening ports with ss). Use sudoers for specific commands:

# /etc/sudoers.d/keepalived-script
keepalived_script ALL=(root) NOPASSWD: /usr/bin/systemctl is-active nginx
keepalived_script ALL=(root) NOPASSWD: /usr/bin/systemctl is-active haproxy
keepalived_script ALL=(root) NOPASSWD: /usr/sbin/ss

Nginx Health Check Script

#!/bin/bash
# /etc/keepalived/scripts/check_nginx.sh

# Method 1: Check if nginx process is running
if ! systemctl is-active --quiet nginx; then
    echo "nginx is not running"
    exit 1
fi

# Method 2: Check if nginx responds to HTTP requests
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
    --max-time 2 http://127.0.0.1/healthz)

if [ "$HTTP_STATUS" != "200" ]; then
    echo "nginx health check failed: HTTP $HTTP_STATUS"
    exit 1
fi

exit 0

chmod +x /etc/keepalived/scripts/check_nginx.sh

HAProxy Health Check Script

#!/bin/bash
# /etc/keepalived/scripts/check_haproxy.sh

# Check HAProxy process
if ! systemctl is-active --quiet haproxy; then
    echo "haproxy is not running"
    exit 1
fi

# Check HAProxy stats socket is responsive
if ! echo "show info" | socat stdio /run/haproxy/admin.sock 2>/dev/null | \
     grep -q "Uptime"; then
    echo "haproxy stats socket not responding"
    exit 1
fi

# Check that at least one backend server is UP
DOWN_SERVERS=$(echo "show stat" | \
    socat stdio /run/haproxy/admin.sock 2>/dev/null | \
    awk -F',' '$18 == "DOWN" && $2 != "BACKEND" {count++} END {print count+0}')

TOTAL_SERVERS=$(echo "show stat" | \
    socat stdio /run/haproxy/admin.sock 2>/dev/null | \
    awk -F',' '$18 ~ /^(UP|DOWN)$/ && $2 != "BACKEND" {count++} END {print count+0}')

if [ "$DOWN_SERVERS" -eq "$TOTAL_SERVERS" ] && [ "$TOTAL_SERVERS" -gt 0 ]; then
    echo "All $TOTAL_SERVERS backend servers are DOWN"
    exit 1
fi

exit 0

Checking Backend Reachability

Sometimes you want to failover based on whether the primary node can actually reach backends — perhaps a network partition has split your infrastructure:

#!/bin/bash
# /etc/keepalived/scripts/check_backends.sh

BACKENDS=("10.0.1.10:8080" "10.0.1.11:8080" "10.0.1.12:8080")
HEALTHY=0
REQUIRED=1   # At least this many must be reachable

for backend in "${BACKENDS[@]}"; do
    HOST="${backend%%:*}"
    PORT="${backend##*:}"

    if curl -s -o /dev/null -w "%{http_code}" \
        --max-time 2 "http://$HOST:$PORT/healthz" | grep -q "^2"; then
        ((HEALTHY++))
    fi
done

if [ "$HEALTHY" -lt "$REQUIRED" ]; then
    echo "Only $HEALTHY/$REQUIRED backends are healthy"
    exit 1
fi

exit 0

Multiple Scripts with Different Weights

Use multiple scripts with different weights to model the severity of each failure:

vrrp_script check_haproxy {
    script "/etc/keepalived/scripts/check_haproxy.sh"
    interval 2
    timeout 2
    rise 2
    fall 2
    weight -30      # Critical: HAProxy down drops priority significantly
}

vrrp_script check_backends {
    script "/etc/keepalived/scripts/check_backends.sh"
    interval 5
    timeout 4
    rise 3
    fall 3
    weight -15      # Backends down drops priority moderately
}

vrrp_script check_disk {
    script "/etc/keepalived/scripts/check_disk.sh"
    interval 30
    timeout 5
    rise 2
    fall 3
    weight -10      # Disk issue drops priority slightly
}

vrrp_instance VI_1 {
    state MASTER
    priority 110    # Start high enough to survive partial failures

    track_script {
        check_haproxy
        check_backends
        check_disk
    }
}

Priority math example:

• Base priority: 110
• HAProxy down: -30 → priority 80
• All backends down: -15 → priority 95
• Both: -45 → priority 65

The backup node has priority 90. In this case:

• HAProxy down alone: 80 < 90 → failover happens
• Backends down alone: 95 > 90 → no failover (backend issue affects both nodes equally)
• Both down: 65 < 90 → failover happens

Disk Space Check Script

#!/bin/bash
# /etc/keepalived/scripts/check_disk.sh

THRESHOLD=90   # Fail if disk usage exceeds this percentage
MOUNT="/"

USAGE=$(df -h "$MOUNT" | awk 'NR==2 {gsub(/%/,""); print $5}')

if [ "$USAGE" -ge "$THRESHOLD" ]; then
    echo "Disk usage on $MOUNT is ${USAGE}% (threshold: ${THRESHOLD}%)"
    exit 1
fi

exit 0

Debugging Script Execution

When scripts aren't triggering failover as expected:

# Run the script manually as the keepalived_script user
sudo -u keepalived_script /etc/keepalived/scripts/check_nginx.sh
echo "Exit code: $?"

# Watch keepalived logs for script results
journalctl -u keepalived -f

# Enable verbose logging
# Add to keepalived.conf:
# global_defs { ... }
# Then restart: systemctl restart keepalived

# View current VRRP priorities including script adjustments
# Keepalived 2.2+:
ps aux | grep keepalived
# Then send SIGUSR2 to dump state:
kill -SIGUSR2 $(cat /var/run/keepalived/keepalived.pid)
# Check /tmp/keepalived.data

Logs to watch for:

keepalived[1234]: VRRP_Script(check_nginx) succeeded
keepalived[1234]: VRRP_Script(check_nginx) failed
keepalived[1234]: (VI_1) Changing effective priority from 100 to 80
keepalived[1234]: (VI_1) Master received advert from 10.0.0.20 with higher priority 90, ours 80
keepalived[1234]: (VI_1) Entering BACKUP STATE

track_interface: Fail Over on Network Issues

Beyond scripts, you can also track network interfaces:

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    priority 100

    # If eth1 (backend-facing network) goes down, reduce priority by 50
    track_interface {
        eth1 weight -50
    }

    track_script {
        check_haproxy
    }
}

If the interface carrying backend traffic fails, this node loses priority and the backup (which presumably still has its eth1 up) takes over.

Script Timeout Pitfalls

Common mistakes with track_script timing:

Mistake	Problem	Fix
timeout too long	Script runs past interval, keepalived queues checks	Set timeout < interval
interval too short	Script runs constantly, high CPU	Use 2-5s for critical, 30s for low-priority
No timeout set	Long-running script blocks keepalived	Always set explicit timeout
Script calls curl without --max-time	Curl hangs on slow backends	Always use --max-time
Script exits 0 on error	Keepalived never sees failure	Use set -e or explicit exit codes

A template for safe scripts:

#!/bin/bash
set -euo pipefail

# Always use timeouts on network calls
RESULT=$(curl -sf --max-time 2 http://127.0.0.1/healthz) || exit 1

# Validate result content
echo "$RESULT" | grep -q '"status":"ok"' || exit 1

exit 0

set -euo pipefail means any unhandled error or undefined variable causes the script to exit non-zero — which Keepalived interprets as a health check failure.