Keepalived VRRP Split-Brain: Diagnosing and Fixing Dual Master State After Network Partition

Split-brain in Keepalived means both nodes think they own the VIP. Traffic gets blackholed or split unpredictably. Here's how to diagnose and fix it fast.

---

Quick Diagnosis Checklist

# Check VRRP state on both nodes simultaneously
systemctl status keepalived
ip addr show | grep -E "inet.*<VIP>"

# If BOTH nodes show the VIP — you have split-brain

Signs you're in split-brain:

• Both nodes respond to ARP requests for the VIP
• Duplicate IP warnings in kernel logs
• Intermittent connection drops for VIP traffic
• Both nodes log Entering MASTER STATE

---

Step 1: Verify Current State

Run this on both nodes:

# Check VRRP state
journalctl -u keepalived --since "1 hour ago" | grep -E "MASTER|BACKUP|FAULT"

# Check who actually has the VIP bound
ip addr show dev eth0 | grep <YOUR_VIP>

# Check ARP table from a third host
arp -n | grep <YOUR_VIP>
arping -I eth0 <YOUR_VIP>   # watch for duplicate replies from different MACs

If arping shows two different MAC addresses responding — confirmed split-brain.

---

Step 2: Root Cause — Why Split-Brain Happens

Cause	Symptom
VRRP multicast blocked by firewall/switch	Both nodes stop seeing each other's advertisements
Network partition (link flap, switch failure)	Backup promotes itself; link restores but master doesn't step down
High CPU/load causing missed advertisements	Intermittent promotion events
Misconfigured advert_int / priority	Race condition on startup
Firewall blocking 224.0.0.18 (VRRP multicast)	Silent split

# Verify VRRP multicast traffic is flowing
tcpdump -i eth0 -n vrrp

# Expected output on working node:
# 10.0.0.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100

If you see no VRRP packets — the multicast is blocked. That's your culprit.

---

Step 3: Fix the Firewall (Most Common Cause)

# iptables — allow VRRP multicast
iptables -I INPUT -p vrrp -j ACCEPT
iptables -I OUTPUT -p vrrp -j ACCEPT

# firewalld
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p vrrp -j ACCEPT
firewall-cmd --direct --add-rule ipv4 filter OUTPUT 0 -p vrrp -j ACCEPT
firewall-cmd --runtime-to-permanent

# nftables
nft add rule inet filter input ip protocol 112 accept
nft add rule inet filter output ip protocol 112 accept

Protocol 112 is VRRP. Don't forget it if you rebuilt your firewall rules.

---

Step 4: Force a Clean State

Once multicast is restored, force a clean election:

# On the node that SHOULD be BACKUP — stop keepalived first
systemctl stop keepalived

# Verify the correct node still holds the VIP
ip addr show | grep <VIP>

# Now restart the backup node — it will see the master's advertisements
# and correctly enter BACKUP state
systemctl start keepalived

# Verify
journalctl -u keepalived -f
# Should see: "Entering BACKUP STATE"

Never restart both nodes simultaneously — you'll just trigger another election race.

---

Step 5: Harden Your Config Against Split-Brain

# /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {
    state BACKUP                  # Always start as BACKUP, let election decide
    interface eth0
    virtual_router_id 51
    priority 100                  # Primary: 100, Secondary: 90
    advert_int 1
    nopreempt                     # CRITICAL: prevent unnecessary failback

    # Use unicast instead of multicast — bypasses multicast issues entirely
    unicast_src_ip 10.0.0.1
    unicast_peer {
        10.0.0.2
    }

    authentication {
        auth_type PASS
        auth_pass yourpassword
    }

    virtual_ipaddress {
        10.0.0.100/24
    }

    # Track the interface — go to FAULT if link drops
    track_interface {
        eth0
    }
}

Key hardening options:

Option	What It Does
nopreempt	Higher-priority node won't yank VIP back after recovery
unicast_peer	Bypasses multicast entirely — works across VLANs/routers
track_interface	Forces FAULT state if NIC goes down
advert_int 1	1-second advertisements (default) — reduce for faster detection

---

Quick-Reference Diagnostic Commands

# Live state watch
watch -n1 "ip addr show | grep -E 'eth0|<VIP>'"

# VRRP packet capture (run on both nodes)
tcpdump -i eth0 -n -v vrrp 2>/dev/null

# Check keepalived logs for state transitions
grep -E "MASTER|BACKUP|FAULT|transition" /var/log/syslog | tail -50

# Verify no duplicate ARP
arping -D -I eth0 <VIP>   # exit code 1 = duplicate found

# Send gratuitous ARP after fixing (forces switches to update MAC tables)
arping -U -I eth0 -c 3 <VIP>

---

The Nuclear Option

If you can't diagnose quickly and need to restore service:

# 1. Stop keepalived on BOTH nodes
systemctl stop keepalived

# 2. Manually remove the VIP from whichever node shouldn't have it
ip addr del <VIP>/24 dev eth0

# 3. Start keepalived on PRIMARY first
systemctl start keepalived
sleep 3

# 4. Start SECONDARY — it will see primary's advertisements
systemctl start keepalived   # on secondary

---

TL;DR

1. Confirm split-brain with arping — two MACs = confirmed
2. Check multicast with tcpdump -n vrrp — no traffic = firewall issue
3. Fix firewall to allow protocol 112
4. Restart BACKUP node only, never both simultaneously
5. Harden with unicast_peer + nopreempt to prevent recurrence

Unicast VRRP is the single biggest reliability improvement you can make — it eliminates an entire class of multicast-related split-brain failures.