Fix: HashiCorp Vault Sealed and Inaccessible

!Symptoms

• Vault API returns 503 with 'Vault is sealed'
• Applications failing to authenticate — secrets unavailable
• Vault UI shows sealed status
• All dependent services failing simultaneously

?Root Causes

• Vault server restarted and auto-unseal not configured
• Auto-unseal KMS key deleted or permissions revoked
• Raft storage corruption after unclean shutdown
• All Vault nodes restarted at the same time (rolling restart skipped)
• Seal migration failed or was interrupted

#Diagnosis Steps

1. 1Check Vault status: `vault status` — look for Sealed: true
2. 2Check Vault logs: `journalctl -u vault` or container logs
3. 3If auto-unseal, check KMS connectivity and permissions
4. 4Check Raft peers: `vault operator raft list-peers`
5. 5Verify the unseal keys are available (stored in secure location)

>Fix

1. 1Unseal manually with unseal keys: `vault operator unseal <key>` (need threshold count)
2. 2Fix auto-unseal KMS permissions and restart Vault
3. 3If Raft corrupted, restore from snapshot: `vault operator raft snapshot restore <file>`
4. 4For HA clusters, unseal one node and let others join via Raft
5. 5If keys are lost: reinitialize Vault (DESTRUCTIVE — all secrets lost)

*Prevention

• Always configure auto-unseal (AWS KMS, GCP KMS, Azure Key Vault, or Transit)
• Never restart all Vault nodes simultaneously — use rolling restarts
• Take regular Raft snapshots: `vault operator raft snapshot save backup.snap`
• Store unseal keys in multiple secure locations (Shamir's Secret Sharing)
• Monitor Vault seal status and alert immediately on seal events

Related Error Messages