Fix: Elasticsearch Cluster Health Red

!Symptoms

• Cluster health API returns red status
• Some indices have unassigned primary shards
• Kibana showing gaps in logs or unable to search
• Index creation failing with 'cluster_block_exception'

?Root Causes

• One or more data nodes down — primary shards lost
• Disk watermark exceeded (85% high, 90% flood stage) blocking allocation
• Corrupted shard from unclean node shutdown
• Insufficient nodes for replica allocation (single-node cluster with replicas)
• JVM heap out of memory causing node to leave the cluster

#Diagnosis Steps

1. 1Check cluster health: `curl localhost:9200/_cluster/health?pretty`
2. 2Find unassigned shards: `curl localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED`
3. 3Check allocation explanation: `curl -X POST localhost:9200/_cluster/allocation/explain?pretty`
4. 4Check node disk usage: `curl localhost:9200/_cat/nodes?v&h=name,disk.total,disk.used,disk.avail,disk.used_percent`
5. 5Check JVM heap: `curl localhost:9200/_nodes/stats/jvm?pretty | grep heap_used_percent`

>Fix

1. 1If disk full, delete old indices: `curl -X DELETE localhost:9200/logs-2024.01.*`
2. 2Reset flood stage watermark: `curl -X PUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.disk.watermark.flood_stage":"95%"}}'`
3. 3Reroute unassigned shards: `curl -X POST localhost:9200/_cluster/reroute?retry_failed=true`
4. 4For corrupted shards, allocate empty primary: `curl -X POST localhost:9200/_cluster/reroute -d '{"commands":[{"allocate_empty_primary":{"index":"<idx>","shard":0,"node":"<node>","accept_data_loss":true}}]}'`
5. 5Bring failed node back online or add a new node

*Prevention

• Monitor disk usage and set alerts at 70% (well before watermarks)
• Implement ILM (Index Lifecycle Management) to auto-delete old indices
• Use dedicated master nodes (3+) for cluster stability
• Set JVM heap to 50% of RAM (max 31GB) and monitor heap pressure
• Use shard allocation awareness for zone/rack distribution

Related Error Messages