Fix: Kubernetes Pods Evicted

!Symptoms

• Pods show status 'Evicted'
• Many Evicted pods accumulating in the namespace
• Application instances suddenly disappear
• kubectl describe pod shows 'The node was low on resource: ephemeral-storage'

?Root Causes

• Node running low on disk space (ephemeral-storage pressure)
• Node running low on memory (memory pressure)
• Pods exceeding their ephemeral-storage limits
• Application writing too many logs or temp files to the container filesystem
• Node conditions triggering eviction thresholds

#Diagnosis Steps

1. 1Check pod status: `kubectl get pods --field-selector=status.phase=Failed`
2. 2Describe evicted pod: `kubectl describe pod <evicted-pod>` for the eviction reason
3. 3Check node conditions: `kubectl describe node <node> | grep -A 5 Conditions`
4. 4Check node disk pressure: `kubectl get nodes -o json | jq '.items[].status.conditions[] | select(.type=="DiskPressure")'`
5. 5Check ephemeral storage usage: `kubectl exec <pod> -- df -h`

>Fix

1. 1Clean up evicted pods: `kubectl get pods --field-selector=status.phase=Failed -o name | xargs kubectl delete`
2. 2Free disk space on the affected node (clean docker images, logs)
3. 3Set ephemeral-storage requests and limits in pod specs
4. 4Move application logs to stdout/stderr instead of writing to container filesystem
5. 5Add more nodes or increase node disk size

*Prevention

• Set ephemeral-storage limits on all pods
• Monitor node disk usage and set alerts before eviction thresholds
• Configure log rotation on nodes and container runtime log limits
• Use PersistentVolumes for data instead of ephemeral container storage
• Set up a CronJob to clean up evicted pods: `kubectl delete pods --field-selector=status.phase=Failed`

Related Error Messages