Fix: Kubernetes Node NotReady

!Symptoms

• kubectl get nodes shows one or more nodes in NotReady state
• Pods on the affected node are being evicted or unreachable
• Cluster capacity drops, new pods cannot be scheduled
• Node conditions show MemoryPressure, DiskPressure, or PIDPressure

?Root Causes

• Kubelet process has crashed or is not running
• Node ran out of disk space, memory, or PIDs
• Network partition: node cannot communicate with the API server
• Container runtime (containerd, Docker) has crashed
• Kernel panic or hardware failure on the node
• Certificate expired on the node

#Diagnosis Steps

1. 1Check node conditions: `kubectl describe node <node-name>` and review Conditions section
2. 2SSH to the node and check kubelet: `systemctl status kubelet`
3. 3Check kubelet logs: `journalctl -u kubelet --since '10 minutes ago'`
4. 4Check container runtime: `systemctl status containerd` or `systemctl status docker`
5. 5Check disk: `df -h`, memory: `free -m`, PIDs: `ps aux | wc -l`

>Fix

1. 1Restart kubelet: `systemctl restart kubelet`
2. 2If disk pressure: free space on the node (clean images, logs)
3. 3If memory pressure: identify and kill memory-intensive processes
4. 4Restart the container runtime: `systemctl restart containerd`
5. 5If the node is unrecoverable: drain and replace it: `kubectl drain <node> --ignore-daemonsets --delete-emptydir-data`

*Prevention

• Monitor node health with node-problem-detector and alerts
• Set up kubelet eviction thresholds appropriately
• Use node auto-repair (GKE, EKS managed node groups)
• Monitor and alert on node conditions (DiskPressure, MemoryPressure)
• Keep nodes updated and rotate/replace them regularly

Related Error Messages