Fix: GKE Node Pool Exhausted — Pods Unschedulable

!Symptoms

• Pods stuck in Pending state with 'Insufficient cpu' or 'Insufficient memory'
• Cluster autoscaler not adding new nodes
• Node pool at max size
• kubectl describe pod shows 'FailedScheduling' events

?Root Causes

• Node pool max size reached and autoscaler cannot add more
• Resource requests over-provisioned — more requested than needed
• Quota limit reached for the machine type in the region
• Pod anti-affinity or node selector preventing scheduling on available nodes
• System pods (kube-system) consuming significant node resources

#Diagnosis Steps

1. 1Check node pool autoscaler status: `gcloud container clusters describe <cluster> --format='yaml(nodePools[].autoscaling)'`
2. 2Check node allocatable vs requested: `kubectl describe nodes | grep -A 5 'Allocated resources'`
3. 3Check pending pods: `kubectl get pods --field-selector=status.phase=Pending`
4. 4Check quotas: `gcloud compute regions describe <region> --format='yaml(quotas)'`
5. 5Review autoscaler events: `kubectl -n kube-system logs -l app=cluster-autoscaler`

>Fix

1. 1Increase node pool max size: `gcloud container clusters update <cluster> --node-pool=<pool> --max-nodes=<new-max>`
2. 2Add a new node pool with different machine types
3. 3Right-size resource requests based on actual usage (VPA recommendations)
4. 4Request quota increase in GCP console
5. 5Relax pod anti-affinity rules if they're too strict

*Prevention

• Set node pool max to handle 2x normal load
• Use Vertical Pod Autoscaler (VPA) recommendations to right-size requests
• Monitor cluster resource utilization and alert at 80%
• Use multiple node pools with different machine types for flexibility
• Plan for quota headroom in all production regions

Related Error Messages