DevOpsil
KubernetesCritical

Fix: Kubernetes Pod OOMKilled

Pod status shows OOMKilled

!Symptoms

  • Pod status shows OOMKilled
  • Container exits with code 137
  • Application suddenly terminates without graceful shutdown logs
  • Node memory pressure events in kubectl describe node

?Root Causes

  • Memory limits set too low for the workload
  • Memory leak in the application
  • JVM heap size exceeding container memory limit
  • Large in-memory caches or data structures growing unbounded
  • Multiple processes inside the container consuming more than allocated

#Diagnosis Steps

  1. 1Run `kubectl describe pod <pod-name>` and check the Last State reason for OOMKilled
  2. 2Check `kubectl top pod <pod-name>` to see current memory usage
  3. 3Review container memory limits in the deployment spec
  4. 4Check node-level memory with `kubectl top nodes`
  5. 5Use `kubectl logs --previous` to find any memory-related messages before the kill

>Fix

  1. 1Increase memory limits in the pod spec if the workload legitimately needs more
  2. 2For JVM apps, set -Xmx to ~75% of the container memory limit
  3. 3Fix memory leaks in the application code
  4. 4Add memory-aware garbage collection tuning
  5. 5Split large workloads across multiple pods with horizontal scaling

*Prevention

  • Right-size memory limits based on profiling under realistic load
  • Set up memory usage alerts before hitting limits (e.g., at 80%)
  • Use Vertical Pod Autoscaler (VPA) to automatically adjust resource limits
  • Implement proper connection pooling and cache eviction policies
  • Load test with realistic data volumes before deploying

Related Error Messages

OOMKilledexit code 137memory cgroup out of memoryCannot allocate memorykilled by signal 9