KubernetesCritical
Fix: Kubernetes Pod OOMKilled
Pod status shows OOMKilled
!Symptoms
- Pod status shows OOMKilled
- Container exits with code 137
- Application suddenly terminates without graceful shutdown logs
- Node memory pressure events in kubectl describe node
?Root Causes
- Memory limits set too low for the workload
- Memory leak in the application
- JVM heap size exceeding container memory limit
- Large in-memory caches or data structures growing unbounded
- Multiple processes inside the container consuming more than allocated
#Diagnosis Steps
- 1Run `kubectl describe pod <pod-name>` and check the Last State reason for OOMKilled
- 2Check `kubectl top pod <pod-name>` to see current memory usage
- 3Review container memory limits in the deployment spec
- 4Check node-level memory with `kubectl top nodes`
- 5Use `kubectl logs --previous` to find any memory-related messages before the kill
>Fix
- 1Increase memory limits in the pod spec if the workload legitimately needs more
- 2For JVM apps, set -Xmx to ~75% of the container memory limit
- 3Fix memory leaks in the application code
- 4Add memory-aware garbage collection tuning
- 5Split large workloads across multiple pods with horizontal scaling
*Prevention
- Right-size memory limits based on profiling under realistic load
- Set up memory usage alerts before hitting limits (e.g., at 80%)
- Use Vertical Pod Autoscaler (VPA) to automatically adjust resource limits
- Implement proper connection pooling and cache eviction policies
- Load test with realistic data volumes before deploying
Related Error Messages
OOMKilledexit code 137memory cgroup out of memoryCannot allocate memorykilled by signal 9