Fix Prometheus High Memory Usage and OOM Kills

The Error: Prometheus Killed by OOM

You check your monitoring server and find Prometheus has been killed:

dmesg | grep -i oom
[12345.678] Out of memory: Killed process 1234 (prometheus) total-vm:16524288kB

Or in journalctl:

prometheus.service: Main process exited, code=killed, status=9/KILL

Prometheus keeps all recently ingested data in memory. When it runs out, the kernel OOM killer steps in.

Root Cause

Prometheus memory usage is directly tied to the number of active time series. Each unique combination of metric name and label values is a separate series. Common causes of memory explosion:

• High cardinality labels like user IDs, request paths, or pod UIDs
• Too many targets being scraped simultaneously
• Long --storage.tsdb.retention.time keeping too much data on disk (and in memory during compaction)
• Missing sample_limit allowing targets to dump unlimited metrics

Step-by-Step Fix

1. Find Your Current Series Count

prometheus_tsdb_head_series

If this number is in the millions, you have a cardinality problem. A healthy Prometheus instance typically handles 1-2 million series per 8 GB of RAM.

2. Identify the Biggest Offenders

Find which jobs produce the most series:

topk(10, count by (job) ({__name__=~".+"}))

Find metrics with the most label combinations:

topk(10, count by (__name__) ({__name__=~".+"}))

3. Drop High-Cardinality Metrics

Add metric_relabel_configs to your prometheus.yml:

scrape_configs:
  - job_name: 'my-app'
    metric_relabel_configs:
      # Drop metrics with high-cardinality labels
      - source_labels: [__name__]
        regex: 'http_request_duration_bucket'
        action: drop
      # Remove a specific high-cardinality label
      - regex: 'instance_id'
        action: labeldrop

4. Set Sample Limits per Target

Protect Prometheus from misbehaving targets:

scrape_configs:
  - job_name: 'my-app'
    sample_limit: 10000
    static_configs:
      - targets: ['app:8080']

If a target returns more than 10,000 samples, the entire scrape is rejected and you'll see a clear error instead of an OOM.

5. Tune Storage Flags

Start Prometheus with memory-conscious flags:

prometheus \
  --storage.tsdb.retention.time=15d \
  --storage.tsdb.retention.size=30GB \
  --storage.tsdb.wal-compression \
  --query.max-samples=50000000

Reducing retention from 30 days to 15 days can cut memory usage significantly during compaction cycles.

6. Restart and Monitor

sudo systemctl restart prometheus

# Watch memory usage
watch -n 5 'ps -o pid,rss,comm -p $(pidof prometheus) | awk "{print \$2/1024 \" MB\"}"'

7. Verify Series Count is Dropping

After config reload, check that series count decreases over the next few scrape cycles:

rate(prometheus_tsdb_head_series_created_total[5m])

Prevention Tips

• Set sample_limit on every scrape job. This is your safety net against cardinality bombs.
• Alert on series count: prometheus_tsdb_head_series > 2000000 should page someone.
• Use recording rules instead of expensive queries that force Prometheus to hold more data in memory.
• Move long-term storage to Thanos or Cortex and keep Prometheus retention at 2-7 days.
• Review new metrics in code review. A label with unbounded values is a production incident waiting to happen.