Fix: Prometheus High Cardinality Causing OOM

!Symptoms

• Prometheus consuming excessive memory and getting OOMKilled
• Slow query responses in Grafana dashboards
• Prometheus WAL replay taking very long after restarts
• TSDB head chunk count growing unbounded

?Root Causes

• Metrics with high-cardinality labels (user IDs, request IDs, UUIDs)
• Uncontrolled label values from dynamic sources (pod names in large clusters)
• Too many targets being scraped with overlapping metrics
• Missing metric_relabel_configs to drop unnecessary series
• Recording rules generating high-cardinality output

#Diagnosis Steps

1. 1Check TSDB stats: `curl localhost:9090/api/v1/status/tsdb` — look at seriesCountByMetricName
2. 2Query top metrics by cardinality: `topk(10, count by (__name__)({__name__=~'.+'}))`
3. 3Check head series count: `prometheus_tsdb_head_series`
4. 4Review recently added scrape configs for new high-cardinality targets
5. 5Check memory usage trend: `process_resident_memory_bytes{job='prometheus'}`

>Fix

1. 1Add metric_relabel_configs to drop high-cardinality labels: `action: labeldrop, regex: request_id`
2. 2Aggregate metrics at the source — don't export per-user or per-request metrics
3. 3Increase Prometheus memory limits temporarily while fixing the root cause
4. 4Use recording rules to pre-aggregate and drop raw high-cardinality series
5. 5Limit scrape targets using relabeling: `action: keep/drop` on service discovery

*Prevention

• Set up cardinality alerts: alert when head series exceeds a threshold
• Review all new metrics in PR reviews for unbounded label values
• Use promtool to lint and analyze metrics before deploying
• Implement label allow-lists in scrape configs
• Plan capacity based on expected series count, not just target count

Related Error Messages