Prometheus Recording Rules for High-Cardinality Metric Aggregation

If you've been running Prometheus in production for more than a few months, you've probably hit the wall. Dashboards that take 30 seconds to load. Alerts that timeout before they fire. Engineers complaining that Grafana is "broken again." Nine times out of ten, the culprit isn't your infrastructure — it's high-cardinality metrics and the expensive queries that try to wrangle them in real time.

Recording rules are Prometheus's answer to this problem. They're one of those features that feels optional until suddenly it's absolutely essential. This guide is the deep dive I wish I had when I first started tearing my hair out over slow queries and overloaded Prometheus servers.

---

What Are Recording Rules, and Why Should You Care?

A recording rule pre-computes a PromQL expression on a schedule and saves the result as a new time series. Instead of running an expensive aggregation every time a dashboard panel refreshes or an alert evaluates, Prometheus does the heavy lifting once per evaluation interval and stores the answer.

Think of it like a materialized view in a database. You're trading storage for query speed — and in almost every production scenario, that's a trade worth making.

Here's the core idea:

# Instead of running this expensive query every 15 seconds across all dashboards:
# sum(rate(http_requests_total[5m])) by (service, status_code)

# You pre-compute it once and store the result as a new metric:
- record: service:http_requests:rate5m
  expr: sum(rate(http_requests_total[5m])) by (service, status_code)

Every query against service:http_requests:rate5m now hits a simple time series lookup instead of scanning millions of raw data points.

---

Understanding High Cardinality (And Why It's Killing Your Prometheus)

Before diving into recording rules, you need to understand why high cardinality is a problem.

Cardinality is the total number of unique time series in your Prometheus instance. Each unique combination of metric name and label set is a separate series. A metric like http_requests_total with labels {method, endpoint, status_code, pod, namespace} can explode fast:

• 3 HTTP methods × 200 endpoints × 15 status codes × 50 pods × 5 namespaces = 2,250,000 series

Every one of those series consumes memory. Every query that aggregates across them burns CPU. High-cardinality metrics aren't inherently bad — they give you granular observability — but querying them raw in real time is a performance disaster.

Recording rules let you have your cake and eat it too: keep the granular raw data, but pre-aggregate to the cardinality you actually need for dashboards and alerts.

---

Anatomy of a Recording Rule File

Recording rules live in YAML files and are loaded by Prometheus via the rule_files configuration in prometheus.yml.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s  # This drives recording rule evaluation

rule_files:
  - "rules/*.yml"
  - "alerts/*.yml"

A rule file looks like this:

# rules/http_aggregations.yml
groups:
  - name: http.rules
    interval: 30s  # Optional: override global evaluation_interval for this group
    rules:
      - record: job:http_requests_total:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

      - record: job:http_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(
            0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le)
          )

Key structural elements:

• groups — A list of rule groups. Groups are evaluated sequentially, but rules within a group are evaluated concurrently.
• name — A unique identifier for the group. Use it meaningfully.
• interval — How often the rules in this group are evaluated. Defaults to global.evaluation_interval.
• record — The name of the new metric to write. Must follow Prometheus naming conventions.
• expr — The PromQL expression to evaluate.

---

The Naming Convention You Need to Follow

Prometheus has an official naming convention for recorded metrics, and you should follow it. The format is:

level:metric:operations

• level — The aggregation level (e.g., job, service, cluster, namespace)
• metric — The base metric name (e.g., http_requests_total, cpu_seconds)
• operations — What was done to produce this metric (e.g., rate5m, sum, p99_1h)

Real examples:

job:http_requests_total:rate5m
namespace:container_cpu_usage_seconds:rate1m
cluster:node_memory_usage_bytes:avg
service:http_request_duration_seconds:p99_5m

This isn't just aesthetics. When you're six months into production and you have 50 recording rules, consistent naming is the difference between maintaining sanity and total chaos. It also makes it immediately obvious what a metric represents without reading the PromQL.

---

Practical Recording Rules: Real-World Scenarios

Let's get into the scenarios you'll actually encounter.

Scenario 1: HTTP Request Rate Aggregation

You have hundreds of pods emitting http_requests_total with {method, endpoint, status_code, pod, instance} labels. Your SLA dashboard needs per-service request rates, but querying all pods in real time is slow.

groups:
  - name: http.aggregations
    interval: 30s
    rules:
      # Total request rate per service
      - record: service:http_requests_total:rate5m
        expr: |
          sum(rate(http_requests_total[5m])) by (service, namespace)

      # Error rate per service (4xx and 5xx)
      - record: service:http_requests_errors:rate5m
        expr: |
          sum(
            rate(http_requests_total{status_code=~"[45].."}[5m])
          ) by (service, namespace)

      # Error ratio — useful for SLO calculations
      - record: service:http_error_ratio:rate5m
        expr: |
          service:http_requests_errors:rate5m
            /
          service:http_requests_total:rate5m

Notice how the last rule builds on the two rules before it. This is a rule chain — using recorded metrics as inputs to other recording rules. When you do this, make sure the rules that produce the inputs appear before the rules that consume them in the same group, or put them in separate groups with the producer group first.

Scenario 2: Histogram Quantile Pre-Computation

Computing histogram quantiles at query time is expensive because it requires fetching all _bucket series. Pre-compute your latency percentiles:

groups:
  - name: latency.aggregations
    interval: 60s
    rules:
      # P50, P95, P99 per service
      - record: service:http_request_duration_seconds:p50_5m
        expr: |
          histogram_quantile(
            0.50,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le)
          )

      - record: service:http_request_duration_seconds:p95_5m
        expr: |
          histogram_quantile(
            0.95,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le)
          )

      - record: service:http_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(
            0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le)
          )

For SLO dashboards that need 30-day percentiles, you'd increase the rate window and evaluation interval accordingly:

      - record: service:http_request_duration_seconds:p99_1h
        expr: |
          histogram_quantile(
            0.99,
            sum(rate(http_request_duration_seconds_bucket[1h])) by (service, le)
          )

Scenario 3: Kubernetes Resource Utilization

Kubernetes environments are cardinality bombs. Container metrics have labels for pod, container, namespace, node, and more. Pre-aggregate for namespace and cluster level views:

groups:
  - name: kubernetes.resource.aggregations
    interval: 60s
    rules:
      # CPU usage rate per namespace
      - record: namespace:container_cpu_usage_seconds_total:rate5m
        expr: |
          sum(
            rate(container_cpu_usage_seconds_total{container!=""}[5m])
          ) by (namespace)

      # Memory usage per namespace (non-aggregated, just sum)
      - record: namespace:container_memory_working_set_bytes:sum
        expr: |
          sum(
            container_memory_working_set_bytes{container!=""}
          ) by (namespace)

      # CPU requests vs. usage ratio per namespace
      - record: namespace:cpu_utilization:ratio
        expr: |
          namespace:container_cpu_usage_seconds_total:rate5m
            /
          sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)

      # Node-level CPU utilization
      - record: node:cpu_utilization:ratio
        expr: |
          1 - avg(
            rate(node_cpu_seconds_total{mode="idle"}[5m])
          ) by (node)

Scenario 4: SLO Burn Rate for Alerting

Multi-window, multi-burn-rate SLO alerting (as recommended by Google's SRE workbook) requires comparing error rates across multiple time windows. This is a perfect recording rule use case:

groups:
  - name: slo.error_rates
    interval: 60s
    rules:
      # Error rate over 5 minutes
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (job)
            /
          sum(rate(http_requests_total[5m])) by (job)

      # Error rate over 30 minutes
      - record: job:slo_errors_per_request:ratio_rate30m
        expr: |
          sum(rate(http_requests_total{status_code=~"5.."}[30m])) by (job)
            /
          sum(rate(http_requests_total[30m])) by (job)

      # Error rate over 1 hour
      - record: job:slo_errors_per_request:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{status_code=~"5.."}[1h])) by (job)
            /
          sum(rate(http_requests_total[1h])) by (job)

      # Error rate over 6 hours
      - record: job:slo_errors_per_request:ratio_rate6h
        expr: |
          sum(rate(http_requests_total{status_code=~"5.."}[6h])) by (job)
            /
          sum(rate(http_requests_total[6h])) by (job)

Then your alert rules become clean and fast:

groups:
  - name: slo.alerts
    rules:
      - alert: HighBurnRate
        expr: |
          job:slo_errors_per_request:ratio_rate1h > 14.4 * 0.001
          AND
          job:slo_errors_per_request:ratio_rate5m > 14.4 * 0.001
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error burn rate for {{ $labels.job }}"

---

Managing and Loading Recording Rules

Validating Rules Before Deployment

Never deploy recording rules without validating them first. Prometheus ships with promtool for exactly this purpose:

# Validate a single rules file
promtool check rules rules/http_aggregations.yml

# Validate all rules files
promtool check rules rules/*.yml

# Test rules against sample data (requires test files)
promtool test rules tests/http_aggregations_test.yml

Hot-Reloading Rules Without Restart

You can reload Prometheus configuration and rules without restarting:

# Send SIGHUP to reload
kill -HUP $(pidof prometheus)

# Or use the HTTP API (requires --web.enable-lifecycle flag)
curl -X POST http://localhost:9090/-/reload

Writing Unit Tests for Recording Rules

This is something most teams skip and then regret. Prometheus supports unit testing recording and alerting rules:

# tests/http_aggregations_test.yml
evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{service="api", status_code="200", pod="api-1"}'
        values: '0 60 120 180 240'
      - series: 'http_requests_total{service="api", status_code="500", pod="api-1"}'
        values: '0 6 12 18 24'

    promql_expr_test:
      - expr: service:http_requests_total:rate5m
        eval_time: 4m
        exp_samples:
          - labels: '{service="api", namespace=""}'
            value: 4  # (240 total / 60 seconds) ≈ 4 req/s over 5m window

      - expr: service:http_error_ratio:rate5m
        eval_time: 4m
        exp_samples:
          - labels: '{service="api", namespace=""}'
            value: 0.1  # 10% error rate

Run tests with:

promtool test rules tests/http_aggregations_test.yml

---

Performance Considerations and Best Practices

Choose Your Evaluation Interval Wisely

Not all recording rules need to run every 15 seconds. Longer evaluation intervals mean less CPU load and fewer writes to storage. A good rule of thumb:

• 15-30s — Operational dashboards, real-time views
• 60s — Standard SLO metrics, most dashboard panels
• 5m — Long-term trend data, capacity planning
• 1h — Very long-window aggregations (daily/weekly summaries)

Match your evaluation interval to the shortest time window in your expression. Running a 1-hour rate calculation every 15 seconds wastes resources — the result barely changes between evaluations.

Avoid Chaining Too Deep

Rule chains are powerful but can bite you. If Rule C depends on Rule B which depends on Rule A, they must all be in the correct order within the same group (since rules in a group are processed sequentially with outputs available to subsequent rules), or in separate ordered groups.

groups:
  # Group 1 runs first
  - name: base.aggregations
    rules:
      - record: service:http_requests_total:rate5m
        expr: sum(rate(http_requests_total[5m])) by (service)
      - record: service:http_requests_errors:rate5m
        expr: sum(rate(http_requests_total{status_code=~"[45].."}[5m])) by (service)

  # Group 2 depends on Group 1 outputs
  - name: derived.aggregations
    rules:
      - record: service:http_error_ratio:rate5m
        expr: |
          service:http_requests_errors:rate5m
            /
          service:http_requests_total:rate5m

In practice, keep chains to two or three levels deep. Beyond that, the dependency graph becomes hard to reason about.

Label Hygiene in Recording Rules

Be deliberate about which labels you preserve in your by() or without() clauses. Every label you keep multiplies your cardinality. Every label you drop loses information that can't be recovered later.

# Too many labels — cardinality explosion
- record: service:http_requests:rate5m
  expr: sum(rate(http_requests_total[5m])) by (service, pod, instance, endpoint)

# Right balance for a service-level dashboard
- record: service:http_requests:rate5m
  expr: sum(rate(http_requests_total[5m])) by (service, namespace)

# If you need pod-level data too, create a separate rule with a different name
- record: pod:http_requests:rate5m
  expr: sum(rate(http_requests_total[5m])) by (pod, service, namespace)

Use without() When Dropping a Few Labels

When you only want to drop one or two labels from a metric that has many, without() is cleaner than listing everything in by():

# Instead of listing all labels you want to keep:
- record: service:http_requests:rate5m
  expr: sum(rate(http_requests_total[5m])) by (service, namespace, method, status_code)

# Drop just the ones you don't want:
- record: service:http_requests:rate5m
  expr: sum without (pod, instance, node) (rate(http_requests_total[5m]))

---

Common Pitfalls and How to Avoid Them

Pitfall 1: Staleness and Lookback Mismatches

If your recording rule evaluation interval is longer than your scrape interval, you might get stale data or gaps. Example: if scrape_interval is 30s and your recording rule uses rate(metric[1m]), you have two data points in the window — just barely enough. Go lower and you'll see gaps.

Rule: The range in your rate() or increase() should be at least 2× your scrape interval, and your recording rule evaluation interval should be ≤ the range vector.

Pitfall 2: Recording Rules That Don't Reduce Cardinality

I've seen teams create recording rules that... produce the same number of series as the original metric. They add all the same labels back in the by() clause. The rule runs, writes data, and saves nothing.

Before creating a rule, calculate the expected cardinality of the result. If it's close to the input cardinality, question whether you actually need that rule.

Pitfall 3: Forgetting to Update Dashboards

When you add recording rules to speed up dashboards, you need to actually update the dashboards to use the recorded metrics. It sounds obvious, but I've seen this skipped. The rule runs, nobody uses it, and you wonder why dashboards are still slow.

Create a checklist:

1. Write and validate the recording rule
2. Deploy and verify data is being produced (prometheus.rule_group_last_evaluation_timestamp_seconds)
3. Update dashboard panels to use the recorded metric
4. Monitor dashboard load time improvements
5. Consider removing the original high-cost panels or queries

Pitfall 4: Using avg() Instead of sum() for Rates

When aggregating rates across replicas, use sum(), not avg(). If you have three pods each handling 100 req/s, the total throughput is 300 req/s — sum() gives you that. avg() gives you 100 req/s, which is misleading for capacity planning and SLO tracking.

# Wrong — this underreports total throughput
- record: service:http_requests:rate5m
  expr: avg(rate(http_requests_total[5m])) by (service)

# Correct — total request rate across all pods
- record: service:http_requests:rate5m
  expr: sum(rate(http_requests_total[5m])) by (service)

Pitfall 5: Not Monitoring the Rules Themselves

Prometheus exposes metrics about rule evaluation. Use them:

# Rules that are taking too long to evaluate
prometheus_rule_evaluation_duration_seconds{rule_group="http.aggregations"} > 1

# Rules that failed to evaluate
increase(prometheus_rule_evaluation_failures_total[5m]) > 0

# Last evaluation time per group
prometheus_rule_group_last_evaluation_timestamp_seconds

Add an alert for rule evaluation failures and slow-running rules. If a recording rule takes longer to evaluate than its evaluation interval, you have a serious problem.

---

Organizing Rules at Scale

When you have more than a handful of rule files, organization matters. Here's a structure that works well:

rules/
├── recording/
│   ├── kubernetes.yml        # K8s resource aggregations
│   ├── http.yml              # HTTP request aggregations
│   ├── database.yml          # DB query metrics
│   └── slo_base.yml          # SLO error rate pre-computations
├── alerts/
│   ├── infrastructure.yml
│   ├── slo.yml
│   └── application.yml
└── tests/
    ├── http_test.yml
    └── slo_test.yml

Keep recording rules and alerting rules in separate files. It makes auditing easier and reduces the risk of accidentally breaking alert rules when tuning recording rules.

---

Putting It All Together: A Complete Example

Here's a production-ready recording rules file for a microservices setup:

# rules/recording/http.yml
groups:
  - name: http.base.aggregations
    interval: 30s
    rules:
      # Base request rates by service
      - record: service:http_requests_total:rate5m
        expr: |
          sum(rate(http_requests_total[5m])) by (service, namespace)

      - record: service:http_requests_total:rate30m
        expr: |
          sum(rate(http_requests_total[30m])) by (service, namespace)

      # Error rates (5xx only for SLO)
      - record: service:http_server_errors_total:rate5m
        expr: |
          sum(
            rate(http_requests_total{status_code=~"5.."}[5m])
          ) by (service, namespace)

      - record: service:http_server_errors_total:rate30m
        expr: |
          sum(
            rate(http_requests_total{status_code=~"5.."}[30m])
          ) by (service, namespace)

      # Latency percentiles
      - record: service:http_request_duration_seconds:p50_5m
        expr: |
          histogram_quantile(0.50,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (service, namespace, le)
          )

      - record: service:http_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (service, namespace, le)
          )

  - name: http.derived.aggregations
    interval: 30s
    rules:
      # Error ratio — built from base rules above
      - record: service:http_error_ratio:rate5m
        expr: |
          service:http_server_errors_total:rate5m
            /
          service:http_requests_total:rate5m

      - record: service:http_error_ratio:rate30m
        expr: |
          service:http_server_errors_total:rate30m
            /
          service:http_requests_total:rate30m

      # Availability (inverse of error ratio)
      - record: service:http_availability:rate5m
        expr: |
          1 - service:http_error_ratio:rate5m

---

Final Thoughts

Recording rules are one of those things where a small investment in setup time pays massive dividends in production stability. A well-crafted set of recording rules will:

• Cut dashboard load times from tens of seconds to under a second
• Dramatically reduce Prometheus memory and CPU usage
• Make your alerting rules faster and more reliable
• Give you a clean, documented layer of aggregated metrics that the whole team can rely on

The pattern I'd encourage you to follow: whenever you're about to create a Grafana dashboard or an alerting rule that involves any complex aggregation over high-cardinality metrics, stop and write a recording rule first. Make the recording rule the single source of truth, and point everything at it.

Start with your highest-traffic metrics and your most-used dashboards. Run promtool check rules before every deploy. Write unit tests. Monitor the rule evaluation metrics. And follow the naming convention — your future self will thank you when you're debugging at 2am.

The raw data will always be there for ad-hoc queries. But your operational system should run on pre-computed, reliable, fast recording rules. That's the shift in thinking that takes your Prometheus setup from "kind of works" to genuinely production-grade.