Prometheus Alerting Rules: From Noisy to Actionable

The biggest alerting problem I see at most companies isn't that they're missing alerts — it's that they have too many alerts firing too often for things that don't actually need human intervention right now. Alert fatigue is real, and it leads directly to missed incidents when a real problem gets buried under a flood of noise. Here's how to write rules that are actually actionable.

The Alert Anatomy

Before writing rules, align on what makes an alert good:

1. It fires on symptoms, not causes — alert on "users are experiencing errors," not "CPU is above 80%"
2. It's actionable — someone waking up at 3am should know what to do
3. It has appropriate urgency — not everything needs to wake someone up
4. It has a runbook link — the alert itself tells you where to start

A good alerting rule structure in Prometheus:

# alert-rules.yaml
groups:
  - name: myapp.rules
    interval: 1m   # how often to evaluate this group
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > 0.05
        for: 5m
        labels:
          severity: critical
          team: backend
          service: myapp
        annotations:
          summary: "High HTTP error rate on {{ $labels.service }}"
          description: >
            Error rate is {{ $value | humanizePercentage }} over the last 5 minutes.
            This exceeds the 5% threshold.
          runbook_url: "https://runbooks.internal/myapp/high-error-rate"
          dashboard_url: "https://grafana.internal/d/myapp/overview"

The for duration is one of the most important knobs. Without it, a 1-second spike fires the alert. With for: 5m, the condition must be true for 5 consecutive minutes before alerting. Almost every alert should have a non-zero for duration.

Severity Levels That Actually Mean Something

Establish severity semantics and stick to them. Here's what works in practice:

Severity	Meaning	Response	Examples
critical	Service down or severely degraded for users right now	Page on-call immediately	Error rate >5%, no healthy pods
warning	Degraded but not broken, or trending toward a problem	Ticket, fix next business day	Disk at 80%, latency P99 elevated
info	Informational, no action required	Log, maybe a Slack notification	Deployment started, cert expiring in 60 days

The discipline is in keeping critical genuinely critical. If you have 20 critical alerts and half of them don't require waking anyone up, you've diluted the signal.

Common Alerting Patterns with Real PromQL

Error Rate Alerts

- alert: HTTPErrorRateCritical
  expr: |
    (
      sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
      /
      sum by (service) (rate(http_requests_total[5m]))
    ) * 100 > 5
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.service }} HTTP error rate is {{ $value | printf \"%.1f\" }}%"

- alert: HTTPErrorRateWarning
  expr: |
    (
      sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
      /
      sum by (service) (rate(http_requests_total[5m]))
    ) * 100 > 1
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.service }} HTTP error rate elevated at {{ $value | printf \"%.1f\" }}%"

Note the two rules: critical fires faster (5m) at a higher threshold (5%), warning fires slower (10m) at a lower threshold (1%). This layering is intentional.

Latency Alerts Using Histograms

- alert: HighLatencyP99
  expr: |
    histogram_quantile(0.99,
      sum by (service, le) (
        rate(http_request_duration_seconds_bucket[5m])
      )
    ) > 2.0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "P99 latency for {{ $labels.service }} is {{ $value | humanizeDuration }}"
    description: "99th percentile latency exceeds 2s SLO threshold"

Always use histograms (not summaries) for percentile alerting. Summaries can't be aggregated across instances; histograms can.

Availability / Pod Health

- alert: DeploymentReplicasMismatch
  expr: |
    kube_deployment_spec_replicas
      != kube_deployment_status_available_replicas
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has unavailable replicas"
    description: >
      Desired: {{ $value }} replicas. Check pod logs and events.

- alert: PodCrashLooping
  expr: |
    increase(kube_pod_container_status_restarts_total[1h]) > 5
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"

The for: 0m on crash looping is intentional — this needs immediate attention, don't wait 5 minutes to confirm it.

Resource Saturation

- alert: NodeMemoryPressure
  expr: |
    (
      node_memory_MemAvailable_bytes
      / node_memory_MemTotal_bytes
    ) * 100 < 10
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Node {{ $labels.instance }} has < 10% memory available"

- alert: DiskSpaceCritical
  expr: |
    (
      node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
      / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}
    ) * 100 < 5
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.instance }} disk {{ $labels.mountpoint }} is {{ $value | printf \"%.1f\" }}% full"

Absence Alerts (When Silence Is the Problem)

- alert: MetricsMissing
  expr: |
    absent(up{job="myapp"})
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "No myapp instances are being scraped"
    description: "Prometheus cannot reach any myapp targets — the service may be down or misconfigured"

absent() is underused. It fires when a metric simply doesn't exist in Prometheus — catching cases where a service is completely gone, not just degraded.

Recording Rules for Performance

Complex PromQL in alerting rules runs at every evaluation. For expensive queries, precompute with recording rules:

groups:
  - name: myapp.precompute
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job, status) (rate(http_requests_total[5m]))

      - record: job:http_error_rate:ratio5m
        expr: |
          sum by (job) (job:http_requests:rate5m{status=~"5.."})
          /
          sum by (job) (job:http_requests:rate5m)

  - name: myapp.alerts
    rules:
      - alert: HighErrorRate
        expr: job:http_error_rate:ratio5m{job="myapp"} > 0.05
        for: 5m
        labels:
          severity: critical

Recording rules reduce query load significantly when many alerts reference the same underlying computation.

Alert Routing with Alertmanager

Writing good rules is half the work — routing them correctly is the other half:

# alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-general'
  routes:
    - matchers:
        - severity = critical
      receiver: 'pagerduty-oncall'
      continue: true   # also send to Slack

    - matchers:
        - severity = critical
        - team = backend
      receiver: 'pagerduty-backend'

    - matchers:
        - severity = warning
      receiver: 'slack-warnings'
      group_interval: 1h   # batch warnings less aggressively

receivers:
  - name: pagerduty-oncall
    pagerduty_configs:
      - service_key: "${PAGERDUTY_KEY}"

  - name: slack-warnings
    slack_configs:
      - api_url: "${SLACK_WEBHOOK}"
        channel: '#alerts-warning'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

The group_wait and group_interval settings matter a lot for reducing noise. group_wait: 30s means Alertmanager waits 30 seconds before sending the first notification for a new group, in case more alerts arrive and can be batched together.

Testing Your Alert Rules

Don't deploy alerting rules without testing them:

# tests/alert_tests.yaml
rule_files:
  - ../alert-rules.yaml

tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{status="500", service="myapp"}'
        values: '0+10x30'   # 10 req/min for 30 minutes
      - series: 'http_requests_total{status="200", service="myapp"}'
        values: '0+90x30'   # 90 req/min (so 10% error rate)

    alert_rule_test:
      - eval_time: 5m
        alertname: HTTPErrorRateCritical
        exp_alerts:
          - exp_labels:
              severity: critical
              service: myapp
            exp_annotations:
              summary: "myapp HTTP error rate is 10.0%"

promtool test rules tests/alert_tests.yaml

promtool ships with Prometheus and validates that your rules fire when they should and don't fire when they shouldn't. Add this to your CI pipeline — rule changes without tests are how alert gaps happen in production.

The key insight that changes how you write alerts: you're not monitoring your infrastructure, you're monitoring the user experience. Start from "what does a user notice when something goes wrong?" and work backwards to the metrics. Everything else — CPU, memory, disk — is supplementary context, not primary signals.