Grafana Alerting: Contact Points, Silences, and Escalation

Grafana's unified alerting system (the one that replaced the old legacy alerting in Grafana 9+) is significantly more capable than it gets credit for. It handles everything from simple Slack notifications to complex escalation policies with on-call rotations — and it works with any Grafana data source, not just Prometheus. Here's how to configure it properly.

Architecture: How Unified Alerting Works

Grafana unified alerting has three main components:

1. Alert rules — PromQL (or any query) expressions with thresholds, evaluated on a schedule
2. Contact points — where notifications go (Slack, PagerDuty, email, webhooks, etc.)
3. Notification policies — routing rules that match alert labels to contact points

The routing is label-based, just like Alertmanager. This matters for how you design your alert rules — the labels on the alert determine where it gets routed.

Setting Up Contact Points

Contact points are configured under Alerting → Contact points. Each contact point can send to one or more integrations.

Slack Contact Point (via API)

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -d '{
    "name": "slack-critical",
    "type": "slack",
    "settings": {
      "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
      "recipient": "#alerts-critical",
      "username": "Grafana Alerts",
      "icon_emoji": ":fire:",
      "title": "{{ template \"slack.title\" . }}",
      "text": "{{ template \"slack.message\" . }}",
      "mentionChannel": "here"
    }
  }' \
  "https://grafana.internal/api/v1/provisioning/contact-points"

PagerDuty Contact Point

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -d '{
    "name": "pagerduty-oncall",
    "type": "pagerduty",
    "settings": {
      "integrationKey": "${PAGERDUTY_INTEGRATION_KEY}",
      "severity": "{{ if eq .CommonLabels.severity \"critical\" }}critical{{ else }}warning{{ end }}",
      "component": "{{ .CommonLabels.service }}",
      "group": "{{ .CommonLabels.namespace }}"
    }
  }' \
  "https://grafana.internal/api/v1/provisioning/contact-points"

Email Contact Point

For email, you need to configure the SMTP settings in grafana.ini first:

[smtp]
enabled = true
host = smtp.yourcompany.com:587
user = [email protected]
password = ${SMTP_PASSWORD}
from_address = [email protected]
from_name = Grafana Alerts

Then create the contact point:

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -d '{
    "name": "email-platform",
    "type": "email",
    "settings": {
      "addresses": "[email protected];[email protected]",
      "singleEmail": false
    }
  }' \
  "https://grafana.internal/api/v1/provisioning/contact-points"

Notification Policies

Notification policies define routing: "alerts matching these labels go to this contact point." They're hierarchical — you start with a root policy and add specific routes underneath.

# notification-policy.yaml (Grafana provisioning format)
apiVersion: 1
policies:
  - orgId: 1
    receiver: slack-general     # default catch-all
    group_by: [alertname, namespace, service]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      # Critical alerts page PagerDuty
      - receiver: pagerduty-oncall
        matchers:
          - name: severity
            value: critical
            matchType: =
        group_by: [alertname, service]
        group_wait: 10s
        group_interval: 5m
        repeat_interval: 1h
        continue: true    # also send to Slack

      # Critical alerts also go to Slack
      - receiver: slack-critical
        matchers:
          - name: severity
            value: critical
            matchType: =

      # Warning alerts go to Slack warnings channel
      - receiver: slack-warnings
        matchers:
          - name: severity
            value: warning
            matchType: =
        group_interval: 1h
        repeat_interval: 8h

      # Team-specific routing
      - receiver: slack-backend-team
        matchers:
          - name: team
            value: backend
            matchType: =
          - name: severity
            value: warning
            matchType: =

Apply via provisioning:

kubectl create configmap grafana-notification-policy \
  --from-file=notification-policy.yaml \
  --namespace monitoring \
  --dry-run=client -o yaml | kubectl apply -f -

Writing Alert Rules in Unified Alerting

Alert rules in Grafana unified alerting can be created via the UI or provisioned as code:

# alert-rules.yaml (Grafana provisioning format)
apiVersion: 1
groups:
  - orgId: 1
    name: myapp.rules
    folder: MyApp Alerts
    interval: 1m
    rules:
      - uid: myapp-error-rate-critical
        title: High Error Rate - Critical
        condition: C
        data:
          - refId: A
            datasourceUid: prometheus
            model:
              expr: |
                (
                  sum(rate(http_requests_total{status=~"5.."}[5m]))
                  / sum(rate(http_requests_total[5m]))
                ) * 100
              legendFormat: Error Rate %
              intervalMs: 15000
              maxDataPoints: 43200
          - refId: C
            datasourceUid: __expr__
            model:
              type: threshold
              conditions:
                - evaluator:
                    params: [5]
                    type: gt
                  operator:
                    type: and
                  query:
                    params: [A]
        for: 5m
        labels:
          severity: critical
          service: myapp
          team: backend
        annotations:
          summary: "Error rate is {{ $value | printf \"%.1f\" }}%"
          runbook_url: "https://runbooks.internal/myapp/errors"
          __dashboardUid__: myapp-overview
          __panelId__: "3"

The __dashboardUid__ and __panelId__ annotations link the alert back to the specific Grafana panel — when the alert fires, there's a "View panel" link in the notification. Small thing, huge time-saver during incidents.

Silences: Muting Without Deleting

Silences suppress notifications for matching alerts without touching the alert rules themselves. Use them during:

• Planned maintenance windows
• Known flapping that you'll fix tomorrow
• Post-incident cleanup when you know alerts will keep firing

# Create a silence via API
curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -d '{
    "matchers": [
      {
        "name": "service",
        "value": "myapp",
        "isRegex": false,
        "isEqual": true
      },
      {
        "name": "alertname",
        "value": "HighErrorRate",
        "isRegex": false,
        "isEqual": true
      }
    ],
    "startsAt": "2026-03-29T02:00:00Z",
    "endsAt": "2026-03-29T04:00:00Z",
    "createdBy": "[email protected]",
    "comment": "Maintenance window for database migration"
  }' \
  "https://grafana.internal/api/alertmanager/grafana/api/v2/silences"

Always set an endsAt. Open-ended silences are how alert fatigue starts — someone silences a noisy alert during an incident and never removes it.

Automation: Create Silences from Deployments

Add silence creation to your deployment pipeline:

#!/bin/bash
DEPLOY_DURATION_MINUTES=30
SERVICE=$1

# Create silence during deployment window
START=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
END=$(date -u -d "+${DEPLOY_DURATION_MINUTES} minutes" +"%Y-%m-%dT%H:%M:%SZ")

SILENCE_ID=$(curl -s -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -d "{
    \"matchers\": [{
      \"name\": \"service\",
      \"value\": \"$SERVICE\",
      \"isRegex\": false,
      \"isEqual\": true
    }],
    \"startsAt\": \"$START\",
    \"endsAt\": \"$END\",
    \"createdBy\": \"ci-pipeline\",
    \"comment\": \"Deployment silence for $SERVICE\"
  }" \
  "https://grafana.internal/api/alertmanager/grafana/api/v2/silences" | jq -r '.silenceID')

echo "Created silence: $SILENCE_ID"

# Run deployment here
deploy_service "$SERVICE"

# Optionally expire silence early if deployment finished quickly
curl -X DELETE \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  "https://grafana.internal/api/alertmanager/grafana/api/v2/silence/$SILENCE_ID"

Escalation with Grafana OnCall

For teams that need proper on-call rotation and escalation beyond basic PagerDuty integration, Grafana OnCall (formerly Amixr) integrates natively with Grafana:

# oncall-integration.yaml
# Route from Grafana Alerting to OnCall
integrations:
  - name: grafana-alerts
    type: grafana
    escalation_chains:
      - name: default-escalation
        steps:
          - type: notify_persons_next_each_time
            persons:
              - user: oncall-engineer-1
              - user: oncall-engineer-2
            important: false
          - type: wait
            duration: 5m
          - type: notify_person_next_each_time
            persons:
              - user: oncall-manager
            important: true
          - type: notify_whole_team_members
            duration: 15m

The escalation chain ensures: first on-call gets paged → if no acknowledgment in 5 minutes, escalate to manager → if still no ack in 15 minutes, page the whole team.

Notification Templates

Default notification messages are generic. Custom templates make them useful:

# templates/slack.tmpl
{{ define "slack.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
{{ end }}

{{ define "slack.message" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Severity:* {{ .Labels.severity }}
*Service:* {{ .Labels.service }}
*Namespace:* {{ .Labels.namespace }}
{{ if .Annotations.description }}*Details:* {{ .Annotations.description }}{{ end }}
{{ if .Annotations.runbook_url }}*Runbook:* <{{ .Annotations.runbook_url }}|Click here>{{ end }}
{{ if .Annotations.dashboard_url }}*Dashboard:* <{{ .Annotations.dashboard_url }}|View dashboard>{{ end }}
{{ end }}
{{ end }}

A good notification should contain enough information for the on-call engineer to start triaging without opening a second tool. The runbook link is not optional — it's what turns a 3am page from chaos into a procedure.

The infrastructure for alerts matters less than the discipline around them: every alert that fires must require human action, every human action must have a documented procedure, and every silence must have an end time. Build the tooling to support that discipline, and alerting becomes a useful system rather than background noise.