Automating Incident Severity Classification With PagerDuty Event Rules and Custom Fields

Stop manually triaging every alert. Here's your quick-reference guide to letting PagerDuty do the heavy lifting.

---

The Core Concept

PagerDuty Event Rules evaluate incoming alert payloads and automatically route, suppress, or classify incidents before anyone gets paged. Pair that with Custom Fields and you have a dynamic severity tagging system that works while you sleep.

---

Event Rule Anatomy

Every rule follows this pattern:

IF [condition on payload field] THEN [action]

Rules live at two levels:

• Global Event Rules — account-wide, evaluated first
• Service Event Rules — scoped to a specific service

---

Payload Structure You're Working With

PagerDuty expects events in this format (PD-CEF):

{
  "routing_key": "YOUR_INTEGRATION_KEY",
  "event_action": "trigger",
  "payload": {
    "summary": "High CPU on prod-web-01",
    "severity": "error",
    "source": "datadog",
    "custom_details": {
      "env": "production",
      "cpu_percent": 95,
      "service_tier": "critical"
    }
  }
}

The custom_details object is your best friend — stuff everything useful in here.

---

Setting Up Event Rules via API

Create a Global Event Rule

curl --request POST \
  --url https://api.pagerduty.com/event_rules \
  --header 'Accept: application/vnd.pagerduty+json;version=2' \
  --header 'Authorization: Token token=YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "rule": {
      "conditions": {
        "operator": "and",
        "subconditions": [
          {
            "operator": "contains",
            "parameters": {
              "value": "production",
              "path": "payload.custom_details.env"
            }
          },
          {
            "operator": "greater_than",
            "parameters": {
              "value": "90",
              "path": "payload.custom_details.cpu_percent"
            }
          }
        ]
      },
      "actions": {
        "severity": [{ "value": "critical" }],
        "priority": [{ "value": "P1XXXXX" }]
      }
    }
  }'

---

Severity Mapping Cheat Sheet

Condition	Mapped Severity	PagerDuty Priority
env=production + metric > 90	critical	P1
env=production + metric 70–90	error	P2
env=staging + any metric	warning	P3
env=dev	info	Suppressed

---

Custom Fields: Tag Incidents Automatically

Custom Fields appear on the incident detail page and can be populated by event rules. Define them first:

curl --request POST \
  --url https://api.pagerduty.com/incidents/custom_fields \
  --header 'Authorization: Token token=YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "custom_field": {
      "name": "service_tier",
      "display_name": "Service Tier",
      "data_type": "string",
      "field_type": "single_value"
    }
  }'

Then reference the field ID in your event rule action:

"actions": {
  "extractions": [
    {
      "target": "incident.custom_fields.service_tier",
      "source": "payload.custom_details.service_tier"
    }
  ]
}

---

Python: Send a Pre-Classified Event

import requests
import json

def trigger_classified_event(host, env, cpu_percent, service_tier):
    severity_map = {
        ("production", lambda x: x > 90): "critical",
        ("production", lambda x: 70 < x <= 90): "error",
        ("staging", lambda x: True): "warning",
    }

    severity = "info"
    for (e, condition), sev in severity_map.items():
        if env == e and condition(cpu_percent):
            severity = sev
            break

    payload = {
        "routing_key": "YOUR_INTEGRATION_KEY",
        "event_action": "trigger",
        "payload": {
            "summary": f"High CPU on {host}",
            "severity": severity,
            "source": host,
            "custom_details": {
                "env": env,
                "cpu_percent": cpu_percent,
                "service_tier": service_tier
            }
        }
    }

    response = requests.post(
        "https://events.pagerduty.com/v2/enqueue",
        headers={"Content-Type": "application/json"},
        data=json.dumps(payload)
    )
    return response.json()

---

Rule Evaluation Order — Don't Get Burned

Rules are evaluated top to bottom and stop at first match by default. Common mistakes:

❌ Wrong order:
  Rule 1: env contains "production" → severity: warning
  Rule 2: env contains "production" AND cpu > 90 → severity: critical
  # Rule 2 never fires for high CPU!

✅ Correct order:
  Rule 1: env contains "production" AND cpu > 90 → severity: critical
  Rule 2: env contains "production" → severity: warning

Always put more specific rules first.

---

Suppression Rule for Dev/Test Noise

{
  "conditions": {
    "operator": "or",
    "subconditions": [
      {
        "operator": "contains",
        "parameters": {
          "value": "dev",
          "path": "payload.custom_details.env"
        }
      },
      {
        "operator": "contains",
        "parameters": {
          "value": "test",
          "path": "payload.custom_details.env"
        }
      }
    ]
  },
  "actions": {
    "suppress": [{ "value": true }]
  }
}

---

Quick Wins Checklist

• Add env and service_tier to every alert payload
• Create suppression rules for dev/test environments
• Map severity → priority using event rules (not manual triage)
• Define Custom Fields for service_tier, runbook_url, team
• Put specific rules before general rules
• Test rules with PagerDuty's Send Test Event before going live

---

Key API Endpoints

Action	Endpoint
List event rules	GET /event_rules
Create event rule	POST /event_rules
Update rule order	PUT /event_rules/{id}
Create custom field	POST /incidents/custom_fields
Send test event	POST https://events.pagerduty.com/v2/enqueue

---

The goal is simple: by the time a human looks at an incident, the severity should already be classified correctly. These rules pay off after the first week.