Runbook Automation with PagerDuty and Rundeck for Faster MTTR

Every minute your systems are down costs money, erodes customer trust, and burns out your on-call engineers. If your incident response process still involves someone frantically searching a wiki for a runbook at 2 AM, you're leaving serious MTTR improvements on the table. Let's fix that.

In this article, I'll walk through how to integrate PagerDuty with Rundeck to automate runbook execution, turning your manual incident response playbooks into automated remediations that fire before a human even rubs the sleep from their eyes.

Why Runbook Automation Matters

The problem with traditional runbooks isn't that they're wrong — it's that they're passive. They sit in Confluence or a Google Doc waiting for someone to read them, interpret them, and execute them under pressure. That's a recipe for mistakes.

Runbook automation flips this model. When PagerDuty detects an incident, it can trigger Rundeck to automatically execute the exact remediation steps defined in your runbook. No human intervention needed for the common cases. Humans stay in the loop for the edge cases that actually require judgment.

Here's what this looks like in practice:

• High CPU alert → Rundeck auto-restarts the problematic service
• Disk space critical → Rundeck clears log files and old temp directories
• Pod crash loop → Rundeck triggers a rolling restart with diagnostic collection

Done right, 40-60% of common incidents never require human escalation at all.

The Architecture

PagerDuty Alert → Webhook → Rundeck Job → Remediation
                                ↓
                         Execution Log + Output
                                ↓
                    PagerDuty Incident Notes/Resolution

PagerDuty acts as the orchestrator — it receives alerts, applies routing rules, and decides what needs to happen. Rundeck acts as the execution engine — it runs your automation jobs securely with proper access controls, audit logs, and error handling.

Setting Up Rundeck for Secure Job Execution

First, let's get Rundeck configured. I strongly recommend running Rundeck with role-based access control from day one. Here's a minimal but solid aclpolicy for your automation service account:

# pagerduty-automation.aclpolicy
description: PagerDuty automation service account policy
context:
  project: '.*'
for:
  job:
    - allow: [read, run]
      match:
        group: 'incident-response/.*'
  node:
    - allow: [read, run]
  resource:
    - kind: job
      allow: [read]
by:
  username: pagerduty-svc

This restricts your PagerDuty service account to only running jobs in the incident-response group — not creating, deleting, or modifying them. Defense in depth starts with least-privilege service accounts.

Generate an API token for this account and store it in your secrets manager. Never hardcode it.

Building Your First Incident Response Job

Here's a practical Rundeck job definition for handling a common scenario — a service that needs a restart with pre/post validation:

# job-restart-service.yaml
- defaultTab: nodes
  description: 'Restart a systemd service with health validation'
  executionEnabled: true
  id: restart-service-job
  loglevel: INFO
  name: Restart Service with Validation
  group: incident-response
  options:
    - name: service_name
      required: true
      description: 'Name of the systemd service to restart'
    - name: health_check_url
      required: true
      description: 'URL to validate service health post-restart'
    - name: max_wait_seconds
      value: '60'
      description: 'Maximum seconds to wait for health check'
  sequence:
    keepgoing: false
    strategy: node-first
    commands:
      - script: |
          #!/bin/bash
          set -euo pipefail
          
          SERVICE="@option.service_name@"
          HEALTH_URL="@option.health_check_url@"
          MAX_WAIT="@option.max_wait_seconds@"
          
          echo "=== Pre-restart status check ==="
          systemctl status "$SERVICE" || true
          
          echo "=== Collecting diagnostics before restart ==="
          journalctl -u "$SERVICE" --since "5 minutes ago" --no-pager
          
          echo "=== Restarting service: $SERVICE ==="
          systemctl restart "$SERVICE"
          
          echo "=== Waiting for health check ==="
          elapsed=0
          until curl -sf "$HEALTH_URL" > /dev/null 2>&1; do
            if [ "$elapsed" -ge "$MAX_WAIT" ]; then
              echo "ERROR: Health check failed after ${MAX_WAIT}s"
              exit 1
            fi
            sleep 5
            elapsed=$((elapsed + 5))
            echo "Waiting... ${elapsed}s elapsed"
          done
          
          echo "=== Service is healthy! ==="
          systemctl status "$SERVICE"

Notice the set -euo pipefail at the top. This is non-negotiable for automation scripts — you want the job to fail loudly if something unexpected happens, not silently continue.

Connecting PagerDuty to Rundeck

PagerDuty's webhook v3 support makes this straightforward. Create a webhook in PagerDuty that points to a lightweight middleware service (I use a small FastAPI app), which translates PagerDuty events into Rundeck API calls.

Here's the core of that middleware:

# pd_rundeck_bridge.py
import httpx
import os
from fastapi import FastAPI, Request, HTTPException
from pydantic import BaseModel

app = FastAPI()

RUNDECK_URL = os.environ["RUNDECK_URL"]
RUNDECK_TOKEN = os.environ["RUNDECK_TOKEN"]  # From secrets manager
PD_WEBHOOK_SECRET = os.environ["PD_WEBHOOK_SECRET"]

JOB_ROUTING = {
    "high_cpu_service_api": {
        "job_id": "restart-service-job",
        "options": {
            "service_name": "api-service",
            "health_check_url": "http://api-service:8080/health"
        }
    },
    "disk_space_critical_app01": {
        "job_id": "cleanup-disk-job",
        "options": {
            "target_dirs": "/var/log,/tmp",
            "threshold_gb": "5"
        }
    }
}

@app.post("/webhook/pagerduty")
async def handle_pagerduty_webhook(request: Request):
    # Always validate the webhook signature first
    signature = request.headers.get("X-PagerDuty-Signature")
    body = await request.body()
    
    if not validate_signature(body, signature, PD_WEBHOOK_SECRET):
        raise HTTPException(status_code=401, detail="Invalid signature")
    
    payload = await request.json()
    
    for event in payload.get("messages", []):
        if event.get("event") == "incident.triggered":
            incident = event["incident"]
            alert_key = incident.get("alert_key", "")
            
            if alert_key in JOB_ROUTING:
                await trigger_rundeck_job(
                    incident["id"],
                    JOB_ROUTING[alert_key]
                )
    
    return {"status": "processed"}

async def trigger_rundeck_job(incident_id: str, job_config: dict):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{RUNDECK_URL}/api/41/job/{job_config['job_id']}/run",
            headers={
                "X-Rundeck-Auth-Token": RUNDECK_TOKEN,
                "Content-Type": "application/json"
            },
            json={
                "options": job_config["options"],
                "runAtTime": None
            }
        )
        response.raise_for_status()
        
        execution_id = response.json()["id"]
        print(f"Triggered job execution {execution_id} for incident {incident_id}")
        return execution_id

The validate_signature function is critical — don't skip it. You're essentially exposing an endpoint that can run code on your infrastructure. Validate every single request.

Closing the Loop: Writing Results Back to PagerDuty

The real power comes from writing execution results back to the PagerDuty incident. Your on-call engineer should open a triggered incident and immediately see "Automated remediation attempted — service restarted successfully, health checks passing" or "Automated remediation failed — escalating."

Add this to your job sequence as a final step:

#!/bin/bash
# Post execution results back to PagerDuty
INCIDENT_ID="@option.pd_incident_id@"
PD_TOKEN="$PD_API_TOKEN"  # Injected via Rundeck Key Storage

STATUS="resolved"
NOTE="Automated remediation completed successfully. 
Service @option.service_name@ restarted at $(date -u +%Y-%m-%dT%H:%M:%SZ).
Health check passed after ${elapsed}s."

# Add a note to the incident
curl -s -X POST \
  "https://api.pagerduty.com/incidents/${INCIDENT_ID}/notes" \
  -H "Authorization: Token token=${PD_TOKEN}" \
  -H "Content-Type: application/json" \
  -H "From: [email protected]" \
  -d "{
    \"note\": {
      \"content\": \"${NOTE}\"
    }
  }"

For auto-resolution, be careful. I'd recommend auto-resolving only for incidents where your confidence in the fix is very high (>95% success rate in your metrics). For everything else, add a note and let the on-call engineer make the resolution call.

Security Considerations You Can't Skip

Running automation that touches production infrastructure deserves serious security attention:

Secrets Management: Store your Rundeck API token and PagerDuty credentials in HashiCorp Vault or AWS Secrets Manager. Rundeck has native Key Storage integration with both. Never put credentials in job definitions.

Audit Everything: Rundeck's execution history is your audit trail. Ensure it's retained for at least 90 days and shipped to your SIEM. Every automated remediation action needs to be attributable and reviewable.

Network Segmentation: Your webhook receiver and Rundeck should not be publicly accessible. Run them inside your VPC/private network and limit PagerDuty webhook delivery to PagerDuty's published IP ranges.

Dry Run Mode: Add a dry_run option to all your jobs. When developing new automation, trigger it in dry-run mode first so you can validate the logic without risking side effects.

Measuring Your Improvement

You can't manage what you don't measure. Track these metrics before and after implementing runbook automation:

• MTTR by alert type: Are automated remediations resolving incidents faster than manual ones?
• Automation success rate: What percentage of triggered jobs result in successful remediation? Target >90%.
• Escalation rate: What percentage of automatically triggered jobs still require human escalation?
• False positive automation: How often does automation fire for an incident that didn't need intervention?

PagerDuty Analytics and Rundeck's built-in reporting give you most of this data out of the box. Build a dashboard in Grafana or Datadog to make trends visible.

Start Small and Iterate

If you're starting from scratch, don't try to automate everything at once. Pick the three incidents your team responds to most frequently, and automate those first. Review the execution logs after every automated run for the first 30 days. Treat your runbook automation like code — it needs testing, review, and iteration.

The teams that do this well aren't necessarily the ones with the most sophisticated tooling. They're the ones that consistently review their automation, measure its effectiveness, and keep improving it. Build that habit and your MTTR will thank you.