Apache mod_status Real-Time Server Monitoring with Grafana and Prometheus Integration

Monitoring Apache without proper observability tooling is like flying blind. You might catch a problem after your users start complaining, but by then, you've already lost the battle. In this guide, I'm going to walk you through building a production-grade monitoring stack for Apache using mod_status, Prometheus, and Grafana — the kind of setup I've deployed on workloads handling millions of requests per day.

This isn't a "hello world" tutorial. We're going to get into the weeds: configuration details, scraping strategies, alerting rules, and the common mistakes that will burn you in production if you're not careful.

---

What Is mod_status and Why Should You Care?

mod_status is an Apache module that exposes real-time server statistics through a dedicated endpoint. It gives you:

• Total accesses and bytes served
• Current requests being processed
• Worker states (idle, busy, reading, writing, etc.)
• CPU load and uptime
• Request-per-second and bytes-per-second rates
• Individual worker scoreboard

Out of the box, this data is presented as an HTML page — useful for a quick look, completely useless for automated alerting or trend analysis. That's where Prometheus and Grafana come in.

The architecture we're building looks like this:

Apache (mod_status) → apache_exporter (Prometheus Exporter) → Prometheus → Grafana

Simple. Effective. Battle-tested.

---

Part 1: Configuring mod_status in Apache

Enabling the Module

First, make sure the module is loaded. On Debian/Ubuntu:

sudo a2enmod status
sudo systemctl reload apache2

On RHEL/CentOS/Amazon Linux, it's typically compiled in by default. Verify:

httpd -M | grep status
# status_module (shared)

Basic mod_status Configuration

Here's a minimal but correct mod_status configuration. Add this to your Apache config — either in httpd.conf or a dedicated include file:

<Location "/server-status">
    SetHandler server-status
    
    # Restrict access — CRITICAL in production
    Require local
    Require ip 10.0.0.0/8
    Require ip 172.16.0.0/12
    Require ip 192.168.0.0/16
</Location>

# Enable extended status — required for Prometheus metrics
ExtendedStatus On

# For virtual host stats (optional but useful)
<IfModule mod_status.c>
    SrvRoot "/etc/apache2"
</IfModule>

ExtendedStatus On is non-negotiable. Without it, you get a stripped-down scoreboard that's missing CPU usage, per-request data, and other metrics Prometheus needs. The performance overhead is negligible — we're talking microseconds per request.

Machine-Readable Output

mod_status supports a ?auto query parameter that returns plain-text output instead of HTML. This is what the Prometheus exporter will scrape:

curl http://localhost/server-status?auto

Expected output:

ServerVersion: Apache/2.4.54 (Ubuntu)
ServerMPM: event
Server Built: 2022-03-15T18:10:40
CurrentTime: Thursday, 01-Jan-2023 12:00:00 UTC
RestartTime: Thursday, 01-Jan-2023 08:00:00 UTC
ParentServerConfigGeneration: 1
ParentServerMPMGeneration: 0
ServerUptimeSeconds: 14400
ServerUptime: 4 hours
Load1: 0.42
Load5: 0.38
Load15: 0.35
Total Accesses: 145823
Total kBytes: 892341
Total Duration: 4829201
CPUUser: 12.4
CPUSystem: 4.2
CPUChildrenUser: 0
CPUChildrenSystem: 0
CPULoad: .0456
Uptime: 14400
ReqPerSec: 10.1266
BytesPerSec: 63398.3
BytesPerReq: 6260.1
DurationPerReq: 33.1
BusyWorkers: 8
IdleWorkers: 92
Scoreboard: ___________W_________...

If you see this output, you're in good shape. If you see an HTML page or a 403, check your access restrictions.

Virtual Host Considerations

If you're running multiple virtual hosts, make sure your mod_status location is in the right context. I typically put it in the main server config or a dedicated monitoring virtual host:

<VirtualHost *:8080>
    ServerName monitoring.internal
    
    <Location "/server-status">
        SetHandler server-status
        Require ip 10.0.0.0/8
    </Location>
    
    ExtendedStatus On
</VirtualHost>

Using a dedicated port for internal monitoring traffic is a pattern I strongly recommend. It keeps monitoring traffic off your main listeners and makes firewall rules cleaner.

---

Part 2: Deploying the Apache Prometheus Exporter

The Apache Exporter by Lusitaniae is the standard tool for bridging mod_status and Prometheus. It scrapes the ?auto endpoint and exposes metrics in Prometheus exposition format.

Installation

Binary install (recommended for production):

# Download the latest release
APACHE_EXPORTER_VERSION="1.0.3"
wget https://github.com/Lusitaniae/apache_exporter/releases/download/v${APACHE_EXPORTER_VERSION}/apache_exporter-${APACHE_EXPORTER_VERSION}.linux-amd64.tar.gz

tar xzf apache_exporter-${APACHE_EXPORTER_VERSION}.linux-amd64.tar.gz
sudo mv apache_exporter-${APACHE_EXPORTER_VERSION}.linux-amd64/apache_exporter /usr/local/bin/
sudo chmod +x /usr/local/bin/apache_exporter

Docker:

# docker-compose.yml snippet
services:
  apache_exporter:
    image: lusotycoon/apache-exporter:latest
    ports:
      - "9117:9117"
    command:
      - '--scrape_uri=http://apache/server-status?auto'
    networks:
      - monitoring

Systemd Service Unit

For bare-metal or VM deployments, run it as a systemd service:

# /etc/systemd/system/apache_exporter.service
[Unit]
Description=Apache Exporter for Prometheus
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/apache_exporter \
    --scrape_uri=http://localhost/server-status?auto \
    --telemetry.address=:9117 \
    --telemetry.endpoint=/metrics \
    --log.level=warn

SyslogIdentifier=apache_exporter
Restart=always
RestartSec=5s

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/apache_exporter

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now apache_exporter
sudo systemctl status apache_exporter

Verify it's working:

curl http://localhost:9117/metrics | grep apache

You should see metrics like:

# HELP apache_accesses_total Current total apache accesses (*)
# TYPE apache_accesses_total counter
apache_accesses_total 145823

# HELP apache_workers Apache worker statuses
# TYPE apache_workers gauge
apache_workers{state="busy"} 8
apache_workers{state="idle"} 92

---

Part 3: Prometheus Configuration

Scrape Configuration

Add the Apache exporter as a scrape target in your prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    environment: 'production'
    region: 'us-east-1'

scrape_configs:
  - job_name: 'apache'
    static_configs:
      - targets:
          - 'web-01.internal:9117'
          - 'web-02.internal:9117'
          - 'web-03.internal:9117'
        labels:
          cluster: 'web-frontend'
    
    # Tune scrape settings
    scrape_interval: 10s
    scrape_timeout: 8s
    metrics_path: /metrics
    
    # Relabeling for cleaner metric labels
    relabel_configs:
      - source_labels: [__address__]
        regex: '([^:]+):\d+'
        target_label: instance
        replacement: '${1}'

For dynamic infrastructure (auto-scaling groups on AWS, GCP managed instance groups), use service discovery instead of static configs:

scrape_configs:
  - job_name: 'apache-ec2'
    ec2_sd_configs:
      - region: us-east-1
        port: 9117
        filters:
          - name: "tag:Role"
            values: ["web"]
          - name: "instance-state-name"
            values: ["running"]
    
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_availability_zone]
        target_label: az
      - source_labels: [__meta_ec2_tag_Environment]
        target_label: environment

This is a game-changer for elastic workloads. No more manually updating scrape configs when instances spin up or down.

Recording Rules

Recording rules pre-compute expensive queries and make your dashboards faster. Define them in a separate rules file:

# /etc/prometheus/rules/apache.yml
groups:
  - name: apache_recording_rules
    interval: 30s
    rules:
      # Request rate over 5 minutes
      - record: job:apache_accesses_total:rate5m
        expr: rate(apache_accesses_total[5m])
      
      # Worker utilization percentage
      - record: job:apache_worker_utilization:ratio
        expr: |
          apache_workers{state="busy"} 
          / 
          (apache_workers{state="busy"} + apache_workers{state="idle"})
      
      # Average response time (requires custom logging — see later section)
      - record: job:apache_request_duration_seconds:avg5m
        expr: |
          rate(apache_duration_ms_total[5m]) / 1000
          /
          rate(apache_accesses_total[5m])

Reference this in your main prometheus.yml:

rule_files:
  - "/etc/prometheus/rules/apache.yml"

---

Part 4: Alerting Rules

This is where monitoring actually earns its keep. Here are the alerting rules I use in production:

# /etc/prometheus/rules/apache_alerts.yml
groups:
  - name: apache_alerts
    rules:
      # High worker utilization — starting to get saturated
      - alert: ApacheHighWorkerUtilization
        expr: job:apache_worker_utilization:ratio > 0.80
        for: 5m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Apache worker pool nearing saturation on {{ $labels.instance }}"
          description: |
            Worker utilization is {{ $value | humanizePercentage }} on {{ $labels.instance }}.
            Busy workers: {{ query "apache_workers{state='busy', instance='" }}{{ $labels.instance }}{{ "'}" | first | value }}
            Consider scaling horizontally or tuning MaxRequestWorkers.
          runbook_url: "https://wiki.internal/runbooks/apache-saturation"

      # Critical worker saturation
      - alert: ApacheWorkerPoolExhausted
        expr: apache_workers{state="idle"} < 5
        for: 2m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "Apache worker pool nearly exhausted on {{ $labels.instance }}"
          description: "Only {{ $value }} idle workers remaining. New connections will be queued or refused."

      # Server not responding (exporter can't reach mod_status)
      - alert: ApacheDown
        expr: up{job="apache"} == 0
        for: 1m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "Apache exporter is down on {{ $labels.instance }}"
          description: "Cannot scrape Apache metrics. Apache may be down or mod_status unreachable."

      # Sudden drop in request rate (could indicate a routing or load balancer issue)
      - alert: ApacheRequestRateDrop
        expr: |
          (
            job:apache_accesses_total:rate5m 
            < 
            job:apache_accesses_total:rate5m offset 10m * 0.3
          )
          and job:apache_accesses_total:rate5m offset 10m > 10
        for: 3m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Significant drop in Apache request rate on {{ $labels.instance }}"
          description: "Request rate dropped by more than 70% compared to 10 minutes ago."

      # High CPU load on the Apache host
      - alert: ApacheHighCPULoad
        expr: apache_cpuload > 0.8
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Apache reporting high CPU load on {{ $labels.instance }}"
          description: "CPU load is {{ $value | humanizePercentage }}. Check for runaway processes or request spikes."

The ApacheRequestRateDrop alert is one I've found particularly valuable in production. Load balancer misconfigurations and upstream DNS issues often manifest as a sudden traffic drop on individual nodes rather than errors — this catches it early.

---

Part 5: Grafana Dashboard Configuration

Data Source Setup

Assuming Prometheus is already added as a data source in Grafana, here's how I structure the Apache dashboard.

Key Panels to Build

Panel 1: Request Rate

# PromQL
sum by (instance) (rate(apache_accesses_total[5m]))

Visualization: Time series, unit: reqps

Panel 2: Worker Utilization Gauge

# PromQL  
apache_workers{state="busy"} / (apache_workers{state="busy"} + apache_workers{state="idle"}) * 100

Visualization: Gauge, thresholds at 70 (yellow), 85 (red)

Panel 3: Worker Scoreboard Breakdown

# Busy workers
apache_workers{state="busy"}

# Idle workers  
apache_workers{state="idle"}

Visualization: Bar gauge or stat panels side by side

Panel 4: Bytes Transferred

# PromQL
rate(apache_sent_kilobytes_total[5m]) * 1024

Visualization: Time series, unit: bytes/sec

Panel 5: Connections by State

# PromQL — requires mod_status scoreboard parsing
apache_scoreboard{state=~"sending|reading|waiting|starting|closing|logging|finishing|idle_cleanup"}

Dashboard JSON Snippet

Here's a practical Grafana panel definition you can import:

{
  "title": "Apache Worker Utilization",
  "type": "gauge",
  "datasource": "Prometheus",
  "targets": [
    {
      "expr": "apache_workers{state=\"busy\",instance=\"$instance\"} / (apache_workers{state=\"busy\",instance=\"$instance\"} + apache_workers{state=\"idle\",instance=\"$instance\"}) * 100",
      "legendFormat": "Worker Utilization %",
      "refId": "A"
    }
  ],
  "options": {
    "reduceOptions": {
      "calcs": ["lastNotNull"]
    },
    "orientation": "auto",
    "showThresholdLabels": false,
    "showThresholdMarkers": true
  },
  "fieldConfig": {
    "defaults": {
      "min": 0,
      "max": 100,
      "unit": "percent",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {"color": "green", "value": null},
          {"color": "yellow", "value": 70},
          {"color": "red", "value": 85}
        ]
      }
    }
  }
}

For a complete dashboard, I recommend importing dashboard ID 3894 from Grafana.com as a starting point, then customizing it for your environment. It covers the core metrics and saves you significant setup time.

---

Part 6: Enhancing Apache Logs for Better Metrics

mod_status gives you aggregate stats, but for per-request latency and status code distribution, you need custom log formats fed into a log-based metrics system.

Custom Log Format with Timing

# Add timing to your log format
LogFormat "%h %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" combined_timing
CustomLog /var/log/apache2/access.log combined_timing

%D gives you request duration in microseconds — essential for latency analysis.

Integrating with Prometheus via mtail or promtail

For log-based metrics, use mtail to parse Apache logs and expose them as Prometheus metrics:

# /etc/mtail/apache.mtail

counter apache_requests_total by status, method
histogram apache_request_duration_seconds by status buckets 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10

/^(?P<host>\S+) \S+ \S+ \[.*\] "(?P<method>\S+) \S+ \S+" (?P<status>\d+) \d+ (?P<duration>\d+)/ {
    apache_requests_total[status][method]++
    apache_request_duration_seconds[status] = float(duration) / 1000000
}

This gives you status code distribution and proper latency histograms — two things mod_status alone can't provide.

---

Part 7: Common Pitfalls and How to Avoid Them

Pitfall 1: Leaving mod_status Open to the Internet

I've seen this in security audits more times than I care to count. The default Apache install on some distributions leaves /server-status accessible without restrictions. Always use Require directives and consider putting it on a non-public port.

# Wrong — don't do this
<Location "/server-status">
    SetHandler server-status
</Location>

# Right
<Location "/server-status">
    SetHandler server-status
    Require ip 127.0.0.1
    Require ip ::1
    Require ip 10.0.0.0/8
</Location>

Pitfall 2: Forgetting ExtendedStatus

Without ExtendedStatus On, many metrics return zero or are missing entirely. The Prometheus exporter will still work, but you'll get incomplete data. Always enable it.

Pitfall 3: Scrape Interval Too High During Incidents

At a 15s scrape interval, you're getting decent resolution for trending but might miss short-lived spikes. For Apache worker pools, a 10s interval is a reasonable balance between resolution and Prometheus storage pressure.

Pitfall 4: Not Accounting for MPM Differences

Apache's event MPM handles connections differently than prefork or worker. The scoreboard states and worker counts mean different things depending on your MPM. In event MPM, a "busy" worker might be handling many concurrent connections via async I/O, so raw busy worker counts can be misleading.

Check your MPM:

apache2ctl -V | grep MPM
# Server MPM: event

Tune your alerting thresholds accordingly. Event MPM can typically sustain much higher concurrency per worker than prefork.

Pitfall 5: The Exporter Running as Root

Run the Apache exporter as a dedicated low-privilege user. The systemd unit file above already handles this, but it's worth calling out explicitly. The exporter only needs network access to scrape mod_status — it doesn't need root.

Pitfall 6: Missing Network Security Groups / Firewall Rules

In cloud environments, don't forget to add the Prometheus server's IP to the security group allowing port 9117. I've troubleshot more than one "metrics missing" incident that turned out to be a security group rule, not a config issue.

---

Part 8: Real-World Scenario — Diagnosing a Slow Leak

Here's a scenario I've dealt with in production: Apache workers slowly accumulate over 48 hours until the server becomes unresponsive, then recovers after a restart.

The mod_status + Prometheus stack is exactly what you need to diagnose this. The key query:

# Watch for workers stuck in "closing" state
apache_scoreboard{state="closing"}

# Correlate with connection rates
rate(apache_connections_total[5m])

# Look for keep-alive connections piling up
apache_connections{state="keepalive"}

If you see closing state workers accumulating, you likely have a keep-alive timeout issue or upstream backend connections not being released properly. The fix is usually tuning:

# Reduce keep-alive timeout
KeepAliveTimeout 5

# Limit keep-alive requests
MaxKeepAliveRequests 100

# Tune worker lifecycle
MaxConnectionsPerChild 1000

The ability to observe these patterns over time — rather than just noticing the server is slow — is what makes this monitoring stack genuinely valuable.

---

Part 9: Production Checklist

Before you call this setup production-ready, run through this checklist:

• ExtendedStatus On is configured
• /server-status is restricted to monitoring IPs only
• Apache exporter runs as non-root user
• Port 9117 is firewalled from public access
• Prometheus scrape interval is 10-15s
• Recording rules are defined for expensive queries
• Alerting rules cover: worker saturation, server down, request rate drops
• Grafana dashboard has a template variable for instance selection
• Alert routing goes to the right team (PagerDuty, Slack, etc.)
• Retention policy on Prometheus covers at least 15 days for trend analysis
• Log-based metrics (mtail or promtail) supplement mod_status for latency data

---

Wrapping Up

Building a solid Apache monitoring stack with mod_status, Prometheus, and Grafana isn't complex, but there are enough moving pieces that getting it right requires attention to detail. The configuration choices you make — ExtendedStatus, scrape intervals, alert thresholds, access restrictions — have real consequences in production.

The key insight here is that mod_status is a foundation, not a complete solution. Combined with the Prometheus exporter, intelligent recording rules, and log-based metrics for latency, you get a monitoring system that can tell you not just that something is wrong, but what is wrong and for how long it's been wrong.

That's the difference between reactive firefighting and proactive operations. And in my experience, that difference is worth the investment every time.