PromQL Queries You'll Actually Use in Production

PromQL has a learning curve, but once it clicks, it's remarkably expressive. The problem is that most learning resources teach the syntax in isolation. Here are the queries I actually use when building dashboards and writing alert rules — organized by use case, with notes on why each is written the way it is.

The Fundamentals You Need First

Before the queries, two concepts that trip people up:

rate() vs irate(): Use rate() for dashboards and alerts — it averages over the time window, which smooths out spikes. Use irate() only when you need per-second instantaneous rates and are fine with noisier graphs. For a 5-minute window, rate(metric[5m]) calculates the per-second average rate over the last 5 minutes.

Counter reset handling: rate() and increase() automatically handle counter resets (when a process restarts and the counter drops to 0). Never use delta() on counters — it doesn't handle resets.

HTTP Request Metrics

Requests Per Second by Status Code

sum by (status) (
  rate(http_requests_total[5m])
)

Overall Error Rate as a Percentage

(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

Error Rate Per Service

sum by (service) (
  rate(http_requests_total{status=~"5.."}[5m])
)
/
sum by (service) (
  rate(http_requests_total[5m])
)
* 100

Request Volume Over Time (for capacity planning)

sum(increase(http_requests_total[1h]))

increase() is equivalent to rate() * duration_in_seconds. I prefer it for "total requests in the last hour" type panels since the number is more intuitive than per-second.

Top 10 Slowest Endpoints

topk(10,
  histogram_quantile(0.95,
    sum by (path, le) (
      rate(http_request_duration_seconds_bucket[5m])
    )
  )
)

Latency Percentiles

Latency queries almost always need histograms. If your app exports http_request_duration_seconds_bucket, you can compute any percentile.

P50/P90/P99 Latency for a Service

# P50
histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket{service="myapp"}[5m])))

# P90
histogram_quantile(0.90, sum by (le) (rate(http_request_duration_seconds_bucket{service="myapp"}[5m])))

# P99
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{service="myapp"}[5m])))

Latency Heatmap (Native Histograms)

If you're on Prometheus 2.40+ with native histograms:

histogram_quantile(0.99, rate(http_request_duration_seconds[5m]))

Native histograms eliminate the le label aggregation complexity and give more accurate results.

Average Request Duration

sum(rate(http_request_duration_seconds_sum[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

Average latency is less useful than percentiles for SLO monitoring, but it's quick to implement before you have histograms everywhere.

Kubernetes Pod and Workload Metrics

These use kube-state-metrics and cAdvisor metrics.

Deployment Availability Ratio

kube_deployment_status_available_replicas
/
kube_deployment_spec_replicas

Values less than 1 mean the deployment is degraded. Good for a status panel.

Pods Not Running by Namespace

count by (namespace, phase) (
  kube_pod_status_phase{phase!="Running", phase!="Succeeded"}
)

Container CPU Usage (Actual vs Request)

# Actual CPU usage
sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
)

# CPU throttling percentage
sum by (namespace, pod, container) (
  rate(container_cpu_throttled_seconds_total[5m])
)
/
sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total[5m])
)
* 100

CPU throttling is one of the most common causes of mysterious latency in Kubernetes. If this number is high (>25%), your CPU limits are too low.

Memory Usage vs Limit

# Memory usage as percentage of limit
sum by (namespace, pod, container) (
  container_memory_working_set_bytes{container!=""}
)
/
sum by (namespace, pod, container) (
  kube_pod_container_resource_limits{resource="memory"}
)
* 100

Use container_memory_working_set_bytes rather than container_memory_usage_bytes. The working set excludes file cache, which is what Kubernetes uses for OOMKill decisions.

OOMKill Events

increase(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[1h])

Pod Restart Rate

sum by (namespace, pod) (
  increase(kube_pod_container_status_restarts_total[1h])
)
> 0

The > 0 filters out pods with no restarts, keeping the result set clean.

Node Resource Metrics

Node CPU Utilization

100 - (
  avg by (instance) (
    rate(node_cpu_seconds_total{mode="idle"}[5m])
  ) * 100
)

Available Memory Percentage

(
  node_memory_MemAvailable_bytes
  / node_memory_MemTotal_bytes
) * 100

Disk I/O Utilization

rate(node_disk_io_time_seconds_total{device!~"dm.*|sr.*"}[5m]) * 100

Network Traffic Per Node

# Inbound
sum by (instance) (rate(node_network_receive_bytes_total{device!~"lo|veth.*"}[5m])) * 8

# Outbound
sum by (instance) (rate(node_network_transmit_bytes_total{device!~"lo|veth.*"}[5m])) * 8

Multiply by 8 to convert bytes/sec to bits/sec for the standard network graph format.

Application-Level SLO Queries

Availability SLO (Success Rate)

# 30-day success rate window
(
  sum(rate(http_requests_total{status!~"5.."}[30d]))
  /
  sum(rate(http_requests_total[30d]))
) * 100

Error Budget Remaining

# Assuming 99.9% SLO target (0.1% error budget)
1 - (
  (
    1 - (
      sum(rate(http_requests_total{status!~"5.."}[30d]))
      /
      sum(rate(http_requests_total[30d]))
    )
  )
  /
  0.001   # 100% - 99.9% = 0.1% = 0.001
)

Values above 1 mean you've spent more than your error budget. Values at 0.5 mean half your budget is gone.

Multi-Window Error Rate (for burn rate alerts)

# 1-hour error rate
(
  sum(rate(http_requests_total{status=~"5.."}[1h]))
  / sum(rate(http_requests_total[1h]))
)

# 5-minute error rate (fast burn detection)
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))
)

Google's SRE Book recommends multi-window burn rate alerts: alert when both the short-window (5m) and long-window (1h) rates are elevated. Short window catches fast burns; long window catches slow burns.

Useful Label Manipulation

Grouping Across Multiple Labels

sum by (namespace, service, version) (
  rate(http_requests_total[5m])
)

Filtering with Regex

# Match multiple values
http_requests_total{env=~"staging|production"}

# Exclude values
http_requests_total{status!~"2..|3.."}

# Match prefix
http_requests_total{path=~"/api/.*"}

Joining Two Metrics

# Add info labels from kube_pod_info to container metrics
container_cpu_usage_seconds_total
* on (pod, namespace) group_left (node)
kube_pod_info

The group_left join is one of the most powerful PromQL features. It lets you enrich a metric with labels from another metric. In this case, adding node from kube_pod_info to container CPU metrics.

Quick Reference Table

Use Case	Key Function	Window
Per-second rate of counter	rate()	5m typical
Total count over window	increase()	1h/24h
Percentile from histogram	histogram_quantile()	same as rate
Value if metric missing	absent()	—
Top N by value	topk()	—
Bottom N by value	bottomk()	—
Average across series	avg()	—
Max across series	max()	—
Predict future value	predict_linear()	lookback window

Disk Full Prediction

predict_linear(
  node_filesystem_avail_bytes{mountpoint="/"}[6h],
  4 * 3600   # predict 4 hours ahead
) < 0

This fires an alert when Prometheus predicts disk will be full in the next 4 hours, based on the current 6-hour trend. Far more useful than a static threshold alert.

PromQL rewards experimentation. The Prometheus UI's "Table" view is great for iterating on a query — you can see the exact label set of each result and adjust filters accordingly. Build the query there first, then paste it into your alert rule or dashboard.