Grafana Dashboard Design: Patterns That Don't Suck

Most Grafana dashboards I've inherited were built by engineers who understood Prometheus but had never thought about information design. Every metric gets a graph, all graphs are the same size, there's no clear answer to "is anything wrong right now?", and navigating the dashboard during an incident is like reading a spreadsheet sideways. Here's how to build dashboards that work when it matters.

Start with the Question, Not the Metric

The most common mistake in dashboard design is starting with "what data do I have?" instead of "what questions does this dashboard need to answer?" Before creating a single panel, write out the three to five questions the dashboard should answer:

• Is the service healthy right now?
• How busy is it?
• Where is the latency going?
• Are there any errors users are experiencing?
• Is it about to run out of resources?

Every panel on the dashboard should answer one of those questions. If you can't explain which question a panel answers, delete it.

Information Hierarchy: The Row Structure

Organize panels in rows from most important to least:

Row 1: STATUS (stat panels, traffic lights — answer "is anything broken?")
Row 2: THE FOUR GOLDEN SIGNALS (rate, errors, latency, saturation)
Row 3: RESOURCE USAGE (CPU, memory, disk)
Row 4: DEPENDENCIES (database, cache, downstream services)
Row 5: DETAILS (granular breakdowns — only needed during deep-dives)

The top row should be readable from 10 feet away. Stat panels with large text and color thresholds work well here:

// Panel config for a "Service Status" stat panel
{
  "type": "stat",
  "title": "Error Rate",
  "options": {
    "colorMode": "background",
    "graphMode": "none",
    "textMode": "value_and_name",
    "reduceOptions": {
      "calcs": ["lastNotNull"]
    }
  },
  "fieldConfig": {
    "defaults": {
      "unit": "percentunit",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.01 },
          { "color": "red", "value": 0.05 }
        ]
      }
    }
  }
}

Green background at a glance = fine. Red background = something needs attention now.

Panel Types: Use the Right Tool

Panel Type	Best For	Avoid For
Time series	Trends over time, multiple series	Single current values
Stat	Single current value with status color	Trend data
Gauge	Single value in a range (% full, etc.)	Multiple time series
Bar chart	Comparing values across categories	Time-based data
Table	Breakdowns with multiple dimensions	Trend visualization
Heatmap	Distribution over time (latency histograms)	Simple rates
State timeline	Boolean/state changes over time	Numeric values
Logs	Log data from Loki	Metric data

The heatmap panel is underused for latency. A latency heatmap shows the full distribution over time — you can see whether your P99 is driven by consistent slow responses or rare outliers, which a percentile line chart can't show you.

Dashboard Variables: Make It Reusable

A dashboard without variables is a dashboard that becomes outdated. Variables let users filter by namespace, service, pod, environment — without editing the dashboard.

Service Selector Variable

# In dashboard JSON (simplified)
variables:
  - name: namespace
    type: query
    datasource: Prometheus
    query: "label_values(kube_pod_info, namespace)"
    refresh: 2   # refresh on time range change
    sort: 1      # alphabetical

  - name: service
    type: query
    datasource: Prometheus
    query: "label_values(kube_pod_info{namespace='$namespace'}, pod)"
    refresh: 2
    includeAll: true
    allValue: ".*"

Then use $namespace and $service in your panel queries:

sum(rate(http_requests_total{namespace="$namespace", service=~"$service"}[5m]))

Environment Selector

- name: env
  type: custom
  options:
    - text: "Staging"
      value: "staging"
    - text: "Production"
      value: "production"
  current:
    text: "Production"
    value: "production"

This lets the same dashboard work for staging and production, reducing the number of dashboards you need to maintain.

The Four Golden Signals Row

This row should be a staple on every service dashboard:

Request Rate — how much traffic is the service handling?

sum(rate(http_requests_total{namespace="$namespace"}[5m]))

Error Rate — what percentage of requests are failing?

(
  sum(rate(http_requests_total{namespace="$namespace", status=~"5.."}[5m]))
  / sum(rate(http_requests_total{namespace="$namespace"}[5m]))
) * 100

Latency — how long do requests take? Show multiple percentiles on one panel:

# Series A: P50
histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket{namespace="$namespace"}[5m])))

# Series B: P90
histogram_quantile(0.90, sum by (le) (rate(http_request_duration_seconds_bucket{namespace="$namespace"}[5m])))

# Series C: P99
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{namespace="$namespace"}[5m])))

Saturation — how close to capacity is the service?

max by (pod) (
  container_memory_working_set_bytes{namespace="$namespace"}
  / container_spec_memory_limit_bytes{namespace="$namespace"}
) * 100

Put these four panels side by side in a row. They tell the complete story of service health.

Annotations: Add Context to Graphs

Annotations overlay deployment events, incidents, and other changes onto your time series graphs. This is invaluable for correlation — "the error rate spiked at exactly the moment this deployment happened."

# Push a Grafana annotation via API after a deployment
curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -d '{
    "dashboardUID": "myapp-overview",
    "time": '"$(date +%s%3N)"',
    "tags": ["deployment", "myapp"],
    "text": "Deployed v3.2.1 to production"
  }' \
  "https://grafana.internal/api/annotations"

Add this to your CI/CD pipeline so every deployment is visible on every relevant dashboard.

Dashboard as Code with Grafonnet

Manual dashboard editing doesn't scale. Use Grafonnet (Jsonnet library for Grafana) to generate dashboards from code:

// dashboard.jsonnet
local grafana = import 'grafonnet/grafana.libsonnet';
local dashboard = grafana.dashboard;
local row = grafana.row;
local prometheus = grafana.prometheus;
local graphPanel = grafana.graphPanel;
local statPanel = grafana.statPanel;

dashboard.new(
  'MyApp Overview',
  schemaVersion=36,
  tags=['myapp', 'production'],
  time_from='now-3h',
  refresh='30s',
)
.addTemplate(
  grafana.template.datasource(
    'datasource',
    'prometheus',
    'Prometheus',
  )
)
.addRow(
  row.new('Status')
  .addPanel(
    statPanel.new(
      'Error Rate',
      datasource='$datasource',
    )
    .addTarget(
      prometheus.target(
        '(sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) * 100',
        legendFormat='Error %',
      )
    )
  )
)

# Render and push to Grafana
jsonnet dashboard.jsonnet | \
  jq '{dashboard: ., overwrite: true}' | \
  curl -X POST \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $GRAFANA_API_KEY" \
    -d @- \
    "https://grafana.internal/api/dashboards/db"

Alternatively, Grafana's built-in provisioning lets you manage dashboards as JSON files in a Git repo:

# grafana/provisioning/dashboards/dashboards.yaml
apiVersion: 1
providers:
  - name: default
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

Mount your dashboard JSON files via a ConfigMap, and Grafana picks them up automatically. Changes to the JSON files are reflected without restarting.

Layout Rules That Help During Incidents

• Width consistency within rows: all panels in a row should be the same height
• Put the most important metric top-left: that's where eyes go first
• Never use more than 6-8 panels per row: cognitive overload kills response time
• Set sensible time ranges as defaults: most service dashboards should default to "last 3 hours"
• Add a link to the runbook in the dashboard description — not just in alerts
• Repeat panels for each service/pod using panel repeat instead of one panel per thing

The "repeat" feature deserves a call-out. Instead of manually creating a memory panel for each of your 10 microservices, create one panel and set it to repeat by a variable:

In the panel options, set "Repeat by variable" to $service. Grafana renders one panel per value in the variable. When you add a new service, it automatically appears.

A dashboard that nobody looks at during incidents isn't worth having. The bar for "is this dashboard useful?" is: can a new on-call engineer who has never seen this system before figure out what's wrong within two minutes of opening it? If not, redesign it.