Fix Grafana 'Datasource Error' When Querying Prometheus

The Error: "datasource error" in Grafana Panels

You open a Grafana dashboard and every panel shows a red exclamation mark with:

Datasource error
Post "http://prometheus:9090/api/v1/query_range": dial tcp: lookup prometheus: no such host

Or a variation like:

Data source is not working: Post "http://localhost:9090/api/v1/query": connection refused

Your dashboards are completely blank. No data is being returned.

Root Cause

Grafana cannot reach the Prometheus API endpoint. This typically happens because:

• Wrong URL in the datasource configuration (hostname, port, or protocol mismatch)
• Network isolation between Grafana and Prometheus containers or pods
• Prometheus is down or restarting
• Reverse proxy stripping paths or headers
• Authentication required but not configured in the datasource

Step-by-Step Fix

1. Verify Prometheus is Running

First, confirm Prometheus is actually up:

# On the Prometheus host
curl -s http://localhost:9090/-/healthy
# Should return: Prometheus Server is Healthy.

# Check the process
systemctl status prometheus

2. Test Connectivity from Grafana's Perspective

The URL Grafana uses must be reachable from the Grafana server, not your laptop:

# If both run in Docker
docker exec grafana-container curl -s http://prometheus:9090/-/healthy

# If both run in Kubernetes
kubectl exec -it deploy/grafana -- curl -s http://prometheus-server.monitoring.svc:9090/-/healthy

# If running on separate VMs
curl -s http://prometheus-ip:9090/-/healthy

3. Fix the Datasource URL

Go to Grafana > Connections > Data sources > Prometheus and update the URL.

Common corrections:

Wrong URL	Correct URL
http://localhost:9090 (in Docker)	http://prometheus:9090 (use container name)
http://prometheus:9090 (in K8s)	http://prometheus-server.monitoring.svc.cluster.local:9090
https://prometheus:9090	http://prometheus:9090 (if no TLS)
http://prometheus:9090/ (trailing slash)	http://prometheus:9090

Click Save & Test after each change. You should see a green "Data source is working" banner.

4. Check for Authentication Issues

If Prometheus is behind a reverse proxy with basic auth:

# In Grafana datasource provisioning
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    basicAuth: true
    basicAuthUser: admin
    secureJsonData:
      basicAuthPassword: your-password

5. Fix Docker Network Issues

If using Docker Compose, ensure both services are on the same network:

services:
  prometheus:
    image: prom/prometheus:latest
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

6. Check Kubernetes Service Resolution

In Kubernetes, verify the Prometheus service exists and has endpoints:

kubectl get svc -n monitoring prometheus-server
kubectl get endpoints -n monitoring prometheus-server

If endpoints are empty, the Prometheus pods aren't matching the service selector.

7. Test the Fix

After updating the datasource, go back to your dashboard. Panels should load within a few seconds. If panels still show "No data," the datasource is connected but the queries may reference metrics that don't exist yet.

Prevention Tips

• Use Grafana datasource provisioning via YAML files so the URL is version-controlled and consistent.
• Use Kubernetes DNS names (service.namespace.svc.cluster.local) instead of IPs that can change.
• Add a health check dashboard that queries up as a canary for datasource connectivity.
• Set up Grafana alerting on datasource health so you know before users report blank dashboards.