Fix Prometheus 'context deadline exceeded' Scrape Errors

The Error: "context deadline exceeded"

You open the Prometheus Targets page and see targets stuck in DOWN state with this error:

Get "http://your-target:9090/metrics": context deadline exceeded

This means Prometheus sent a scrape request but the target didn't respond before the configured timeout. Your metrics have gaps, and your alerts are probably firing on stale data.

Root Cause

Prometheus has a default scrape timeout of 10 seconds. When a target takes longer than that to respond, Prometheus cancels the request and logs the error. Common reasons include:

• Target is overloaded and generating metrics too slowly
• Network latency between Prometheus and the target (especially cross-region)
• Too many metrics being exposed on a single endpoint (hundreds of thousands of time series)
• DNS resolution delays adding seconds to each scrape
• Firewall or security group intermittently blocking traffic

Step-by-Step Fix

1. Check the Actual Scrape Duration

Query Prometheus itself to see how long scrapes are taking:

scrape_duration_seconds{job="your-job-name"}

If scrapes regularly take 8-9 seconds on a 10-second timeout, you've found the bottleneck.

2. Increase the Scrape Timeout

Edit your prometheus.yml to give slow targets more time:

scrape_configs:
  - job_name: 'slow-target'
    scrape_interval: 30s
    scrape_timeout: 25s
    static_configs:
      - targets: ['your-target:9090']

The scrape_timeout must always be less than or equal to scrape_interval. If your scrape takes 20 seconds, set the interval to at least 30 seconds.

3. Verify Network Connectivity

From the Prometheus server, test that you can reach the target:

# Test basic connectivity
curl -w "Time: %{time_total}s\n" -o /dev/null -s http://your-target:9090/metrics

# Check DNS resolution time
dig your-target | grep "Query time"

# Test with a specific timeout
curl --max-time 15 http://your-target:9090/metrics | wc -l

If curl takes more than a few seconds, the problem is network-level, not Prometheus configuration.

4. Reduce Metrics Cardinality on the Target

If the target exposes too many metrics, reduce them at the source or filter in Prometheus:

scrape_configs:
  - job_name: 'high-cardinality-target'
    scrape_timeout: 20s
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_.*'
        action: drop

This drops all Go runtime metrics, which can significantly reduce scrape payload size.

5. Reload Prometheus

After editing the config, reload without restarting:

# Send SIGHUP
kill -HUP $(pidof prometheus)

# Or use the lifecycle API if enabled
curl -X POST http://localhost:9090/-/reload

6. Confirm the Fix

Check the Targets page again at http://localhost:9090/targets. The target should show UP and the Last Scrape column should show a duration well under your timeout.

Prevention Tips

• Set realistic timeouts from the start. If you know a target is slow, configure scrape_timeout when you add it.
• Monitor scrape durations. Alert when scrape_duration_seconds exceeds 80% of the timeout: scrape_duration_seconds > 0.8 * 10.
• Use Prometheus federation for remote targets instead of scraping across high-latency links.
• Keep metrics endpoints lean. Avoid exposing high-cardinality labels that inflate response size.
• Use service discovery with relabeling to drop targets that are known to be flaky or retired.