Prometheus Scrape Target Down: Diagnosing and Fixing "connection refused" Errors Step by Step

If you've spent any time with Prometheus, you've seen it. That red DOWN label in the Targets page, accompanied by the dreaded connection refused error. Your dashboards go dark, your alerts start firing, and suddenly you're the most popular person in the incident channel.

The good news: connection refused is one of the more honest error messages in networking. Unlike timeout which could mean a dozen different things, connection refused is telling you exactly what happened — the TCP connection was actively rejected. Something isn't listening where Prometheus thinks it should be. Let's fix it systematically.

What "Connection Refused" Actually Means

Before diving into fixes, understand what's happening at the network layer. When Prometheus tries to scrape a target, it opens a TCP connection to <host>:<port>. A connection refused means the kernel on the target machine sent back a TCP RST packet. This happens for one of two reasons:

1. Nothing is listening on that port — the exporter isn't running
2. A firewall is actively rejecting the connection — note: a drop would cause a timeout, not a refused

This distinction matters. If you're getting connection refused, you can rule out dropped packets immediately.

Step 1: Verify What's Actually Happening

Start at the Prometheus UI. Hit http://<prometheus-host>:9090/targets and find your failing target. You'll see something like:

State: DOWN
Labels: {instance="10.0.1.42:9100", job="node-exporter"}
Last Error: Get "http://10.0.1.42:9100/metrics": dial tcp 10.0.1.42:9100: connect: connection refused

Screenshot the last scrape time. If it just went down, something changed. If it's been down since it was added, there's a configuration problem.

Step 2: Test Connectivity From Prometheus Itself

This is the step most people skip, and it costs them 30 minutes. Don't test from your laptop — test from the Prometheus server.

# SSH into your Prometheus server first, then:
curl -v http://10.0.1.42:9100/metrics

If that also fails with connection refused, you've confirmed the issue isn't Prometheus — the target itself is the problem. If curl works but Prometheus can't scrape, you've got a configuration issue on the Prometheus side.

Also try:

# Quick TCP check without HTTP overhead
nc -zv 10.0.1.42 9100

# Or with telnet if nc isn't available
telnet 10.0.1.42 9100

Step 3: Check if the Exporter is Actually Running

On the target machine:

# Check if node_exporter is running
systemctl status node_exporter

# Or check by process
ps aux | grep node_exporter

# Check what's listening on port 9100
ss -tlnp | grep 9100
# or the older netstat
netstat -tlnp | grep 9100

The ss output will tell you immediately if anything is bound to that port:

State   Recv-Q  Send-Q  Local Address:Port  Peer Address:Port  Process
LISTEN  0       128     0.0.0.0:9100       0.0.0.0:*          users:(("node_exporter",pid=12345,fd=3))

If that output is empty — nothing is listening. The exporter is down. Start there.

# Start it back up
systemctl start node_exporter

# Check why it failed
journalctl -u node_exporter -n 50 --no-pager

Step 4: Check the Listening Address

This is a sneaky one. Your exporter might be running but bound to the wrong interface. If node_exporter starts with --web.listen-address=127.0.0.1:9100, it's only reachable locally. Prometheus scraping from a different host will get connection refused.

# See exactly what address it's bound to
ss -tlnp | grep 9100

Compare these two outputs:

• 127.0.0.1:9100 → Only localhost can reach it
• 0.0.0.0:9100 → All interfaces, reachable remotely
• 10.0.1.42:9100 → Bound to specific interface

If it's bound to localhost, fix the service configuration:

# /etc/systemd/system/node_exporter.service
[Service]
ExecStart=/usr/local/bin/node_exporter \
    --web.listen-address=0.0.0.0:9100

Then reload and restart:

systemctl daemon-reload
systemctl restart node_exporter

Step 5: Firewall Rules

If the exporter is running and listening on the right address but you still can't connect from the Prometheus server, a firewall is the likely culprit.

# Check iptables rules on the target
iptables -L INPUT -n -v | grep 9100

# Check firewalld if that's what you're using
firewall-cmd --list-all

# On Ubuntu with ufw
ufw status verbose

To add the rule with iptables directly:

iptables -A INPUT -p tcp --dport 9100 -s <prometheus-server-ip> -j ACCEPT

Or more permanently with firewall-cmd:

firewall-cmd --permanent --add-port=9100/tcp
firewall-cmd --reload

If you're in a cloud environment (AWS, GCP, Azure), don't forget to check your security groups or VPC firewall rules. These operate outside the OS and won't show up in iptables. I've wasted an embarrassing amount of time before remembering to check the AWS console.

Step 6: Verify Your Prometheus Scrape Configuration

Sometimes the problem is in prometheus.yml itself — wrong IP, wrong port, or a typo that's been there since day one.

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets:
          - '10.0.1.42:9100'  # Double check this IP and port
    # If your exporter uses HTTPS or a custom path:
    scheme: http
    metrics_path: /metrics

Common mistakes I see regularly:

• Port 9100 vs 9090 (that's Prometheus itself)
• Forgetting the port entirely (defaults to 80)
• Using a hostname that doesn't resolve from the Prometheus server
• Mixing up IPv4 and IPv6 — if you're targeting [::1]:9100 but the exporter only binds IPv4, you'll get connection refused

Test DNS resolution from Prometheus:

dig +short your-target-hostname
# or
nslookup your-target-hostname

Step 7: Check for Port Conflicts

Another scenario: something else grabbed port 9100 before the exporter could. This is more common in containerized environments where port allocation can get chaotic.

# What's using port 9100?
ss -tlnp | grep 9100
fuser 9100/tcp

If something else is there, either reconfigure the exporter to use a different port or sort out why the port conflict exists.

Step 8: Docker and Kubernetes Specific Issues

If you're running exporters in containers, the network model adds complexity.

Docker: If Prometheus is on the host and the exporter is in a container, make sure the port is published:

docker run -d \
  --name node-exporter \
  -p 9100:9100 \
  prom/node-exporter

Without -p 9100:9100, the port isn't accessible from outside the container network.

Kubernetes: Check if the Pod is actually running and the Service is correctly defined:

kubectl get pods -n monitoring
kubectl describe pod <node-exporter-pod>

# Check the Service
kubectl get svc -n monitoring
kubectl describe svc node-exporter

For Kubernetes, your Prometheus scrape config typically needs to target the Service ClusterIP or use Kubernetes service discovery. If you're using prometheus-operator or the Prometheus Helm chart, check your ServiceMonitor resource:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  endpoints:
    - port: metrics
      interval: 30s

Make sure the port name matches what's defined in your Service spec.

Quick Diagnostic Checklist

When you hit connection refused, run through this in order:

1. [ ] Test connectivity from Prometheus server: curl http://<target>:<port>/metrics
2. [ ] Check if exporter is running: systemctl status <exporter>
3. [ ] Check what's listening: ss -tlnp | grep <port>
4. [ ] Verify bind address (not just 127.0.0.1): ss -tlnp
5. [ ] Check OS firewall: iptables -L or ufw status
6. [ ] Check cloud firewall/security groups (if applicable)
7. [ ] Verify prometheus.yml has correct IP, port, and path
8. [ ] Check for port conflicts: fuser <port>/tcp
9. [ ] If containerized: verify port mapping and network config

Setting Up an Alert for Scrape Failures

While you're fixing this, add an alert so you catch it faster next time:

# In your alerting rules file
groups:
  - name: prometheus-meta
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Prometheus target {{ $labels.instance }} is down"
          description: "Job {{ $labels.job }} target {{ $labels.instance }} has been down for more than 5 minutes."

The up metric is set to 0 by Prometheus itself when it can't scrape a target, so this catches connection refused, timeouts, and any other scrape failure.

Wrapping Up

connection refused follows a predictable diagnostic path: is the process running, is it listening on the right address, is anything blocking the connection, and is Prometheus configured correctly. Work through that checklist in order and you'll resolve it 95% of the time.

The remaining 5% is usually some environment-specific weirdness — a corporate proxy intercepting connections, an SELinux policy blocking the bind, or a misconfigured CNI plugin in Kubernetes. Those warrant their own articles. But start with the basics. In my experience, it's almost always either the exporter crashed or it's bound to localhost when it shouldn't be.

Packets don't lie — the kernel told you the connection was refused. Now you know exactly where to look.