Prometheus Service Discovery in Kubernetes: Auto-Scrape Everything

One of the best things about running Prometheus in Kubernetes is that you never have to manually maintain a list of targets. Kubernetes service discovery lets Prometheus automatically find and scrape any pod, service, endpoint, or node — and update in real time as things scale up and down. The challenge is configuring it correctly so you're scraping what you intend and filtering out what you don't.

How Kubernetes Service Discovery Works

Prometheus's Kubernetes SD watches the Kubernetes API and generates scrape targets from the objects it finds. There are five role types:

Role	What Gets Scraped
pod	Each pod, using its IP and a specified port
endpoints	Each endpoint address in an Endpoints object
service	Each service port
node	Each node
ingress	Each ingress rule

In practice, you'll use pod and endpoints most often for application metrics, and node for host-level metrics. The endpoints role is usually preferred over service because it gives you per-pod granularity rather than hitting the service VIP.

Basic Prometheus Config for Kubernetes

If you're running Prometheus directly (not via the Operator), here's a complete scrape config:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # Scrape the Prometheus server itself
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  # Node exporter on each Kubernetes node
  - job_name: kubernetes-nodes
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics

  # Scrape metrics from annotated pods
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Only scrape pods with the annotation prometheus.io/scrape: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      # Use the port from annotation if provided
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: $(__meta_kubernetes_pod_ip):${1}
      # Use the path from annotation if provided
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      # Copy pod labels to Prometheus labels
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      # Add namespace label
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

Annotation-Based Scraping

The annotation approach is the simplest way to get apps scraped. Add these to your pod spec:

# In your Deployment/Pod template
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"

Any pod with prometheus.io/scrape: "true" gets picked up automatically. The port and path annotations let different apps expose metrics on different ports without changing Prometheus config.

This is fine for smaller setups, but it has a problem at scale: anyone can add those annotations and start getting scraped. The Prometheus Operator pattern (using ServiceMonitor CRDs) gives you better access control.

The Prometheus Operator Way

The Prometheus Operator is the standard way to run Prometheus in Kubernetes. It introduces CRDs — ServiceMonitor, PodMonitor, and PrometheusRule — that let application teams define their own scrape configs without touching the central Prometheus configuration.

Install via Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm upgrade --install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --values monitoring-values.yaml

ServiceMonitor Example

A ServiceMonitor tells Prometheus to scrape an existing Kubernetes Service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
  namespace: monitoring
  labels:
    release: kube-prometheus-stack  # must match Prometheus's serviceMonitorSelector
spec:
  namespaceSelector:
    matchNames:
      - default
      - production
  selector:
    matchLabels:
      app.kubernetes.io/name: myapp
  endpoints:
    - port: metrics        # the Service port name (not number)
      interval: 30s
      path: /metrics
      scheme: http
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_node_name]
          targetLabel: node
      metricRelabelings:
        - sourceLabels: [__name__]
          regex: 'go_.*'    # drop Go runtime metrics if you don't need them
          action: drop

The Service itself needs a named port:

apiVersion: v1
kind: Service
metadata:
  name: myapp
  labels:
    app.kubernetes.io/name: myapp
spec:
  selector:
    app.kubernetes.io/name: myapp
  ports:
    - name: metrics     # ServiceMonitor references this name
      port: 9090
      targetPort: 8080

PodMonitor Example

When you don't have a Service (headless pods, batch jobs), use PodMonitor:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: myapp-workers
  namespace: monitoring
spec:
  namespaceSelector:
    matchNames:
      - workers
  selector:
    matchLabels:
      app: myapp-worker
  podMetricsEndpoints:
    - port: metrics
      interval: 60s
      path: /metrics

Relabeling: The Most Powerful Part

Relabeling transforms label metadata before it's stored. It's also how you filter targets. Understanding it unlocks most of the power of Kubernetes SD.

Each relabel rule has:

• source_labels: which labels to read from
• target_label: where to write the result
• regex: a regex to match/capture against the source value
• action: what to do (keep, drop, replace, labelmap, etc.)

Common Relabeling Patterns

relabel_configs:
  # Keep only pods in specific namespaces
  - source_labels: [__meta_kubernetes_namespace]
    regex: (default|production|staging)
    action: keep

  # Drop pods with a specific label
  - source_labels: [__meta_kubernetes_pod_label_monitoring]
    regex: disabled
    action: drop

  # Add an environment label based on namespace
  - source_labels: [__meta_kubernetes_namespace]
    regex: production
    target_label: env
    replacement: prod

  # Rename a pod label
  - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
    target_label: app

  # Set job name from pod annotation (override default)
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_job]
    regex: (.+)
    target_label: job

Metric Relabeling (After Scrape)

metricRelabelings applies after the metrics are scraped — useful for dropping high-cardinality or unwanted metrics:

metricRelabelings:
  # Drop internal Go runtime metrics
  - sourceLabels: [__name__]
    regex: 'go_(gc|goroutine|memstats).*'
    action: drop

  # Drop metrics with too many labels (high cardinality)
  - sourceLabels: [path]
    regex: '/api/v1/items/\d+'  # per-item paths
    action: drop

  # Truncate long label values
  - sourceLabels: [user_agent]
    regex: '(.{50}).*'
    replacement: '${1}...'
    targetLabel: user_agent

High cardinality is a major Prometheus performance issue. Dropping or truncating problematic labels at the relabeling stage keeps your time series count manageable.

Scraping cAdvisor and kube-state-metrics

Two indispensable metric sources for Kubernetes monitoring:

cAdvisor ships with kubelet and exposes container resource usage. It's automatically scraped by the Prometheus Operator's node scraper. If you're configuring manually:

- job_name: kubernetes-cadvisor
  kubernetes_sd_configs:
    - role: node
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
    - target_label: __address__
      replacement: kubernetes.default.svc:443
    - source_labels: [__meta_kubernetes_node_name]
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

kube-state-metrics exposes Kubernetes object state (deployment replicas, pod phase, etc.). Deploy it separately:

helm upgrade --install kube-state-metrics \
  prometheus-community/kube-state-metrics \
  --namespace monitoring

Then add a ServiceMonitor for it, or let kube-prometheus-stack manage it automatically.

Verifying Discovery Is Working

# Port-forward to Prometheus
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090

# Then open http://localhost:9090/targets in your browser
# Or via the API:
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, instance: .labels.instance, health: .health}'

Check the Targets page in the Prometheus UI to see which targets are UP, DOWN, or in UNKNOWN state. The "Labels" column shows what metadata was discovered; the "Last Scrape" column shows whether it's working.

For debugging label issues:

# See all labels a specific target would have before relabeling
curl 'http://localhost:9090/api/v1/targets?state=active' | \
  jq '.data.activeTargets[] | select(.labels.job == "kubernetes-pods")'

Getting service discovery right means you never have a "where are my metrics?" question after deploying a new service. Everything that announces itself gets scraped, everything that doesn't stays silent — and that's exactly how it should work.