Prometheus Federation: Scraping Metrics Across Multiple Data Centers with a Global View

If you're running Kubernetes clusters across multiple data centers or cloud regions, you've probably felt the pain of fragmented observability. Each cluster has its own Prometheus instance, each team has their own dashboards, and getting a unified view of your entire infrastructure feels like herding cats. Prometheus federation is the answer — but it's also a feature that's easy to get wrong.

I've set up federation for organizations running anywhere from two data centers to fifteen, and the patterns that work in production are very different from what looks elegant in architecture diagrams. Let me walk you through a setup that actually holds up under real-world conditions.

What Federation Actually Is (and Isn't)

Prometheus federation allows a global Prometheus instance to scrape aggregated metrics from multiple local Prometheus instances. The global instance doesn't replace the locals — it pulls a curated subset of metrics upward for cross-datacenter querying, alerting on infrastructure-wide SLOs, and centralized dashboards.

Think of it this way:

• Local Prometheus: Scrapes everything in its datacenter, handles local alerting, stores high-cardinality data
• Global Prometheus: Scrapes pre-aggregated or high-value metrics from locals, handles global alerting, powers executive dashboards

The key insight is that you're not replicating all data. You're curating it. This is where most federation setups go sideways — people try to pull everything to the global tier and end up with a Prometheus instance that's chronically overloaded and perpetually behind.

Setting Up Local Prometheus Instances

Your local Prometheus instances are your workhorses. For each datacenter, you want them scraping everything with high resolution. Here's a production-ready configuration:

# local-prometheus.yml (per datacenter)
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    datacenter: 'us-east-1'        # Critical: label everything at the source
    environment: 'production'
    cluster: 'k8s-prod-east'

rule_files:
  - '/etc/prometheus/rules/*.yml'

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

Notice the external_labels block. This is non-negotiable. Every metric scraped by this instance will get these labels attached. When your global Prometheus aggregates data from multiple locals, these labels are what let you tell us-east-1 apart from eu-west-2. Skip this and you'll spend hours debugging label collisions.

Configuring the Global Prometheus for Federation

The global Prometheus scrapes the /federate endpoint on each local instance. This endpoint accepts a match[] parameter that filters which metrics to export — it's your curation mechanism.

# global-prometheus.yml
global:
  scrape_interval: 30s          # Less frequent — you're aggregating, not real-time monitoring
  evaluation_interval: 30s
  external_labels:
    tier: 'global'

scrape_configs:
  - job_name: 'federate-us-east'
    scrape_interval: 30s
    honor_labels: true           # Preserve labels from local instances
    metrics_path: '/federate'
    params:
      match[]:
        - '{job="kubernetes-pods",__name__=~"container_cpu.*"}'
        - '{job="kubernetes-pods",__name__=~"container_memory.*"}'
        - 'up'
        - 'kube_pod_status_phase'
        - 'kube_deployment_status_replicas_unavailable'
        - '{__name__=~"sli:.*"}'   # Pull all SLI recording rules
    static_configs:
      - targets:
          - 'prometheus-us-east.internal:9090'
    relabel_configs:
      - source_labels: [datacenter]
        target_label: datacenter
        regex: (.*)
        replacement: '$1'

  - job_name: 'federate-eu-west'
    scrape_interval: 30s
    honor_labels: true
    metrics_path: '/federate'
    params:
      match[]:
        - '{job="kubernetes-pods",__name__=~"container_cpu.*"}'
        - '{job="kubernetes-pods",__name__=~"container_memory.*"}'
        - 'up'
        - 'kube_pod_status_phase'
        - 'kube_deployment_status_replicas_unavailable'
        - '{__name__=~"sli:.*"}'
    static_configs:
      - targets:
          - 'prometheus-eu-west.internal:9090'
    relabel_configs:
      - source_labels: [datacenter]
        target_label: datacenter

The honor_labels: true setting is critical here. Without it, the global Prometheus will overwrite the datacenter labels with its own job/instance labels, and you'll lose the origin information you need for cross-datacenter queries.

The Secret Weapon: Recording Rules for Federation

Here's the pattern that separates good federation setups from great ones: use recording rules on local instances to pre-aggregate data before federation.

Instead of pulling raw pod-level metrics to the global tier (which creates enormous cardinality), create recording rules on each local Prometheus that produce service-level aggregations:

# local-recording-rules.yml
groups:
  - name: sli_rules
    interval: 30s
    rules:
      # Service-level CPU aggregation
      - record: sli:container_cpu_usage_seconds:rate5m
        expr: |
          sum by (namespace, service, datacenter) (
            rate(container_cpu_usage_seconds_total{container!=""}[5m])
          )

      # Service error rate
      - record: sli:http_requests_error_rate:rate5m
        expr: |
          sum by (namespace, service, datacenter) (
            rate(http_requests_total{status=~"5.."}[5m])
          )
          /
          sum by (namespace, service, datacenter) (
            rate(http_requests_total[5m])
          )

      # Deployment health score
      - record: sli:deployment_availability:ratio
        expr: |
          1 - (
            kube_deployment_status_replicas_unavailable
            /
            kube_deployment_status_replicas
          )

Then in your global federation config, you only pull these pre-aggregated SLI metrics:

params:
  match[]:
    - '{__name__=~"sli:.*"}'
    - 'up'

This approach reduces the data volume going to your global tier by orders of magnitude while still giving you everything you need for cross-datacenter queries and SLO dashboards.

Handling the honor_labels Conflict Problem

One gotcha I see constantly: when you set honor_labels: true and your local Prometheus instances expose a metric with an instance label, the global Prometheus won't overwrite it with the local Prometheus's address. This is what you want for real application metrics, but it can cause confusion.

My recommendation is to always rename potentially conflicting labels at the federation scrape level:

relabel_configs:
  # Rename the local prometheus address to 'prometheus_source'
  - source_labels: [__address__]
    target_label: prometheus_source
    regex: '(.*):.*'
    replacement: '$1'

This gives you a clean audit trail of which Prometheus instance originally scraped each metric, which is invaluable for debugging when a datacenter goes dark or starts returning stale data.

Cross-Datacenter Alerting on the Global Instance

The real payoff of federation is global alerting. Here's an example alerting rule that wouldn't be possible without federation — alerting when service availability drops across multiple datacenters simultaneously:

# global-alerting-rules.yml
groups:
  - name: global_slo_alerts
    rules:
      - alert: ServiceUnavailableMultipleRegions
        expr: |
          count by (namespace, service) (
            sli:deployment_availability:ratio < 0.9
          ) >= 2
        for: 5m
        labels:
          severity: critical
          tier: global
        annotations:
          summary: "{{ $labels.service }} is degraded in multiple regions"
          description: |
            Service {{ $labels.namespace }}/{{ $labels.service }} has availability
            below 90% in {{ $value }} datacenters simultaneously.

      - alert: GlobalErrorRateHigh
        expr: |
          avg by (service) (sli:http_requests_error_rate:rate5m) > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Global error rate above 5% for {{ $labels.service }}"

These alerts only fire based on the global picture — a single datacenter blip won't wake someone up at 3am.

Operational Considerations That Will Save Your Sanity

Stagger your federation scrapes. If you have ten local Prometheus instances and they all get hit at the same second, you'll see coordinated spikes in your metrics pipeline. Add scrape_offset or simply deploy global instances that aren't perfectly time-synchronized.

Set appropriate timeouts. Federation scrapes can be slow, especially if a local instance is under load. Set explicit timeouts that are longer than your normal application scrapes:

- job_name: 'federate-us-east'
  scrape_interval: 30s
  scrape_timeout: 20s    # Give it time — this is aggregating a lot of data

Monitor the monitoring. Add alerts on your global Prometheus for federation lag and scrape failures:

- alert: FederationScrapeFailed
  expr: up{job=~"federate-.*"} == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Lost federation connection to {{ $labels.instance }}"

Consider Thanos or Cortex for serious scale. Federation works well for 2-10 datacenters with reasonable cardinality. If you're running 20+ regions or dealing with hundreds of millions of time series, you'll hit the limits of the federation pull model. Thanos Sidecar or Cortex's remote_write approach handles this more gracefully. But don't over-engineer — federation covers the majority of real-world use cases with far less operational complexity.

A Quick Note on Security

Your federation endpoints expose a significant portion of your metrics. Don't leave them open. At minimum, use mutual TLS between global and local Prometheus instances:

scrape_configs:
  - job_name: 'federate-us-east'
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/certs/ca.crt
      cert_file: /etc/prometheus/certs/global-prometheus.crt
      key_file: /etc/prometheus/certs/global-prometheus.key
    static_configs:
      - targets: ['prometheus-us-east.internal:9090']

Putting It All Together

The Prometheus federation pattern I've described here follows a clear hierarchy: local instances scrape everything with high fidelity, recording rules on locals produce SLI-level aggregations, and the global tier pulls only those aggregations for cross-datacenter visibility and alerting.

The setup isn't complicated, but it requires discipline about what you pull to the global tier. Every time someone asks "can we add this metric to federation?", the answer should be "do we have a recording rule that aggregates it first?" If you maintain that discipline, your global Prometheus stays lean, responsive, and actually useful — rather than becoming the monolithic bottleneck I've watched take down more than a few well-intentioned observability platforms.

Start with the minimal match pattern, add recording rules as your alerting needs evolve, and resist the temptation to just pull {__name__=~".*"} to the global tier. Future-you will be grateful.