Prometheus Remote Write Tuning for High-Throughput Long-Throughput Storage with Thanos

If you're running Prometheus at scale with Thanos for long-term storage, misconfigured remote write settings are silently killing your performance — and your cloud bill. Here's everything you need to know, organized for fast lookup.

---

Remote Write Configuration Cheat Sheet

Baseline High-Throughput Config

remote_write:
  - url: "http://thanos-receive.monitoring.svc.cluster.local:19291/api/v1/receive"
    name: thanos-receive
    
    # Queue management — the most impactful settings
    queue_config:
      capacity: 10000           # In-memory queue size per shard
      max_shards: 200           # Max parallel shards (increase for high cardinality)
      min_shards: 20            # Prevents cold-start bottlenecks
      max_samples_per_send: 2000  # Batch size per HTTP request
      batch_send_deadline: 5s     # Max wait before sending partial batch
      min_backoff: 30ms           # Initial retry delay
      max_backoff: 5s             # Cap exponential backoff
      retry_on_http_429: true     # Respect rate limiting signals

    # Connection tuning
    remote_timeout: 30s
    
    metadata_config:
      send: true
      send_interval: 1m
      max_samples_per_send: 500

When to Increase max_shards

Metric	Action
prometheus_remote_storage_pending_samples > 10000	Increase max_shards
prometheus_remote_storage_failed_samples_total rising	Check Thanos receive logs
prometheus_remote_storage_queue_highest_sent_timestamp_seconds lagging	Increase capacity + max_shards
CPU on Prometheus pod > 80%	Reduce max_shards, increase max_samples_per_send

---

TLS + Compression (Don't Skip This)

remote_write:
  - url: "https://thanos-receive.example.com/api/v1/receive"
    tls_config:
      ca_file: /etc/prometheus/certs/ca.crt
      cert_file: /etc/prometheus/certs/client.crt
      key_file: /etc/prometheus/certs/client.key
      insecure_skip_verify: false  # Never true in production
    
    # Snappy compression is enabled by default in Prometheus 2.x
    # Force protobuf encoding for better Thanos compatibility
    protobuf_message: "io.prometheus.write.v2.Request"  # Prometheus 2.54+

---

Thanos Receive Tuning

Receiver Deployment Flags

thanos receive \
  --tsdb.path="/data/thanos/receive" \
  --tsdb.retention=2h \                        # Short retention — Sidecar handles long-term
  --receive.replication-factor=3 \
  --receive.hashrings-file=/etc/thanos/hashring.json \
  --receive.local-endpoint=thanos-receive-0.thanos-receive.monitoring.svc.cluster.local:10901 \
  --receive.grpc-compression=snappy \          # Reduce inter-node bandwidth
  --tsdb.max-block-duration=2h \
  --receive.limits-config-file=/etc/thanos/limits.yaml

Limits Config (Prevent Cardinality Explosions)

# /etc/thanos/limits.yaml
default:
  request_limits:
    samples_per_series: 120       # Max samples per remote write request
    series_per_request: 10000     # Hard cap per tenant
  active_series_limit: 2000000    # Protects Thanos from cardinality bombs
  
tenant_overrides:
  high-cardinality-tenant:
    active_series_limit: 5000000

---

Key Prometheus Metrics to Monitor

# Remote write lag (should stay under 30s)
time() - prometheus_remote_storage_queue_highest_sent_timestamp_seconds

# Pending sample backlog
prometheus_remote_storage_pending_samples{remote_name="thanos-receive"}

# Shard utilization — if consistently near max_shards, scale up
prometheus_remote_storage_shards / prometheus_remote_storage_shards_max

# Dropped samples — this is data loss, investigate immediately
increase(prometheus_remote_storage_samples_dropped_total[5m])

# Failed sends — retryable vs permanent errors
rate(prometheus_remote_storage_failed_samples_total[5m])

---

Common Problems & Fast Fixes

Problem: Remote write lag keeps growing

# Fix: Increase shards and batch size together
queue_config:
  max_shards: 300
  max_samples_per_send: 5000
  capacity: 50000

Problem: Thanos receive OOMing

# Fix: Throttle at the Prometheus side
queue_config:
  max_shards: 50           # Reduce concurrency
  max_samples_per_send: 500

Problem: High CPU on Prometheus from remote write

# Fix: Increase batch sizes, reduce shard count
queue_config:
  max_shards: 30
  max_samples_per_send: 10000   # Fewer, larger requests
  batch_send_deadline: 10s      # Allow more batching time

---

Hashring Configuration for Thanos Receive HA

// /etc/thanos/hashring.json
[
  {
    "hashring": "default",
    "endpoints": [
      "thanos-receive-0.thanos-receive.monitoring.svc.cluster.local:10901",
      "thanos-receive-1.thanos-receive.monitoring.svc.cluster.local:10901",
      "thanos-receive-2.thanos-receive.monitoring.svc.cluster.local:10901"
    ],
    "tenants": []
  }
]

---

Cost Optimization Tips

• Set tsdb.retention on Thanos Receive to 2h — object storage is 10-50x cheaper than block storage
• Enable block compaction in Thanos Compactor with aggressive downsampling for metrics older than 30 days
• Filter at the source — use write_relabel_configs to drop high-cardinality, low-value metrics before they hit remote write:

remote_write:
  - url: "http://thanos-receive:19291/api/v1/receive"
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_gc_.*|process_virtual_memory_.*"
        action: drop
      - source_labels: [__name__]
        regex: "apiserver_request_duration_seconds_bucket"
        action: drop        # Drop high-cardinality histograms you're not using

Dropping unused metrics before remote write directly reduces Thanos storage costs, compaction CPU, and query latency. It's the highest-ROI tuning move in this entire guide.