Envoy Observability: Distributed Tracing and Metrics

Envoy is uniquely positioned in your infrastructure — it handles every service-to-service call and every inbound request. That makes it the ideal place to instrument observability: traces, metrics, and access logs that tell you exactly what's happening across your entire system without touching application code.

This guide wires up Envoy with Jaeger for distributed tracing and Prometheus for metrics, then shows you what to actually look at.

The Observability Stack

Client → Envoy (edge) → Service A → Envoy (sidecar) → Service B
              ↓                            ↓
         Jaeger Agent                Jaeger Agent
              ↓                            ↓
         Jaeger Collector ← ← ← ← ← ← ← ←
              ↓
         Jaeger Query UI

Envoy Admin → Prometheus scrape → Grafana dashboards

Every hop through Envoy generates a trace span. Jaeger stitches them together by trace ID propagated in HTTP headers.

Deploy Jaeger

# jaeger.yaml — All-in-one for dev; use distributed for prod
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
        - name: jaeger
          image: jaegertracing/all-in-one:1.54
          env:
            - name: COLLECTOR_OTLP_ENABLED
              value: "true"
            - name: SPAN_STORAGE_TYPE
              value: badger      # Use Cassandra/Elasticsearch in prod
          ports:
            - containerPort: 5775    # Zipkin compact thrift
              protocol: UDP
            - containerPort: 6831    # Jaeger compact thrift (agent)
              protocol: UDP
            - containerPort: 6832    # Jaeger binary thrift
              protocol: UDP
            - containerPort: 5778    # Config/sampling HTTP
            - containerPort: 16686   # Query UI
            - containerPort: 14268   # Collector HTTP
            - containerPort: 4317    # OTLP gRPC
            - containerPort: 4318    # OTLP HTTP
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger
  namespace: observability
spec:
  selector:
    app: jaeger
  ports:
    - name: jaeger-compact
      port: 6831
      protocol: UDP
    - name: collector-http
      port: 14268
    - name: query
      port: 16686
    - name: otlp-grpc
      port: 4317

Configure Envoy Tracing

# envoy.yaml — Tracing configuration
static_resources:
  listeners:
    - name: main
      address:
        socket_address: { address: 0.0.0.0, port_value: 8080 }
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                generate_request_id: true
                tracing:
                  provider:
                    name: envoy.tracers.zipkin
                    typed_config:
                      "@type": type.googleapis.com/envoy.config.trace.v3.ZipkinConfig
                      collector_cluster: jaeger
                      collector_endpoint: /api/v2/spans
                      collector_endpoint_version: HTTP_JSON
                      shared_span_context: false
                  random_sampling:
                    value: 100.0    # 100% in dev; 1-10% in prod
                  custom_tags:
                    - tag: env
                      literal:
                        value: production
                    - tag: user.id
                      request_header:
                        name: x-user-id
                        default_value: anonymous
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: backend
                      domains: ["*"]
                      routes:
                        - match: { prefix: "/" }
                          route:
                            cluster: backend_service
                          decorator:
                            operation: backend.request   # Span operation name
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
    - name: jaeger
      type: STRICT_DNS
      connect_timeout: 1s
      load_assignment:
        cluster_name: jaeger
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: jaeger.observability.svc.cluster.local
                      port_value: 9411    # Zipkin-compatible endpoint

    - name: backend_service
      type: STRICT_DNS
      connect_timeout: 5s
      load_assignment:
        cluster_name: backend_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: backend.default.svc.cluster.local
                      port_value: 8080

Trace Context Propagation

Envoy generates the initial trace but your services must forward the trace headers to maintain the chain:

# Headers Envoy injects and your apps must propagate
x-request-id          # Envoy's request ID (ties logs to traces)
x-b3-traceid          # Zipkin/B3 trace ID
x-b3-spanid           # Current span ID
x-b3-parentspanid     # Parent span ID
x-b3-sampled          # Sampling decision (0 or 1)
x-b3-flags            # Debug flag
traceparent           # W3C Trace Context (if using OTLP)

In Node.js, forwarding headers is straightforward:

// middleware/tracing.js
const TRACE_HEADERS = [
  'x-request-id',
  'x-b3-traceid',
  'x-b3-spanid',
  'x-b3-parentspanid',
  'x-b3-sampled',
  'x-b3-flags',
  'traceparent',
  'tracestate',
];

function forwardTraceHeaders(incomingReq, outgoingHeaders = {}) {
  for (const header of TRACE_HEADERS) {
    const value = incomingReq.headers[header];
    if (value) {
      outgoingHeaders[header] = value;
    }
  }
  return outgoingHeaders;
}

// Usage in an Express route making a downstream call
app.get('/data', async (req, res) => {
  const response = await fetch('http://downstream-service/api', {
    headers: forwardTraceHeaders(req),
  });
  res.json(await response.json());
});

Without this forwarding, each service appears as an independent trace with no parent-child relationship.

Envoy Metrics with Prometheus

Envoy exposes hundreds of metrics via its admin interface. Scrape them with Prometheus:

# prometheus-config.yaml
scrape_configs:
  - job_name: envoy
    metrics_path: /stats/prometheus
    static_configs:
      - targets:
          - envoy-edge:9901
          - envoy-sidecar-service-a:9901
          - envoy-sidecar-service-b:9901
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

Key Envoy metrics to monitor:

Metric	What It Tells You
envoy_cluster_upstream_rq_total	Total requests to upstream cluster
envoy_cluster_upstream_rq_time	Request latency histogram
envoy_cluster_upstream_rq_5xx	5xx errors from upstream
envoy_cluster_upstream_cx_active	Active connections to upstream
envoy_http_downstream_rq_time	End-to-end latency seen by clients
envoy_cluster_upstream_rq_pending_total	Requests queued (circuit breaker filling)
envoy_cluster_ejections_total	Outlier detection ejections
envoy_http_ratelimit_ok	Requests passed rate limiting
envoy_http_ratelimit_over_limit	Requests rate-limited

Grafana Dashboard Queries

# P99 latency for a specific cluster
histogram_quantile(0.99,
  rate(envoy_cluster_upstream_rq_time_bucket{
    envoy_cluster_name="backend_service"
  }[5m])
)

# Error rate (5xx) per cluster
sum(rate(envoy_cluster_upstream_rq_5xx[5m])) by (envoy_cluster_name)
/
sum(rate(envoy_cluster_upstream_rq_total[5m])) by (envoy_cluster_name)

# Active connection count
sum(envoy_cluster_upstream_cx_active) by (envoy_cluster_name)

# Circuit breaker overflow events
rate(envoy_cluster_upstream_rq_pending_overflow[5m])

Envoy Access Logs in JSON

Structured access logs make log queries fast:

access_log:
  - name: envoy.access_loggers.stdout
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
      log_format:
        json_format:
          timestamp: "%START_TIME%"
          method: "%REQ(:METHOD)%"
          path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
          protocol: "%PROTOCOL%"
          response_code: "%RESPONSE_CODE%"
          response_flags: "%RESPONSE_FLAGS%"
          bytes_received: "%BYTES_RECEIVED%"
          bytes_sent: "%BYTES_SENT%"
          duration_ms: "%DURATION%"
          upstream_host: "%UPSTREAM_HOST%"
          upstream_cluster: "%UPSTREAM_CLUSTER%"
          trace_id: "%REQ(X-B3-TRACEID)%"
          request_id: "%REQ(X-REQUEST-ID)%"
          user_agent: "%REQ(USER-AGENT)%"
          forwarded_for: "%REQ(X-FORWARDED-FOR)%"

The trace_id field in access logs is the link between your traces in Jaeger and your logs in your log aggregator. When you find a slow trace in Jaeger, copy the trace ID and search your logs for it.

Sampling Strategy for Production

100% sampling is expensive at scale. Use a sensible sampling strategy:

tracing:
  random_sampling:
    value: 1.0    # 1% random sample

# Override sampling for specific paths:
# - Always sample errors (done via custom sampler or OTel collector)
# - Always sample admin/auth endpoints
# - Never sample health checks

For fine-grained control, route health check endpoints to a separate listener with tracing disabled:

# Separate listener for health checks — no tracing
- name: health_listener
  address:
    socket_address: { address: 0.0.0.0, port_value: 8081 }
  filter_chains:
    - filters:
        - name: envoy.filters.network.http_connection_manager
          typed_config:
            stat_prefix: health_check
            # No tracing block — tracing disabled for this listener
            route_config:
              virtual_hosts:
                - name: health
                  domains: ["*"]
                  routes:
                    - match: { prefix: /healthz }
                      direct_response:
                        status: 200
                        body:
                          inline_string: "OK"

Putting It All Together

With this setup you get: every request through Envoy generates a trace, Jaeger shows you the full request path across services, Prometheus collects latency histograms and error rates, and JSON access logs with trace IDs let you correlate logs to traces in seconds. Your application code only needs to forward trace headers — Envoy handles the rest.