Envoy Traffic Management: Retries, Timeouts, Canary Deployments, and Rate Limiting

Traffic Management Is Envoy's Superpower

HAProxy and Nginx can load balance and health check. Envoy goes further: it can retry failed requests with exponential backoff, route 5% of traffic to a canary, inject faults for chaos testing, and enforce per-user rate limits — all in configuration, no code changes.

These features are why Istio uses Envoy under the hood. This guide shows how to configure them directly in Envoy's YAML.

---

Retries and Backoff

Automatically retry failed requests before returning an error to the client:

routes:
  - match:
      prefix: "/api/"
    route:
      cluster: api_cluster
      timeout: 30s
      retry_policy:
        retry_on: "5xx,gateway-error,connect-failure,retriable-4xx"
        num_retries: 3
        per_try_timeout: 10s
        retry_back_off:
          base_interval: 0.25s
          max_interval: 5s

retry_on conditions:

• 5xx — retry any 5xx response
• gateway-error — retry 502, 503, 504
• connect-failure — retry on TCP connection failure
• retriable-4xx — retry 409 Conflict
• reset — retry on TCP reset from upstream

With num_retries: 3 and exponential backoff, a failing backend gets up to 3 attempts with delays of 0.25s, 0.5s, 1s before Envoy returns an error to the client.

---

Timeouts

Three timeout types matter in Envoy:

routes:
  - match:
      prefix: "/"
    route:
      cluster: web_cluster
      # Total request timeout (from request received to response sent)
      timeout: 60s

      # Individual retry attempt timeout
      retry_policy:
        per_try_timeout: 15s
        ...

Cluster-level connection timeout:

clusters:
  - name: web_cluster
    connect_timeout: 5s    # TCP connection timeout to upstream
    ...

Setting timeout: 0s disables the timeout entirely (useful for streaming/WebSockets).

---

Weighted Traffic Splitting (Canary Deployments)

Route a percentage of traffic to a new version:

routes:
  - match:
      prefix: "/"
    route:
      weighted_clusters:
        clusters:
          - name: app_v1
            weight: 95
          - name: app_v2_canary
            weight: 5
        total_weight: 100

Clusters in the config:

clusters:
  - name: app_v1
    load_assignment:
      cluster_name: app_v1
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address: { address: 10.0.1.10, port_value: 8080 }
            - endpoint:
                address:
                  socket_address: { address: 10.0.1.11, port_value: 8080 }

  - name: app_v2_canary
    load_assignment:
      cluster_name: app_v2_canary
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address: { address: 10.0.2.10, port_value: 8080 }

To gradually shift traffic (0% → 5% → 25% → 100%), update the weights and reload. With the xDS API (Istio), this happens without Envoy restarts.

Header-Based Canary Routing

Route specific users (e.g., internal testers) to the new version:

routes:
  # Internal testers get v2
  - match:
      prefix: "/"
      headers:
        - name: "x-canary"
          string_match:
            exact: "true"
    route:
      cluster: app_v2_canary

  # Everyone else gets v1
  - match:
      prefix: "/"
    route:
      cluster: app_v1

---

Fault Injection

Inject errors and delays to test resilience without breaking actual backends:

http_filters:
  - name: envoy.filters.http.fault
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault
      delay:
        fixed_delay: 2s
        percentage:
          numerator: 10        # 10% of requests get a 2s delay
          denominator: HUNDRED
      abort:
        http_status: 503
        percentage:
          numerator: 5         # 5% of requests get a 503 error
          denominator: HUNDRED
  - name: envoy.filters.http.router
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

Use fault injection to:

• Verify circuit breakers trigger at the right thresholds
• Test client retry behavior
• Validate timeout handling end-to-end

---

Rate Limiting

Local Rate Limiting (Per Envoy Instance)

Limit requests per second on a single Envoy without an external service:

http_filters:
  - name: envoy.filters.http.local_ratelimit
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
      stat_prefix: local_rate_limiter
      token_bucket:
        max_tokens: 1000
        tokens_per_fill: 1000
        fill_interval: 1s
      filter_enabled:
        runtime_key: local_rate_limit_enabled
        default_value:
          numerator: 100
          denominator: HUNDRED
      filter_enforced:
        runtime_key: local_rate_limit_enforced
        default_value:
          numerator: 100
          denominator: HUNDRED
      response_headers_to_add:
        - append_action: OVERWRITE_IF_EXISTS_OR_ADD
          header:
            key: x-local-rate-limit
            value: "true"

Global Rate Limiting (External RateLimit Service)

For rate limiting across multiple Envoy instances (e.g., per-user limits):

http_filters:
  - name: envoy.filters.http.ratelimit
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
      domain: my_service
      request_type: external
      rate_limit_service:
        grpc_service:
          envoy_grpc:
            cluster_name: rate_limit_cluster
        transport_api_version: V3

The rate limit service (e.g., Lyft's ratelimit service or go-micro/ratelimit) enforces limits and responds to Envoy with allow/deny decisions.

---

Request/Response Manipulation

Add, modify, or remove headers:

routes:
  - match:
      prefix: "/"
    route:
      cluster: web_cluster
    request_headers_to_add:
      - header:
          key: "x-request-id"
          value: "%REQ(X-REQUEST-ID)%"
        keep_empty_value: false
      - header:
          key: "x-forwarded-for"
          value: "%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%"
    response_headers_to_add:
      - header:
          key: "x-envoy-upstream-service-time"
          value: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
    response_headers_to_remove:
      - "server"
      - "x-powered-by"

---

Access Logging with Custom Format

access_log:
  - name: envoy.access_loggers.file
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
      path: /dev/stdout
      log_format:
        json_format:
          timestamp: "%START_TIME%"
          method: "%REQ(:METHOD)%"
          path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
          protocol: "%PROTOCOL%"
          status: "%RESPONSE_CODE%"
          response_flags: "%RESPONSE_FLAGS%"
          duration_ms: "%DURATION%"
          upstream_host: "%UPSTREAM_HOST%"
          upstream_cluster: "%UPSTREAM_CLUSTER%"
          bytes_received: "%BYTES_RECEIVED%"
          bytes_sent: "%BYTES_SENT%"

%RESPONSE_FLAGS% is particularly useful — it indicates why a request failed:

Flag	Meaning
UH	No healthy upstream
UF	Upstream connection failure
UO	Upstream overflow (circuit breaker)
NR	No route configured
RL	Rate limited

---

Admin API Reference

# Dump current config
curl http://localhost:9901/config_dump

# Check cluster health
curl http://localhost:9901/clusters

# Per-cluster stats
curl "http://localhost:9901/stats?filter=cluster.api_cluster"

# View routing table
curl http://localhost:9901/routes

# Drain connections gracefully (for rolling deploys)
curl -X POST http://localhost:9901/drain_listeners

# Health check endpoint
curl http://localhost:9901/healthcheck/ok
curl -X POST http://localhost:9901/healthcheck/fail   # mark unhealthy
curl -X POST http://localhost:9901/healthcheck/ok     # mark healthy

These admin endpoints are essential for debugging traffic issues in production — cluster health, retry rates, and request flags tell you precisely what's failing and why.