Fix Envoy Proxy 'upstream connect error or disconnect/reset before headers'

The Error: "upstream connect error or disconnect/reset before headers"

Your application returns a generic error page with:

upstream connect error or disconnect/reset before headers. reset reason: connection failure

The HTTP response code is typically 503. This is Envoy telling you it tried to connect to a backend service but couldn't establish or maintain the connection.

Root Cause

Envoy acts as a proxy between clients and upstream services. This error means the connection between Envoy and the upstream (backend) service failed. Common causes:

• Upstream service is down or not listening on the expected port
• Connection timeout -- Envoy gave up waiting for the upstream to respond
• Incorrect cluster configuration -- wrong host, port, or protocol
• mTLS mismatch in a service mesh (Istio/Envoy) -- one side expects TLS, the other doesn't
• Circuit breaker tripped -- too many pending requests or connections

Step-by-Step Fix

1. Check Envoy's View of Upstream Health

# If running standalone Envoy
curl -s http://localhost:15000/clusters | grep health_flags

# In Istio service mesh
kubectl exec -it deploy/my-app -c istio-proxy -- \
  curl -s localhost:15000/clusters | grep "my-upstream-service"

Look for clusters with health_flags::/failed_active_hc or health_flags::/failed_outlier_check. These upstreams are marked unhealthy.

2. Verify the Upstream Service is Running

# Check the service exists and has endpoints
kubectl get endpoints my-service -n my-namespace

# Check pods are running
kubectl get pods -n my-namespace -l app=my-service

# Test direct connectivity from the sidecar
kubectl exec -it deploy/my-app -c istio-proxy -- \
  curl -v http://my-service.my-namespace.svc.cluster.local:8080/health

If endpoints are empty, the service selector doesn't match any pods.

3. Check for Port Mismatch

A common issue: the Kubernetes Service targets one port, but the container listens on another.

kubectl get svc my-service -n my-namespace -o yaml

Verify targetPort matches the actual port your application listens on:

spec:
  ports:
    - name: http
      port: 80
      targetPort: 8080  # Must match the container's listening port
      protocol: TCP

4. Fix mTLS Issues (Istio)

In Istio, if STRICT mTLS is enabled but a service doesn't have a sidecar:

# Check mTLS status
istioctl x describe pod my-app-pod-xyz -n my-namespace

# Check PeerAuthentication policies
kubectl get peerauthentication -n my-namespace -o yaml

If the upstream service doesn't have an Istio sidecar, configure a DestinationRule to disable mTLS for that service:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-external-service
  namespace: my-namespace
spec:
  host: my-external-service.my-namespace.svc.cluster.local
  trafficPolicy:
    tls:
      mode: DISABLE

5. Increase Connection Timeouts

If the upstream is slow to accept connections:

# Envoy static config
clusters:
  - name: my_cluster
    connect_timeout: 10s
    type: STRICT_DNS
    load_assignment:
      cluster_name: my_cluster
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: my-service
                    port_value: 8080

In Istio, use a VirtualService timeout:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-service
spec:
  hosts:
    - my-service
  http:
    - timeout: 30s
      route:
        - destination:
            host: my-service
            port:
              number: 8080

6. Check Circuit Breaker Settings

Envoy may be rejecting connections because circuit breakers tripped:

# Check for overflow (circuit breaker trips)
curl -s http://localhost:15000/clusters | grep "overflow"

Increase limits if needed:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-service
spec:
  host: my-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1000
      http:
        h2UpgradePolicy: DEFAULT
        http1MaxPendingRequests: 1000
        http2MaxRequests: 1000
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s

7. Check Envoy Logs

# Standalone Envoy
docker logs envoy-container 2>&1 | grep "upstream connect error" | tail -20

# Istio sidecar
kubectl logs deploy/my-app -c istio-proxy | grep "UF\|UH\|UR" | tail -20

Response flags to look for: UF (upstream connection failure), UH (no healthy upstream), UR (upstream reset).

Prevention Tips

• Always name your ports in Kubernetes Services (e.g., name: http). Istio uses port names to determine the protocol.
• Set appropriate circuit breaker limits based on your traffic patterns, not the defaults.
• Use Istio's istioctl analyze to catch configuration issues before they hit production.
• Monitor Envoy metrics like envoy_cluster_upstream_cx_connect_fail and alert on sustained failures.
• Implement proper health check endpoints in all services so Envoy can detect unhealthy upstreams quickly.