Fix: Envoy Proxy 503 No Healthy Upstream

!Symptoms

• All requests returning 503 Service Unavailable
• Envoy logs show 'no healthy upstream'
• Health checks failing for all endpoints
• Service mesh traffic completely broken

?Root Causes

• All backend pods failed health checks simultaneously
• Health check configuration too aggressive (low timeout, high threshold)
• Backend service port mismatch in Envoy cluster configuration
• DNS resolution failing for upstream service names
• mTLS misconfiguration — Envoy can't establish TLS with backends

#Diagnosis Steps

1. 1Check Envoy admin endpoint: `curl localhost:15000/clusters` — look for health flags
2. 2Check upstream health: `curl localhost:15000/clusters | grep health_flags`
3. 3Review Envoy access logs for upstream connection errors
4. 4Verify backend pods are running and passing their own health checks
5. 5Check Envoy config dump: `curl localhost:15000/config_dump`

>Fix

1. 1Fix backend service health — restart unhealthy pods or fix the underlying issue
2. 2Relax health check parameters: increase timeout and unhealthy_threshold
3. 3Fix port mismatch between Envoy cluster config and backend service
4. 4Fix DNS: ensure CoreDNS is running and service names resolve
5. 5Fix mTLS certificates: regenerate or rotate if expired

*Prevention

• Set reasonable health check intervals and thresholds
• Use outlier detection instead of aggressive health checks
• Monitor upstream health metrics in your dashboards
• Implement circuit breaking to handle partial failures gracefully
• Test health check configuration changes in staging

Related Error Messages