Istio Pilot Discovery xDS Push Errors: Fixing "No Healthy Upstream" in Multi-Cluster Mesh Deployments

Quick reference for debugging no healthy upstream errors caused by Pilot xDS push failures in multi-cluster Istio setups.

---

The Problem at a Glance

You're running a multi-cluster Istio mesh and services start returning 503 no healthy upstream. The root cause is usually Pilot (istiod) failing to push xDS configuration to Envoy sidecars — endpoints go stale, Envoy thinks there's nowhere to route traffic.

---

Quick Diagnostic Commands

Run these first, in this order:

# Check istiod health across clusters
kubectl get pods -n istio-system -l app=istiod --context cluster1
kubectl get pods -n istio-system -l app=istiod --context cluster2

# Check xDS push errors in istiod logs
kubectl logs -n istio-system -l app=istiod --context cluster1 | grep -E "pushError|NoPush|rejected"

# Check Envoy's endpoint state for a specific service
istioctl proxy-config endpoints <pod-name>.<namespace> --cluster "outbound|80||my-service.default.svc.cluster.local"

# Check for EDS (Endpoint Discovery) rejections
istioctl proxy-config clusters <pod-name>.<namespace> | grep -i "my-service"

---

Common Root Causes & Fixes

1. Remote Secret Missing or Stale

The most common culprit in multi-cluster setups.

# Verify remote secrets exist on both clusters
kubectl get secrets -n istio-system --context cluster1 | grep "istio-remote-secret"
kubectl get secrets -n istio-system --context cluster2 | grep "istio-remote-secret"

# Recreate the remote secret if missing or stale
istioctl create-remote-secret \
  --context=cluster2 \
  --name=cluster2 | kubectl apply -f - --context=cluster1

2. Endpoint Not Propagating (EDS Issue)

# Dump full endpoint table from istiod
istioctl proxy-config endpoints <pod>.<ns> --context cluster1

# Force istiod to re-sync endpoints
kubectl rollout restart deployment/istiod -n istio-system --context cluster1

# Check if service entry is missing for remote cluster services
kubectl get serviceentries -A --context cluster1

3. xDS Push Backpressure / Rate Limiting

When istiod is overwhelmed, it drops or delays pushes.

# Check istiod push queue metrics
kubectl exec -n istio-system deployment/istiod -- \
  curl -s localhost:15014/metrics | grep -E "pilot_xds_push|pilot_push_trigger"

# Look for push errors specifically
kubectl exec -n istio-system deployment/istiod -- \
  curl -s localhost:15014/metrics | grep "pilot_xds_push_errors"

If pilot_xds_push_errors is climbing, increase istiod resources:

# istiod resource patch
apiVersion: apps/v1
kind: Deployment
metadata:
  name: istiod
  namespace: istio-system
spec:
  template:
    spec:
      containers:
      - name: discovery
        resources:
          requests:
            cpu: "500m"
            memory: "2Gi"
          limits:
            cpu: "2"
            memory: "4Gi"

---

Envoy-Side Debugging

# Check Envoy's live cluster health directly
kubectl exec <pod-name> -c istio-proxy -- \
  curl -s localhost:15000/clusters | grep "my-service"

# Look for cx_connect_fail or cx_active = 0
kubectl exec <pod-name> -c istio-proxy -- \
  curl -s localhost:15000/clusters | grep -A5 "my-service"

# Check Envoy config dump for CDS/EDS entries
kubectl exec <pod-name> -c istio-proxy -- \
  curl -s localhost:15000/config_dump | python3 -m json.tool | grep -A10 "my-service"

---

DestinationRule Gotchas in Multi-Cluster

A misconfigured DestinationRule is a silent killer.

# BAD: outlierDetection too aggressive kicks out healthy endpoints
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-service
spec:
  host: my-service.default.svc.cluster.local
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 1      # Too aggressive
      interval: 10s
      baseEjectionTime: 5m         # Too long

---
# GOOD: Sane defaults for multi-cluster
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-service
spec:
  host: my-service.default.svc.cluster.local
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50       # Never eject all endpoints

---

ServiceEntry for Cross-Cluster Visibility

If a service only exists in one cluster but needs to be routable from another:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: remote-my-service
  namespace: default
spec:
  hosts:
  - my-service.default.svc.cluster.local
  location: MESH_INTERNAL
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  endpoints:
  - address: my-service.default.svc.cluster2.local
    locality: us-east1/us-east1-b

---

Healthcheck Commands Cheat Sheet

What to Check	Command
istiod xDS push errors	kubectl logs -n istio-system -l app=istiod | grep pushError
Envoy cluster health	curl localhost:15000/clusters (from sidecar)
Endpoint table	istioctl proxy-config endpoints <pod>
Proxy sync status	istioctl proxy-status
Full config dump	istioctl proxy-config all <pod> -o json
Istiod debug endpoint	kubectl exec deploy/istiod -- curl localhost:15014/debug/endpointz

---

Fast Recovery Playbook

1. Verify remote secrets — recreate if anything looks off
2. Check istioctl proxy-status — look for STALE next to any proxy
3. Inspect outlierDetection — loosen ejection thresholds temporarily
4. Restart istiod — clears push queue buildup: kubectl rollout restart deployment/istiod -n istio-system
5. Restart affected pods — forces fresh xDS connection and endpoint registration

# Nuclear option: restart everything in the affected namespace
kubectl rollout restart deployment -n <your-namespace> --context cluster1

---

Bottom line: Most no healthy upstream errors in multi-cluster Istio trace back to stale endpoint data from failed xDS pushes. Start with istioctl proxy-status, look for STALE proxies, and work backwards from there.