KubernetesHigh
Fix: Kubernetes Service Not Reachable
curl to ClusterIP service returns 'Connection refused' or times out
!Symptoms
- curl to ClusterIP service returns 'Connection refused' or times out
- Service name does not resolve inside the cluster
- External LoadBalancer IP shows no response
- Ingress returns 404 or 503 for the configured path
?Root Causes
- Service selector labels do not match pod labels
- Target port in service does not match container port
- No endpoints: pods behind the service are not ready
- Network policy blocking traffic between namespaces or pods
- Ingress controller not configured or misconfigured
#Diagnosis Steps
- 1Check service endpoints: `kubectl get endpoints <service-name>`
- 2Verify labels match: compare `kubectl get svc <name> -o yaml` selector with pod labels
- 3Check pod readiness: `kubectl get pods -l <selector>` — all should be Running and Ready
- 4Test from inside the cluster: `kubectl run tmp --rm -i --image=busybox -- wget -qO- http://<service>:<port>`
- 5Check network policies: `kubectl get networkpolicy -n <namespace>`
>Fix
- 1Fix label mismatch between service selector and pod template labels
- 2Correct the targetPort in the service spec to match the container port
- 3Fix failing readiness probes so pods become Ready endpoints
- 4Update or remove NetworkPolicies that block the required traffic
- 5For Ingress, verify the backend service name and port number
*Prevention
- Use consistent labeling conventions across deployments and services
- Always define readiness probes for pods behind services
- Test service connectivity as part of deployment verification
- Document network policy requirements for each service
- Use service mesh (Istio, Linkerd) for better observability of service traffic
Related Error Messages
Connection refusedno endpoints availableservice has no endpointsupstream connect error or disconnect/reset before headers503 Service Temporarily Unavailable