DevOpsil
KubernetesHigh

Fix: Kubernetes Service Not Reachable

curl to ClusterIP service returns 'Connection refused' or times out

!Symptoms

  • curl to ClusterIP service returns 'Connection refused' or times out
  • Service name does not resolve inside the cluster
  • External LoadBalancer IP shows no response
  • Ingress returns 404 or 503 for the configured path

?Root Causes

  • Service selector labels do not match pod labels
  • Target port in service does not match container port
  • No endpoints: pods behind the service are not ready
  • Network policy blocking traffic between namespaces or pods
  • Ingress controller not configured or misconfigured

#Diagnosis Steps

  1. 1Check service endpoints: `kubectl get endpoints <service-name>`
  2. 2Verify labels match: compare `kubectl get svc <name> -o yaml` selector with pod labels
  3. 3Check pod readiness: `kubectl get pods -l <selector>` — all should be Running and Ready
  4. 4Test from inside the cluster: `kubectl run tmp --rm -i --image=busybox -- wget -qO- http://<service>:<port>`
  5. 5Check network policies: `kubectl get networkpolicy -n <namespace>`

>Fix

  1. 1Fix label mismatch between service selector and pod template labels
  2. 2Correct the targetPort in the service spec to match the container port
  3. 3Fix failing readiness probes so pods become Ready endpoints
  4. 4Update or remove NetworkPolicies that block the required traffic
  5. 5For Ingress, verify the backend service name and port number

*Prevention

  • Use consistent labeling conventions across deployments and services
  • Always define readiness probes for pods behind services
  • Test service connectivity as part of deployment verification
  • Document network policy requirements for each service
  • Use service mesh (Istio, Linkerd) for better observability of service traffic

Related Error Messages

Connection refusedno endpoints availableservice has no endpointsupstream connect error or disconnect/reset before headers503 Service Temporarily Unavailable