Istio Observability and Authorization: Distributed Tracing, Metrics, and Access Policies

Istio's Observability Is Automatic

Every Envoy sidecar in your mesh emits three types of telemetry without any application code changes:

1. Metrics: Per-service request rates, error rates, latency (in Prometheus format)
2. Distributed traces: End-to-end request traces across service boundaries (Jaeger/Zipkin)
3. Access logs: Per-request structured logs from every sidecar

This is the main operational value of Istio beyond mTLS — you get a service topology and performance data for every service in your cluster for free.

---

Prometheus Metrics

Istio exports metrics automatically to Prometheus. The key metrics:

Metric	Description
istio_requests_total	Total request count with labels for source, destination, method, status
istio_request_duration_milliseconds	Request latency histogram
istio_request_bytes	Request body size
istio_response_bytes	Response body size
istio_tcp_connections_opened_total	TCP connection count

Querying Istio Metrics

# Request rate per second for a service
rate(istio_requests_total{destination_service="myapp.default.svc.cluster.local"}[5m])

# 99th percentile latency for a service
histogram_quantile(0.99,
  sum(rate(istio_request_duration_milliseconds_bucket{
    destination_service="myapp.default.svc.cluster.local"
  }[5m])) by (le)
)

# Error rate (5xx responses) per service
sum(rate(istio_requests_total{
  destination_service="myapp.default.svc.cluster.local",
  response_code=~"5.."
}[5m])) / 
sum(rate(istio_requests_total{
  destination_service="myapp.default.svc.cluster.local"
}[5m]))

# Top services by request rate
topk(10, sum(rate(istio_requests_total[5m])) by (destination_service))

Custom Prometheus Rules

# prometheusrule.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: istio-slos
  namespace: monitoring
spec:
  groups:
    - name: istio.slos
      interval: 30s
      rules:
        - alert: ServiceHighErrorRate
          expr: |
            sum(rate(istio_requests_total{response_code=~"5.."}[5m])) by (destination_service)
            /
            sum(rate(istio_requests_total[5m])) by (destination_service)
            > 0.01
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "High error rate on {{ $labels.destination_service }}"
            description: "Error rate is {{ $value | humanizePercentage }}"

---

Distributed Tracing

Istio propagates trace context headers automatically, but your application must forward the following headers when calling downstream services:

x-request-id
x-b3-traceid
x-b3-spanid
x-b3-parentspanid
x-b3-sampled
x-b3-flags
b3

Example in Node.js:

app.get('/api/orders', async (req, res) => {
  const traceHeaders = {
    'x-request-id': req.headers['x-request-id'],
    'x-b3-traceid': req.headers['x-b3-traceid'],
    'x-b3-spanid': req.headers['x-b3-spanid'],
    'x-b3-parentspanid': req.headers['x-b3-parentspanid'],
    'x-b3-sampled': req.headers['x-b3-sampled'],
  };

  // Forward headers to downstream service
  const items = await axios.get('http://inventory-service/items', {
    headers: traceHeaders
  });
  
  res.json(items.data);
});

Configuring Trace Sampling

By default Istio samples 1% of traces. Adjust in the Istio config:

# Patch istiod to change sampling rate
kubectl patch configmap istio -n istio-system --type merge \
  -p '{"data":{"mesh":"defaultConfig:\n  tracing:\n    sampling: 10.0\n"}}'

# Or via IstioOperator
kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  meshConfig:
    defaultConfig:
      tracing:
        sampling: 10.0    # 10% sampling
    extensionProviders:
      - name: jaeger
        opentelemetry:
          service: jaeger-collector.istio-system.svc.cluster.local
          port: 4317
EOF

---

Kiali: Service Topology

Kiali provides a visual service graph showing:

• Which services call which
• Request rates and error rates on each edge
• mTLS status (lock icon)
• Circuit breaker status

# Access Kiali
kubectl port-forward -n istio-system svc/kiali 20001:20001
# Open: http://localhost:20001

# Or expose via Istio Gateway
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kiali
  namespace: istio-system
spec:
  hosts: ["kiali.example.com"]
  gateways: ["istio-ingressgateway"]
  http:
    - route:
        - destination:
            host: kiali
            port:
              number: 20001
EOF

---

AuthorizationPolicy: Zero-Trust Access Control

AuthorizationPolicy lets you define exactly which services can call which services — enforced by Envoy sidecars, not application code.

Default Deny All

# deny-all.yaml — deny all traffic in namespace
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: default
spec: {}    # empty spec = deny everything

Allow Specific Services

# allow-frontend-to-backend.yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-frontend
  namespace: default
spec:
  selector:
    matchLabels:
      app: backend    # applies to pods with this label
  rules:
    - from:
        - source:
            principals:
              # Only allow the frontend service account
              - "cluster.local/ns/default/sa/frontend"
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/api/*"]

Allow by Namespace

# Allow all services in the 'monitoring' namespace to scrape metrics
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-monitoring
  namespace: default
spec:
  selector:
    matchLabels:
      app: myapp
  rules:
    - from:
        - source:
            namespaces: ["monitoring"]
      to:
        - operation:
            paths: ["/metrics"]
            methods: ["GET"]

JWT-Based Authorization

# Require a valid JWT for API access
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-auth
  namespace: default
spec:
  selector:
    matchLabels:
      app: myapp
  jwtRules:
    - issuer: "https://auth.example.com"
      jwksUri: "https://auth.example.com/.well-known/jwks.json"

---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-jwt
  namespace: default
spec:
  selector:
    matchLabels:
      app: myapp
  rules:
    - from:
        - source:
            requestPrincipals: ["https://auth.example.com/*"]
      to:
        - operation:
            methods: ["GET"]
            paths: ["/api/v1/*"]

---

Egress Control

By default Istio blocks outbound traffic to external services. Configure ServiceEntry to allow it:

# Allow outbound access to an external API
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-api
  namespace: default
spec:
  hosts:
    - api.external-service.com
  ports:
    - number: 443
      name: https
      protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL

---
# Apply TLS to the egress traffic
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: external-api-tls
  namespace: default
spec:
  host: api.external-service.com
  trafficPolicy:
    tls:
      mode: SIMPLE    # standard TLS (not mTLS)

---

Istio Upgrade

# Check current version
istioctl version

# Download new istioctl
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.24.1 sh -

# In-place upgrade (canary upgrade recommended for production)
istioctl upgrade

# Canary upgrade (runs new control plane alongside old)
istioctl install --set revision=1-24-1 --set profile=default

# Migrate namespaces to new revision
kubectl label namespace default istio.io/rev=1-24-1 istio-injection-

# Restart pods to pick up new sidecar
kubectl rollout restart deployment -n default

# Remove old control plane once all pods migrated
istioctl uninstall --revision default