Distributed Tracing with Jaeger: Pinpointing Latency Bottlenecks in Microservices

Microservices give you deployment flexibility and team autonomy, but they'll absolutely destroy your ability to debug latency issues if you don't have the right observability tooling in place. When a request touches eight services before returning a response, "the API is slow" tells you nothing. Distributed tracing tells you everything.

Jaeger is my go-to open-source tracing solution. It's battle-tested, integrates cleanly with OpenTelemetry, and the UI is genuinely useful for flame-graph style trace analysis. Let's walk through setting it up, instrumenting your services, and — more importantly — actually using it to find where your latency is hiding.

Why Distributed Tracing, Not Just Logs

Before we jump into code, let me make the case clearly. You already have logs. You probably have metrics. Why do you need tracing?

Logs are isolated per service. Metrics show you that something is slow. Traces show you where and why. A trace follows a single request through every service it touches, recording timing data at each hop. When your P99 latency spikes, traces let you drill into a specific slow request and see that order-service waited 800ms for a database query while everything else completed in under 50ms.

That's the difference between hours of debugging and minutes.

Standing Up Jaeger Locally

For development, Docker Compose is the fastest path to a working Jaeger instance:

version: '3.8'
services:
  jaeger:
    image: jaegertracing/all-in-one:1.52
    ports:
      - "16686:16686"   # Jaeger UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
      - "6831:6831/udp" # Jaeger Thrift compact
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    networks:
      - observability

networks:
  observability:
    driver: bridge

The all-in-one image bundles the collector, query service, and UI — perfect for development and testing. In production you'll want these as separate deployments with a proper backend store (Elasticsearch or Cassandra) instead of the default in-memory storage.

Hit http://localhost:16686 after startup and you'll have the Jaeger UI ready to receive traces.

Instrumenting a Go Service with OpenTelemetry

I strongly recommend instrumenting via OpenTelemetry rather than Jaeger's native client libraries. OTel is vendor-neutral — if you ever swap Jaeger for Tempo or Honeycomb, your instrumentation code doesn't change. Only the exporter configuration does.

Here's a complete tracer setup for a Go service:

package telemetry

import (
    "context"
    "fmt"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
    "google.golang.org/grpc"
)

func InitTracer(serviceName, jaegerEndpoint string) (*sdktrace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(
        context.Background(),
        otlptracegrpc.WithEndpoint(jaegerEndpoint),
        otlptracegrpc.WithInsecure(), // Use WithTLSClientConfig in production
        otlptracegrpc.WithDialOption(grpc.WithBlock()),
    )
    if err != nil {
        return nil, fmt.Errorf("failed to create exporter: %w", err)
    }

    res, err := resource.New(
        context.Background(),
        resource.WithAttributes(
            semconv.ServiceName(serviceName),
            semconv.ServiceVersion("1.0.0"),
            semconv.DeploymentEnvironment("production"),
        ),
    )
    if err != nil {
        return nil, fmt.Errorf("failed to create resource: %w", err)
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.1)), // 10% sampling in prod
    )

    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(
        propagation.NewCompositeTextMapPropagator(
            propagation.TraceContext{},
            propagation.Baggage{},
        ),
    )

    return tp, nil
}

Notice the TraceIDRatioBased(0.1) sampler — in production at scale, sampling 100% of requests is expensive. Ten percent gives you statistically significant data without the storage overhead. For debugging a specific issue, you can bump this to 1.0 temporarily.

Creating Meaningful Spans

Initializing the tracer is the easy part. The value comes from creating spans that tell a coherent story. Here's how to instrument an HTTP handler with meaningful child spans:

package handlers

import (
    "net/http"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
)

var tracer = otel.Tracer("order-service")

func CreateOrderHandler(w http.ResponseWriter, r *http.Request) {
    ctx, span := tracer.Start(r.Context(), "CreateOrder")
    defer span.End()

    // Add contextual attributes — these are searchable in Jaeger UI
    span.SetAttributes(
        attribute.String("user.id", r.Header.Get("X-User-ID")),
        attribute.String("http.method", r.Method),
        attribute.String("http.route", "/orders"),
    )

    orderID, err := processOrder(ctx, r)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        http.Error(w, "failed to create order", http.StatusInternalServerError)
        return
    }

    span.SetAttributes(attribute.String("order.id", orderID))
    w.WriteHeader(http.StatusCreated)
}

func processOrder(ctx context.Context, r *http.Request) (string, error) {
    // Each logical operation gets its own span
    ctx, dbSpan := tracer.Start(ctx, "ValidateInventory")
    if err := checkInventory(ctx); err != nil {
        dbSpan.RecordError(err)
        dbSpan.End()
        return "", err
    }
    dbSpan.End()

    ctx, paymentSpan := tracer.Start(ctx, "ProcessPayment")
    orderID, err := callPaymentService(ctx)
    if err != nil {
        paymentSpan.RecordError(err)
        paymentSpan.End()
        return "", err
    }
    paymentSpan.End()

    return orderID, nil
}

The key discipline here: span names should be operation names, not implementation details. ValidateInventory is good. checkInventory_v2_new is noise. You want trace UIs that read like a business process, not a call stack.

Propagating Context Across Service Boundaries

Here's where most teams get tripped up. Spans only form a connected trace if context is correctly propagated between services. For HTTP calls between services, you need to inject trace headers into outgoing requests and extract them on the receiving end.

Outgoing HTTP client with context propagation:

package clients

import (
    "context"
    "net/http"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

// Wrap your HTTP client with OTel instrumentation
func NewInstrumentedClient() *http.Client {
    return &http.Client{
        Transport: otelhttp.NewTransport(http.DefaultTransport),
    }
}

// Usage — context carries the trace, OTel handles header injection automatically
func CallInventoryService(ctx context.Context, productID string) error {
    client := NewInstrumentedClient()
    req, err := http.NewRequestWithContext(
        ctx,
        http.MethodGet,
        "http://inventory-service/products/"+productID,
        nil,
    )
    if err != nil {
        return err
    }

    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    return nil
}

On the server side, use otelhttp.NewHandler to automatically extract incoming trace context:

mux := http.NewServeMux()
mux.HandleFunc("/products/", InventoryHandler)

// This middleware extracts trace context from incoming requests
handler := otelhttp.NewHandler(mux, "inventory-service")
http.ListenAndServe(":8080", handler)

The otelhttp instrumentation library handles traceparent header injection and extraction automatically. Don't try to do this manually — you'll get it wrong and end up with disconnected traces.

Reading Traces and Identifying Bottlenecks

Now the fun part. Once traces are flowing into Jaeger, here's how to systematically hunt latency:

1. Use the search to find slow traces first. In the Jaeger UI, filter by service and set a minimum duration threshold. If your SLA is 500ms, start by looking at traces over 1 second. Don't start with average traces — outliers reveal the worst problems.

2. Look for sequential calls that could be parallel. If you see this pattern in a trace:

[OrderService - CreateOrder: 900ms]
  ├─ [InventoryService - Check: 300ms]
  ├─ [UserService - GetProfile: 250ms]
  └─ [PricingService - Calculate: 350ms]

Those three child calls are executing sequentially when they don't need to be. Parallelizing them drops your total latency to roughly 350ms (the slowest of the three). This is one of the highest-ROI findings distributed tracing surfaces.

3. Watch for "wide" database spans. A database query span taking 500ms is almost always an N+1 query or a missing index. The trace tells you exactly which service and which operation — you don't have to guess.

4. Identify retry storms. If you see the same span repeating multiple times with short gaps, you're looking at a retry loop. Traces make this blindingly obvious in a way that metrics dashboards often obscure.

Production Considerations

A few things I've learned the hard way:

Use a proper backend store. Jaeger's in-memory store loses data on restart. For production, deploy with Elasticsearch. The Jaeger Helm chart makes this straightforward:

helm install jaeger jaegertracing/jaeger \
  --set storage.type=elasticsearch \
  --set storage.elasticsearch.host=elasticsearch-master \
  --set collector.replicaCount=2

Set retention policies. Traces are high-cardinality data. 7-14 days of retention is usually sufficient — you're not doing historical trend analysis with traces, you're debugging active or recent incidents.

Sample intelligently. Consider using tail-based sampling for production workloads. With head-based sampling (what we configured above), you sample randomly at trace ingestion. Tail-based sampling evaluates the completed trace and keeps it only if something interesting happened (errors, high latency). The Jaeger Collector supports tail-based sampling configuration.

Add baggage for user context. OpenTelemetry baggage lets you propagate key-value pairs through the entire trace without manually adding them to every span. User IDs, tenant IDs, and feature flag states belong in baggage — it makes filtering traces in production dramatically easier.

The Payoff

I've used distributed tracing to find a payment service that was making a synchronous database call for every line item in an order (classic N+1, surfaced in 10 minutes with tracing, would have taken days with logs). I've found a service that was waiting for a downstream timeout on every request because someone had set http.DefaultClient with no timeout. I've caught circular dependencies between services that were adding 400ms of latency on every request.

The setup cost is maybe half a day for a mature microservices platform. The return on that investment shows up the first time you're on-call at 2am and you can tell your team "it's the inventory service database query on /orders/checkout" within three minutes of the alert firing.

That's what distributed tracing is for. Jaeger makes it approachable and production-ready without locking you into a vendor. Get it instrumented, get it deployed, and stop guessing where your latency lives.