Distributed Tracing Basics — OpenTelemetry in .NET

When a user reports that a request was slow, how do you find the bottleneck? In a monolith, you check a single log file. In microservices, the request might touch 5–10 services. Distributed tracing is how you follow a single request across all of them.

What is Distributed Tracing?

A trace represents a single end-to-end request. It's made up of spans — units of work within that request.

Trace: "POST /api/checkout" — 450ms total
  ├── Span: OrderService.PlaceOrder — 380ms
  │     ├── Span: InventoryService.Reserve — 120ms
  │     ├── Span: PaymentService.Charge — 200ms
  │     │     └── Span: Stripe API call — 180ms
  │     └── Span: DB.Insert(orders) — 25ms
  └── Span: NotificationService.Send — 70ms (parallel)

Each span has:

A name
Start time and duration
Parent span ID (to build the tree)
Tags (key-value metadata)
Status (OK, Error)

The trace ID is propagated between services via HTTP headers — every service that participates adds its spans to the same trace.

OpenTelemetry

OpenTelemetry (OTel) is the open standard for observability data — traces, metrics, and logs. It provides vendor-neutral instrumentation: you instrument once with OTel, then send data to any backend (Jaeger, Zipkin, Azure Monitor, Datadog, Grafana Tempo).

The alternative is vendor-specific SDKs (Application Insights SDK, Datadog tracer). OTel avoids lock-in.

Adding OpenTelemetry to a .NET API

Bash

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Instrumentation.EntityFrameworkCore
dotnet add package OpenTelemetry.Exporter.Console      # dev: print to console
dotnet add package OpenTelemetry.Exporter.Jaeger       # optional: Jaeger
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol  # OTLP: Azure/Grafana

// Program.cs
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .SetResourceBuilder(ResourceBuilder.CreateDefault()
                .AddService("OrderService", serviceVersion: "1.0"))
            .AddAspNetCoreInstrumentation(options =>
            {
                options.RecordException = true;
            })
            .AddHttpClientInstrumentation()
            .AddEntityFrameworkCoreInstrumentation(options =>
            {
                options.SetDbStatementForText = true;
            })
            .AddSource("OrderService")  // custom spans (see below)
            .AddConsoleExporter()       // dev: view in terminal
            .AddJaegerExporter();       // send to Jaeger
    });

That's the minimum to get automatic tracing for:

All incoming HTTP requests (ASP.NET Core)
All outgoing HTTP requests (HttpClient)
All EF Core database queries

Custom Spans

Automatic instrumentation gives you the outer shell. Add custom spans for important business operations.

using System.Diagnostics;

public class OrderService
{
    // One ActivitySource per component — reuse it
    private static readonly ActivitySource _activitySource = new("OrderService");

    private readonly IInventoryClient _inventory;
    private readonly IPaymentClient _payment;

    public async Task<Order> PlaceOrderAsync(PlaceOrderRequest request)
    {
        // Start a custom span — automatically parented to current span
        using var activity = _activitySource.StartActivity("PlaceOrder");

        // Add tags: key business context on the span
        activity?.SetTag("order.customer_id", request.CustomerId);
        activity?.SetTag("order.item_count", request.Items.Count);
        activity?.SetTag("order.total_amount", request.TotalAmount);

        try
        {
            var reservation = await _inventory.ReserveAsync(request.Items);
            activity?.SetTag("order.reservation_id", reservation.Id);

            var payment = await _payment.ChargeAsync(request.PaymentMethod, request.TotalAmount);
            activity?.SetTag("order.payment_id", payment.Id);

            var order = await _repository.CreateAsync(request, reservation, payment);
            activity?.SetTag("order.id", order.Id);

            return order;
        }
        catch (Exception ex)
        {
            // Mark span as failed — shows red in trace viewer
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            throw;
        }
    }
}

The key principle: add tags for the IDs and values you'd want to search for later. If the trace shows a slow span, you want to immediately see which order, which customer, which payment was affected.

Trace Context Propagation

For tracing to work across services, the trace ID must be passed between them. OpenTelemetry does this automatically for HttpClient — it adds traceparent and tracestate headers.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             version  trace-id (16 bytes)               span-id       flags

If you're calling another service, just use HttpClient — OTel handles the headers. The receiving service needs OTel too (it reads the header and continues the trace).

// Service A: sends HTTP request
var response = await _httpClient.GetAsync("/api/inventory/reserve");
// OTel automatically adds traceparent header

// Service B: receives the request
// OTel automatically reads traceparent and continues the trace
// All spans in Service B appear under the same trace in Jaeger

Running Jaeger Locally

Jaeger is the simplest way to view traces during development — it runs in a single Docker container.

Bash

docker run -d --name jaeger \
  -p 16686:16686 \   # Jaeger UI
  -p 14268:14268 \   # Jaeger HTTP collector
  jaegertracing/all-in-one:latest

Open http://localhost:16686 to see the trace UI.

Configure your app to export to Jaeger:

.AddJaegerExporter(options =>
{
    options.AgentHost = "localhost";
    options.AgentPort = 6831;
})

Or use the OTLP exporter (recommended — works with most backends):

.AddOtlpExporter(options =>
{
    options.Endpoint = new Uri("http://localhost:4317");
})

Reading a Trace

In Jaeger, select your service name and click "Find Traces". Each trace shows as a horizontal timeline:

[OrderService] POST /api/checkout                          450ms
  [OrderService] PlaceOrder                               380ms
    [InventoryService] POST /api/inventory/reserve         120ms
    [PaymentService]   POST /api/payments/charge           200ms
      [PaymentService] Stripe.CreateCharge                 180ms
    [OrderService]    orders INSERT                         25ms
  [NotificationService] POST /api/notifications/send       70ms

A wide span = slow operation. Red span = error. This view immediately shows you where the 450ms went.

Correlating Traces with Logs

Link your logs to the current trace so you can jump from a log line to the trace:

// Serilog enricher that adds trace ID to every log line
builder.Host.UseSerilog((context, services, config) =>
{
    config
        .Enrich.WithProperty("TraceId", Activity.Current?.TraceId.ToString())
        .Enrich.WithProperty("SpanId", Activity.Current?.SpanId.ToString())
        // ... other config
});

Now every log line has the trace ID. In your log viewer, click the trace ID to jump to the full trace in Jaeger. Or in Jaeger, find the trace ID and search for it in your log viewer.

Sampling

Tracing every request in production has CPU and storage cost. Use sampling to reduce volume:

.AddAspNetCoreInstrumentation()
.SetSampler(new TraceIdRatioBasedSampler(0.1))  // sample 10% of traces

For production, 1–10% sampling is typical. Always sample:

Requests that resulted in errors (ParentBasedSampler with error-biased logic)
Slow requests above a threshold
All requests in development (AlwaysOnSampler)

Summary

A trace = one end-to-end request; a span = one operation within it
OpenTelemetry is vendor-neutral — instrument once, export anywhere
Automatic instrumentation covers HTTP, HttpClient, and EF Core with 5 lines of setup
Add custom spans for important business operations using ActivitySource
Trace context propagates automatically via traceparent header in HttpClient
Run Jaeger locally in Docker for development trace viewing
Enrich logs with TraceId to link logs and traces

Distributed Tracing Basics — OpenTelemetry in .NET

Distributed Tracing Basics — OpenTelemetry in .NET

What is Distributed Tracing?

OpenTelemetry

Adding OpenTelemetry to a .NET API

Custom Spans

Trace Context Propagation

Running Jaeger Locally

Reading a Trace

Correlating Traces with Logs

Sampling

Summary

Enjoyed this article?

Leave a comment