OpenTelemetry — Traces, Metrics, and Logs for Distributed Systems

The Three Pillars of Observability

Logs:    discrete events — "Patient MRN-001 registered"
Metrics: numeric measurements over time — "API p99 latency = 142ms"
Traces:  end-to-end request path — "POST /api/patients took 340ms:
          ↳ Validation: 2ms
          ↳ DB: ExistsByMRN: 12ms
          ↳ DB: SaveChanges: 48ms
          ↳ Redis: InvalidateTag: 5ms"

OpenTelemetry is the standard for collecting all three. .NET Aspire wires it up automatically. In a standalone project, you configure it manually.

Installation

XML

<!-- ServiceDefaults or Api.csproj -->
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.EntityFrameworkCore" Version="1.*" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.*" />
<PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.*" />

ServiceDefaults — Aspire's Shared Configuration

// ServiceDefaults/Extensions.cs
public static IHostApplicationBuilder AddServiceDefaults(
    this IHostApplicationBuilder builder)
{
    builder.ConfigureOpenTelemetry();
    builder.AddDefaultHealthChecks();
    builder.Services.AddServiceDiscovery();
    return builder;
}

public static IHostApplicationBuilder ConfigureOpenTelemetry(
    this IHostApplicationBuilder builder)
{
    builder.Logging.AddOpenTelemetry(logging =>
    {
        logging.IncludeFormattedMessage = true;
        logging.IncludeScopes = true;
    });

    builder.Services.AddOpenTelemetry()
        .WithMetrics(metrics =>
        {
            metrics.AddAspNetCoreInstrumentation();
            metrics.AddHttpClientInstrumentation();
            metrics.AddRuntimeInstrumentation();
        })
        .WithTracing(tracing =>
        {
            tracing.AddAspNetCoreInstrumentation();
            tracing.AddHttpClientInstrumentation();
            tracing.AddEntityFrameworkCoreInstrumentation(ef =>
            {
                ef.SetDbStatementForText = true;   // include SQL in traces
            });
        });

    builder.AddOpenTelemetryExporters();
    return builder;
}

public static IHostApplicationBuilder AddOpenTelemetryExporters(
    this IHostApplicationBuilder builder)
{
    var useOtlpExporter = !string.IsNullOrWhiteSpace(
        builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]);

    if (useOtlpExporter)
        builder.Services.AddOpenTelemetry().UseOtlpExporter();

    return builder;
}

Adding Custom Spans to Handlers

// Application/Patients/Commands/CreatePatient/CreatePatientCommandHandler.cs
using System.Diagnostics;

public sealed class CreatePatientCommandHandler
{
    private static readonly ActivitySource _activitySource =
        new("SystemForge.Application");

    private readonly IPatientRepository _patients;
    private readonly IUnitOfWork _unitOfWork;

    public async Task<Result<PatientId>> Handle(
        CreatePatientCommand command, CancellationToken ct)
    {
        using var activity = _activitySource.StartActivity(
            "CreatePatient",
            ActivityKind.Internal);

        activity?.SetTag("patient.mrn", command.MRN);

        if (await _patients.ExistsByMRNAsync(command.MRN, ct))
        {
            activity?.SetStatus(ActivityStatusCode.Error, "MRN already exists");
            return Result.Failure<PatientId>(PatientErrors.MRNAlreadyExists);
        }

        var patientResult = Patient.Create(command.Name, command.DateOfBirth, command.MRN);
        if (patientResult.IsFailure)
        {
            activity?.SetStatus(ActivityStatusCode.Error, patientResult.Error.Description);
            return Result.Failure<PatientId>(patientResult.Error);
        }

        await _patients.AddAsync(patientResult.Value, ct);
        await _unitOfWork.SaveChangesAsync(ct);

        activity?.SetTag("patient.id", patientResult.Value.Id.Value.ToString());
        activity?.SetStatus(ActivityStatusCode.Ok);

        return Result.Success(patientResult.Value.Id);
    }
}

Custom Metrics

// Infrastructure/Telemetry/SystemForgeMeter.cs
using System.Diagnostics.Metrics;

public sealed class SystemForgeMeter : IDisposable
{
    private readonly Meter _meter;
    private readonly Counter<long> _patientsRegistered;
    private readonly Counter<long> _prescriptionsAdded;
    private readonly Histogram<double> _handlerDuration;

    public SystemForgeMeter()
    {
        _meter = new Meter("SystemForge.Application", "1.0.0");

        _patientsRegistered = _meter.CreateCounter<long>(
            "systemforge.patients.registered",
            description: "Number of patients registered");

        _prescriptionsAdded = _meter.CreateCounter<long>(
            "systemforge.prescriptions.added",
            description: "Number of prescriptions added");

        _handlerDuration = _meter.CreateHistogram<double>(
            "systemforge.handler.duration",
            unit: "ms",
            description: "Command/query handler execution time in milliseconds");
    }

    public void RecordPatientRegistered() =>
        _patientsRegistered.Add(1);

    public void RecordPrescriptionAdded(string medicationCode) =>
        _prescriptionsAdded.Add(1, new KeyValuePair<string, object?>("medication.code", medicationCode));

    public void RecordHandlerDuration(string handlerName, double milliseconds) =>
        _handlerDuration.Record(milliseconds,
            new KeyValuePair<string, object?>("handler", handlerName));

    public void Dispose() => _meter.Dispose();
}

// Registration
services.AddSingleton<SystemForgeMeter>();

// Usage in handler
_meter.RecordPatientRegistered();

EF Core Query Tracing

With SetDbStatementForText = true, every EF Core query appears as a child span in the trace:

POST /api/patients
  └── SystemForge.Application/CreatePatient (2ms)
        └── sql: SELECT COUNT(*) FROM Patients WHERE MRN = 'MRN-001' (8ms)
        └── sql: INSERT INTO Patients (...) VALUES (...) (31ms)
        └── sql: UPDATE ... (saved domain events) (4ms)

Production issue I've seen: A team was experiencing slow API responses but could not identify where the time was going. After enabling EF Core OpenTelemetry instrumentation, they discovered one handler was executing 47 separate SQL queries (N+1 problem — iterating through patients and loading prescriptions one-by-one). The trace made it obvious. The fix was a single Include(p => p.Prescriptions).

Aspire Dashboard

When running with .NET Aspire, the Aspire Dashboard (accessible at http://localhost:18888) shows:

✓ All traces with waterfall view (nested spans, timing, SQL queries)
✓ All metrics (request rate, error rate, p50/p95/p99 latency)
✓ Structured logs with correlation (click a trace → see all logs for that request)
✓ Resource health (which services are running, which have errors)

No extra configuration needed — Aspire's OTLP collector receives everything from the application automatically.

Production Export

// For production, export to a real collector (Jaeger, Datadog, Azure Monitor)
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing.AddAspNetCoreInstrumentation();
        tracing.AddEntityFrameworkCoreInstrumentation();

        if (builder.Environment.IsProduction())
        {
            tracing.AddOtlpExporter(otlp =>
            {
                otlp.Endpoint = new Uri(configuration["OpenTelemetry:Endpoint"]!);
                otlp.Headers  = $"api-key={configuration["OpenTelemetry:ApiKey"]}";
            });
        }
        else
        {
            tracing.AddConsoleExporter();
        }
    });

PRO TIP — Sampling

In production with high traffic, you do not want to export 100% of traces — storage costs and overhead add up. Configure sampling to capture all errors and a percentage of successful requests:

tracing.SetSampler(new ParentBasedSampler(
    new TraceIdRatioBasedSampler(0.1)));   // 10% of traces
// Errors are always sampled regardless of the rate

Key Takeaway

OpenTelemetry gives you the three signals — logs, metrics, traces — through a single standardized API. Traces are the most powerful: they show you the exact path of every request, including every database query, every cache hit, and every downstream HTTP call. When a patient registration is slow, you do not guess — you look at the trace and see exactly which operation took how long.