.NET & C# Development · Lesson 217 of 229
Microservices in .NET — Inter-Service Communication, API Gateway & Distributed Tracing
Microservices in .NET — Inter-Service Communication, API Gateway, and Distributed Tracing
Microservices solve real problems — independent deployability, team autonomy, targeted scaling — but they introduce an entirely new category of failure modes: network timeouts, partial availability, message ordering, and distributed debugging. This article covers the mechanics that make .NET microservices reliable in production.
What you'll build:
- Typed HTTP clients with Polly retry + circuit breaker
- gRPC service contracts with strongly-typed clients
- Event-driven messaging with MassTransit and RabbitMQ
- YARP API Gateway with route-based forwarding
- OpenTelemetry distributed tracing across services
- Health checks with dependency awareness
Architecture used throughout:
┌─────────────────────────────────────┐
│ YARP API Gateway │
│ (auth, routing, rate limiting) │
└────┬─────────────┬──────────────────┘
│ │
┌──────────┘ ┌─────┘
▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Orders │ │ Inventory │ │ Payments │
│ Service │ │ Service │ │ Service │
│ :5001 │ │ :5002 │ │ :5003 │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
└──────────────────┴───────────────────┘
│ RabbitMQ
(domain events)1. Typed HTTP Clients with Resilience
The wrong way to call another service is new HttpClient() inside a method. The right way is a typed client registered in DI, wrapped with Polly policies.
Define the client interface
// Services/Clients/IInventoryClient.cs
public interface IInventoryClient
{
Task<StockLevelDto?> GetStockAsync(string sku, CancellationToken ct = default);
Task<bool> ReserveStockAsync(ReserveStockRequest request, CancellationToken ct = default);
}
public record StockLevelDto(string Sku, int Available, int Reserved);
public record ReserveStockRequest(string Sku, int Quantity, string OrderId);Implement the typed client
// Services/Clients/InventoryClient.cs
public class InventoryClient : IInventoryClient
{
private readonly HttpClient _http;
private readonly ILogger<InventoryClient> _logger;
public InventoryClient(HttpClient http, ILogger<InventoryClient> logger)
{
_http = http;
_logger = logger;
}
public async Task<StockLevelDto?> GetStockAsync(string sku, CancellationToken ct = default)
{
try
{
return await _http.GetFromJsonAsync<StockLevelDto>($"/api/stock/{sku}", ct);
}
catch (HttpRequestException ex)
{
_logger.LogWarning(ex, "Failed to get stock for SKU {Sku}", sku);
return null;
}
}
public async Task<bool> ReserveStockAsync(ReserveStockRequest request, CancellationToken ct = default)
{
var response = await _http.PostAsJsonAsync("/api/stock/reserve", request, ct);
return response.IsSuccessStatusCode;
}
}Register with Polly resilience
// Program.cs
builder.Services
.AddHttpClient<IInventoryClient, InventoryClient>(client =>
{
client.BaseAddress = new Uri(builder.Configuration["Services:Inventory:BaseUrl"]!);
client.Timeout = TimeSpan.FromSeconds(10);
})
.AddResilienceHandler("inventory", pipeline =>
{
// Retry: 3 attempts, exponential backoff with jitter
pipeline.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 3,
Delay = TimeSpan.FromMilliseconds(200),
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.Handle<HttpRequestException>()
.HandleResult(r => r.StatusCode >= HttpStatusCode.InternalServerError),
});
// Circuit breaker: open after 5 failures in 30s, half-open after 15s
pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
FailureRatio = 0.5,
SamplingDuration = TimeSpan.FromSeconds(30),
MinimumThroughput = 5,
BreakDuration = TimeSpan.FromSeconds(15),
OnOpened = args =>
{
// Log when circuit opens — this is a signal for on-call
return ValueTask.CompletedTask;
},
});
// Timeout per attempt
pipeline.AddTimeout(TimeSpan.FromSeconds(5));
});The Polly 8 / Microsoft.Extensions.Resilience API (AddResilienceHandler) is the .NET 9 recommended approach — it replaces the older AddPolicyHandler extension from Polly 7.
2. gRPC for Internal Service Contracts
HTTP+JSON is fine for client-facing APIs. For internal service-to-service calls where you control both ends, gRPC gives you: schema-enforced contracts, binary serialization (much faster), bi-directional streaming, and generated strongly-typed clients.
Define the contract (proto file)
// Protos/payments.proto
syntax = "proto3";
option csharp_namespace = "PaymentsService.Grpc";
package payments;
service Payments {
rpc Charge (ChargeRequest) returns (ChargeResponse);
rpc Refund (RefundRequest) returns (RefundResponse);
rpc GetStatus (GetStatusRequest) returns (PaymentStatusResponse);
}
message ChargeRequest {
string order_id = 1;
string customer_id = 2;
int64 amount_cents = 3;
string currency = 4;
}
message ChargeResponse {
string payment_id = 1;
string status = 2;
string failure_reason = 3;
}
message RefundRequest {
string payment_id = 1;
int64 amount_cents = 2;
}
message RefundResponse {
bool success = 1;
}
message GetStatusRequest {
string payment_id = 1;
}
message PaymentStatusResponse {
string payment_id = 1;
string status = 2;
int64 amount_cents = 3;
string created_at = 4;
}Server implementation (Payments Service)
// Services/PaymentsGrpcService.cs
public class PaymentsGrpcService : Payments.PaymentsBase
{
private readonly IPaymentProcessor _processor;
private readonly ILogger<PaymentsGrpcService> _logger;
public PaymentsGrpcService(IPaymentProcessor processor, ILogger<PaymentsGrpcService> logger)
{
_processor = processor;
_logger = logger;
}
public override async Task<ChargeResponse> Charge(ChargeRequest request, ServerCallContext context)
{
_logger.LogInformation("Charging {AmountCents} for order {OrderId}", request.AmountCents, request.OrderId);
var result = await _processor.ChargeAsync(new ChargeCommand(
request.OrderId,
request.CustomerId,
request.AmountCents,
request.Currency
), context.CancellationToken);
return new ChargeResponse
{
PaymentId = result.PaymentId,
Status = result.Status.ToString(),
FailureReason = result.FailureReason ?? string.Empty,
};
}
public override async Task<RefundResponse> Refund(RefundRequest request, ServerCallContext context)
{
var success = await _processor.RefundAsync(request.PaymentId, request.AmountCents, context.CancellationToken);
return new RefundResponse { Success = success };
}
}// Program.cs (Payments Service)
builder.Services.AddGrpc(options =>
{
options.EnableDetailedErrors = builder.Environment.IsDevelopment();
options.MaxReceiveMessageSize = 4 * 1024 * 1024; // 4 MB
});
// ...
app.MapGrpcService<PaymentsGrpcService>();Client usage (Orders Service calling Payments)
// Program.cs (Orders Service)
builder.Services.AddGrpcClient<Payments.PaymentsClient>(options =>
{
options.Address = new Uri(builder.Configuration["Services:Payments:GrpcUrl"]!);
})
.ConfigureChannel(options =>
{
options.HttpHandler = new SocketsHttpHandler
{
PooledConnectionIdleTimeout = TimeSpan.FromMinutes(5),
KeepAlivePingDelay = TimeSpan.FromSeconds(60),
KeepAlivePingTimeout = TimeSpan.FromSeconds(30),
EnableMultipleHttp2Connections = true,
};
})
.AddResilienceHandler("payments-grpc", pipeline =>
{
pipeline.AddRetry(new RetryStrategyOptions
{
MaxRetryAttempts = 2,
Delay = TimeSpan.FromMilliseconds(100),
ShouldHandle = new PredicateBuilder().Handle<RpcException>(ex =>
ex.StatusCode is StatusCode.Unavailable or StatusCode.DeadlineExceeded),
});
});// Application/Commands/PlaceOrderCommandHandler.cs
public class PlaceOrderCommandHandler : IRequestHandler<PlaceOrderCommand, OrderResult>
{
private readonly Payments.PaymentsClient _payments;
private readonly IInventoryClient _inventory;
private readonly IOrderRepository _orders;
public async Task<OrderResult> Handle(PlaceOrderCommand command, CancellationToken ct)
{
// 1. Reserve inventory
var reserved = await _inventory.ReserveStockAsync(
new ReserveStockRequest(command.Sku, command.Quantity, command.OrderId), ct);
if (!reserved)
return OrderResult.Fail("Insufficient stock");
// 2. Charge via gRPC
var charge = await _payments.ChargeAsync(new ChargeRequest
{
OrderId = command.OrderId,
CustomerId = command.CustomerId,
AmountCents = command.AmountCents,
Currency = command.Currency,
}, cancellationToken: ct);
if (charge.Status != "succeeded")
return OrderResult.Fail(charge.FailureReason);
// 3. Persist order
var order = Order.Create(command.OrderId, command.CustomerId, charge.PaymentId);
await _orders.AddAsync(order, ct);
return OrderResult.Ok(order.Id);
}
}3. Event-Driven Messaging with MassTransit
Synchronous calls (HTTP, gRPC) create tight coupling — if Inventory is down, Orders can't place orders. Domain events via a message bus decouple services for workflows that don't require an immediate response.
dotnet add package MassTransit.RabbitMQDefine shared contracts (separate NuGet package or shared project)
// Shared/Contracts/OrderEvents.cs
namespace SystemForge.Contracts;
public record OrderPlaced(
string OrderId,
string CustomerId,
string Sku,
int Quantity,
long AmountCents,
DateTimeOffset PlacedAt
);
public record OrderCancelled(
string OrderId,
string Reason,
DateTimeOffset CancelledAt
);
public record StockReserved(
string OrderId,
string Sku,
int Quantity
);
public record StockReservationFailed(
string OrderId,
string Sku,
string Reason
);Publisher (Orders Service)
// Application/Commands/PlaceOrderCommandHandler.cs
public class PlaceOrderCommandHandler : IRequestHandler<PlaceOrderCommand, OrderResult>
{
private readonly IPublishEndpoint _bus;
private readonly IOrderRepository _orders;
public async Task<OrderResult> Handle(PlaceOrderCommand command, CancellationToken ct)
{
var order = Order.Create(command.OrderId, command.CustomerId, command.Sku, command.Quantity);
await _orders.AddAsync(order, ct);
// Publish event — Inventory Service will consume this asynchronously
await _bus.Publish(new OrderPlaced(
order.Id,
order.CustomerId,
order.Sku,
order.Quantity,
command.AmountCents,
DateTimeOffset.UtcNow
), ct);
return OrderResult.Ok(order.Id);
}
}Consumer (Inventory Service)
// Consumers/OrderPlacedConsumer.cs
public class OrderPlacedConsumer : IConsumer<OrderPlaced>
{
private readonly IStockRepository _stock;
private readonly IPublishEndpoint _bus;
private readonly ILogger<OrderPlacedConsumer> _logger;
public async Task Consume(ConsumeContext<OrderPlaced> context)
{
var msg = context.Message;
_logger.LogInformation("Reserving stock for order {OrderId}", msg.OrderId);
var available = await _stock.GetAvailableAsync(msg.Sku);
if (available >= msg.Quantity)
{
await _stock.ReserveAsync(msg.Sku, msg.Quantity, msg.OrderId);
await context.Publish(new StockReserved(msg.OrderId, msg.Sku, msg.Quantity));
}
else
{
await context.Publish(new StockReservationFailed(msg.OrderId, msg.Sku,
$"Only {available} available, {msg.Quantity} requested"));
}
}
}Registration (Inventory Service Program.cs)
builder.Services.AddMassTransit(x =>
{
x.AddConsumer<OrderPlacedConsumer>(cfg =>
{
// Retry 3 times before moving to error queue
cfg.UseMessageRetry(r => r.Intervals(
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(30)
));
});
x.UsingRabbitMq((context, cfg) =>
{
cfg.Host(builder.Configuration["RabbitMq:Host"], "/", h =>
{
h.Username(builder.Configuration["RabbitMq:Username"]!);
h.Password(builder.Configuration["RabbitMq:Password"]!);
});
// Configure quorum queues for durability
cfg.ReceiveEndpoint("inventory-order-placed", e =>
{
e.ConfigureConsumeTopology = false;
e.UseRawJsonDeserializer();
e.ConfigureConsumer<OrderPlacedConsumer>(context);
});
cfg.ConfigureEndpoints(context);
});
});Message idempotency: Consumers must handle duplicate messages (at-least-once delivery). Add an OutboxMessage table and check before processing:
public async Task Consume(ConsumeContext<OrderPlaced> context)
{
var messageId = context.MessageId?.ToString() ?? context.Message.OrderId;
if (await _outbox.AlreadyProcessedAsync(messageId))
return;
// ... process ...
await _outbox.MarkProcessedAsync(messageId);
}4. YARP API Gateway
Every microservices system needs a gateway: one ingress point for clients, where you handle authentication, rate limiting, and routing before requests hit individual services.
dotnet add package Yarp.ReverseProxy
dotnet add package Microsoft.AspNetCore.Authentication.JwtBearer// Program.cs (Gateway project)
var builder = WebApplication.CreateBuilder(args);
builder.Services
.AddReverseProxy()
.LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"));
builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddJwtBearer(options =>
{
options.Authority = builder.Configuration["Auth:Authority"];
options.Audience = builder.Configuration["Auth:Audience"];
});
builder.Services.AddAuthorization();
// Rate limiting
builder.Services.AddRateLimiter(options =>
{
options.AddFixedWindowLimiter("api", cfg =>
{
cfg.PermitLimit = 100;
cfg.Window = TimeSpan.FromMinutes(1);
cfg.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
cfg.QueueLimit = 10;
});
});
var app = builder.Build();
app.UseRateLimiter();
app.UseAuthentication();
app.UseAuthorization();
app.MapReverseProxy();
app.Run();// appsettings.json
{
"ReverseProxy": {
"Routes": {
"orders-route": {
"ClusterId": "orders-cluster",
"AuthorizationPolicy": "default",
"RateLimiterPolicy": "api",
"Match": { "Path": "/api/orders/{**catch-all}" }
},
"inventory-route": {
"ClusterId": "inventory-cluster",
"AuthorizationPolicy": "default",
"RateLimiterPolicy": "api",
"Match": { "Path": "/api/inventory/{**catch-all}" }
},
"payments-route": {
"ClusterId": "payments-cluster",
"AuthorizationPolicy": "default",
"Match": { "Path": "/api/payments/{**catch-all}" }
}
},
"Clusters": {
"orders-cluster": {
"Destinations": {
"primary": { "Address": "http://orders-service:5001" }
},
"HealthCheck": {
"Active": { "Enabled": true, "Interval": "00:00:10", "Path": "/health" }
}
},
"inventory-cluster": {
"Destinations": {
"primary": { "Address": "http://inventory-service:5002" }
},
"HealthCheck": {
"Active": { "Enabled": true, "Interval": "00:00:10", "Path": "/health" }
}
},
"payments-cluster": {
"Destinations": {
"primary": { "Address": "http://payments-service:5003" }
}
}
}
}
}Request transformation at the gateway
// Middleware/TenantHeaderTransform.cs — inject tenant_id from JWT into downstream headers
public class TenantHeaderTransformProvider : ITransformProvider
{
public void Apply(TransformBuilderContext context)
{
context.AddRequestTransform(async transformCtx =>
{
var user = transformCtx.HttpContext.User;
var tenantId = user.FindFirst("tenant_id")?.Value;
if (tenantId is not null)
transformCtx.ProxyRequest.Headers.TryAddWithoutValidation("X-Tenant-Id", tenantId);
});
}
public void ValidateCluster(TransformClusterValidationContext context) { }
public void ValidateRoute(TransformRouteValidationContext context) { }
}
// Program.cs
builder.Services.AddSingleton<ITransformProvider, TenantHeaderTransformProvider>();5. Distributed Tracing with OpenTelemetry
When a request fails, you need to see the full chain: gateway → Orders → Inventory → RabbitMQ → Payments. OpenTelemetry propagates trace context across all of this automatically.
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Instrumentation.GrpcNetClient
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package MassTransit.OpenTelemetry// Extensions/OpenTelemetryExtensions.cs
public static class OpenTelemetryExtensions
{
public static IServiceCollection AddObservability(
this IServiceCollection services,
IConfiguration config,
string serviceName)
{
services.AddOpenTelemetry()
.ConfigureResource(resource => resource
.AddService(serviceName)
.AddAttributes(new Dictionary<string, object>
{
["deployment.environment"] = config["ASPNETCORE_ENVIRONMENT"] ?? "production",
}))
.WithTracing(tracing => tracing
.AddAspNetCoreInstrumentation(options =>
{
options.RecordException = true;
options.Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health");
})
.AddHttpClientInstrumentation()
.AddGrpcClientInstrumentation()
.AddSource("MassTransit") // traces MassTransit publish/consume
.AddOtlpExporter(otlp =>
{
otlp.Endpoint = new Uri(config["Otlp:Endpoint"] ?? "http://localhost:4317");
}))
.WithMetrics(metrics => metrics
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddOtlpExporter());
return services;
}
}// Program.cs (each service)
builder.Services.AddObservability(builder.Configuration, "orders-service");Custom spans for business events
// Application/Commands/PlaceOrderCommandHandler.cs
public class PlaceOrderCommandHandler : IRequestHandler<PlaceOrderCommand, OrderResult>
{
private static readonly ActivitySource _activitySource = new("Orders.Application");
public async Task<OrderResult> Handle(PlaceOrderCommand command, CancellationToken ct)
{
using var activity = _activitySource.StartActivity("PlaceOrder");
activity?.SetTag("order.id", command.OrderId);
activity?.SetTag("order.sku", command.Sku);
activity?.SetTag("order.quantity", command.Quantity);
try
{
// ... processing ...
activity?.SetTag("order.result", "success");
return OrderResult.Ok(orderId);
}
catch (Exception ex)
{
activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
activity?.RecordException(ex);
throw;
}
}
}Register your custom ActivitySource:
// In AddObservability extension, add:
.AddSource("Orders.Application")
.AddSource("Inventory.Application")
.AddSource("Payments.Application")6. Health Checks with Dependency Awareness
A service is only healthy if its dependencies are healthy. Report granular health so the gateway and load balancer can make accurate routing decisions.
dotnet add package AspNetCore.HealthChecks.NpgSql
dotnet add package AspNetCore.HealthChecks.RabbitMQ
dotnet add package AspNetCore.HealthChecks.Redis// Program.cs (Orders Service)
builder.Services
.AddHealthChecks()
.AddNpgSql(
connectionString: builder.Configuration.GetConnectionString("Postgres")!,
name: "postgres",
failureStatus: HealthStatus.Unhealthy,
tags: ["db", "ready"])
.AddRabbitMQ(
rabbitConnectionString: builder.Configuration["RabbitMq:Host"]!,
name: "rabbitmq",
failureStatus: HealthStatus.Degraded, // degraded, not unhealthy — can still take orders
tags: ["messaging", "ready"])
.AddRedis(
builder.Configuration.GetConnectionString("Redis")!,
name: "redis",
failureStatus: HealthStatus.Degraded,
tags: ["cache", "ready"])
.AddUrlGroup(
new Uri(builder.Configuration["Services:Inventory:BaseUrl"] + "/health/live"),
name: "inventory-service",
failureStatus: HealthStatus.Degraded,
tags: ["dependencies"]);
// Expose two endpoints:
// /health/live — liveness: is the process alive? (no dependency checks)
// /health/ready — readiness: are dependencies ready? (for load balancer)
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
Predicate = _ => false, // no checks = just "I'm alive"
});
app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("ready"),
ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse,
});7. docker-compose for Local Development
Running all services locally without Kubernetes:
# docker-compose.yml
services:
gateway:
build: ./src/Gateway
ports: ["5000:8080"]
environment:
- Services__Orders__BaseUrl=http://orders:8080
- Services__Inventory__BaseUrl=http://inventory:8080
- Services__Payments__GrpcUrl=http://payments:8080
depends_on: [orders, inventory, payments]
orders:
build: ./src/Orders.Api
environment:
- ConnectionStrings__Postgres=Host=postgres;Database=orders;Username=app;Password=secret
- RabbitMq__Host=amqp://rabbitmq
- Services__Inventory__BaseUrl=http://inventory:8080
- Services__Payments__GrpcUrl=http://payments:8080
- Otlp__Endpoint=http://jaeger:4317
depends_on: [postgres, rabbitmq]
inventory:
build: ./src/Inventory.Api
environment:
- ConnectionStrings__Postgres=Host=postgres;Database=inventory;Username=app;Password=secret
- RabbitMq__Host=amqp://rabbitmq
- Otlp__Endpoint=http://jaeger:4317
depends_on: [postgres, rabbitmq]
payments:
build: ./src/Payments.Api
environment:
- ConnectionStrings__Postgres=Host=postgres;Database=payments;Username=app;Password=secret
- Otlp__Endpoint=http://jaeger:4317
depends_on: [postgres]
postgres:
image: postgres:16
environment:
POSTGRES_PASSWORD: secret
POSTGRES_USER: app
volumes: [pgdata:/var/lib/postgresql/data]
rabbitmq:
image: rabbitmq:3-management
ports: ["15672:15672"] # management UI
jaeger:
image: jaegertracing/all-in-one:latest
ports: ["16686:16686"] # Jaeger UI
volumes:
pgdata:Run everything with docker compose up --build. Jaeger UI at http://localhost:16686 shows complete request traces across services.
8. Service Contracts and Versioning
Services must evolve without breaking consumers. Two strategies:
Additive-only changes: New fields in gRPC proto files are backwards-compatible (protobuf ignores unknown fields). New optional fields in JSON DTOs are backwards-compatible. Never remove or rename fields without a version bump.
Versioned routes at the gateway:
// appsettings.json
"Routes": {
"orders-v1": {
"ClusterId": "orders-v1-cluster",
"Match": { "Path": "/api/v1/orders/{**catch-all}" }
},
"orders-v2": {
"ClusterId": "orders-v2-cluster",
"Match": { "Path": "/api/v2/orders/{**catch-all}" }
}
}This lets you run v1 and v2 of a service simultaneously during the migration window.
Consumer-driven contract testing with Pact ensures that when the Orders Service publishes an OrderPlaced event, the Inventory Service consumer gets what it expects — before either deploys to production.
Decisions Summary
| Scenario | Recommended approach |
|---|---|
| Client to API | HTTP+JSON through YARP gateway |
| Internal synchronous call | Typed HttpClient + Polly resilience |
| Internal synchronous, performance-sensitive | gRPC with shared proto contracts |
| Async workflow, no immediate response needed | MassTransit domain events |
| Async workflow, ordered processing required | MassTransit with sequential consumer |
| Cross-service debugging | OpenTelemetry traces → Jaeger/Tempo |
| Deployment gate | /health/ready with dependency checks |
The boundary between synchronous and asynchronous decomposition is the hardest design decision. Synchronous calls are easier to reason about but create availability coupling. Events are more resilient but introduce eventual consistency. Model your domain first — the communication pattern follows from whether operations need an immediate answer or not.