System Design · Lesson 15 of 26

Data Management in Microservices — No Shared DB

Data management is where most microservices architectures break down. You can get the service boundaries right, set up Kubernetes, and deploy everything cleanly — then run headfirst into the wall of distributed data consistency.

This article is the guide that will save you months of pain.

Database Per Service — The Rule and Its Cost

Every microservice must own its data exclusively. No other service may directly access its database.

Correct:
  Order Service → Orders DB (PostgreSQL)
  User Service  → Users DB (PostgreSQL)
  Product Service → Products DB (MongoDB)

Incorrect:
  ┌──────────────────────────────────────┐
  │             Shared DB                │
  │  orders | users | products | reviews │
  └──────────────────────────────────────┘
       ↑          ↑          ↑
  Order Svc  User Svc  Product Svc

Why the shared database pattern fails:

A schema migration in one service can silently break another
Services are coupled at the data layer — not truly independent
You can't replace one service's database technology without affecting all services
One slow query from Service A degrades Service B's database performance

The cost you pay:

Cross-service queries require API calls instead of SQL JOINs
No ACID transactions spanning two services
Data duplication (each service stores a subset of what it needs)

This cost is real. But it's the price of independence.

Eventual Consistency — Making Peace With "Later"

When data spans multiple services, you can't have strong consistency without distributed locks or two-phase commit (both terrible in practice). The practical answer is eventual consistency.

What eventually consistent means:
  Time T:    Order Service creates order
  Time T+0:  Orders DB has new order
  Time T+0:  Event "OrderPlaced" published
  Time T+50ms: Inventory Service receives event, reserves stock
  Time T+200ms: Notification Service sends email
  
  Between T and T+200ms, if you query "is the email sent?" — no.
  Eventually (T+200ms), all systems are consistent.

When eventual consistency is acceptable:

Sending a confirmation email (100ms delay is fine)
Updating product view count (±1 second accuracy is fine)
Propagating profile updates (5 seconds lag is fine)
User activity feeds (few-second delay is fine)

When eventual consistency is NOT acceptable:

Charging a credit card (must know current balance synchronously)
Reserving the last unit of inventory (must prevent overselling right now)
Financial account debits (must be atomic with the corresponding credit)

For the cases that require strong consistency, use the Saga pattern or keep the operation within a single service's transaction boundary.

CQRS — Separate Read and Write Models

Command Query Responsibility Segregation (CQRS) separates the model for writing data (command side) from the model for reading data (query side).

Without CQRS:
  All operations use the same model
  Your Order object serves: create order, update status, display order history
  The model becomes a compromise — not optimized for any use case

With CQRS:
  Write side (Commands):
    CreateOrderCommand → OrderAggregate → Orders DB
    (normalized, enforces business rules, ACID)
    
  Read side (Queries):
    GetOrderHistoryQuery → OrderSummaryView → Read DB (denormalized)
    (denormalized, optimized for display, no JOINs needed)

CQRS in practice (.NET example):

// Write side — rich domain model
public class Order
{
    private readonly List<OrderItem> _items = new();
    public Guid Id { get; private set; }
    public OrderStatus Status { get; private set; }
    
    public void PlaceOrder(IEnumerable<OrderItemDto> items)
    {
        if (!items.Any()) throw new DomainException("Order must have items");
        foreach (var item in items)
            _items.Add(new OrderItem(item.ProductId, item.Quantity, item.Price));
        Status = OrderStatus.Pending;
        AddDomainEvent(new OrderPlacedEvent(Id));
    }
    
    public void Ship(string trackingNumber)
    {
        if (Status != OrderStatus.Confirmed) throw new DomainException("Can only ship confirmed orders");
        Status = OrderStatus.Shipped;
        AddDomainEvent(new OrderShippedEvent(Id, trackingNumber));
    }
}

// Read side — flat, optimized for queries
public class OrderSummaryView
{
    public Guid OrderId { get; set; }
    public string UserName { get; set; }   // denormalized from User
    public string UserEmail { get; set; }   // denormalized from User
    public int ItemCount { get; set; }
    public decimal Total { get; set; }
    public string Status { get; set; }
    public DateTime CreatedAt { get; set; }
}

// Read model is updated by handling domain events
public class OrderSummaryProjection
{
    public async Task HandleAsync(OrderPlacedEvent @event)
    {
        var user = await _userService.GetAsync(@event.UserId);
        await _readDb.UpsertAsync(new OrderSummaryView
        {
            OrderId = @event.OrderId,
            UserName = user.Name,
            UserEmail = user.Email,
            ItemCount = @event.Items.Count,
            Total = @event.Items.Sum(i => i.Price * i.Quantity),
            Status = "Pending",
            CreatedAt = @event.OccurredAt
        });
    }
}

CQRS benefits:

Query side can be a different database (e.g., Elasticsearch for full-text search)
Read models can be pre-computed and denormalized — zero-JOIN queries
Scale reads and writes independently
Read side can be rebuilt from event history

CQRS cost:

More code and complexity
Eventual consistency between write and read models
Requires events to keep models in sync

The Saga Pattern — Distributed Transactions

A Saga is a sequence of local transactions with compensating transactions to handle failures.

Without Saga (naive approach — broken):
  BEGIN DISTRIBUTED TRANSACTION  ← doesn't exist across services
    OrderService.CreateOrder()
    InventoryService.ReserveStock()
    PaymentService.Charge()
  COMMIT
  
With Saga:
  Step 1: OrderService.CreateOrder()           → emit OrderCreated
  Step 2: InventoryService.ReserveStock()      → emit StockReserved
  Step 3: PaymentService.ChargeCard()          → emit PaymentProcessed
  
  If Step 3 fails:
    Compensation 3: PaymentService.RefundCharge()       (if partial)
    Compensation 2: InventoryService.ReleaseReservation()
    Compensation 1: OrderService.CancelOrder()

Saga: Choreography Style

Each service reacts to events and emits the next event. No central coordinator.

OrderService:    OrderPlaced →
InventoryService:            → StockReserved →
PaymentService:                               → PaymentProcessed →
OrderService:                                                     → OrderConfirmed

Failure handling:
PaymentService:  → PaymentFailed →
InventoryService: → (release reservation) → StockReleased →
OrderService:                                             → OrderCancelled

Pros: No central coordinator. Services are loosely coupled.

Cons: Workflow logic is scattered across services. Hard to track overall state. Complex failure handling.

Saga: Orchestration Style

A dedicated saga orchestrator drives the workflow and handles compensations.

public class CreateOrderSaga : ISaga
{
    private SagaState _state = SagaState.Started;
    private Guid? _reservationId;
    private Guid? _paymentId;
    
    public async Task ExecuteAsync(CreateOrderCommand cmd)
    {
        try
        {
            // Step 1
            var orderId = await _orderService.CreatePendingOrderAsync(cmd);
            _state = SagaState.OrderCreated;
            
            // Step 2
            _reservationId = await _inventoryService.ReserveAsync(cmd.Items);
            _state = SagaState.InventoryReserved;
            
            // Step 3
            _paymentId = await _paymentService.ChargeAsync(cmd.UserId, cmd.Total);
            _state = SagaState.PaymentProcessed;
            
            // Confirm
            await _orderService.ConfirmAsync(orderId);
            _state = SagaState.Completed;
        }
        catch (PaymentFailedException)
        {
            await CompensateAsync();
        }
    }
    
    private async Task CompensateAsync()
    {
        if (_state >= SagaState.InventoryReserved)
            await _inventoryService.ReleaseAsync(_reservationId!.Value);
        if (_state >= SagaState.OrderCreated)
            await _orderService.CancelAsync();
        _state = SagaState.Compensated;
    }
}

Compensating transactions must be idempotent. The saga may retry a compensation step. Calling ReleaseReservation twice must not double-release.

Event Sourcing — Events as the Source of Truth

Instead of storing the current state, event sourcing stores the full history of events. Current state is derived by replaying events.

Traditional (state-based):
  orders table: { id: 123, status: "Shipped", updated_at: "2026-04-15" }
  (you know the current state, but not how it got there)

Event sourcing:
  order_events table:
    { id: 1, order_id: 123, type: "OrderPlaced",   data: {...}, timestamp: T1 }
    { id: 2, order_id: 123, type: "PaymentReceived", data: {...}, timestamp: T2 }
    { id: 3, order_id: 123, type: "OrderShipped",  data: {...}, timestamp: T3 }
  
  Current state = replay all events in order

// Event sourced aggregate
public class OrderAggregate
{
    private readonly List<DomainEvent> _events = new();
    
    public Guid Id { get; private set; }
    public OrderStatus Status { get; private set; }
    
    // Load from event history
    public static OrderAggregate Rehydrate(IEnumerable<DomainEvent> history)
    {
        var order = new OrderAggregate();
        foreach (var @event in history)
            order.Apply(@event);
        return order;
    }
    
    private void Apply(DomainEvent @event)
    {
        switch (@event)
        {
            case OrderPlacedEvent e:
                Id = e.OrderId;
                Status = OrderStatus.Pending;
                break;
            case PaymentReceivedEvent:
                Status = OrderStatus.Confirmed;
                break;
            case OrderShippedEvent:
                Status = OrderStatus.Shipped;
                break;
        }
    }
    
    public void Ship(string trackingNumber)
    {
        if (Status != OrderStatus.Confirmed) throw new DomainException("...");
        var @event = new OrderShippedEvent(Id, trackingNumber, DateTime.UtcNow);
        Apply(@event);
        _events.Add(@event);  // pending events to be saved
    }
}

Event sourcing benefits:

Full audit log — you know exactly what happened and when
Time travel — replay events to any point in time
Rebuilt read models — add a new projection and replay all history
Natural fit with CQRS

Event sourcing costs:

Complex to implement correctly
Eventual consistency in read models
Event schema evolution is hard (past events can't be changed)
Snapshots required for aggregates with long event histories

Use event sourcing when: Audit trail is a core requirement (finance, healthcare, compliance), or when you need the ability to rebuild read models from scratch.

API Composition — Queries That Span Services

When a query needs data from multiple services, you compose the response in an API layer.

GET /orders/{id}/details
  Needs: order data + user data + product data

Option 1: API Gateway composition
  API Gateway:
    1. GET order from Order Service
    2. GET user from User Service (using order.userId)
    3. GET products from Product Service (using order.itemProductIds)
    4. Compose and return

Option 2: Dedicated read model (CQRS projection)
  Order Placed event triggers projection that pre-joins:
    order + user + product names into OrderDetailsView table
  Query: SELECT * FROM order_details_view WHERE id = ?
  (fast, no runtime composition, slightly stale)

// API Composition in a BFF (Backend for Frontend)
public class OrderDetailsQueryHandler
{
    public async Task<OrderDetailsDto> HandleAsync(Guid orderId)
    {
        // Parallel fetches
        var orderTask = _orderClient.GetAsync(orderId);
        var userTask = _userClient.GetAsync(orderId);  // from correlation in order
        
        await Task.WhenAll(orderTask, userTask);
        
        var order = await orderTask;
        var user = await userTask;
        
        // Fetch products for all items in parallel
        var productTasks = order.Items
            .Select(item => _productClient.GetAsync(item.ProductId))
            .ToList();
        var products = await Task.WhenAll(productTasks);
        
        return new OrderDetailsDto
        {
            OrderId = order.Id,
            Status = order.Status,
            CustomerName = user.Name,
            Items = order.Items.Zip(products, (item, product) => new OrderItemDto
            {
                ProductName = product.Name,
                Quantity = item.Quantity,
                Price = item.Price
            }).ToList()
        };
    }
}

Data Consistency in Practice — What's "Good Enough"

Not everything needs perfect consistency. A pragmatic consistency hierarchy:

Strong consistency required:
  - Financial transactions
  - Inventory reservation (prevent overselling)
  - Security operations (password change, revoke token)

Eventual consistency acceptable (seconds):
  - Order confirmation email
  - Analytics and reporting
  - Profile data propagation
  - Cache invalidation

Eventual consistency acceptable (minutes or hours):
  - Search index updates
  - Data warehouse ETL
  - Non-critical notifications

Design your system so that operations requiring strong consistency happen within a single service's transaction boundary. Cross-service operations should be designed to tolerate eventual consistency.

Key Takeaways

Database per service is mandatory for independence. Cross-service data access breaks isolation.
Eventual consistency is the norm for cross-service data. Design for it explicitly.
CQRS separates write (commands) and read (queries) models — enables optimized read projections and independent scaling.
Saga pattern handles distributed transactions with local transactions and compensating actions.
Orchestration sagas are easier to reason about and debug than choreography sagas for complex workflows.
Compensating transactions must be idempotent — they may be called more than once.
Event sourcing provides a full audit trail and enables read model rebuilds. Use it when audit/compliance is core, not as a default.
API Composition or pre-built read projections handle cross-service queries.
Know which operations need strong consistency — keep them within one service boundary.

Sync vs Async Communication — REST, gRPC & Message Queues

Next Lesson

Service Discovery & API Gateway Patterns