Introduction to Message Queues

Message queues are one of the most powerful tools in distributed systems. They're also one of the most overused. This article explains what they solve, when to reach for one, and how to get started.

The Problem They Solve

Imagine your e-commerce API takes an order, then needs to:

Send a confirmation email
Notify the warehouse
Update the analytics dashboard
Apply a loyalty points credit

If you do all of this synchronously in the order handler, the user waits for every step. If the email service is slow, the user waits longer. If the analytics service is down, the whole order fails.

A message queue decouples the sender from the receivers:

POST /orders → save order → respond to user (fast!)
                    ↓
              Publish "OrderPlaced" to queue
                    ↓
       ┌──────────┬───────────┬──────────────┐
  Email Service  Warehouse  Analytics   Loyalty Points
  (processes      (processes  (processes   (processes
   independently)  independently) ...)      ...)

The order API returns immediately. Each downstream service processes the message independently and at its own pace.

Queue vs Pub/Sub

These are two delivery patterns, often confused:

Queue (point-to-point):

Each message is consumed by exactly one consumer
Multiple consumers compete for messages — good for load balancing work
Use case: background job processing, task distribution

Producer → [Queue] → Consumer A
                   → Consumer B  (only one gets each message)
                   → Consumer C

Pub/Sub (publish-subscribe):

Each message is delivered to all subscribers
Each subscriber gets its own copy
Use case: event broadcasting (OrderPlaced → email + warehouse + analytics all get it)

Producer → [Topic] → Subscription A → Consumer A (email)
                   → Subscription B → Consumer B (warehouse)
                   → Subscription C → Consumer C (analytics)

Azure Service Bus supports both. Kafka is primarily pub/sub. RabbitMQ supports both via exchanges.

When to Use a Message Queue

Good reasons:

Async side effects — email, notifications, analytics, audit logs: don't block the main flow
Load levelling — your API gets 1,000 orders/min but your email service handles 100/min; queue absorbs the spike
Decoupling — the order service doesn't need to know about the loyalty service
Resilience — if the email service is down, messages queue up and are processed when it recovers
Retry handling — failed messages can be retried automatically with backoff

Bad reasons (when you don't need a queue):

You need the result synchronously — if you publish a message and then immediately wait for the response, you've just built a slow HTTP call with extra steps
Low volume — a queue for 10 events/day is unnecessary complexity
Simple single-service apps — a queue is a distributed system component; it adds failure modes

The simplest rule: if the operation is a side effect that doesn't affect the response, make it async via a queue.

Getting Started with Azure Service Bus

Azure Service Bus is the managed queue/topic service on Azure — no infrastructure to run.

Bash

dotnet add package Azure.Messaging.ServiceBus
dotnet add package Microsoft.Extensions.Azure

// Program.cs
builder.Services.AddAzureClients(azureBuilder =>
{
    azureBuilder.AddServiceBusClient(
        builder.Configuration.GetConnectionString("ServiceBus"));
});

Sending a message:

public class OrderService
{
    private readonly ServiceBusClient _serviceBus;

    public OrderService(ServiceBusClient serviceBus)
    {
        _serviceBus = serviceBus;
    }

    public async Task PlaceOrderAsync(Order order)
    {
        await _repository.SaveAsync(order);

        // Publish event after saving — do not publish before
        var sender = _serviceBus.CreateSender("orders");
        var message = new ServiceBusMessage(JsonSerializer.Serialize(new OrderPlacedEvent
        {
            OrderId = order.Id,
            CustomerId = order.CustomerId,
            TotalAmount = order.TotalAmount,
            PlacedAt = order.PlacedAt
        }))
        {
            MessageId = order.Id,         // prevents duplicate processing
            ContentType = "application/json",
            Subject = "OrderPlaced"
        };

        await sender.SendMessageAsync(message);
    }
}

Receiving messages (background worker):

public class OrderPlacedConsumer : BackgroundService
{
    private readonly ServiceBusClient _serviceBus;
    private readonly IEmailService _emailService;
    private readonly ILogger<OrderPlacedConsumer> _logger;

    public OrderPlacedConsumer(ServiceBusClient serviceBus, IEmailService emailService,
        ILogger<OrderPlacedConsumer> logger)
    {
        _serviceBus = serviceBus;
        _emailService = emailService;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var processor = _serviceBus.CreateProcessor(
            "orders",
            "email-subscription",  // subscription name on the topic
            new ServiceBusProcessorOptions
            {
                MaxConcurrentCalls = 5,
                AutoCompleteMessages = false  // we complete manually after processing
            });

        processor.ProcessMessageAsync += HandleMessageAsync;
        processor.ProcessErrorAsync += HandleErrorAsync;

        await processor.StartProcessingAsync(stoppingToken);
        await Task.Delay(Timeout.Infinite, stoppingToken);
        await processor.StopProcessingAsync();
    }

    private async Task HandleMessageAsync(ProcessMessageEventArgs args)
    {
        var orderEvent = JsonSerializer.Deserialize<OrderPlacedEvent>(
            args.Message.Body.ToString());

        try
        {
            await _emailService.SendOrderConfirmationAsync(orderEvent!);

            // Only complete (delete from queue) after successful processing
            await args.CompleteMessageAsync(args.Message);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to send email for order {OrderId}", orderEvent?.OrderId);

            // Abandon — message returns to queue for retry
            // After max delivery count, moves to dead-letter queue
            await args.AbandonMessageAsync(args.Message);
        }
    }

    private Task HandleErrorAsync(ProcessErrorEventArgs args)
    {
        _logger.LogError(args.Exception, "Service Bus processor error: {Source}", args.ErrorSource);
        return Task.CompletedTask;
    }
}

Dead-Letter Queue

When a message fails processing too many times (default: 10 in Azure Service Bus), it goes to the dead-letter queue (DLQ). This prevents poison messages from blocking the queue forever.

You should:

Monitor the DLQ — a growing DLQ means something is consistently failing
Inspect DLQ messages to find bugs
Replay or discard them after fixing the bug

Bash

# Azure CLI: check dead-letter count
az servicebus topic subscription show \
  --resource-group my-rg \
  --namespace-name my-ns \
  --topic-name orders \
  --name email-subscription \
  --query deadLetterMessageCount

Getting Started with RabbitMQ (Local Dev)

RabbitMQ is a good choice for local development and self-hosted deployments.

Bash

# Run RabbitMQ with management UI
docker run -d --name rabbitmq \
  -p 5672:5672 \
  -p 15672:15672 \
  rabbitmq:3-management

# Management UI: http://localhost:15672 (guest/guest)

Bash

dotnet add package MassTransit
dotnet add package MassTransit.RabbitMQ

MassTransit is the recommended .NET library for message-based systems — it abstracts over RabbitMQ, Azure Service Bus, and Amazon SQS with the same API.

// Program.cs
builder.Services.AddMassTransit(x =>
{
    x.AddConsumer<OrderPlacedConsumer>();

    x.UsingRabbitMq((context, cfg) =>
    {
        cfg.Host("localhost", "/", h =>
        {
            h.Username("guest");
            h.Password("guest");
        });

        cfg.ConfigureEndpoints(context);
    });
});

// Publishing — same API regardless of transport
public class OrderService
{
    private readonly IPublishEndpoint _publish;

    public async Task PlaceOrderAsync(Order order)
    {
        await _repository.SaveAsync(order);
        await _publish.Publish(new OrderPlacedEvent { OrderId = order.Id, ... });
    }
}

Message Design Best Practices

Include the event type in the message:

// Good: self-describing
public record OrderPlacedEvent
{
    public string OrderId { get; init; }
    public string CustomerId { get; init; }
    public decimal TotalAmount { get; init; }
    public DateTimeOffset PlacedAt { get; init; }
}

Don't send the full entity — send an event with the ID:

// Bad: large payload, stale by the time it's consumed
{ "order": { /* entire order with 50 fields */ } }

// Good: lightweight event — consumer fetches what it needs
{ "orderId": "ord_123", "placedAt": "2026-04-21T10:00:00Z" }

Make consumers idempotent: Queues guarantee at-least-once delivery. Your consumer may receive the same message twice (network retry, restart during processing). Design consumers so processing the same message twice produces the same result.

// Check if already processed before doing the work
if (await _emailLog.AlreadySentAsync(orderEvent.OrderId, "OrderConfirmation"))
    return; // idempotent: safe to skip

await _emailService.SendConfirmationAsync(orderEvent);
await _emailLog.RecordSentAsync(orderEvent.OrderId, "OrderConfirmation");

Summary

Message queues decouple producers from consumers and enable async processing
Queue = one consumer per message; Pub/Sub = all subscribers get a copy
Use queues for side effects (email, notifications, analytics), not for operations you need results from synchronously
Azure Service Bus: managed, production-ready, supports queues and topics
RabbitMQ: self-hosted, good for local dev and on-premises
MassTransit: unified .NET API over both
Always handle failures: complete on success, abandon on failure, monitor the DLQ
Make consumers idempotent — at-least-once delivery means duplicates happen

Introduction to Message Queues — Why, When, and How

Introduction to Message Queues

The Problem They Solve

Queue vs Pub/Sub

When to Use a Message Queue

Getting Started with Azure Service Bus

Dead-Letter Queue

Getting Started with RabbitMQ (Local Dev)

Message Design Best Practices

Summary

RabbitMQ & Messaging Knowledge Check

Enjoyed this article?

Leave a comment