Kafka & Event-Driven Architecture

Kafka is a distributed event streaming platform. It decouples services, handles millions of events per second, and provides durable event logs that consumers can replay. This lesson covers the core concepts and how to apply them in production architectures.

Why Kafka? The Core Problem

Without Kafka, services call each other directly:

OrderService → PaymentService (HTTP)
             → InventoryService (HTTP)
             → EmailService (HTTP)
             → AnalyticsService (HTTP)

Problems:

OrderService must know about every downstream service
If PaymentService is down, the order fails
Scaling is coupled — more orders = more load on all services simultaneously
Replaying events (for new services, debugging) is impossible

With Kafka:

OrderService → Kafka (order.placed topic)
                   ← PaymentService (subscribes)
                   ← InventoryService (subscribes)
                   ← EmailService (subscribes)
                   ← AnalyticsService (subscribes)

OrderService publishes one event. Each consumer processes it independently.

Core Concepts

Topics

A topic is a named, ordered, immutable log of events. Think of it as a category:

topics:
  order.placed
  payment.processed
  inventory.updated
  user.registered

Events are appended to the end. Consumers track their position (offset) — they can read from the beginning, current position, or any specific offset.

Partitions

Topics are split into partitions for parallelism and throughput:

order.placed topic:
  Partition 0: [event0, event1, event4, event7, ...]
  Partition 1: [event2, event5, event8, ...]
  Partition 2: [event3, event6, event9, ...]

Messages with the same key always go to the same partition (ensuring ordering per key). Events with no key are round-robined.

Producers

Publish events to topics:

JAVASCRIPT

// Node.js with kafkajs
const { Kafka } = require('kafkajs')

const kafka = new Kafka({ brokers: ['localhost:9092'] })
const producer = kafka.producer()

await producer.connect()
await producer.send({
  topic: 'order.placed',
  messages: [
    {
      key: orderId.toString(),  // same orderId → same partition → ordered
      value: JSON.stringify({
        orderId,
        customerId,
        items,
        total,
        placedAt: new Date().toISOString(),
      }),
    },
  ],
})
await producer.disconnect()

Consumers & Consumer Groups

Multiple consumers in a group share partitions — scale horizontally:

JAVASCRIPT

const consumer = kafka.consumer({ groupId: 'payment-service' })

await consumer.connect()
await consumer.subscribe({ topic: 'order.placed', fromBeginning: false })

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value.toString())
    await paymentService.processOrder(event)
    // Kafka auto-commits offset after eachMessage returns
  },
})

With 3 partitions and 3 consumers in the group, each consumer owns one partition. With 6 consumers in the group and 3 partitions, 3 consumers are idle — you can't have more active consumers than partitions.

Event Schema — Envelope Pattern

Always include envelope metadata with every event:

TYPESCRIPT

interface KafkaEvent<T> {
  eventId:    string;      // UUID — for idempotency
  eventType:  string;      // 'order.placed', 'payment.processed'
  version:    string;      // '1.0' — for schema evolution
  occurredAt: string;      // ISO timestamp
  aggregateId: string;     // the primary entity ID
  payload:    T;           // event-specific data
}

const event: KafkaEvent<OrderPlacedPayload> = {
  eventId:    randomUUID(),
  eventType:  'order.placed',
  version:    '1.0',
  occurredAt: new Date().toISOString(),
  aggregateId: order.id,
  payload:    { items: order.items, total: order.total, customerId: order.customerId },
}

Idempotency — Handle Duplicate Events

Kafka guarantees at-least-once delivery — the same event may be delivered more than once (consumer crash before committing offset). Your consumers must be idempotent:

TYPESCRIPT

async function handleOrderPlaced(event: KafkaEvent<OrderPlacedPayload>) {
  // Check if already processed using event ID
  const alreadyProcessed = await redis.exists(`processed:${event.eventId}`)
  if (alreadyProcessed) return  // skip duplicate

  // Process the event
  await paymentService.charge(event.payload.customerId, event.payload.total)

  // Mark as processed (TTL: 24h is enough — retries don't happen after that)
  await redis.setex(`processed:${event.eventId}`, 86400, '1')
}

The Outbox Pattern — Guaranteed Event Publishing

The biggest risk: your service updates the database AND publishes to Kafka. If Kafka is down after the DB commit, the event is never published:

// DANGEROUS
await db.order.create(orderData)           // committed
await kafka.send('order.placed', event)    // FAILS — event lost!

Fix: the Outbox Pattern — write to a DB table and Kafka atomically:

TYPESCRIPT

// In one database transaction:
await db.$transaction(async (tx) => {
  const order = await tx.order.create({ data: orderData })

  // Write event to outbox table (same transaction)
  await tx.outboxEvent.create({
    data: {
      eventId:    randomUUID(),
      eventType:  'order.placed',
      aggregateId: order.id,
      payload:    JSON.stringify({ ...order }),
      status:     'PENDING',
    },
  })
})

// A separate process (Outbox Relay) polls the outbox and publishes to Kafka
// Then marks events as PUBLISHED

This guarantees: if the DB transaction commits, the event will eventually be published. If the transaction rolls back, the event is never written.

Kafka in .NET (Confluent.Kafka)

// Producer
var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

using var producer = new ProducerBuilder<string, string>(config).Build();

var message = new Message<string, string>
{
    Key   = order.Id.ToString(),
    Value = JsonSerializer.Serialize(new { order.Id, order.Total, order.CustomerId }),
};

await producer.ProduceAsync("order.placed", message);

// Consumer (hosted service)
public class OrderConsumer : BackgroundService
{
    protected override Task ExecuteAsync(CancellationToken ct)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers  = "localhost:9092",
            GroupId           = "payment-service",
            AutoOffsetReset   = AutoOffsetReset.Earliest,
            EnableAutoCommit  = false,   // manual commit for at-least-once guarantee
        };

        using var consumer = new ConsumerBuilder<string, string>(config).Build();
        consumer.Subscribe("order.placed");

        while (!ct.IsCancellationRequested)
        {
            var result = consumer.Consume(ct);
            var order  = JsonSerializer.Deserialize<OrderPlacedEvent>(result.Message.Value);

            await _paymentService.ProcessAsync(order);
            consumer.Commit(result);   // commit only after successful processing
        }

        return Task.CompletedTask;
    }
}

Kafka vs RabbitMQ vs SQS

| | Kafka | RabbitMQ | AWS SQS | |---|---|---|---| | Model | Event log (pull) | Message queue (push) | Message queue (pull) | | Retention | Days/weeks (configurable) | Until consumed | 14 days max | | Replay | Yes — consumers can re-read | No | No | | Throughput | Millions/sec | 100K/sec | 3,000/sec standard | | Ordering | Per partition | Per queue | Per FIFO queue | | Best for | Event streaming, analytics, audit log | Task queues, RPC-style | Cloud-native AWS services |

Choose Kafka for: event sourcing, streaming analytics, audit trails, event replay. Choose RabbitMQ/SQS for: simple task queues where replay isn't needed.

Quick Reference

Topic:         Named log of events — append-only
Partition:     Ordered sub-log within a topic for parallelism
Producer:      Publishes to a topic (with optional key for ordering)
Consumer:      Reads from a topic, tracks its own offset
Consumer group: Multiple consumers sharing partitions — scale horizontally
Offset:        Position of a consumer in a partition
At-least-once: Same message may be delivered twice → consumers must be idempotent
Outbox pattern: Write event to DB + Kafka atomically via outbox table

Key decisions:
  Partition count:    = max parallelism you'll ever want
  Replication factor: 3 for production (survives 2 broker failures)
  Retention:          Based on replay needs (hours for tasks, days/weeks for events)
  Idempotency:        Store processed event IDs in Redis with TTL

Kafka & Event-Driven Architecture

Why Kafka? The Core Problem

Core Concepts

Topics

Partitions

Producers

Consumers & Consumer Groups

Event Schema — Envelope Pattern

Idempotency — Handle Duplicate Events

The Outbox Pattern — Guaranteed Event Publishing

Kafka in .NET (Confluent.Kafka)

Kafka vs RabbitMQ vs SQS

Quick Reference

Enjoyed this article?

Leave a comment