Learnixo
Back to blog
Cloud & DevOpsintermediate

Docker Compose for Data Engineering Dev Environments

Build a complete local data engineering stack — Airflow, PostgreSQL, Redis, Kafka, Schema Registry — with Docker Compose. Covers compose syntax, healthchecks, named volumes, override files, and profiles.

LearnixoMay 7, 202610 min read
DockerDocker ComposeAirflowKafkaPostgreSQLData EngineeringDevOps
Share:𝕏

What Docker Compose Solves

Running a modern data pipeline locally requires at least five services: an orchestrator (Airflow), a metadata database (PostgreSQL), a message broker (Kafka), a cache/queue (Redis), and a monitoring UI. Starting them by hand with docker run is error-prone and hard to share with teammates.

Docker Compose lets you define the entire stack in one YAML file. One command starts everything in the right order with the right configuration.

Bash
# Entire local data engineering stack
docker compose up -d

# Tear everything down
docker compose down

docker-compose.yml Structure

YAML
# Top-level keys
version: "3.9"           # Compose file format version

services:                # Container definitions
  postgres:
    image: postgres:16
    ...

  airflow-web:
    build: .             # Build from local Dockerfile
    ...

volumes:                 # Named volumes
  postgres-data:

networks:                # Named networks
  data-net:
    driver: bridge

Note on version: Compose v2 (Docker Desktop 4.x+) treats version as informational. You can omit it. Including it is harmless and communicates your minimum supported Compose version.

build vs image

YAML
services:
  # Use a published image
  postgres:
    image: postgres:16           # pulls from registry

  # Build from local Dockerfile
  pipeline:
    build:
      context: .                 # build context (directory)
      dockerfile: Dockerfile     # optional — defaults to Dockerfile
      args:
        PYTHON_VERSION: "3.11"   # build-time args
      target: runtime            # stop at a named stage (multi-stage)
    image: my-pipeline:local     # tag the built image

  # Build from a subdirectory
  airflow:
    build:
      context: ./airflow
      dockerfile: Dockerfile.airflow

depends_on and healthchecks

depends_on alone only waits for the container to start — not for the service inside it to be ready. Combine with healthcheck to wait for readiness.

YAML
services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U airflow"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s   # grace period before health checks begin

  redis:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

  airflow-webserver:
    image: apache/airflow:2.9.0
    depends_on:
      postgres:
        condition: service_healthy   # waits for healthcheck to pass
      redis:
        condition: service_healthy

The three condition values:

  • service_started: container is running (default)
  • service_healthy: healthcheck is passing
  • service_completed_successfully: container exited with code 0 (for init containers)

Environment Variables and env_file

YAML
services:
  airflow-webserver:
    image: apache/airflow:2.9.0
    environment:
      # Inline values
      AIRFLOW__CORE__EXECUTOR: CeleryExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
      AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
      AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
      AIRFLOW__CORE__FERNET_KEY: ""
      AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: "true"
      AIRFLOW__CORE__LOAD_EXAMPLES: "false"
      # Pass from host environment (value from shell)
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
    env_file:
      - .env                # loaded first
      - .env.local          # overrides (gitignored)

.env (committed, no secrets):

Bash
AIRFLOW_IMAGE_NAME=apache/airflow:2.9.0
AIRFLOW_UID=50000
COMPOSE_PROJECT_NAME=data-stack

.env.local (gitignored, has secrets):

Bash
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
SNOWFLAKE_PASSWORD=my_snowflake_password

Volumes for Persistent Data

YAML
services:
  postgres:
    image: postgres:16
    volumes:
      # Named volume — Docker manages location
      - postgres-data:/var/lib/postgresql/data
      # Bind mount — init scripts run at first start
      - ./init-sql:/docker-entrypoint-initdb.d:ro

  airflow-webserver:
    image: apache/airflow:2.9.0
    volumes:
      # DAGs folder: bind mount for live editing
      - ./dags:/opt/airflow/dags
      # Logs: named volume so they persist across restarts
      - airflow-logs:/opt/airflow/logs
      # Plugins: bind mount
      - ./plugins:/opt/airflow/plugins

volumes:
  postgres-data:          # Docker manages lifecycle
    driver: local
  airflow-logs:
    driver: local
  kafka-data:
    driver: local

Named Networks for Service Communication

Services on the same network reach each other by service name as hostname.

YAML
services:
  postgres:
    image: postgres:16
    networks:
      - backend

  airflow-webserver:
    image: apache/airflow:2.9.0
    networks:
      - backend
      - frontend     # also reachable from the frontend network

  nginx:
    image: nginx:alpine
    networks:
      - frontend

networks:
  backend:
    driver: bridge
  frontend:
    driver: bridge

airflow-webserver connects to postgres with hostname postgres:

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
#                                                                               ^^^^^^^^
#                                                                          service name = hostname

Override Files for Local Dev

docker-compose.override.yml is automatically merged with docker-compose.yml when you run docker compose up. Use it for dev-specific settings that you don't want in the base file.

docker-compose.yml (base — production-like):

YAML
services:
  pipeline:
    image: my-pipeline:${TAG:-latest}
    restart: unless-stopped
    environment:
      LOG_LEVEL: INFO

docker-compose.override.yml (local dev — gitignored or committed):

YAML
services:
  pipeline:
    build: .             # build locally instead of pulling
    volumes:
      - .:/app           # live code reload
    environment:
      LOG_LEVEL: DEBUG
      PYTHONDONTWRITEBYTECODE: "1"
    command: ["python", "-m", "debugpy", "--listen", "0.0.0.0:5678", "pipeline.py"]
    ports:
      - "5678:5678"      # debugpy port

Use named override files explicitly:

Bash
# Staging
docker compose -f docker-compose.yml -f docker-compose.staging.yml up -d

# Production
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

Profiles for Selective Service Startup

Profiles let you start subsets of the stack. Only services with no profile (or a matching profile) start.

YAML
services:
  postgres:
    image: postgres:16
    # no profile — always starts

  airflow-webserver:
    image: apache/airflow:2.9.0
    profiles: ["airflow"]

  airflow-scheduler:
    image: apache/airflow:2.9.0
    profiles: ["airflow"]

  flower:
    image: mher/flower
    profiles: ["airflow", "monitoring"]

  kafka:
    image: confluentinc/cp-kafka:7.6.0
    profiles: ["kafka"]

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    profiles: ["kafka", "monitoring"]
Bash
# Start only postgres
docker compose up -d

# Start Airflow stack (includes postgres)
docker compose --profile airflow up -d

# Start Kafka stack
docker compose --profile kafka up -d

# Start everything with monitoring
docker compose --profile airflow --profile kafka --profile monitoring up -d

Core Compose Commands

Bash
# Start all services (detached)
docker compose up -d

# Start and rebuild images
docker compose up -d --build

# Start a specific service
docker compose up -d postgres

# Stop all services (keep volumes)
docker compose down

# Stop and remove volumes (DESTRUCTIVE — deletes data)
docker compose down -v

# View logs
docker compose logs

# Follow specific service logs
docker compose logs -f airflow-scheduler

# Execute command in running service
docker compose exec postgres psql -U airflow

# Run one-off command (new container, then removed)
docker compose run --rm pipeline python -c "from src import pipeline; print('ok')"

# Check service status
docker compose ps

# Restart a service
docker compose restart airflow-webserver

# Pull latest images
docker compose pull

# Scale a service (multiple replicas)
docker compose up -d --scale airflow-worker=3

Practical Stack 1: Airflow + PostgreSQL + Redis + Flower

This is a production-grade local Airflow environment with CeleryExecutor.

YAML
# docker-compose.yml — Airflow CeleryExecutor Stack

x-airflow-common: &airflow-common
  image: apache/airflow:2.9.0
  environment: &airflow-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: "46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLt0nIe3aI="
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: "true"
    AIRFLOW__CORE__LOAD_EXAMPLES: "false"
    AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"
    AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: "true"
    _PIP_ADDITIONAL_REQUIREMENTS: ""
  env_file:
    - .env
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - ./config/airflow.cfg:/opt/airflow/airflow.cfg:ro
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on: &airflow-deps
    postgres:
      condition: service_healthy
    redis:
      condition: service_healthy
  networks:
    - data-net

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - ./initdb:/docker-entrypoint-initdb.d
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U airflow"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s
    restart: unless-stopped
    networks:
      - data-net

  redis:
    image: redis:7-alpine
    expose:
      - 6379
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped
    networks:
      - data-net

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: unless-stopped

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8974/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: unless-stopped

  airflow-worker:
    <<: *airflow-common
    command: celery worker
    environment:
      <<: *airflow-env
      DUMB_INIT_SETSID: "0"
    restart: unless-stopped
    healthcheck:
      test:
        - "CMD-SHELL"
        - 'celery --app airflow.providers.celery.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}" || celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s

  airflow-triggerer:
    <<: *airflow-common
    command: triggerer
    restart: unless-stopped

  airflow-init:
    <<: *airflow-common
    entrypoint: /bin/bash
    command:
      - -c
      - |
        airflow db migrate
        airflow users create \
          --username admin \
          --firstname Admin \
          --lastname User \
          --role Admin \
          --email admin@example.com \
          --password admin
    environment:
      <<: *airflow-env
      _AIRFLOW_DB_MIGRATE: "true"
      _AIRFLOW_WWW_USER_CREATE: "true"
      _AIRFLOW_WWW_USER_USERNAME: admin
      _AIRFLOW_WWW_USER_PASSWORD: admin
    profiles: ["init"]   # only runs when explicitly requested

  flower:
    <<: *airflow-common
    command: celery flower
    profiles: ["monitoring"]
    ports:
      - "5555:5555"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 10s
    restart: unless-stopped

volumes:
  postgres-data:
  airflow-logs:

networks:
  data-net:
    driver: bridge

First-time setup:

Bash
# Initialise DB and create admin user
docker compose --profile init run --rm airflow-init

# Start the stack
docker compose up -d

# Check status
docker compose ps

# Access Airflow UI: http://localhost:8080 (admin/admin)
# Access Flower:     http://localhost:5555

Practical Stack 2: Kafka + Zookeeper + Schema Registry

YAML
# docker-compose.kafka.yml — Confluent Kafka stack

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.6.0
    hostname: zookeeper
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    healthcheck:
      test: ["CMD-SHELL", "echo ruok | nc localhost 2181 | grep imok"]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - zookeeper-data:/var/lib/zookeeper/data
      - zookeeper-log:/var/lib/zookeeper/log
    networks:
      - kafka-net

  broker:
    image: confluentinc/cp-kafka:7.6.0
    hostname: broker
    container_name: broker
    depends_on:
      zookeeper:
        condition: service_healthy
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_LOG_RETENTION_HOURS: 168
      KAFKA_LOG_SEGMENT_BYTES: 1073741824
    healthcheck:
      test: ["CMD-SHELL", "kafka-broker-api-versions --bootstrap-server localhost:9092"]
      interval: 15s
      timeout: 10s
      retries: 5
      start_period: 30s
    ports:
      - "9092:9092"
      - "9101:9101"
    volumes:
      - kafka-data:/var/lib/kafka/data
    networks:
      - kafka-net

  schema-registry:
    image: confluentinc/cp-schema-registry:7.6.0
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      broker:
        condition: service_healthy
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: "broker:29092"
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8081/subjects"]
      interval: 15s
      timeout: 10s
      retries: 5
      start_period: 20s
    ports:
      - "8081:8081"
    networks:
      - kafka-net

  kafka-connect:
    image: confluentinc/cp-kafka-connect:7.6.0
    hostname: kafka-connect
    container_name: kafka-connect
    depends_on:
      broker:
        condition: service_healthy
      schema-registry:
        condition: service_healthy
    environment:
      CONNECT_BOOTSTRAP_SERVERS: "broker:29092"
      CONNECT_REST_ADVERTISED_HOST_NAME: kafka-connect
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
    ports:
      - "8083:8083"
    networks:
      - kafka-net

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    depends_on:
      broker:
        condition: service_healthy
      schema-registry:
        condition: service_healthy
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:29092
      KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://schema-registry:8081
      KAFKA_CLUSTERS_0_KAFKACONNECT_0_NAME: local-connect
      KAFKA_CLUSTERS_0_KAFKACONNECT_0_ADDRESS: http://kafka-connect:8083
    ports:
      - "8090:8080"
    networks:
      - kafka-net

  akhq:
    image: tchiotludo/akhq:0.24.0
    container_name: akhq
    profiles: ["monitoring"]
    environment:
      AKHQ_CONFIGURATION: |
        akhq:
          connections:
            local:
              properties:
                bootstrap.servers: "broker:29092"
              schema-registry:
                url: "http://schema-registry:8081"
    ports:
      - "8080:8080"
    depends_on:
      broker:
        condition: service_healthy
    networks:
      - kafka-net

volumes:
  zookeeper-data:
  zookeeper-log:
  kafka-data:

networks:
  kafka-net:
    driver: bridge
Bash
# Start Kafka stack
docker compose -f docker-compose.kafka.yml up -d

# Create a topic
docker compose -f docker-compose.kafka.yml exec broker \
  kafka-topics --create \
  --bootstrap-server localhost:9092 \
  --topic raw-events \
  --partitions 3 \
  --replication-factor 1

# List topics
docker compose -f docker-compose.kafka.yml exec broker \
  kafka-topics --list --bootstrap-server localhost:9092

# Produce test messages
docker compose -f docker-compose.kafka.yml exec broker \
  kafka-console-producer --bootstrap-server localhost:9092 --topic raw-events

# Consume messages
docker compose -f docker-compose.kafka.yml exec broker \
  kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic raw-events \
  --from-beginning

# Kafka UI: http://localhost:8090

Troubleshooting Common Issues

Bash
# Service won't start — check logs
docker compose logs --tail 50 airflow-scheduler

# Port already in use
lsof -i :8080   # or: netstat -tulnp | grep 8080

# Container keeps restarting
docker compose ps                    # check status column
docker inspect <container_id>        # check exit code, restart policy

# Volume data seems stale
docker compose down -v               # WARNING: deletes all volume data
docker compose up -d

# Out of disk space (common with Kafka)
docker system df                     # see disk usage by type
docker volume prune                  # remove unused volumes
docker image prune -a                # remove unused images

# Network issues between services
docker compose exec airflow-webserver curl http://postgres:5432
docker network inspect data-stack_data-net

What's Next

Your local data engineering environment is fully containerised and reproducible. Every team member runs docker compose up -d and gets an identical stack. The next lesson covers production hardening: non-root users, distroless images, secrets management, health checks in Dockerfiles, image scanning, and GitHub Actions CI/CD pipelines that build, scan, and push images to ECR.

Enjoyed this article?

Explore the Cloud & DevOps learning path for more.

Subscribe for weekly tutorials

Found this helpful?

Share:𝕏

Leave a comment

Have a question, correction, or just found this helpful? Leave a note below.