Docker in Production: Security, CI/CD, and Data Engineering Patterns

Why Production Docker Is Different

A Dockerfile that works in development can be a security liability in production. The default Python image runs as root. Secrets baked into images leak in docker history. Unscanned images carry known CVEs into live systems. Build context accidentally includes .env files.

This lesson covers the practices that separate hobby containers from production-grade images — and builds a complete GitHub Actions pipeline to enforce them automatically on every push.

Base Image Strategy

The Tradeoff Ladder

| Base Image | Size | Attack Surface | Build Complexity | |---|---|---|---| | python:3.11 (full Debian) | ~900 MB | Largest | None | | python:3.11-slim (Debian minimal) | ~130 MB | Moderate | None | | python:3.11-alpine (Alpine Linux) | ~55 MB | Small | Medium (musl libc) | | gcr.io/distroless/python3 | ~50 MB | Minimal | High (no shell) |

slim vs alpine for Python

alpine uses musl libc instead of glibc. Many Python packages with compiled extensions (numpy, pandas, psycopg2) need to be compiled from source on Alpine — dramatically increasing build time and image size.

DOCKERFILE

# Alpine — often LARGER for data pipelines due to compilation
FROM python:3.11-alpine
RUN apk add --no-cache gcc musl-dev postgresql-dev
RUN pip install pandas psycopg2   # compiles from source — 10+ minutes

# slim — smaller for data pipelines, no compilation issues
FROM python:3.11-slim
RUN apt-get update && apt-get install -y --no-install-recommends libpq5 && \
    rm -rf /var/lib/apt/lists/*
RUN pip install pandas psycopg2-binary   # pre-compiled wheels

Recommendation for data pipelines: use python:3.11-slim. Use Alpine only for simple services with no native extensions.

Distroless Images

Google's distroless images contain only the runtime — no package manager, no shell, no debugging tools. Zero unnecessary attack surface.

DOCKERFILE

# Multi-stage: build on slim, run on distroless
FROM python:3.11-slim AS builder
WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

FROM gcr.io/distroless/python3-debian12 AS runtime
COPY --from=builder /install /usr/local
COPY --from=builder /usr/local/lib/python3.11 /usr/local/lib/python3.11
COPY src/ /app/src/
COPY pipeline.py /app/
WORKDIR /app
USER nonroot
ENTRYPOINT ["python3", "pipeline.py"]

Tradeoff: distroless has no shell, so docker exec -it container bash does not work. Debug with ephemeral debug containers:

Bash

# Attach a debug sidecar to an existing distroless container
docker run --rm -it \
  --pid container:my-pipeline \
  --network container:my-pipeline \
  --volumes-from my-pipeline \
  busybox sh

Non-Root Users

Every container runs as root by default. A process exploit in a root container can escape to the host. Always add and switch to a dedicated user.

DOCKERFILE

FROM python:3.11-slim

# Create group and user (no home dir, no shell)
RUN groupadd --gid 1001 appgroup && \
    useradd \
      --uid 1001 \
      --gid appgroup \
      --no-create-home \
      --shell /usr/sbin/nologin \
      appuser

WORKDIR /app

# Install dependencies as root (before USER switch)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy files with correct ownership
COPY --chown=appuser:appgroup src/ /app/src/
COPY --chown=appuser:appgroup pipeline.py /app/

# Switch to non-root
USER appuser

CMD ["python", "pipeline.py"]

Verify in compose:

YAML

services:
  pipeline:
    image: my-pipeline:latest
    user: "1001:1001"   # explicit override — belt and suspenders

Secrets Management: Never Use ENV for Secrets

DOCKERFILE

# WRONG — secret visible in docker inspect, docker history, logs
ENV DB_PASSWORD=supersecret

# WRONG — ARG is also visible in docker history
ARG DB_PASSWORD
ENV DB_PASSWORD=${DB_PASSWORD}

Correct Approaches

1. Runtime environment injection (most common)

Pass secrets at container start time — never store them in the image:

Bash

# From a secrets manager at runtime
DB_PASSWORD=$(aws secretsmanager get-secret-value \
  --secret-id prod/db/password \
  --query SecretString --output text | jq -r .password)

docker run \
  -e DB_PASSWORD="${DB_PASSWORD}" \
  my-pipeline:latest

2. Docker Secrets (Swarm)

YAML

# docker-compose.yml with secrets
services:
  pipeline:
    image: my-pipeline:latest
    secrets:
      - db_password
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt   # dev only
    # external: true                  # production: created by swarm/k8s

In the application, read from the file:

Python

import os

def get_secret(env_var: str) -> str:
    """Read secret from file (Docker secrets) or environment variable."""
    file_path = os.environ.get(f"{env_var}_FILE")
    if file_path:
        with open(file_path) as f:
            return f.read().strip()
    return os.environ[env_var]

DB_PASSWORD = get_secret("DB_PASSWORD")

3. BuildKit secret mounts (build-time secrets)

DOCKERFILE

# syntax=docker/dockerfile:1
FROM python:3.11-slim AS builder

# Mount a secret during build — never stored in image layers
RUN --mount=type=secret,id=pip_token \
    PIP_INDEX_URL=$(cat /run/secrets/pip_token) \
    pip install --no-cache-dir -r requirements.txt

Bash

DOCKER_BUILDKIT=1 docker build \
  --secret id=pip_token,src=./pip_token.txt \
  -t my-pipeline:latest .

Health Checks in Dockerfile

Define health checks in the Dockerfile so they apply regardless of how the container is started.

DOCKERFILE

# HTTP service health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=20s --retries=3 \
    CMD curl -f http://localhost:8000/health || exit 1

# Python pipeline health check (verifies imports and connectivity)
HEALTHCHECK --interval=60s --timeout=15s --start-period=10s --retries=3 \
    CMD python -c "
import sys
try:
    import psycopg2
    import boto3
    print('Dependencies OK')
    sys.exit(0)
except ImportError as e:
    print(f'Import failed: {e}', file=sys.stderr)
    sys.exit(1)
"

# Script-based health check (most readable)
COPY healthcheck.sh /healthcheck.sh
RUN chmod +x /healthcheck.sh
HEALTHCHECK --interval=30s --timeout=10s --retries=3 CMD /healthcheck.sh

healthcheck.sh:

Bash

#!/bin/bash
set -e

# Check that the pipeline process is running
if ! pgrep -f "python pipeline.py" > /dev/null; then
    echo "Pipeline process not found" >&2
    exit 1
fi

# Check database connectivity
python -c "
import psycopg2, os
conn = psycopg2.connect(os.environ['DATABASE_URL'])
conn.close()
" || { echo "Database unreachable" >&2; exit 1; }

echo "Healthy"
exit 0

Resource Limits

In Docker Compose

YAML

services:
  pipeline:
    image: my-pipeline:latest
    deploy:
      resources:
        limits:
          cpus: "2.0"       # max 2 CPU cores
          memory: 4G        # max 4 GB RAM
        reservations:
          cpus: "0.5"       # guaranteed 0.5 cores
          memory: 512M      # guaranteed 512 MB

  airflow-worker:
    image: apache/airflow:2.9.0
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 2G

In docker run

Bash

docker run \
  --memory=4g \
  --memory-swap=4g \     # same as memory = no swap
  --cpus=2.0 \
  --memory-reservation=512m \
  my-pipeline:latest

Container Registries

AWS ECR

Bash

#!/bin/bash
# ecr-push.sh — push an image to ECR

set -euo pipefail

IMAGE_NAME="${1:?Image name required}"
IMAGE_TAG="${2:-latest}"
AWS_REGION="${AWS_REGION:-us-east-1}"

# Get account ID
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
ECR_REGISTRY="${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"
ECR_REPO="${ECR_REGISTRY}/${IMAGE_NAME}"

# Authenticate
echo "Authenticating with ECR..."
aws ecr get-login-password --region "${AWS_REGION}" | \
  docker login --username AWS --password-stdin "${ECR_REGISTRY}"

# Create repository if it doesn't exist
aws ecr describe-repositories --repository-names "${IMAGE_NAME}" 2>/dev/null || \
  aws ecr create-repository \
    --repository-name "${IMAGE_NAME}" \
    --image-scanning-configuration scanOnPush=true \
    --encryption-configuration encryptionType=AES256

# Tag and push
docker tag "${IMAGE_NAME}:${IMAGE_TAG}" "${ECR_REPO}:${IMAGE_TAG}"
docker push "${ECR_REPO}:${IMAGE_TAG}"

# Also push as latest if not already latest
if [[ "${IMAGE_TAG}" != "latest" ]]; then
  docker tag "${IMAGE_NAME}:${IMAGE_TAG}" "${ECR_REPO}:latest"
  docker push "${ECR_REPO}:latest"
fi

echo "Pushed: ${ECR_REPO}:${IMAGE_TAG}"

Azure Container Registry (ACR)

Bash

# Login to ACR
az acr login --name myregistry

# Tag and push
docker tag my-pipeline:latest myregistry.azurecr.io/my-pipeline:1.0.0
docker push myregistry.azurecr.io/my-pipeline:1.0.0

# Pull (with authentication)
az acr login --name myregistry
docker pull myregistry.azurecr.io/my-pipeline:1.0.0

Image Scanning with Trivy

Trivy is the industry standard for container image vulnerability scanning. Install it in CI to gate deployments on CVE severity.

Bash

# Install Trivy (Linux)
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin

# Scan a local image
trivy image my-pipeline:latest

# Scan with severity filter — fail CI on CRITICAL or HIGH
trivy image \
  --severity CRITICAL,HIGH \
  --exit-code 1 \
  my-pipeline:latest

# Scan and output as JSON (for SARIF upload to GitHub Security)
trivy image \
  --format sarif \
  --output trivy-results.sarif \
  my-pipeline:latest

# Scan only OS packages (not Python packages)
trivy image \
  --vuln-type os \
  --severity CRITICAL \
  my-pipeline:latest

# Scan from ECR (uses AWS credentials automatically)
trivy image \
  123456789012.dkr.ecr.us-east-1.amazonaws.com/my-pipeline:latest

Docker Scout (Docker's built-in scanner)

Bash

# Enable Docker Scout (requires Docker Hub account)
docker scout quickview my-pipeline:latest

# CVE report
docker scout cves my-pipeline:latest

# Filter by severity
docker scout cves --only-severity critical,high my-pipeline:latest

# Compare image versions
docker scout compare my-pipeline:1.0.0 --to my-pipeline:1.0.1

Complete GitHub Actions Workflow: Build, Scan, Push to ECR

This workflow runs on every push to main and every pull request. On PRs it scans only. On main merges it pushes to ECR.

YAML

# .github/workflows/pipeline-image.yml

name: Build, Scan, and Push Pipeline Image

on:
  push:
    branches: [main]
    paths:
      - "src/**"
      - "pipeline.py"
      - "Dockerfile"
      - "requirements*.txt"
      - ".github/workflows/pipeline-image.yml"
  pull_request:
    branches: [main]
    paths:
      - "src/**"
      - "pipeline.py"
      - "Dockerfile"
      - "requirements*.txt"

env:
  AWS_REGION: us-east-1
  ECR_REPOSITORY: my-pipeline
  IMAGE_NAME: my-pipeline

jobs:
  build-and-scan:
    name: Build and Security Scan
    runs-on: ubuntu-latest
    permissions:
      contents: read
      security-events: write    # for uploading SARIF to GitHub Security

    outputs:
      image-tag: ${{ steps.meta.outputs.version }}
      image-digest: ${{ steps.build.outputs.digest }}

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver-opts: image=moby/buildkit:master

      - name: Extract metadata (tags, labels)
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.IMAGE_NAME }}
          tags: |
            type=sha,prefix=sha-,format=short
            type=ref,event=branch
            type=semver,pattern={{version}}
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Build image (no push yet)
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile
          target: runtime
          push: false
          load: true              # load into local Docker daemon for scanning
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha   # GitHub Actions cache
          cache-to: type=gha,mode=max
          build-args: |
            BUILD_DATE=${{ github.event.repository.updated_at }}
            GIT_SHA=${{ github.sha }}

      - name: Run Trivy vulnerability scan
        id: trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.IMAGE_NAME }}:latest
          format: "sarif"
          output: "trivy-results.sarif"
          severity: "CRITICAL,HIGH"
          exit-code: "1"          # fail the job on CRITICAL/HIGH CVEs
          ignore-unfixed: true    # skip CVEs with no available fix

      - name: Upload Trivy results to GitHub Security
        if: always()              # upload even if scan failed
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: "trivy-results.sarif"

      - name: Run Dockerfile lint (hadolint)
        uses: hadolint/hadolint-action@v3.1.0
        with:
          dockerfile: Dockerfile
          failure-threshold: warning

  push-to-ecr:
    name: Push to ECR
    runs-on: ubuntu-latest
    needs: build-and-scan
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    permissions:
      contents: read
      id-token: write             # for OIDC AWS authentication

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS credentials (OIDC — no long-lived keys)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions-ecr-push
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ steps.login-ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}
          tags: |
            type=sha,prefix=sha-,format=short
            type=raw,value=latest,enable={{is_default_branch}}
            type=raw,value=prod-${{ github.run_number }}

      - name: Build and push to ECR
        id: push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile
          target: runtime
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          provenance: true        # SLSA provenance attestation
          sbom: true              # Software Bill of Materials

      - name: Output pushed image URI
        run: |
          echo "Image pushed to ECR:"
          echo "${{ steps.meta.outputs.tags }}" | tr ',' '\n'
          echo "Digest: ${{ steps.push.outputs.digest }}"

      - name: Notify on success
        if: success()
        run: |
          echo "::notice title=Image Pushed::Successfully pushed to ECR"
          echo "IMAGE_URI=${{ steps.login-ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}:latest" >> $GITHUB_ENV

Required AWS IAM Policy for the GitHub Actions role

JSON

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "ecr:InitiateLayerUpload",
        "ecr:UploadLayerPart",
        "ecr:CompleteLayerUpload",
        "ecr:PutImage",
        "ecr:DescribeRepositories",
        "ecr:CreateRepository"
      ],
      "Resource": "arn:aws:iam::123456789012:repository/my-pipeline"
    }
  ]
}

Running Python Pipelines as One-Shot Containers

Data pipelines often run on a schedule — not as persistent servers. The container runs, does the work, and exits. Kubernetes CronJobs and ECS Scheduled Tasks both support this pattern.

DOCKERFILE

# Dockerfile for a one-shot pipeline
FROM python:3.11-slim AS builder
WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

FROM python:3.11-slim AS runtime
RUN groupadd --gid 1001 pipeline && \
    useradd --uid 1001 --gid pipeline --no-create-home pipeline

COPY --from=builder /install /usr/local
WORKDIR /app
COPY --chown=pipeline:pipeline . /app/

USER pipeline

# No EXPOSE — this is not a server
# No restart policy — let the orchestrator handle retries

ENTRYPOINT ["python", "pipeline.py"]
CMD ["--mode", "incremental", "--date", "today"]

Run locally:

Bash

docker run --rm \
  --env-file .env \
  -e PIPELINE_DATE=2026-05-07 \
  my-pipeline:latest \
  --mode full --date 2026-05-07

ECS Scheduled Task (Terraform snippet):

HCL

resource "aws_ecs_task_definition" "pipeline" {
  family                   = "my-pipeline"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = 1024   # 1 vCPU
  memory                   = 2048   # 2 GB

  container_definitions = jsonencode([{
    name  = "pipeline"
    image = "${aws_ecr_repository.pipeline.repository_url}:latest"
    command = ["--mode", "incremental"]
    environment = [
      { name = "DB_HOST", value = var.db_host },
      { name = "S3_BUCKET", value = var.s3_bucket }
    ]
    secrets = [
      { name = "DB_PASSWORD", valueFrom = aws_secretsmanager_secret.db_password.arn }
    ]
    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = "/ecs/my-pipeline"
        "awslogs-region"        = var.aws_region
        "awslogs-stream-prefix" = "pipeline"
      }
    }
  }])
}

Kubernetes Pod Spec from Docker Image

When moving from Docker to Kubernetes, your Dockerfile settings map to pod spec fields:

YAML

apiVersion: v1
kind: Pod
metadata:
  name: my-pipeline
  labels:
    app: my-pipeline
spec:
  restartPolicy: OnFailure    # Never / Always / OnFailure

  # Equivalent to Dockerfile USER
  securityContext:
    runAsNonRoot: true
    runAsUser: 1001
    runAsGroup: 1001
    fsGroup: 1001

  containers:
    - name: pipeline
      image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-pipeline:sha-abc1234
      imagePullPolicy: Always

      # Equivalent to Dockerfile CMD override
      args: ["--mode", "incremental", "--date", "2026-05-07"]

      # Equivalent to -e / --env-file
      env:
        - name: DB_HOST
          value: "postgres-service"
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: pipeline-secrets
              key: db-password
        - name: S3_BUCKET
          valueFrom:
            configMapKeyRef:
              name: pipeline-config
              key: s3-bucket

      # Equivalent to --memory / --cpus
      resources:
        requests:
          memory: "512Mi"
          cpu: "250m"
        limits:
          memory: "2Gi"
          cpu: "1000m"

      # Equivalent to Dockerfile HEALTHCHECK
      livenessProbe:
        exec:
          command: ["python", "-c", "import src.pipeline; print('ok')"]
        initialDelaySeconds: 10
        periodSeconds: 30
        failureThreshold: 3

      # Volume mounts
      volumeMounts:
        - name: pipeline-output
          mountPath: /app/output

  volumes:
    - name: pipeline-output
      persistentVolumeClaim:
        claimName: pipeline-pvc

Data Engineering Patterns: dbt in Docker

Run dbt transformations in a container — no local Python environment required.

DOCKERFILE

# Dockerfile.dbt
FROM python:3.11-slim AS runtime

RUN pip install --no-cache-dir \
    dbt-core==1.8.0 \
    dbt-postgres==1.8.0 \
    dbt-snowflake==1.8.0

RUN groupadd --gid 1001 dbt && \
    useradd --uid 1001 --gid dbt --no-create-home dbt

WORKDIR /dbt
COPY --chown=dbt:dbt profiles/ /root/.dbt/
COPY --chown=dbt:dbt . /dbt/

USER dbt

ENTRYPOINT ["dbt"]
CMD ["run", "--profiles-dir", "/root/.dbt", "--project-dir", "/dbt"]

profiles/profiles.yml (injected at runtime via volume or secrets):

YAML

my_project:
  target: prod
  outputs:
    prod:
      type: postgres
      host: "{{ env_var('DB_HOST') }}"
      port: 5432
      user: "{{ env_var('DB_USER') }}"
      password: "{{ env_var('DB_PASSWORD') }}"
      dbname: "{{ env_var('DB_NAME') }}"
      schema: analytics
      threads: 4

Bash

# Run dbt models
docker run --rm \
  --env-file .env \
  -v $(pwd)/dbt-project:/dbt \
  my-dbt:latest run --select tag:daily

# Run dbt tests
docker run --rm \
  --env-file .env \
  -v $(pwd)/dbt-project:/dbt \
  my-dbt:latest test

# Generate and serve docs
docker run --rm \
  --env-file .env \
  -v $(pwd)/dbt-project:/dbt \
  -p 8080:8080 \
  my-dbt:latest docs serve --host 0.0.0.0

In docker-compose:

YAML

  dbt:
    build:
      context: ./dbt
      dockerfile: Dockerfile.dbt
    volumes:
      - ./dbt:/dbt
    env_file: .env
    profiles: ["dbt"]
    depends_on:
      postgres:
        condition: service_healthy
    networks:
      - data-net
    command: ["run", "--profiles-dir", "/root/.dbt", "--select", "tag:daily"]

Data Engineering Patterns: Airflow Workers in Docker

When using DockerOperator or KubernetesPodOperator, each task runs in its own container with isolated dependencies.

Python

# dags/pipeline_dag.py
from airflow import DAG
from airflow.providers.docker.operators.docker import DockerOperator
from datetime import datetime, timedelta

default_args = {
    "owner": "data-engineering",
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
}

with DAG(
    dag_id="daily_pipeline",
    default_args=default_args,
    schedule="0 6 * * *",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    tags=["production", "daily"],
) as dag:

    extract = DockerOperator(
        task_id="extract",
        image="123456789012.dkr.ecr.us-east-1.amazonaws.com/my-pipeline:latest",
        command=["--mode", "extract", "--date", "{{ ds }}"],
        environment={
            "DB_HOST": "{{ var.value.db_host }}",
            "S3_BUCKET": "{{ var.value.s3_bucket }}",
        },
        secrets=[
            Secret("env", "DB_PASSWORD", "pipeline-secrets", "db_password"),
        ],
        docker_url="unix://var/run/docker.sock",
        network_mode="data-stack_data-net",
        auto_remove=True,
        mount_tmp_dir=False,
    )

    transform = DockerOperator(
        task_id="transform",
        image="123456789012.dkr.ecr.us-east-1.amazonaws.com/my-pipeline:latest",
        command=["--mode", "transform", "--date", "{{ ds }}"],
        environment={
            "DB_HOST": "{{ var.value.db_host }}",
        },
        auto_remove=True,
    )

    load = DockerOperator(
        task_id="load",
        image="123456789012.dkr.ecr.us-east-1.amazonaws.com/my-pipeline:latest",
        command=["--mode", "load", "--date", "{{ ds }}"],
        auto_remove=True,
    )

    extract >> transform >> load

Production Hardening Checklist

Before deploying any container image to production:

Bash

# 1. Verify non-root user
docker inspect my-pipeline:latest | jq '.[0].Config.User'

# 2. Check no secrets in image
docker history my-pipeline:latest           # look for ENV with secret-like values
docker inspect my-pipeline:latest | jq '.[0].Config.Env'

# 3. Scan for CVEs
trivy image --severity CRITICAL,HIGH --exit-code 1 my-pipeline:latest

# 4. Check image size
docker images my-pipeline:latest

# 5. Verify healthcheck is defined
docker inspect my-pipeline:latest | jq '.[0].Config.Healthcheck'

# 6. Lint the Dockerfile
hadolint Dockerfile

# 7. Check .dockerignore is present and complete
cat .dockerignore

# 8. Verify multi-stage build (no dev deps in final image)
docker history my-pipeline:latest --no-trunc | grep -E "pip install|apt-get"

| Check | Command | Pass condition | |---|---|---| | Non-root user | docker inspect \| jq '.[0].Config.User' | "1001:1001" not "" | | No secrets in ENV | docker inspect \| jq '.[0].Config.Env' | No passwords/keys | | No CRITICAL CVEs | trivy image --severity CRITICAL | Exit code 0 | | Health check defined | docker inspect \| jq '.[0].Config.Healthcheck' | Not null | | Image size reasonable | docker images | < 500 MB for Python apps |

Summary

Production Docker for data engineering rests on five pillars:

Minimal images: slim or distroless base, multi-stage builds, .dockerignore. Smaller images pull faster and have fewer CVEs.
Non-root users: add a dedicated user, switch before CMD. One line — no excuse to skip it.
Secrets at runtime: never in ENV, never in ARG, never in COPY. Inject at container start from a secrets manager.
Scanning in CI: Trivy or Docker Scout on every build. Gate merges to main on CRITICAL/HIGH severity.
Health checks: defined in the Dockerfile, not only in Compose. Every container should be self-describing about its health.

The GitHub Actions workflow in this lesson enforces all five automatically. Copy it into your repository, configure the ECR repository ARN and the OIDC role, and you have a production-grade container supply chain on day one.

Docker in Production: Security, CI/CD, and Data Engineering Patterns

Why Production Docker Is Different

Base Image Strategy

The Tradeoff Ladder

slim vs alpine for Python

Distroless Images

Non-Root Users

Secrets Management: Never Use ENV for Secrets

Correct Approaches

Health Checks in Dockerfile

Resource Limits

In Docker Compose

In docker run

Container Registries

AWS ECR

Azure Container Registry (ACR)

Image Scanning with Trivy

Docker Scout (Docker's built-in scanner)

Complete GitHub Actions Workflow: Build, Scan, Push to ECR

Required AWS IAM Policy for the GitHub Actions role

Running Python Pipelines as One-Shot Containers

Kubernetes Pod Spec from Docker Image

Data Engineering Patterns: dbt in Docker

Data Engineering Patterns: Airflow Workers in Docker

Production Hardening Checklist

Summary

Enjoyed this article?

Leave a comment