Skill 10 — Production Delivery: CI/CD Pipeline, Logging & Azure Monitor

What Production Delivery Actually Means

Pushing code to main and hoping for the best is not production delivery. Production delivery means:

Automated testing — every push runs the test suite before it can merge
Automated deployment — a successful merge deploys automatically
Structured logging — every event is searchable and filterable
Health monitoring — you know when the system is degraded before users do
Rollback — if something goes wrong, you can revert in under 2 minutes

GitHub Actions CI/CD Pipeline

YAML

# .github/workflows/ci.yml
name: PharmaBot CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  REGISTRY: pharmabotacr.azurecr.io
  IMAGE: pharmabot

jobs:
  # ── Job 1: Test ───────────────────────────────────────────────────────────
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: pgvector/pgvector:pg16
        env:
          POSTGRES_PASSWORD: test
          POSTGRES_DB: pharmabot_test
        ports: ["5432:5432"]
      redis:
        image: redis:7
        ports: ["6379:6379"]

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }

      - name: Install dependencies
        run: pip install -e ".[dev]"

      - name: Run tests
        run: pytest tests/ -v --tb=short
        env:
          DATABASE_URL: postgresql+asyncpg://postgres:test@localhost/pharmabot_test
          REDIS_URL: redis://localhost:6379
          MOCK_AZURE: "true"   # use mock Azure client in CI

  # ── Job 2: Build & Push ──────────────────────────────────────────────────
  build:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'

    steps:
      - uses: actions/checkout@v4

      - name: Log in to Azure Container Registry
        uses: azure/docker-login@v1
        with:
          login-server: ${{ env.REGISTRY }}
          username: ${{ secrets.ACR_USERNAME }}
          password: ${{ secrets.ACR_PASSWORD }}

      - name: Build and push Docker image
        run: |
          docker build -t $REGISTRY/$IMAGE:${{ github.sha }} .
          docker tag $REGISTRY/$IMAGE:${{ github.sha }} $REGISTRY/$IMAGE:latest
          docker push $REGISTRY/$IMAGE:${{ github.sha }}
          docker push $REGISTRY/$IMAGE:latest

  # ── Job 3: Deploy ────────────────────────────────────────────────────────
  deploy:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    environment: production

    steps:
      - name: Azure login
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}

      - name: Deploy to Container Apps
        run: |
          az containerapp update \
            --name pharmabot-api \
            --resource-group pharmabot-rg \
            --image ${{ env.REGISTRY }}/${{ env.IMAGE }}:${{ github.sha }}

      - name: Verify deployment health
        run: |
          sleep 15   # wait for container to start
          STATUS=$(curl -s https://pharmabot-api.example.azurecontainerapps.io/health | jq -r '.status')
          if [ "$STATUS" != "healthy" ]; then
            echo "Health check failed: $STATUS"
            exit 1
          fi

Structured Logging with structlog

Structured logs are JSON — they're searchable, filterable, and parseable by Azure Monitor:

Python

# pharmabot/logging.py
import structlog, logging, sys

def configure_logging(environment: str = "development"):
    processors = [
        structlog.contextvars.merge_contextvars,
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
    ]

    if environment == "production":
        # JSON output for Azure Monitor ingestion
        processors.append(structlog.processors.JSONRenderer())
    else:
        # Human-readable output for local development
        processors.append(structlog.dev.ConsoleRenderer())

    structlog.configure(
        processors=processors,
        wrapper_class=structlog.make_filtering_bound_logger(logging.INFO),
        logger_factory=structlog.PrintLoggerFactory(sys.stdout),
    )

Usage throughout the codebase:

Python

import structlog
log = structlog.get_logger()

# Bind context variables for the request lifetime
structlog.contextvars.bind_contextvars(session_id=request.session_id)

log.info("chat.request",  message_length=len(message), intent="interaction")
log.info("rag.retrieved", chunk_count=3, top_score=0.92, source="azure_search")
log.info("llm.response",  tokens_used=412, ttft_ms=380, cached=False)
log.warning("rate.limited", session_id=session_id, remaining=0)
log.error("llm.failed",   error=str(e), attempt=2)

Every log line is a JSON object with session_id bound — you can filter all events for a specific session in Azure Monitor.

Azure Monitor Integration

Python

# pharmabot/monitoring.py
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace

def setup_monitoring(connection_string: str):
    configure_azure_monitor(connection_string=connection_string)
    tracer = trace.get_tracer("pharmabot")
    return tracer

Add to main.py startup:

Python

if settings.azure_monitor_connection_string:
    setup_monitoring(settings.azure_monitor_connection_string)

Now every FastAPI request automatically appears in Azure Application Insights with:

Request duration and status code
Dependency calls (Azure OpenAI, Redis, PostgreSQL)
Exception traces
Custom events from log.info(...) calls

Key Metrics to Monitor

Set Azure Monitor alerts on:

| Metric | Alert threshold | Meaning | |---|---|---| | requests/failed | > 5% per 5 min | Something is broken | | dependencies/duration (Azure OpenAI) | p95 > 10s | LLM latency too high | | Rate limit responses (429) | > 20/min | Possible abuse or bot traffic | | Container restarts | > 0 per hour | Application is crashing | | Redis memory | > 80% | Session cache about to fill |

Rolling Back a Bad Deployment

If the health check fails post-deploy, roll back to the previous image:

Bash

# Get the previous commit SHA
PREV_SHA=$(git rev-parse HEAD~1)

# Redeploy previous image
az containerapp update \
  --name pharmabot-api \
  --resource-group pharmabot-rg \
  --image pharmabotacr.azurecr.io/pharmabot:$PREV_SHA

With Container Apps, this takes ~15 seconds — zero downtime because old containers keep running until new ones are healthy.

Checkpoint

Push a small change to main and watch the full CI/CD pipeline:

Bash

echo "# Production deployed $(date)" >> CHANGELOG.md
git add CHANGELOG.md && git commit -m "test: verify CI/CD pipeline"
git push origin main

Watch the Actions tab in GitHub — you should see three jobs complete in sequence: test → build → deploy. The final job verifies the health check passes before marking the deployment complete.

Skill 10 — Production Delivery: CI/CD Pipeline, Logging & Azure Monitor

What Production Delivery Actually Means

GitHub Actions CI/CD Pipeline

Structured Logging with structlog

Azure Monitor Integration

Key Metrics to Monitor

Rolling Back a Bad Deployment

Checkpoint

Enjoyed this article?

Leave a comment