Platform Engineering: Building Kubernetes Operators with Kubebuilder — Controller Patterns, Reconcilers, and Production Hardening

When to Build an Operator

A Kubernetes operator is a custom controller that extends the Kubernetes API with domain-specific knowledge. Before building one, answer these questions:

Build an operator when:

The resource has a complex lifecycle — create, configure, monitor, upgrade, failover, clean up
You need a feedback loop — the controller reacts to runtime events (not just apply-time)
You're encoding domain expertise — a PostgreSQL operator knows how to do pg_upgrade; Helm doesn't
You want drift correction — someone manually edits a resource, the operator resets it

Don't build an operator when:

Helm or Kustomize can handle it (stateless config rendering)
A community operator already exists (CloudNativePG for PostgreSQL, Strimzi for Kafka, Argo CD for GitOps)
The lifecycle is simple enough for a CronJob

Prefer community operators over building your own. Every operator you build is a 24/7 on-call engineer embedded in code — it needs to handle every edge case, be tested, upgraded, and monitored.

Kubebuilder: Project Structure

Bash

# Scaffold a new operator project
kubebuilder init --domain platform.example.com --repo github.com/org/my-operator

# Create an API (CRD + controller)
kubebuilder create api \
  --group infra \
  --version v1alpha1 \
  --kind DatabaseClaim \
  --resource \
  --controller

Generated structure:

my-operator/
├── api/
│   └── v1alpha1/
│       ├── databaseclaim_types.go      # CRD schema (Go struct → Kubernetes CRD)
│       └── groupversion_info.go
├── config/
│   ├── crd/                            # Generated CRD YAML
│   ├── rbac/                           # Generated ClusterRole YAML
│   └── manager/                        # Controller deployment
├── controllers/
│   └── databaseclaim_controller.go     # Your reconciler logic
├── main.go                             # Manager setup and startup
└── Makefile                            # generate, manifests, build, test targets

Defining the CRD

The CRD schema is derived from a Go struct with // +kubebuilder: markers:

// api/v1alpha1/databaseclaim_types.go

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
// +kubebuilder:validation:XValidation:rule="self.spec.storageGB >= 1",message="storageGB must be at least 1"
type DatabaseClaim struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   DatabaseClaimSpec   `json:"spec,omitempty"`
	Status DatabaseClaimStatus `json:"status,omitempty"`
}

type DatabaseClaimSpec struct {
	// +kubebuilder:validation:Enum=postgres;mysql;redis
	Engine string `json:"engine"`

	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=10000
	StorageGB int `json:"storageGB"`

	// +kubebuilder:validation:Enum=staging;production
	Environment string `json:"environment"`

	// +optional
	// +kubebuilder:default=false
	DeletionProtection bool `json:"deletionProtection,omitempty"`
}

type DatabaseClaimStatus struct {
	// +optional
	// +kubebuilder:validation:Enum=Provisioning;Ready;Failed;Deleting
	Phase string `json:"phase,omitempty"`

	// +optional
	Conditions []metav1.Condition `json:"conditions,omitempty"`

	// +optional
	ConnectionSecretRef string `json:"connectionSecretRef,omitempty"`

	// +optional
	LastProvisionedAt *metav1.Time `json:"lastProvisionedAt,omitempty"`
}

Generate CRD YAML from markers:

Bash

make manifests   # runs controller-gen, writes config/crd/*.yaml
make generate    # runs controller-gen for DeepCopy methods

The Reconciler: Core Patterns

// controllers/databaseclaim_controller.go

type DatabaseClaimReconciler struct {
	client.Client
	Scheme   *runtime.Scheme
	Recorder record.EventRecorder
	// inject your external clients (AWS SDK, Vault client, etc.)
	RDSClient rds.Client
}

// +kubebuilder:rbac:groups=infra.platform.example.com,resources=databaseclaims,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=infra.platform.example.com,resources=databaseclaims/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=infra.platform.example.com,resources=databaseclaims/finalizers,verbs=update
// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch;create;update;patch

func (r *DatabaseClaimReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := log.FromContext(ctx)

	// 1. Fetch the resource
	claim := &infrav1alpha1.DatabaseClaim{}
	if err := r.Get(ctx, req.NamespacedName, claim); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// 2. Handle deletion (finalizer pattern)
	if !claim.DeletionTimestamp.IsZero() {
		return r.handleDeletion(ctx, claim)
	}

	// 3. Ensure finalizer is registered
	if !controllerutil.ContainsFinalizer(claim, "infra.platform.example.com/cleanup") {
		controllerutil.AddFinalizer(claim, "infra.platform.example.com/cleanup")
		if err := r.Update(ctx, claim); err != nil {
			return ctrl.Result{}, err
		}
		return ctrl.Result{Requeue: true}, nil
	}

	// 4. Observe-diff-act: check external resource state
	exists, dbInstance, err := r.RDSClient.DescribeDB(ctx, claim.Name)
	if err != nil {
		r.setCondition(claim, "Ready", metav1.ConditionFalse, "DescribeFailed", err.Error())
		r.Recorder.Event(claim, corev1.EventTypeWarning, "DescribeFailed", err.Error())
		if err := r.Status().Update(ctx, claim); err != nil {
			return ctrl.Result{}, err
		}
		// Transient error: requeue with backoff
		return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
	}

	if !exists {
		return r.createDatabase(ctx, claim)
	}

	// 5. Reconcile desired vs actual state
	if !r.isUpToDate(claim, dbInstance) {
		return r.updateDatabase(ctx, claim, dbInstance)
	}

	// 6. Update status to reflect current state
	claim.Status.Phase = "Ready"
	r.setCondition(claim, "Ready", metav1.ConditionTrue, "Provisioned", "Database is available")
	if err := r.Status().Update(ctx, claim); err != nil {
		return ctrl.Result{}, err
	}

	// 7. Requeue periodically to detect out-of-band changes
	return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

The Observe-Diff-Act Pattern

Every reconcile loop must be idempotent — assume nothing about the current state. The loop always runs as if it's the first time:

1. Read the desired state (DatabaseClaim.Spec)
2. Read the actual state (AWS RDS describe call)
3. Compute the diff
4. Apply the smallest change needed to converge

Never assume the previous reconcile completed. If the reconciler crashed mid-way through creating a database, the next reconcile may find a partially-created database. Handle this gracefully.

Status Conditions

Use metav1.Condition (standardized condition type) instead of custom status fields:

func (r *DatabaseClaimReconciler) setCondition(
	claim *infrav1alpha1.DatabaseClaim,
	conditionType string,
	status metav1.ConditionStatus,
	reason, message string,
) {
	meta.SetStatusCondition(&claim.Status.Conditions, metav1.Condition{
		Type:               conditionType,
		Status:             status,
		Reason:             reason,
		Message:            message,
		ObservedGeneration: claim.Generation,
	})
}

This enables:

Bash

# Wait for the condition to become true
kubectl wait databaseclaim/my-db \
  --for=condition=Ready=True \
  --timeout=5m

Standard condition types to implement: Ready, Synced, Degraded.

Finalizers: Safe External Resource Cleanup

Finalizers prevent Kubernetes from deleting an object until the controller confirms cleanup:

func (r *DatabaseClaimReconciler) handleDeletion(
	ctx context.Context,
	claim *infrav1alpha1.DatabaseClaim,
) (ctrl.Result, error) {
	log := log.FromContext(ctx)

	if !controllerutil.ContainsFinalizer(claim, "infra.platform.example.com/cleanup") {
		return ctrl.Result{}, nil
	}

	// Check deletion protection
	if claim.Spec.DeletionProtection {
		log.Info("Deletion protection enabled, skipping RDS deletion")
		r.Recorder.Event(claim, corev1.EventTypeWarning, "DeletionProtected",
			"DeletionProtection is enabled. Remove it before deleting.")
		// Keep requeueing — user must explicitly disable protection
		return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
	}

	// Delete the external resource
	log.Info("Deleting RDS instance", "name", claim.Name)
	if err := r.RDSClient.DeleteDB(ctx, claim.Name); err != nil {
		// Check if already gone
		if !rds.IsNotFound(err) {
			return ctrl.Result{}, err
		}
	}

	// Remove finalizer — Kubernetes GC will delete the object now
	controllerutil.RemoveFinalizer(claim, "infra.platform.example.com/cleanup")
	if err := r.Update(ctx, claim); err != nil {
		return ctrl.Result{}, err
	}

	log.Info("RDS instance deleted and finalizer removed")
	return ctrl.Result{}, nil
}

Never remove the finalizer before the external resource is cleaned up. If the controller panics between external deletion and finalizer removal, the external resource is already gone — the next reconcile will find IsNotFound and remove the finalizer cleanly.

Owner References: Automatic Child Resource Cleanup

When the controller creates child resources (Secrets, ConfigMaps, Services), set owner references so Kubernetes GC cleans them up when the parent is deleted:

// Create the connection Secret with an owner reference to the DatabaseClaim
secret := &corev1.Secret{
	ObjectMeta: metav1.ObjectMeta{
		Name:      claim.Name + "-credentials",
		Namespace: claim.Namespace,
	},
	StringData: map[string]string{
		"host":     dbEndpoint,
		"port":     "5432",
		"username": username,
		"password": password,
	},
}

// Set owner reference — Secret is garbage-collected when DatabaseClaim is deleted
if err := controllerutil.SetControllerReference(claim, secret, r.Scheme); err != nil {
	return ctrl.Result{}, err
}

if err := r.Create(ctx, secret); err != nil {
	if !apierrors.IsAlreadyExists(err) {
		return ctrl.Result{}, err
	}
}

Event Filtering: Avoid Unnecessary Reconciles

The controller watches all DatabaseClaim objects. By default, every update (including status updates) triggers a reconcile. Filter to avoid loops:

func (r *DatabaseClaimReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&infrav1alpha1.DatabaseClaim{},
			// Only reconcile on Spec changes, not Status updates
			builder.WithPredicates(predicate.GenerationChangedPredicate{}),
		).
		// Also watch owned Secrets — if the Secret is deleted, reconcile to recreate it
		Owns(&corev1.Secret{}).
		// Watch with a rate limiter to prevent thundering herd on restart
		WithOptions(controller.Options{
			MaxConcurrentReconciles: 5,
			RateLimiter: workqueue.NewItemExponentialFailureRateLimiter(
				5*time.Second,   // base delay
				5*time.Minute,   // max delay
			),
		}).
		Complete(r)
}

GenerationChangedPredicate is critical: it prevents the controller from reconciling when only the status changes (which the controller itself writes), breaking infinite loops.

Exponential Backoff for External API Errors

External API calls (AWS, Azure, Vault) can be flaky. Use exponential backoff, not fixed delays:

// In the reconciler, return errors for unrecoverable issues (operator will back off)
// Return ctrl.Result{RequeueAfter: ...} for expected waits

func (r *DatabaseClaimReconciler) createDatabase(
	ctx context.Context,
	claim *infrav1alpha1.DatabaseClaim,
) (ctrl.Result, error) {
	// Set phase to Provisioning
	claim.Status.Phase = "Provisioning"
	r.setCondition(claim, "Ready", metav1.ConditionFalse, "Provisioning", "Creating database")
	if err := r.Status().Update(ctx, claim); err != nil {
		return ctrl.Result{}, err
	}

	// Initiate creation
	if err := r.RDSClient.CreateDB(ctx, claim); err != nil {
		// Return error — controller-runtime's exponential backoff will handle retry
		return ctrl.Result{}, fmt.Errorf("creating RDS: %w", err)
	}

	// RDS creation is async — requeue to check status
	return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

The workqueue rate limiter handles retries with exponential backoff when errors are returned.

Testing Operators

Unit tests with envtest

// controllers/suite_test.go

var (
	cfg       *rest.Config
	k8sClient client.Client
	testEnv   *envtest.Environment
)

func TestAPIs(t *testing.T) {
	RegisterFailHandler(Fail)
	RunSpecs(t, "Controller Suite")
}

var _ = BeforeSuite(func() {
	testEnv = &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
	}
	var err error
	cfg, err = testEnv.Start()
	Expect(err).NotTo(HaveOccurred())

	k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
	Expect(err).NotTo(HaveOccurred())
})

Controller integration test

var _ = Describe("DatabaseClaim controller", func() {
	const claimName = "test-db"
	const namespace = "default"
	ctx := context.Background()

	It("should set status.phase to Provisioning after creation", func() {
		// Arrange
		claim := &infrav1alpha1.DatabaseClaim{
			ObjectMeta: metav1.ObjectMeta{
				Name:      claimName,
				Namespace: namespace,
			},
			Spec: infrav1alpha1.DatabaseClaimSpec{
				Engine:    "postgres",
				StorageGB: 20,
				Environment: "staging",
			},
		}
		Expect(k8sClient.Create(ctx, claim)).To(Succeed())

		// Act + Assert
		Eventually(func() string {
			found := &infrav1alpha1.DatabaseClaim{}
			_ = k8sClient.Get(ctx, types.NamespacedName{Name: claimName, Namespace: namespace}, found)
			return found.Status.Phase
		}, timeout, interval).Should(Equal("Provisioning"))
	})
})

Production Hardening Checklist

Before shipping an operator to production:

✓ Status conditions (metav1.Condition) — enables kubectl wait and monitoring
✓ Finalizers — prevents orphaned external resources
✓ Owner references — automatic child resource cleanup
✓ Idempotent reconciler — safe to run multiple times on same state
✓ GenerationChangedPredicate — no reconcile on status-only updates
✓ Rate limiter (exponential backoff) — handles flaky external APIs
✓ Recorder.Event() — visible in kubectl describe for debugging
✓ Structured logging (log.FromContext) — with resource name in context
✓ Metrics: reconcile duration, error count, queue depth
✓ Leader election — safe multi-replica deployment (only one leader reconciles)
✓ Graceful shutdown — finish in-flight reconciles before exit
✓ envtest integration tests — test reconcile logic against real K8s API
✓ Chaos test — what happens if the operator crashes mid-reconcile?

Leader election

// main.go
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
	LeaderElection:          true,
	LeaderElectionID:        "database-claim-operator.platform.example.com",
	LeaderElectionNamespace: "platform-system",
})

With leader election, you can run 3 replicas of the operator. Only the leader reconciles; the others take over immediately if the leader's lease expires. This gives HA without duplicate reconciles.

Platform Engineering: Building Kubernetes Operators with Kubebuilder — Controller Patterns, Reconcilers, and Production Hardening

When to Build an Operator

Kubebuilder: Project Structure

Defining the CRD

The Reconciler: Core Patterns

The Observe-Diff-Act Pattern

Status Conditions

Finalizers: Safe External Resource Cleanup

Owner References: Automatic Child Resource Cleanup

Event Filtering: Avoid Unnecessary Reconciles

Exponential Backoff for External API Errors

Testing Operators

Unit tests with envtest

Controller integration test

Production Hardening Checklist

Leader election

Enjoyed this article?

Leave a comment