[alex-hunt-materialize]

The balancerd and console Deployments had no PDB, so a drain could evict all replicas together. Each controller now applies a PDB with maxUnavailable: 1 over its Deployment's pods. It is created unconditionally, since at one replica it never blocks a drain. Externally scaled balancers get one too.

The operator ClusterRole gains policy/poddisruptionbudgets. Any RBAC managed outside the Helm chart needs the same rule.

Tests: unit test pdb_selects_deployment_pods, and PDB checks in the BalancerdEnabled and ConsoleEnabled modifications of test/orchestratord/mzcompose.py.

Closes: CLO-395

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- PDB blocks every eviction once all pods are unready, including the single-replica case

src/orchestratord/src/k8s.rs:110

The new PDB leaves unhealthyPodEvictionPolicy unset, so Kubernetes applies IfHealthyBudget. Under that policy a Running-but-not-Ready pod (CrashLoopBackOff, failing readiness) cannot be evicted whenever no pod of the Deployment is healthy. So a broken balancerd or console now blocks kubectl drain, cluster-autoscaler scale-down, Karpenter disruption and managed node-pool upgrades indefinitely, where before this PR those pods never blocked anything. This also holds at one replica, which contradicts the doc comment at k8s.rs:109 and the PR's reason for creating the PDB unconditionally.

Details

With maxUnavailable: 1, the disruption controller sets desiredHealthy = expectedPods - 1 and disruptionsAllowed = max(0, currentHealthy - desiredHealthy). The eviction subresource (pkg/registry/core/pod/storage/eviction.go) only waves an unready pod through without spending budget if currentHealthy >= desiredHealthy && desiredHealthy > 0. Otherwise it falls through to checkAndDecrement, which returns 429 when disruptionsAllowed == 0. Only Pending, terminal, or already-deleting pods skip the PDB entirely.

• 1 replica, pod crashlooping: desiredHealthy = 0 fails the > 0 guard and disruptionsAllowed = 0, so eviction is refused.
• 2 replicas (the default for both), both unready: currentHealthy 0 < desiredHealthy 1, so both evictions are refused.

Realistic triggers include a balancerd or console image that crashes on startup, and balancerd's /api/readyz returning 503 on every pod at the connection high watermark (src/balancerd/src/lib.rs:428). In each case the frontend is already down, and the PDB only adds a stuck node drain or a failed node-pool upgrade on top. The mzcompose check only asserts the all-healthy state, so it does not exercise this.

Fix: set unhealthy_pod_eviction_policy: Some("AlwaysAllow".into()) in PodDisruptionBudgetSpec. This is the policy the Kubernetes docs recommend for exactly this drain-blocking case. It is GA since 1.31 and on by default since 1.27, and the chart requires 1.29+. Healthy pods stay serialized by maxUnavailable: 1, and the "never blocks a drain at one replica" claim then holds.

[alex-hunt-materialize]

Fixed in ea90e2a3bb: the PDB now sets unhealthyPodEvictionPolicy: AlwaysAllow, asserted in the unit test and the mzcompose check.

[rjimeno-mz]

CI passes now. AlwaysAllow addresses the unhealthy-pod drain issue.