orchestratord: use leader election and run multiple operator replicas#37806
Draft
alex-hunt-materialize wants to merge 1 commit into
Draft
orchestratord: use leader election and run multiple operator replicas#37806alex-hunt-materialize wants to merge 1 commit into
alex-hunt-materialize wants to merge 1 commit into
Conversation
Run each of orchestratord's controllers (materialize, balancer, console) under k8s-controller's lease-based leader election, so that multiple replicas of orchestratord can run while only one replica reconciles each controller at a time. The other replicas keep serving the conversion webhook, which avoids webhook downtime during operator rollouts and node drains. When a replica loses a leadership lease it exits and lets Kubernetes restart it, so no in-flight reconciliation outlives the lease. Helm chart changes: default to 2 operator replicas, create a PodDisruptionBudget when running more than one replica, and grant the operator get/create/update on coordination.k8s.io leases. Add a leader-failover workflow to test/orchestratord that verifies the leases are held by operator pods, that leadership fails over when the leader pod is deleted and after a rolling restart of the operator deployment, and that reconciliation completes after both. Enable it in the Nightly pipeline. NOTE: this requires a k8s-controller release with leader election support. The dependency bump lands separately once the library is released. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Rolling out orchestratord updates causes short downtime of the CRD conversion webhook when running a single replica. Running multiple replicas fixes that, but requires that only one replica reconciles at a time.
Changes
coordination.k8s.io/v1Lease in the operator's namespace; standby replicas keep serving the conversion webhook while waiting on the lease. When a replica loses a leadership lease it exits and lets Kubernetes restart it, so no in-flight reconciliation outlives the lease.operator.podDisruptionBudgetvalues), and grant the operatorget/create/updateonleasesincoordination.k8s.io.Testing
leader-failoverworkflow intest/orchestratord/mzcompose.py, enabled in the Nightly pipeline. It verifies that all controller leases are held by operator pods, that leadership fails over both when the leader pod is deleted and after a rolling restart of the operator deployment, and that reconciliation completes after each failover.Blocked on
Draft until MaterializeInc/k8s-controller#51 merges and is released: this PR does not build against the released k8s-controller 0.11.0. Once the library is released, the
k8s-controllerversion bump inCargo.toml/Cargo.lockwill be added here.🤖 Generated with Claude Code