Conversation
The gRPC server runnable was added to the controller-runtime manager without implementing LeaderElectionRunnable, so it only ran on the leader. With more than one replica the other pods never listened on the plugin port, failed the readiness probe and stayed NotReady, and the Service could only ever route to the leader. The gRPC handlers only read from the informer cache, so every replica can serve them. Make the runnable opt out of leader election; the ObjectStore controller keeps using leader election as before. Add an e2e test that scales the deployment to two replicas, checks both become ready, deletes the leader pod and verifies that a new leader is elected and clusters still reconcile. Document how to run multiple replicas. Closes #620 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #620
Problem
Running the plugin Deployment with more than one replica does not work today. The gRPC server is added to the controller-runtime manager as a plain runnable, without implementing
LeaderElectionRunnable, so controller-runtime only starts it on the pod holding the leader lease. The other replicas never listen on port 9090, fail the TCP readiness probe and stayNotReadyforever. The Service can only route to the leader, and a leader crash means waiting for the lease to expire before anything answers again.Fix
Make the gRPC runnable opt out of leader election. Every replica now serves the plugin interface, so the Service load-balances across all ready pods. The handlers only read
ObjectStoreobjects from the informer cache and never write, so serving them from every replica is safe. TheObjectStorecontroller keeps using leader election as before, so there is still a single writer.The Deployment strategy stays
Recreate. Switching toRollingUpdatewould run two plugin versions side by side during an upgrade and turn gRPC compatibility between consecutive releases into a contract, which deserves its own discussion. Multiple replicas therefore protect against pod or node loss, not against upgrade downtime; the docs say so.Testing
NeedLeaderElection().Seriale2e spec: scale the Deployment to two replicas, check both become ready (this fails onmain), find the leader through the lease, create anObjectStoreand aClusterthrough the load-balanced Service, delete the leader pod, wait for a new leader and both replicas ready again, then create a secondClusterto check reconciliation still works after the failover. The Deployment is scaled back to one replica afterwards.