Conversation
Hetzner answers 423 locked while an action is still running on the balancer, and the operator skipped the target on that answer as if it were permanently invalid. A node added right after another change could stay out of the balancer until the next reconcile. Tell temporary rejections (locked, conflict, robot_unavailable, 5xx) from permanent ones and retry the former up to twice, after 1s and 2s. A target that still fails is skipped as before, and a 429 is still returned at once for the rate limit gate. Once one target has used up its retries, the rest of the run does not retry: the lock is on the whole balancer, and waiting per target would stall the service for minutes on a large balancer. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
This was referenced Sep 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A target added while another action is still running on the balancer gets skipped as if the IP were invalid (#54). Hetzner answers 423
lockedin that window.The add-target loop now retries a temporary rejection (
locked,conflict,robot_unavailableor a 5xx) twice, after 1s and 2s. A permanent one, like an IP outside the vSwitch subnet, is skipped as before. A 429 is still returned at once for the rate limit gate. After one target has used up its retries, the other targets in that run are not retried, because the lock is on the whole balancer.I did not add waiting for the running action to finish. That costs extra API calls per target, and the retry plus the existing 30 second requeue covers the short window. Other balancer calls (removing targets, changing services) still fail on
lockedas before.tokio::timerelies on the tokiotimefeature; #52 declares it explicitly.Stacked on #49.
Closes #54