fix(deploy): tolerate transient Get errors in wait loops - #5097
Draft
lubien wants to merge 1 commit into
Draft
Conversation
lubien
force-pushed
the
retry/wait-loops
branch
from
August 19, 2026 18:30
41d5467 to
05d5230
Compare
lubien
force-pushed
the
retry/wait-loops
branch
from
August 20, 2026 17:11
05d5230 to
a582fd0
Compare
Four poll loops \u2014 WaitForSmokeChecksToPass, WaitForHealthchecksToPass, WaitForEventType, WaitForEventTypeAfterType, plus the bluegreen strategy's WaitForGreenMachinesToBeHealthy \u2014 all called flapsClient.Get inside a for-loop and aborted the whole wait on any error. A single transient 408 or 5xx during health polling would fail a deploy that was otherwise about to succeed. Get is idempotent, and these loops already exist because state propagation takes time. The loops now check IsTransientFlapsError and keep polling on transient failures; only genuinely permanent errors (4xx other than 408/429, or non-flaps errors that aren't network hiccups) still abort the wait. The outer wait context caps total time.
lubien
force-pushed
the
retry/wait-loops
branch
from
August 20, 2026 18:56
a582fd0 to
61dacf9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four poll loops \u2014 WaitForSmokeChecksToPass, WaitForHealthchecksToPass,
WaitForEventType, WaitForEventTypeAfterType, plus the bluegreen
strategy's WaitForGreenMachinesToBeHealthy \u2014 all called flapsClient.Get
inside a for-loop and aborted the whole wait on any error. A single
transient 408 or 5xx during health polling would fail a deploy that was
otherwise about to succeed.
Get is idempotent, and these loops already exist because state
propagation takes time. The loops now check IsTransientFlapsError and
keep polling on transient failures; only genuinely permanent errors
(4xx other than 408/429, or non-flaps errors that aren't network
hiccups) still abort the wait. The outer wait context caps total time.