Clear attempted regions when a connect failover cycle ends - #1027
Open
xianshijing-lk wants to merge 4 commits into
Open
xianshijing-lk wants to merge 4 commits into
xianshijing-lk wants to merge 4 commits into
Conversation
`clearAttemptedRegions()` had a single call site — a successful reconnect — so the initial-connect failover loop in `Room.connect` never cleared the set it fills. The provider outlives one connect: `connect` reuses an existing provider for the same url and only creates one when it is null. Two consequences follow. After a successful connect, a region that failed transiently before another one succeeded stays excluded for the life of the provider, and a later failover skips it even once it has recovered. After an exhausted connect the set holds every region, so the next `connect` call resolves no region at all — it fails on the provided url and rethrows without trying a single fallback. Clear the set at both ends of the cycle, matching client-sdk-flutter, which resets on success, on exhaustion, and on a certificate-pinning failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
xianshijing-lk
requested review from
MaxHeimbrock and
davidliu
as code owners
September 21, 2026 22:19
|
MaxHeimbrock
force-pushed
the
sxian/reset-attempted-regions-on-connect-cycle-end
branch
from
September 22, 2026 08:36
1f5a97f to
aac93e9
Compare
Contributor
|
Diffuse output: AARJAR |
`getNextBestRegionUrl` refreshes region settings once the cache expires and propagates request and decoding failures, so it can throw from inside the catch and carry the exception out of the loop — past both exit-specific clears. The cycle would then end with regions still marked attempted, and `connect` reuses the provider for the same url, so a later failover skips them. Scoping the loop with try/finally covers every exit at once: connected, out of regions, a settings refresh that threw, and cancellation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`regionUrlProvider` is a mutable field and `prepareConnection` replaces it without holding `stateLock`, so reading it separately in the loop and in `finally` can touch two different instances: the engine keeps the one the cycle actually used while the clear lands on its replacement, leaving the engine's provider holding regions a later failover then skips. Read it once into a local and use that for the engine handoff, the next-region lookup, and the clear. Serializing provider installation against the connection state transition is the other half of that race and is left alone here — it changes `prepareConnection`'s locking, which is outside this fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
davidliu
approved these changes
Sep 22, 2026
davidliu
left a comment
Contributor
There was a problem hiding this comment.
LGTM, just needs a changeset
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while auditing the equivalent path in client-sdk-swift (#1120), where review surfaced the same class of bug.
Problem
RegionUrlProvider.clearAttemptedRegions()has exactly one call site —RTCEngine.kt:751, on a successful reconnect. The initial-connect failover loop inRoom.connectfillsattemptedRegionsand never clears it:The provider outlives a single connect —
connectreuses an existing one for the same url and creates one only when it isnull(Room.kt:530-534) — so the set leaks across cycles two ways.A region that failed transiently stays excluded. Region A fails, B connects. A remains in
attemptedRegionsfor the life of the provider. Later B fails while A has recovered; failover skips A and reports failure instead of connecting.An exhausted cycle poisons the next connect. After
throw ethe set holds every region. The nextconnecttries the provided url, fails, getsnullfromgetNextBestRegionUrl(), and rethrows — without attempting a single fallback region. This is the more serious of the two: region failover is silently disabled from then on.prepareConnectionmasks it in some flows, since it builds a fresh provider (Room.kt:442-443) — but it also callsgetNextBestRegionUrl(), so it adds to the set of the provider it creates. Apps that don't call it get the unmasked behaviour.Fix
Scope the failover loop with
try/finallyand clear the captured provider there, so every exit ends the cycle: connected, out of regions, a settings refresh that threw, or cancellation.Two review rounds shaped this (thanks @MaxHeimbrock, Devin):
getNextBestRegionUrl()refreshes settings once the cache expires and propagates request/decode failures. It is called from inside thecatch, so a throw carries the exception out of the loop past any exit-specific clear.finallycovers that.regionUrlProvideris aprivate varthatprepareConnectionwrites withoutstateLock, so the loop and the clear could touch different instances, leaving the engine's provider dirty. It is now captured once per cycle and used for the engine handoff, the lookup, and the clear.Not addressed here: serializing provider installation against the connection state transition. That changes
prepareConnection's locking on a path this PR doesn't otherwise touch, and the gap is pre-existing.client-sdk-flutter resets at three exits — success, exhaustion, certificate-pinning failure (
engine.dart:1302,1311,1320) — but itsgetNextBestRegionUrl()can also throw (region_url_provider.dart:43-49, viafetchRegionSettings) and its call site atengine.dart:1311is unprotected, so it has the same gap this PR closes. Worth a follow-up there. Swift is converging on the same scoping in livekit/client-sdk-swift#1120.Test
clearAttemptedRegionsMakesEveryRegionEligibleAgain— takes two regions, clears, and asserts the best region is offered again rather than the next unattempted one, with no extra settings fetch.Verification
Not built or run locally — this checkout can't compile the SDK:
protocrejects(logger.redact)inlivekit_models.protoat the pinned protocol submodule, and the newer submodule my checkout had fails onDataTrackSchema.kt. Both failures predate this change and are unrelated to it. Relying on CI; happy to iterate if it finds anything.