Skip to content

Clear attempted regions when a connect failover cycle ends - #1027

Open
xianshijing-lk wants to merge 4 commits into
mainfrom
sxian/reset-attempted-regions-on-connect-cycle-end
Open

xianshijing-lk wants to merge 4 commits into
mainfrom
sxian/reset-attempted-regions-on-connect-cycle-end

Conversation

@xianshijing-lk

@xianshijing-lk xianshijing-lk commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Found while auditing the equivalent path in client-sdk-swift (#1120), where review surfaced the same class of bug.

Problem

RegionUrlProvider.clearAttemptedRegions() has exactly one call site — RTCEngine.kt:751, on a successful reconnect. The initial-connect failover loop in Room.connect fills attemptedRegions and never clears it:

while (nextUrl != null) {
    try { engine.join(connectUrl, token, options, roomOptions) }
    catch (e: Exception) {
        nextUrl = regionUrlProvider?.getNextBestRegionUrl()   // ← adds to attemptedRegions
        if (nextUrl != null) { ... } else { throw e }         // ← no clear
    }
}
// ← no clear on success either

The provider outlives a single connect — connect reuses an existing one for the same url and creates one only when it is null (Room.kt:530-534) — so the set leaks across cycles two ways.

A region that failed transiently stays excluded. Region A fails, B connects. A remains in attemptedRegions for the life of the provider. Later B fails while A has recovered; failover skips A and reports failure instead of connecting.

An exhausted cycle poisons the next connect. After throw e the set holds every region. The next connect tries the provided url, fails, gets null from getNextBestRegionUrl(), and rethrows — without attempting a single fallback region. This is the more serious of the two: region failover is silently disabled from then on.

prepareConnection masks it in some flows, since it builds a fresh provider (Room.kt:442-443) — but it also calls getNextBestRegionUrl(), so it adds to the set of the provider it creates. Apps that don't call it get the unmasked behaviour.

Fix

Scope the failover loop with try/finally and clear the captured provider there, so every exit ends the cycle: connected, out of regions, a settings refresh that threw, or cancellation.

Two review rounds shaped this (thanks @MaxHeimbrock, Devin):

  • Clearing per exit wasn't enough. getNextBestRegionUrl() refreshes settings once the cache expires and propagates request/decode failures. It is called from inside the catch, so a throw carries the exception out of the loop past any exit-specific clear. finally covers that.
  • Re-reading the field wasn't safe. regionUrlProvider is a private var that prepareConnection writes without stateLock, so the loop and the clear could touch different instances, leaving the engine's provider dirty. It is now captured once per cycle and used for the engine handoff, the lookup, and the clear.

Not addressed here: serializing provider installation against the connection state transition. That changes prepareConnection's locking on a path this PR doesn't otherwise touch, and the gap is pre-existing.

client-sdk-flutter resets at three exits — success, exhaustion, certificate-pinning failure (engine.dart:1302,1311,1320) — but its getNextBestRegionUrl() can also throw (region_url_provider.dart:43-49, via fetchRegionSettings) and its call site at engine.dart:1311 is unprotected, so it has the same gap this PR closes. Worth a follow-up there. Swift is converging on the same scoping in livekit/client-sdk-swift#1120.

Test

clearAttemptedRegionsMakesEveryRegionEligibleAgain — takes two regions, clears, and asserts the best region is offered again rather than the next unattempted one, with no extra settings fetch.

Verification

Not built or run locally — this checkout can't compile the SDK: protoc rejects (logger.redact) in livekit_models.proto at the pinned protocol submodule, and the newer submodule my checkout had fails on DataTrackSchema.kt. Both failures predate this change and are unrelated to it. Relying on CI; happy to iterate if it finds anything.

`clearAttemptedRegions()` had a single call site — a successful reconnect — so the
initial-connect failover loop in `Room.connect` never cleared the set it fills.

The provider outlives one connect: `connect` reuses an existing provider for the
same url and only creates one when it is null. Two consequences follow.

After a successful connect, a region that failed transiently before another one
succeeded stays excluded for the life of the provider, and a later failover skips
it even once it has recovered.

After an exhausted connect the set holds every region, so the next `connect` call
resolves no region at all — it fails on the provided url and rethrows without
trying a single fallback.

Clear the set at both ends of the cycle, matching client-sdk-flutter, which
resets on success, on exhaustion, and on a certificate-pinning failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Sep 21, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 2ef1a30

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

devin-ai-integration[bot]

This comment was marked as resolved.

@MaxHeimbrock
MaxHeimbrock force-pushed the sxian/reset-attempted-regions-on-connect-cycle-end branch from 1f5a97f to aac93e9 Compare September 22, 2026 08:36
@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Diffuse output:

OLD: diffuse-source-file
NEW: livekit-android-sdk-release.aar

 AAR      │ old      │ new      │ diff   
──────────┼──────────┼──────────┼────────
      jar │    3 MiB │    3 MiB │ +204 B 
 manifest │  1.5 KiB │  1.5 KiB │    0 B 
 lint-jar │ 12.7 KiB │ 12.7 KiB │    0 B 
    other │  2.4 KiB │  2.4 KiB │    0 B 
──────────┼──────────┼──────────┼────────
    total │    3 MiB │    3 MiB │ +204 B 

 JAR     │ old   │ new   │ diff       
─────────┼───────┼───────┼────────────
 classes │  1702 │  1702 │  0 (+0 -0) 
 methods │ 21934 │ 21934 │  0 (+0 -0) 
  fields │  5685 │  5686 │ +1 (+1 -0)
AAR
 size  │ diff   │ path          
───────┼────────┼───────────────
 3 MiB │ +204 B │ ∆ classes.jar 
───────┼────────┼───────────────
 3 MiB │ +204 B │ (total)
JAR
FIELDS:

   old  │ new  │ diff       
  ──────┼──────┼────────────
   5685 │ 5686 │ +1 (+1 -0) 
  
  + io.livekit.android.room.Room_connect_2_connectJob_1 L_5: Object

`getNextBestRegionUrl` refreshes region settings once the cache expires and
propagates request and decoding failures, so it can throw from inside the catch
and carry the exception out of the loop — past both exit-specific clears. The
cycle would then end with regions still marked attempted, and `connect` reuses the
provider for the same url, so a later failover skips them.

Scoping the loop with try/finally covers every exit at once: connected, out of
regions, a settings refresh that threw, and cancellation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

`regionUrlProvider` is a mutable field and `prepareConnection` replaces it without
holding `stateLock`, so reading it separately in the loop and in `finally` can
touch two different instances: the engine keeps the one the cycle actually used
while the clear lands on its replacement, leaving the engine's provider holding
regions a later failover then skips.

Read it once into a local and use that for the engine handoff, the next-region
lookup, and the clear.

Serializing provider installation against the connection state transition is the
other half of that race and is left alone here — it changes `prepareConnection`'s
locking, which is outside this fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@davidliu davidliu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just needs a changeset

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants