wasmtime host: peer-connection drivers never share the embedder's runtime - #159
Merged
Conversation
…time Every driver now runs on webrtc's dedicated reactor pool (with_dedicated_reactor_pool_size: a bounded set of single-threaded runtimes on their own OS threads, drivers pinned round-robin, I/O bound to the pool thread) instead of the ambient tokio runtime. A driver is a third-party event loop; on the shared runtime it held the whole embedder hostage. Observed live from wosh's freeze drill: a sans-IO core bug (lann/rtc#2 -- a failed ICE agent pins its wake-up deadline in the past) span the driver at ~4.5M loop passes/s once a frozen peer's connection failed, starving the embedding wasm component -- endpoint deaf, zombie sessions never idle-timed-out -- and starving the very teardown that would have ended the spin: the resource owner never ran, never dropped the connection, so the driver's closing check never fired. On the pool, the same wedge costs a pool thread and then heals itself: the owner keeps running, observes the failure, drops the connection, and the closing check ends the loop. Verified against the live bug (unfixed webrtc pin): the embedder stays responsive through the freeze, sessions park on schedule, new dials are accepted mid-wedge, and the spinning thread goes quiet within seconds. A pool of two bounds the thread cost while keeping one wedged driver from pausing every other connection's driver. The regression test asserts the pool is actually engaged (a webrtc-rx reactor thread exists once a connection is built) -- the engagement is one easily-lost builder call, and nothing else observes it. Fixes #158.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #158.
One builder call plus a constant: every peer-connection driver now runs on
webrtc's dedicated reactor pool (with_dedicated_reactor_pool_size(2)) instead of the embedder's ambient runtime, so a stalled or spinning driver costs a pool thread rather than the host.Why now: wosh's freeze drill (SIGSTOP a connected peer 45s) hit a sans-IO core bug — lann/rtc#2, a failed ICE agent pinning its wake-up deadline in the past — that span the driver at ~4.5M loop passes/s. On the shared runtime that froze the embedding wasm component solid and starved the teardown that would have ended the spin (the owner never ran, never dropped the connection, so the driver's
closingcheck never fired). Details and instrumentation on the issue.Verified against the live bug (deliberately unfixed
webrtcpin), driving the full wosh listener natively: through the 45s freeze the embedder stays responsive — the zombie session idle-times-out and parks on schedule, fresh dials are accepted mid-wedge — and the spin self-heals within seconds (busy-thread count returns to zero) because the still-running owner drops the connection. Thawed peer then resumes its session cleanly.just checkgreen.Residual: while a driver spins, other drivers pinned to the same pool thread stall until the self-heal; two threads bound the cost while keeping one wedge from pausing every other connection. This stays correct — and keeps webrtc#101's throughput-isolation property — after the root fix lands upstream, so it is defense-in-depth rather than a revert-later shim.
The regression test asserts the pool is actually engaged (a
webrtc-rxreactor thread appears once a connection is built): the engagement is one easily-lost builder call, and nothing else observes it. Verified to bite (fails in ~10s with the pool disabled).