Repository navigation
fix(fleet): fan-in serves every listener — a second view no longer evicts the first #1772
Description
Important
Problem — Each fan-in service accepts exactly one global-stream listener: a new listener tears down the previous connection with a clean end of stream. With two views onto the same remote hub (two windows, a Chat Deck pane, a split pane), each view's once-per-second reconcile re-opens its arm and evicts the other, indefinitely — the Studio engine logged ~340 reconnects in 15 minutes on Oct 12 and ~234 per minute for 110 minutes on Sep 29. Every eviction silently drops whatever the evicted view would have received.
Approach — One fan-in connection per listener: each listener gets its own aggregator, upstream arms, and reconcile; the driver tracks the set and reconciles every member. A cap bounds concurrent listeners, and a listener beyond the cap is refused with a retryable error instead of evicting an existing one.
Scope — in: the driver's listener model, per-connection reconcile, the cap, and teardown that closes only the departing listener's own arms. out: sharing upstream arms across listeners (an optimization once cursors are meaningful); end-of-stream and gap semantics (the loud-gap slice).
Assumptions — Upstream arms per listener are cheap (loopback to the local engine; one SSH channel per peer arm); a service rarely has more than a handful of concurrent views.
Acceptance Criteria
- Two concurrent global-stream listeners on one service both stay connected, and both receive every frame from the local arm and from each peer arm.
- Closing one listener closes only that listener's upstream arms; the other keeps streaming without interruption.
- Reconcile updates every live connection's membership on its cadence.
- A listener beyond the cap is refused with a retryable status and a retry hint; no existing listener is closed to make room.
- A service with two listeners opens one local arm per listener and keeps it open — no repeating re-open pattern (asserted in the two-listener harness; also visible in the census once it lands).
Testing Decisions
- Extend the driver pump suite with a two-listener test and a close-one-of-two test; extend the observe event-routing suite so the real dispatch path accepts concurrent listeners.
- New: the beyond-cap refusal case.
Key Decisions
- Per-listener connections rather than a shared upstream with multicast — the simplest correct model; multicast can come later, once the engine's ids make per-listener cursors meaningful.
- Default cap of 16 concurrent listeners per service, configurable; refusal beyond the cap is explicit, never an eviction.
Constraints & Invariants
- Fleet-of-one byte identity (Fleet thin client — Slice 4: lossless SSE reconnect (cursor resume) #1264) holds per listener.
- The local-arm re-open (Fan-in /event driver never re-opens the LOCAL arm → local session events freeze until reload #1601) and the backpressure re-arm (Fan-in relay never re-arms backpressure (flowing stuck false) → session transcript + rail go stale until reload #1617) hold per connection.
Prior Art
- The fan-in driver's single active connection and its eviction of the previous listener; the per-connection reconcile timer; the single-origin assumption (ADR 0027) that the app opens exactly one stream — no longer true with deck panes, split panes, and multiple windows.
Source
Notes
- Deployment: the cross-window storm happens at the remote machine's hub, so this fix takes effect there on a hub restart; same-window eviction (deck, split pane) needs the window's extension build.
Activity
jeonghun-jj-lee commentedon Oct 12, 2026
Merged into the #1770 integration branch
amico/issue-1770-fleet-event-streamas 46c208c (PR #1781, draft until the campaign completes). Director gates: extension tsc exit 0; fast suite 322 files / 5365 passed / 0 failed / 53 skipped (baseline 5354); all 11 new tests fail against the baseline driver and pass on the branch. Not live until the Studio hub restarts (human-scheduled).