Fleet event stream is lossy by construction — remote sessions lag and desync during live activity #1770

Important

Problem — Sessions viewed across the fleet lag behind, or silently fall out of sync with, the machine that owns them — mostly during live activity. Live events cross up to four relays (owner engine, owner hub fan-in, SSH peer link, viewing window's fan-in, webview), and no frame on that path carries a sequence id, so nothing can be replayed and no hop can tell what it missed. Three mechanisms are confirmed on the current two-machine fleet. Listener eviction: each fan-in service serves exactly one listener, so a second view onto the same hub (another window, a Chat Deck pane, a split pane) ends the first view's stream, and the two re-open against each other once per second — the Studio engine logged ~340 reconnects in 15 minutes on Oct 12 as two interleaved ~1 s series, and ~234 per minute for 110 minutes on Sep 29. Silent gaps: the engine emits no SSE ids and ignores the resume cursor the client already sends; the fan-in's reconnect gap signal only fires for arms that delivered an id'd frame, which never happens in production; relays close listeners with a clean end of stream; and the client treats a clean end as normal, reconnecting without refetching the viewed session. Burst volume: per-turn summary events carry full patch text (up to 12.3 MB on the Studio, 10 MB on the laptop), every stored event is emitted twice, a running shell tool republishes its last 30 KB on every output chunk, and both stream readers re-scan their whole partial buffer per network chunk — quadratic cost on the webview's main thread. Amplifiers: the session-ownership map drops a peer's sessions on one failed refresh (closing its event arm and routing its sessions to the local engine), relay buffers cannot pause their upstream and heartbeats carry no timestamp (lag is invisible), and the peer SSH forward is hand-installed with stale keepalives and never health-checked.
Approach — Make the stream lossless by construction and every relay honest. The engine stamps each global-stream frame with a monotonic id (process epoch plus sequence) and keeps a bounded replay buffer that honors the resume cursor, answering a stale or foreign cursor with an explicit reset. Relays serve every listener independently, never end a listener silently, and resume arms from real ids. The client tracks the SSE id, checks contiguity, and resyncs on reset, unexpected end, or a hole. Burst volume is cut at the source (sync copies opt-in, patch bodies off the wire, shell output throttled) and frame splitting becomes linear. A census line on every fan-in makes evictions, re-opens, drops, gaps, and ownership refreshes visible.
Approaches considered — Keep patching per hop (#1601, #1617, #1638, #1646, #1649, #1766): rejected — each fixed one hop's symptom while the root (no ids, no replay) kept producing new ones; the fan-in's lossless-resume design (#1511 §D3) assumed ids the engine never sent, and its tests feed synthetic id'd frames, so they pass while production cannot. Sequence and replay in the hub service instead of the engine: rejected — leaves the fork untouched but needs a buffer at every relay, leaves the engine-to-hub hop unsequenced, and an engine restart still forces a full resync; the engine's bus already assigns ascending payload ids, so the patch is small. Client-side gap detection without replay: kept as the fallback (reset leads to a full resync), rejected as the primary — it turns every link blip into a full refetch over the same congested link.
Scope — in: the global event route (ids, replay, reset, heartbeat time, subscription release on abort); the fan-in driver and aggregator (per-listener connections, honest close, cursor resume, census); the session-ownership feed (last-good retention); the client stream loop (cursor, contiguity, reset and end-of-stream resync); burst-volume cuts; linear frame splitting; installer-managed peer forwards and their healthcheck. out: the per-directory event route used by the extension host's status client; the v2 per-session event API; compaction of the stored event log; the Python frontdoor deployment tracked in #1757 and #1751.
Assumptions — The reporter's workflow routinely has two or more views onto the same remote hub (confirmed). A replay window of about five minutes covers ordinary link blips and window reloads; longer outages fall back to a full resync. Engine changes reach the hubs only through scratch-engine validation followed by a human-scheduled hub restart per machine, because a hub restart ends that machine's in-flight sessions (#1552).