Repository navigation
fix(fleet): lossless resume end to end — relays and client resume from engine ids #1777
Description
Important
Problem — Once the engine sequences and replays, the relays and the client still won't use it: the client's cursor is the JSON payload id rather than the SSE id; the fan-in only learns cursors from SSE ids it has never seen; and nothing checks contiguity, so a hole the replay could not fill would still be silent.
Approach — The fan-in advances each namespace's cursor from the engine's ids and re-opens arms from the last delivered id, so the upstream replays the gap; an upstream reset becomes a gap signal for that namespace. Solo pass-through forwards engine ids untouched. The client tracks the SSE id as its cursor (composite when fanned in), resumes with it, treats a reset as a full resync, and checks sequence contiguity per origin — a hole forces a resync.
Scope — in: fan-in cursor advance from engine ids; upstream reset handling; the client's cursor source, reset handling, and contiguity check; the two-peer E2E with a link kill. out: the listener model (per-listener slice); the loud-gap fallbacks (kept as defense in depth).
Assumptions — Filtering (sync copies) never creates sequence holes on the global stream, per the engine slice's contract.
Acceptance Criteria
- A peer arm that drops and re-opens within the replay window resumes from its last delivered id, and the viewer receives every missed frame — no gap signal, no full resync.
- An upstream reset on any arm produces exactly one gap signal for that namespace downstream, and the client fully resyncs it.
- In fleet-of-one mode the listener's stream is byte-identical to the engine's, ids included.
- After a reconnect or a reload, the client resumes with the SSE id (composite when fanned in), and a sequence hole within one origin triggers a resync.
- Replayed frames are never applied twice: the client drops any frame whose sequence it has already applied.
- Two-peer fleet E2E: killing the peer link mid-turn and restoring it leaves the observer's transcript identical to a fresh load.
Testing Decisions
- Extend the aggregator and driver suites with epoch-and-sequence fixtures (replacing the id-less fixtures where ids now exist), the app's server-sdk and stream suites (cursor source, reset, hole, duplicate), and the slow two-peer fleet E2E (link kill). Extends the loud-gap slice's and the engine slice's test surface.
Key Decisions
- Contiguity is checked per origin namespace; heartbeats count because they carry the sequence, so a hole is detectable even on an idle stream.
- The client's existing session-storage cursor persists the SSE id.
Constraints & Invariants
- Fleet thin client — Slice 4: lossless SSE reconnect (cursor resume) #1264, Fan-in relay never re-arms backpressure (flowing stuck false) → session transcript + rail go stale until reload #1617, and Fan-in relay resumes re-opened arm from connect-time cursor + no gap frame on reconnect → silent event loss #1638 are preserved.
Prior Art
- The aggregator's composite cursor and per-namespace cursor accessor (Fan-in relay resumes re-opened arm from connect-time cursor + no gap frame on reconnect → silent event loss #1638); the client's session-storage cursor and origin resolution (SSE fan-in contaminates local directory store with remote sessions, dropping fleet badge #1539); the SDK stream reader's last-event-id handling.
Source
- Part of Fleet event stream is lossy by construction — remote sessions lag and desync during live activity #1770 · Blocked by fix(fleet): every lost stretch of events forces a resync — no silent gaps #1773, feat(engine): sequenced, replayable global event stream (epoch:seq ids, replay buffer, reset) #1776.
Activity
fix(fleet): a re-opened arm resumes from its last engine id — the ups…
fix(fleet): the client resumes from the SSE id and checks contiguity …
jeonghun-jj-lee commentedon Oct 13, 2026
Merged into the #1770 integration branch as 8965094 (PR #1781 — HITL review there). The client's cursor is now the SSE id on one carrier; a re-opened arm within the replay window replays with no gap and no resync; an upstream reset becomes exactly one gap; fleet-of-one stays byte-identical, ids included. AC6 ran for real: two compiled scratch engines + a TCP peer link killed mid-turn and restored → replay, 0 gaps / resyncs / duplicates, transcript identical to a fresh load. Director gates: drift PASS, overlay typecheck clean, tsc 0, suite 323/5470/0/51, 5 extension files 0 failures in 20 runs; 13 migrated tests audited (legacy engines still assert the #1773 gap/sever fallback). Integrated tip: suite 323/5494/0/53.
test(fleet): an upstream reset on any arm is exactly one gap for its …
test(fleet): fleet-of-one over real sockets — across a link loss and …
test(fleet): a reconnecting listener's last SSE id — bare engine id o…
test(fleet): two-peer E2E on two scratch engines — killing the peer l…
docs(fleet): fan-in comments name the resume path and say what an ove…
Merge #1777: lossless resume end to end — relays and client resume fr…