Repository navigation
fix(fleet): every lost stretch of events forces a resync — no silent gaps #1773
Description
Important
Problem — Every way the relay chain loses events is silent. The reconnect gap signal fires only for arms that delivered a frame with an SSE id — which the engine never sends — so a re-opened arm never announces its gap. Relays close listeners with a clean end of stream. The client treats a clean end as normal and reconnects 250 ms later without refetching the viewed session. Until the engine slice adds replay, the only honest recovery is a forced resync, and today it never triggers.
Approach — Make every loss loud. In composite mode, re-opening any arm that has delivered at least one data frame emits the gap signal. In fleet-of-one mode, where synthetic frames are not allowed (#1264), a re-opened local arm severs the listener with a transport error so the client registers a disconnect. Relays end listeners with an error for every reason except graceful shutdown, and the arm reader distinguishes upstream end from upstream error. The client treats an unexpected end of the global stream as a gap and advances its forced-resync signal.
Scope — in: the aggregator's notion of "delivered"; fleet-of-one re-open behavior; listener close semantics in the fan-in and in the engine and hub proxies; end-versus-error in the arm reader; the client's end-of-stream handling. out: replay (the engine and end-to-end resume slices); the transcript and entity-rail heal hooks themselves (#1646 — reused, unchanged).
Assumptions — The global stream never ends intentionally except on shutdown, so an unexpected clean end is always a loss; the forced-resync signal already drives the transcript and entity-rail refetch.
Acceptance Criteria
- With engine-shaped (id-less) frames, a non-initial re-open of an arm that delivered data in composite mode emits exactly one gap signal naming that namespace.
- In fleet-of-one mode, a re-opened local arm ends the listener's stream with a transport error and never injects a synthetic frame; while connected, the stream stays byte-identical to the local source.
- A fan-in connection closed for any reason other than graceful service shutdown errors the listener's stream instead of ending it cleanly; the engine and hub proxies do the same when response headers were already sent.
- The arm reader reports upstream end and upstream error as distinct reasons.
- In the client, an unexpected normal end of the global stream advances the forced-resync signal, so the viewed transcript and the entity rail refetch; a user-initiated stop or a page hide does not.
Testing Decisions
- Extend the aggregator suite (re-open gap with the engine-shaped id-less fixtures), the driver pump suite (error-close semantics), the engine-proxy suite (error after headers), and the app's gap-frame and stream-liveness suites (unexpected end leads to resync). Extends the census slice's fixture surface.
Key Decisions
- "Delivered" means any data frame, not only a frame carrying an SSE id.
- Solo-mode loss is signalled by severing the connection, never by injecting a frame — preserves Fleet thin client — Slice 4: lossless SSE reconnect (cursor resume) #1264.
- The client keeps treating an unexpected end as a gap even after replay exists (defense in depth).
Constraints & Invariants
- Fleet thin client — Slice 4: lossless SSE reconnect (cursor resume) #1264 byte identity, Fan-in relay never re-arms backpressure (flowing stuck false) → session transcript + rail go stale until reload #1617 drain flush, and Fan-in relay resumes re-opened arm from connect-time cursor + no gap frame on reconnect → silent event loss #1638 cursor bookkeeping are all preserved.
- A gap signal never advances a cursor.
Prior Art
- Fan-in relay resumes re-opened arm from connect-time cursor + no gap frame on reconnect → silent event loss #1638's composite-only re-open gap and its id-gated delivered set; Fan-in /event driver never re-opens the LOCAL arm → local session events freeze until reload #1601's local-arm re-open; Chat message transcript has no reconnect/resync self-heal — stale rail heals only on reload (the #1617 fourth root) #1646's transcript heal hooks; Client: honor status seq + re-seed status on reconnect/gap + turn-level working floor (non-text turns) #1637's status re-seed on gap; the proxies' end-after-headers behavior; the client stream loop's normal-completion path.
Source
- Part of Fleet event stream is lossy by construction — remote sessions lag and desync during live activity #1770 · Blocked by feat(fleet): fan-in census line + engine-shaped stream fixtures #1771 (engine-shaped fixtures).
Activity
jeonghun-jj-lee commentedon Oct 13, 2026
Merged into the #1770 integration branch as bf64938 (PR #1781, draft). Director gates: drift PASS, overlay typecheck clean, tsc 0, suite 323/5421/0/53, 4 extension test files 0 failures in 20 runs, touched app tests 0 failures in 20 runs, full app sweep 1635/23 with 0 new failures vs the tip (CI does not run app tests). Three existing tests were rewritten because they asserted the silent fleet-of-one re-open this issue removes; audited — every original guarantee (no gap frame, no namespacing, byte identity while connected, cursor bookkeeping) is kept, and the composite test is now parametrized over legacy and sequenced engines.
fix(fleet): any data frame counts as delivered — a re-opened arm from…
fix(fleet): fleet-of-one local-arm loss severs the listener with a tr…
fix(fleet): fan-in and proxies error-close a lost stream — only grace…
fix(fleet): the arm reader treats a non-200 upstream as an error, nev…
fix(fleet): the client treats an unexpected end of the global stream …
docs(fleet): fan-in comments name the #1773 sever path (#1773)
Merge #1773: every lost stretch of events forces a resync — no silent…
fix(fleet): a re-opened arm resumes from its last engine id — the ups…