Skip to content

fix(fleet): every lost stretch of events forces a resync — no silent gaps #1773

Closed

Description

@jeonghun-jj-lee

Important

Problem — Every way the relay chain loses events is silent. The reconnect gap signal fires only for arms that delivered a frame with an SSE id — which the engine never sends — so a re-opened arm never announces its gap. Relays close listeners with a clean end of stream. The client treats a clean end as normal and reconnects 250 ms later without refetching the viewed session. Until the engine slice adds replay, the only honest recovery is a forced resync, and today it never triggers.
Approach — Make every loss loud. In composite mode, re-opening any arm that has delivered at least one data frame emits the gap signal. In fleet-of-one mode, where synthetic frames are not allowed (#1264), a re-opened local arm severs the listener with a transport error so the client registers a disconnect. Relays end listeners with an error for every reason except graceful shutdown, and the arm reader distinguishes upstream end from upstream error. The client treats an unexpected end of the global stream as a gap and advances its forced-resync signal.
Scope — in: the aggregator's notion of "delivered"; fleet-of-one re-open behavior; listener close semantics in the fan-in and in the engine and hub proxies; end-versus-error in the arm reader; the client's end-of-stream handling. out: replay (the engine and end-to-end resume slices); the transcript and entity-rail heal hooks themselves (#1646 — reused, unchanged).
Assumptions — The global stream never ends intentionally except on shutdown, so an unexpected clean end is always a loss; the forced-resync signal already drives the transcript and entity-rail refetch.

Acceptance Criteria

  • With engine-shaped (id-less) frames, a non-initial re-open of an arm that delivered data in composite mode emits exactly one gap signal naming that namespace.
  • In fleet-of-one mode, a re-opened local arm ends the listener's stream with a transport error and never injects a synthetic frame; while connected, the stream stays byte-identical to the local source.
  • A fan-in connection closed for any reason other than graceful service shutdown errors the listener's stream instead of ending it cleanly; the engine and hub proxies do the same when response headers were already sent.
  • The arm reader reports upstream end and upstream error as distinct reasons.
  • In the client, an unexpected normal end of the global stream advances the forced-resync signal, so the viewed transcript and the entity rail refetch; a user-initiated stop or a page hide does not.

Testing Decisions

  • Extend the aggregator suite (re-open gap with the engine-shaped id-less fixtures), the driver pump suite (error-close semantics), the engine-proxy suite (error after headers), and the app's gap-frame and stream-liveness suites (unexpected end leads to resync). Extends the census slice's fixture surface.

Key Decisions

Constraints & Invariants

Prior Art

Source

Activity

  1. jeonghun-jj-lee commented on Oct 13, 2026

    @jeonghun-jj-lee
    ContributorAuthor

    Merged into the #1770 integration branch as bf64938 (PR #1781, draft). Director gates: drift PASS, overlay typecheck clean, tsc 0, suite 323/5421/0/53, 4 extension test files 0 failures in 20 runs, touched app tests 0 failures in 20 runs, full app sweep 1635/23 with 0 new failures vs the tip (CI does not run app tests). Three existing tests were rewritten because they asserted the silent fleet-of-one re-open this issue removes; audited — every original guarantee (no gap frame, no namespacing, byte identity while connected, cursor bookkeeping) is kept, and the composite test is now parametrized over legacy and sequenced engines.

  2. added 3 commits that reference this issue on Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

afkImplementable without human interactionarea:nodearea:webviewbugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions