Skip to content

fix(fleet): lossless resume end to end — relays and client resume from engine ids #1777

Closed

Description

@jeonghun-jj-lee

Important

Problem — Once the engine sequences and replays, the relays and the client still won't use it: the client's cursor is the JSON payload id rather than the SSE id; the fan-in only learns cursors from SSE ids it has never seen; and nothing checks contiguity, so a hole the replay could not fill would still be silent.
Approach — The fan-in advances each namespace's cursor from the engine's ids and re-opens arms from the last delivered id, so the upstream replays the gap; an upstream reset becomes a gap signal for that namespace. Solo pass-through forwards engine ids untouched. The client tracks the SSE id as its cursor (composite when fanned in), resumes with it, treats a reset as a full resync, and checks sequence contiguity per origin — a hole forces a resync.
Scope — in: fan-in cursor advance from engine ids; upstream reset handling; the client's cursor source, reset handling, and contiguity check; the two-peer E2E with a link kill. out: the listener model (per-listener slice); the loud-gap fallbacks (kept as defense in depth).
Assumptions — Filtering (sync copies) never creates sequence holes on the global stream, per the engine slice's contract.

Acceptance Criteria

  • A peer arm that drops and re-opens within the replay window resumes from its last delivered id, and the viewer receives every missed frame — no gap signal, no full resync.
  • An upstream reset on any arm produces exactly one gap signal for that namespace downstream, and the client fully resyncs it.
  • In fleet-of-one mode the listener's stream is byte-identical to the engine's, ids included.
  • After a reconnect or a reload, the client resumes with the SSE id (composite when fanned in), and a sequence hole within one origin triggers a resync.
  • Replayed frames are never applied twice: the client drops any frame whose sequence it has already applied.
  • Two-peer fleet E2E: killing the peer link mid-turn and restoring it leaves the observer's transcript identical to a fresh load.

Testing Decisions

  • Extend the aggregator and driver suites with epoch-and-sequence fixtures (replacing the id-less fixtures where ids now exist), the app's server-sdk and stream suites (cursor source, reset, hole, duplicate), and the slow two-peer fleet E2E (link kill). Extends the loud-gap slice's and the engine slice's test surface.

Key Decisions

  • Contiguity is checked per origin namespace; heartbeats count because they carry the sequence, so a hole is detectable even on an idle stream.
  • The client's existing session-storage cursor persists the SSE id.

Constraints & Invariants

Prior Art

Source

Activity

  1. jeonghun-jj-lee commented on Oct 13, 2026

    @jeonghun-jj-lee
    ContributorAuthor

    Merged into the #1770 integration branch as 8965094 (PR #1781 — HITL review there). The client's cursor is now the SSE id on one carrier; a re-opened arm within the replay window replays with no gap and no resync; an upstream reset becomes exactly one gap; fleet-of-one stays byte-identical, ids included. AC6 ran for real: two compiled scratch engines + a TCP peer link killed mid-turn and restored → replay, 0 gaps / resyncs / duplicates, transcript identical to a fresh load. Director gates: drift PASS, overlay typecheck clean, tsc 0, suite 323/5470/0/51, 5 extension files 0 failures in 20 runs; 13 migrated tests audited (legacy engines still assert the #1773 gap/sever fallback). Integrated tip: suite 323/5494/0/53.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions