Repository navigation
feat(engine): sequenced, replayable global event stream (epoch:seq ids, replay buffer, reset) #1776
Description
Important
Problem — The engine's global event stream has no SSE ids, ignores the resume cursor the client already sends, and keeps nothing to replay, so every relay hop that drops turns into permanent loss. Separately, the laptop engine logs global-stream connects without matching disconnects (18 against 0 in a 13.5 h run, 3.7 GB resident) while the Studio's engine balances them — possibly subscriptions that are not released when a client aborts.
Approach — One sequencer at the global bus stamps every emitted event with an epoch-and-sequence id. The global stream writes it as the SSE id and keeps a replay buffer bounded by bytes and age. On connect with a cursor (standard header or the existing query parameter), it replays the frames after the cursor and then continues live, registering the live listener before reading the buffer and dropping duplicates by sequence. A cursor from another epoch, or older than the buffer's oldest entry, gets an explicit reset as the first frame. Heartbeats carry server time and the current sequence. Verify, on the compiled binary's runtime, that a client abort releases its subscription, and fix it if not.
Scope — in: bus sequencing; the global stream's ids, replay, reset, and heartbeat contents; subscription release on abort; environment-configurable buffer bounds. out: the per-directory event route; the v2 per-session API; wire volume (the volume slice).
Assumptions — A sync copy shares the id of the event it mirrors, so filtering sync copies later leaves no sequence holes; about 32 MB and 5 minutes of history covers link blips and window reloads.
Acceptance Criteria
- Every global-stream frame, including heartbeats and the connected frame, carries an SSE id of the form epoch-and-sequence; the sequence strictly increases across all subscribers of one engine process; a sync copy carries its source event's id.
- Reconnecting with a cursor inside the replay window delivers exactly the frames after it and then live frames — no duplicate and no missing sequence, including events published while the reconnect is in flight.
- Reconnecting with a cursor from a different epoch, or older than the buffer's oldest entry, yields a reset frame first (carrying the current epoch and sequence), then live frames.
- The buffer evicts oldest-first when it exceeds its byte or age bound; both bounds are configurable by environment.
- Heartbeats carry server time in milliseconds and the current sequence.
- A client that disconnects mid-stream releases its subscription — listener unregistered, queued frames freed, disconnect logged — verified against the compiled binary on a scratch engine.
Testing Decisions
- Extend the engine server suite in the overlay with an in-process client (id format, replay, reset, bounds, heartbeat contents); add a scratch-engine test against the compiled binary for replay and abort release.
Key Decisions
- The epoch is a random token per process start; the sequence is a per-process counter assigned at bus emit.
- The cursor is accepted from both the standard header and the existing query parameter.
- Reset is a named event type that the client handles (end-to-end resume slice); it is never silent.
Data Contracts
- SSE id:
<epoch>:<seq>— the epoch a short random token, the sequence a decimal integer. - Reset frame: a named "cursor unservable" event whose properties are the current epoch and sequence.
- Heartbeat properties: server time in milliseconds and the current sequence.
Constraints & Invariants
- Live delivery latency is unchanged for connected subscribers.
- Buffer memory is bounded regardless of subscriber count (one shared buffer).
Prior Art
- The global bus's ascending payload-id assignment; the global stream handler's queue, heartbeat merge, and finalizer; the HTTP server adapter's client-abort interrupt; the event-v2 bridge's twin emission.
Source
Notes
- Validate on a scratch engine (spare port, temporary database) before any hub restart; hub restarts are human-scheduled per machine because they end in-flight sessions (The hub engine bounce kills all in-flight campaign loops — sessions must survive their own deploy step (interrupted-session re-dispatch) #1552).
Activity
jeonghun-jj-lee commentedon Oct 12, 2026
Merged into the #1770 integration branch
amico/issue-1770-fleet-event-streamas 30289ae (PR #1781 — HITL review happens there before main). Director gates: drift PASS (925), materialize 925 hashes verified, engine + overlay typecheck gates clean, extension suite green, engine tests 13/13 with 0 failures in 40 consecutive runs after one harness repair cycle (root cause:layerTestwildcard bind colliding with leaked 127.0.0.1 listeners from server_detach.test.ts), AC6 on the compiled binary 3/3 (baseline binary: 6 connects / 0 disconnects). Engine deploy still pending: scratch engine first, then human-scheduled hub restarts.fix(fleet): a re-opened arm resumes from its last engine id — the ups…
fix(fleet): the client resumes from the SSE id and checks contiguity …