Skip to content

feat(engine): sequenced, replayable global event stream (epoch:seq ids, replay buffer, reset) #1776

Closed

Description

@jeonghun-jj-lee

Important

Problem — The engine's global event stream has no SSE ids, ignores the resume cursor the client already sends, and keeps nothing to replay, so every relay hop that drops turns into permanent loss. Separately, the laptop engine logs global-stream connects without matching disconnects (18 against 0 in a 13.5 h run, 3.7 GB resident) while the Studio's engine balances them — possibly subscriptions that are not released when a client aborts.
Approach — One sequencer at the global bus stamps every emitted event with an epoch-and-sequence id. The global stream writes it as the SSE id and keeps a replay buffer bounded by bytes and age. On connect with a cursor (standard header or the existing query parameter), it replays the frames after the cursor and then continues live, registering the live listener before reading the buffer and dropping duplicates by sequence. A cursor from another epoch, or older than the buffer's oldest entry, gets an explicit reset as the first frame. Heartbeats carry server time and the current sequence. Verify, on the compiled binary's runtime, that a client abort releases its subscription, and fix it if not.
Scope — in: bus sequencing; the global stream's ids, replay, reset, and heartbeat contents; subscription release on abort; environment-configurable buffer bounds. out: the per-directory event route; the v2 per-session API; wire volume (the volume slice).
Assumptions — A sync copy shares the id of the event it mirrors, so filtering sync copies later leaves no sequence holes; about 32 MB and 5 minutes of history covers link blips and window reloads.

Acceptance Criteria

  • Every global-stream frame, including heartbeats and the connected frame, carries an SSE id of the form epoch-and-sequence; the sequence strictly increases across all subscribers of one engine process; a sync copy carries its source event's id.
  • Reconnecting with a cursor inside the replay window delivers exactly the frames after it and then live frames — no duplicate and no missing sequence, including events published while the reconnect is in flight.
  • Reconnecting with a cursor from a different epoch, or older than the buffer's oldest entry, yields a reset frame first (carrying the current epoch and sequence), then live frames.
  • The buffer evicts oldest-first when it exceeds its byte or age bound; both bounds are configurable by environment.
  • Heartbeats carry server time in milliseconds and the current sequence.
  • A client that disconnects mid-stream releases its subscription — listener unregistered, queued frames freed, disconnect logged — verified against the compiled binary on a scratch engine.

Testing Decisions

  • Extend the engine server suite in the overlay with an in-process client (id format, replay, reset, bounds, heartbeat contents); add a scratch-engine test against the compiled binary for replay and abort release.

Key Decisions

  • The epoch is a random token per process start; the sequence is a per-process counter assigned at bus emit.
  • The cursor is accepted from both the standard header and the existing query parameter.
  • Reset is a named event type that the client handles (end-to-end resume slice); it is never silent.

Data Contracts

  • SSE id: <epoch>:<seq> — the epoch a short random token, the sequence a decimal integer.
  • Reset frame: a named "cursor unservable" event whose properties are the current epoch and sequence.
  • Heartbeat properties: server time in milliseconds and the current sequence.

Constraints & Invariants

  • Live delivery latency is unchanged for connected subscribers.
  • Buffer memory is bounded regardless of subscriber count (one shared buffer).

Prior Art

  • The global bus's ascending payload-id assignment; the global stream handler's queue, heartbeat merge, and finalizer; the HTTP server adapter's client-abort interrupt; the event-v2 bridge's twin emission.

Source

Notes

Activity

  1. jeonghun-jj-lee commented on Oct 12, 2026

    @jeonghun-jj-lee
    ContributorAuthor

    Merged into the #1770 integration branch amico/issue-1770-fleet-event-stream as 30289ae (PR #1781 — HITL review happens there before main). Director gates: drift PASS (925), materialize 925 hashes verified, engine + overlay typecheck gates clean, extension suite green, engine tests 13/13 with 0 failures in 40 consecutive runs after one harness repair cycle (root cause: layerTest wildcard bind colliding with leaked 127.0.0.1 listeners from server_detach.test.ts), AC6 on the compiled binary 3/3 (baseline binary: 6 connects / 0 disconnects). Engine deploy still pending: scratch engine first, then human-scheduled hub restarts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or requesthitlNeeds human review before merge

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions