Repository navigation
feat(fleet): fan-in census line + engine-shaped stream fixtures #1771
Description
Important
Problem — A wedged, evicted, or flapping event arm looks identical to a quiet one in every log we have: the fan-in doesn't record who subscribes, why an arm re-opened, or whether a session-ownership refresh failed. The Oct 12 eviction storm was only found by reverse-engineering engine connect timestamps (see the parent for the full failure analysis).
Approach — Give every fan-in service a census: counters per listener connection and per arm, logged as one line every 30 s and immediately on each transition, plus each ownership refresh's per-peer outcome. Replace the fan-in suites' synthetic upstream frames with engine-shaped fixtures that later slices reuse.
Scope — in: census counters and log lines in the fan-in driver, the aggregator, and the owner-map feed; engine-shaped upstream fixtures (id-less data frames, heartbeats, the plain-plus-sync twin emission, a multi-MB frame delivered in chunks). out: any delivery-behavior change; lag measured from heartbeat time (arrives with the engine slice).
Assumptions — The hub's log and the window's Amicode output channel are the right sinks; one line per 30 s per service is cheap enough to leave on permanently.
Acceptance Criteria
- A fan-in service logs a census line every 30 s with: active listeners; listeners accepted and closed since boot; evictions; and per arm (local and each peer) frames, bytes, largest frame, time since last frame, opens, closes by reason, pending depth, overflow drops, and gaps emitted.
- Close reasons are an enumerated set: upstream end, upstream error, membership change, listener closed, evicted, shutdown.
- Each transition (listener accepted or closed, arm opened or closed, gap emitted, overflow drop) logs one line immediately, with the connection and arm identity.
- Each ownership refresh logs, per peer, the outcome (ok, timeout, error, unauthorized), its duration, and its entry count — a failed peer refresh is visible even though the feed never throws.
- The census adds no frames to any stream and changes no delivery behavior; existing fan-in suites pass with only fixture updates.
- Engine-shaped fixtures exist and the fan-in suites use them: id-less data frames, heartbeat frames, the plain-plus-sync twin emission of a stored event, and a 10 MB frame delivered in 64 KB chunks.
Testing Decisions
- Extend the fan-in aggregator and driver pump suites to assert counters after scripted sequences, and the owner-map feed tests to assert a per-peer outcome record for a failed refresh.
- The engine-shaped fixtures are new shared test surface; the loud-gap slice and the linear-splitting slice build on them.
Key Decisions
- Census reasons are enumerated so later slices and the verification slice can assert on them.
- Cadence: 30 s steady state, transitions immediately, never per frame.
Constraints & Invariants
- Zero wire changes; fleet-of-one byte identity (Fleet thin client — Slice 4: lossless SSE reconnect (cursor resume) #1264) holds.
Prior Art
- The fan-in driver's connection and arm lifecycle and its pump; the aggregator's flowing, pending, drop, and gap state; the owner-map feed's refresh loop and its never-throw projection builder; the hub runner's log and the window's Amicode output channel.
Activity
feat(fleet): fan-in census line every 30 s — listeners, accepted/clos…
test(fleet): census close reasons are the enumerated six; the real re…
test(fleet): census transitions log one line immediately with conn + …
feat(fleet): log each ownership refresh's per-peer outcome, duration …
test(fleet): the census adds no frame and changes no delivered byte —…
jeonghun-jj-lee commentedon Oct 12, 2026
Merged into the #1770 integration branch as de4002b (PR #1781, draft). Director gates: tsc 0; suite 323 files / 5396 passed / 0 failed / 53 skipped (+31 tests); 7 touched files 20 consecutive runs, 0 failures; no wire change proven (the census writes nothing to any response, and the 70 pre-census fan-in tests pass unchanged against the new code); no test removed (fixture migration only, assertions grew).
test(fleet): engine-shaped stream fixtures pinned to the engine wire;…
test(fleet): fan-in suites feed engine-shaped frames — id-less, seque…
test(fleet): both fan-in boot branches log owner-map refresh outcomes…
Merge #1771: fan-in census line + engine-shaped stream fixtures (S0 of …