Skip to content

feat(fleet): fan-in census line + engine-shaped stream fixtures #1771

Closed

Description

@jeonghun-jj-lee

Important

Problem — A wedged, evicted, or flapping event arm looks identical to a quiet one in every log we have: the fan-in doesn't record who subscribes, why an arm re-opened, or whether a session-ownership refresh failed. The Oct 12 eviction storm was only found by reverse-engineering engine connect timestamps (see the parent for the full failure analysis).
Approach — Give every fan-in service a census: counters per listener connection and per arm, logged as one line every 30 s and immediately on each transition, plus each ownership refresh's per-peer outcome. Replace the fan-in suites' synthetic upstream frames with engine-shaped fixtures that later slices reuse.
Scope — in: census counters and log lines in the fan-in driver, the aggregator, and the owner-map feed; engine-shaped upstream fixtures (id-less data frames, heartbeats, the plain-plus-sync twin emission, a multi-MB frame delivered in chunks). out: any delivery-behavior change; lag measured from heartbeat time (arrives with the engine slice).
Assumptions — The hub's log and the window's Amicode output channel are the right sinks; one line per 30 s per service is cheap enough to leave on permanently.

Acceptance Criteria

  • A fan-in service logs a census line every 30 s with: active listeners; listeners accepted and closed since boot; evictions; and per arm (local and each peer) frames, bytes, largest frame, time since last frame, opens, closes by reason, pending depth, overflow drops, and gaps emitted.
  • Close reasons are an enumerated set: upstream end, upstream error, membership change, listener closed, evicted, shutdown.
  • Each transition (listener accepted or closed, arm opened or closed, gap emitted, overflow drop) logs one line immediately, with the connection and arm identity.
  • Each ownership refresh logs, per peer, the outcome (ok, timeout, error, unauthorized), its duration, and its entry count — a failed peer refresh is visible even though the feed never throws.
  • The census adds no frames to any stream and changes no delivery behavior; existing fan-in suites pass with only fixture updates.
  • Engine-shaped fixtures exist and the fan-in suites use them: id-less data frames, heartbeat frames, the plain-plus-sync twin emission of a stored event, and a 10 MB frame delivered in 64 KB chunks.

Testing Decisions

  • Extend the fan-in aggregator and driver pump suites to assert counters after scripted sequences, and the owner-map feed tests to assert a per-peer outcome record for a failed refresh.
  • The engine-shaped fixtures are new shared test surface; the loud-gap slice and the linear-splitting slice build on them.

Key Decisions

  • Census reasons are enumerated so later slices and the verification slice can assert on them.
  • Cadence: 30 s steady state, transitions immediately, never per frame.

Constraints & Invariants

Prior Art

  • The fan-in driver's connection and arm lifecycle and its pump; the aggregator's flowing, pending, drop, and gap state; the owner-map feed's refresh loop and its never-throw projection builder; the hub runner's log and the window's Amicode output channel.

Source

Activity

  1. jeonghun-jj-lee commented on Oct 12, 2026

    @jeonghun-jj-lee
    ContributorAuthor

    Merged into the #1770 integration branch as de4002b (PR #1781, draft). Director gates: tsc 0; suite 323 files / 5396 passed / 0 failed / 53 skipped (+31 tests); 7 touched files 20 consecutive runs, 0 failures; no wire change proven (the census writes nothing to any response, and the 70 pre-census fan-in tests pass unchanged against the new code); no test removed (fixture migration only, assertions grew).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

afkImplementable without human interactionarea:node

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions