Repository navigation
fix(fleet): ownership map keeps a peer's sessions across a failed refresh #1774
Description
Important
Problem — Every 5 s each service rebuilds the session-ownership map from every peer's full session list, with a 10 s timeout. A failed peer fetch yields an empty slice instead of an error, so the feed's keep-last-good guard never engages: the peer's sessions vanish from the map, its event arm closes without notice, and reads and writes for its sessions fall through to the local engine (404 or empty) until a later refresh succeeds and the arm silently re-opens. Heavy live traffic on the same SSH link makes these timeouts likelier exactly when it matters.
Approach — Retain each peer's last-good owner entries across failed refreshes for a bounded time; keep its arm open and its routing unchanged during that window; after the window, expire the entries and close the arm with the existing named-unavailable comment. For a session-scoped request with no known owner, try the local engine first; a local miss returns a retryable "owner unknown" error instead of a 404.
Scope — in: per-peer retention in the ownership map update, expiry, and the unknown-owner routing outcome. out: the refresh cadence and fan-out cost (unchanged); the merged projection behind the fleet session list UI.
Assumptions — A peer's session set changes slowly relative to a 60 s retention window; a spurious retryable error is better than a wrong 404.
Acceptance Criteria
- A refresh in which one peer times out or errors keeps that peer's previous owner entries; its event arm stays open and its sessions keep routing to it.
- After the retention window passes with no successful refresh, the peer's entries expire, its arm closes, and listeners receive the named unavailable comment — not silence.
- A successful refresh replaces the peer's entries authoritatively, so sessions that ended disappear.
- A session-scoped read or write for an id with no known owner is tried locally; if the local engine does not have it, the response is a retryable error with a retry hint, not a 404 or an empty result.
- A just-created local session still routes locally before the next refresh, and the pending binding for a just-created remote session keeps working.
Testing Decisions
- Extend the session-multiplexer and owner-routing suites (failed peer refresh, expiry, unknown-owner outcome) and the observe read and write routing suites.
Key Decisions
- Retention window 60 s, configurable. Timeouts, network errors, and 5xx responses retain; an unauthorized response does not (it is a credential problem and surfaces as unavailable).
- Unknown owner: local first, then a retryable error.
Constraints & Invariants
- The projection builder still never throws; retention lives in the map update, keyed per peer.
Prior Art
- The owner-map feed's never-throw projection builder and its clear-then-repopulate update; the pending-binding overlay; the read and write routers' local fallback; the aggregator's named unavailable comment.
Activity
fix(fleet): a failed peer refresh keeps that peer's owner entries — r…
fix(fleet): past the retention window a peer's owner entries expire a…
test(fleet): a successful refresh replaces a peer's owner entries aut…
fix(fleet): an unknown-owner session request is tried locally first; …
test(fleet): a just-created local session routes locally, and a remot…
fix(fleet): the owner-map refresh line says what the map did with a f…
refactor(fleet): an outcome-less owner-map update just forgets peer r…
jeonghun-jj-lee commentedon Oct 13, 2026
Merged into the #1770 integration branch as 1491f3e (PR #1781, draft). Director gates: tsc 0, suite 323/5427/0/53, 3 test files 0 failures in 20 runs. Combined with #1773 on the integration tip: suite 323/5452/0/53 and the 8 overlapping fan-in + owner-map files 0 failures in 20 runs. The only removed assertions were two
toBeUndefined()checks encoding the old unknown-owner fallback, replaced at the same call sites by the newowner-unknownresult.Merge #1774: ownership map keeps a peer's sessions across a failed re…