Repository navigation
feat(hub): kernel-level corpse detection — SO_KEEPALIVE + TCP_USER_TIMEOUT on every SSE socket (#1810) - #1811
Merged
aarontrowbridge merged 2 commits intoOct 16, 2026
Conversation
…1810) The #1798 corpse class (half-dead sockets blocking fan-out writes) is detected today by application-level dead-men on a 60-second cadence. This is the layer-4 half on the frontdoor: aggressive keepalives (idle 30s, interval 5s, count 3 -> the OS reaps a FIN-less corpse in <=45s) plus TCP_USER_TIMEOUT (~30s -- a write to a socket whose data cannot drain errors instead of blocking). Armed on every engine-facing dial (upstream_loop's connect helper, backend_get, the passthrough proxy socket) and every accepted member socket (the fleet clients' SSE streams -- the real network path where a dead tunnel leaves a FIN-less corpse the fan-out writer would block on). Knobs env-tunable (AMICODE_TCP_KEEPIDLE_S / KEEPINTVL_S / KEEPCNT / USER_TIMEOUT_MS); invalid values keep the briefed defaults; platforms without a knob degrade per-knob to keepalives-only, and a failed SO_KEEPALIVE arm is loud -- never a silent full skip. Tests extend the #1757 real-pair family: the module-runner reads the options back from the kernel on a real socket pair (getsockopt), the e2e test reads them back through the live frontdoor subprocess via the kernel's own per-socket keepalive timer (ss), for both the merged-group upstreams and the accepted member sockets.
…ntime gap (#1810) The engine half of #1810: a shared tcp-keepalive module (env-tunable knobs AMICODE_TCP_KEEPIDLE_S/KEEPINTVL_S/KEEPCNT/USER_TIMEOUT_MS with the briefed defaults, invalid -> default; Linux-ABI FFI setsockopt over the socket fd with kernel read-back telemetry) wired into BOTH SSE subscriber routes at connection setup: /event (overlay handlers/ event.ts) and /global/event (new overlay copy of handlers/global.ts, base 3104c1428e + the same one-line arm). Measured runtime boundary (bun 1.4.2, same family as the 2026-10-14 res.destroy-sends-no-FIN gap): raw node:net sockets expose _handle.fd and arm fully (read-back proven: keepalive=1 idle 30 intvl 5 cnt 3 user_timeout 30000), but bun's node:http server hands routes a NodeHTTPServerSocket whose native handle exposes NO fd and NO option surface, and its setKeepAlive is a silent kernel no-op (verified with ss). The arm therefore degrades LOUDLY there -- once per process, the SSE-KEEPALIVE-GAP line names what is missing and what carries corpse detection instead -- never a crash, never a silent full skip. The route-level tests pin that honest degradation on the real /event and /global/event routes; the module tests pin the full kernel read-back on real sockets, the env tuning, the invalid->default rule, and the AC4 portable degrade (missing TCP_USER_TIMEOUT optname -> ENOPROTOOPT -> keepalives still armed; plan() linux-both / other-keepalives-only; the Linux ABI optnames are linux-gated so darwin sockets take the portable node-API keepalive instead of foreign option numbers).
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1810 (AC1 waived with reason — see below). #1802 refinement 2, the kernel half of #1803's P2.
The frontdoor — where the physics are real — is fully armed with kernel-read-back proof: every upstream dial (build_preamble, backend_get, passthrough_capture) AND every accepted member socket (the only place in the deployed topology with real corpse physics — remote clients over tunnels; loopback hops are ACK'd by the living host's kernel, so keepalives can't detect process death there) carry SO_KEEPALIVE idle 30s / intvl 5s / count 3 + TCP_USER_TIMEOUT 30s, verified end-to-end via ss timer read-back through the live frontdoor subprocess (tuned env reads back 12/4/9/45000; invalid values keep defaults; forced ENOPROTOOPT degrades to keepalives-only without a crash). Frontdoor suite 56 OK (+5).
The engine side: bun's node:http accepted sockets are fd-less native shims (measured three ways — no fd, no option surface, setKeepAlive is a silent kernel no-op; same runtime-gap family as the 2026-10-14 res.destroy-sends-no-FIN finding). The shared arm module (tcp-keepalive.ts) fully arms fd-bearing sockets with kernel-read-back telemetry ({keepalive:true, keepIdleS:30, keepIntvlS:5, keepCntS:3, userTimeoutMs:30000} on a real accepted net socket) and degrades LOUDLY on the shim path (once-per-process SSE-KEEPALIVE-GAP line naming exactly what carries corpse detection instead). Wired into both SSE routes. AC1's letter is waived on this branch: the runtime wall is measured, and the loopback physics make the engine-side knobs decorative even if the serving stack swapped to raw node:net (option b rejected — an architecture change for physically-decorative benefit).
Engine suites: tcp-keepalive 8/8; test/server 688 pass / 2 fail (the environmental corpse pair, byte-identical at base); test/session 599/5; typecheck 0 new; drift PASS.
Director adjudication: the accepted posture is the loud-degrade one — the engine's corpse detection on loopback hops stays the bounded writer (SSE-DROP) + the frontdoor's FIN discipline + the recycler, all armed; the kernel knobs guard the member edge where the fleet's real networks live.