Repository navigation
fix: the SSE heartbeat tick stall — bounded fan-out + tick telemetry + FIN-closing frontdoor (#1798) - #1800
Merged
aarontrowbridge merged 14 commits intoOct 15, 2026
Conversation
…atchdog (#1798 phase A)
…s guarantee (#1798 phase A)
…STALL watchdog (#1798 phase A)
…t crashes (#1798 phase A)
…ibers (#1798 phase C) The named mechanism: the platform response writer parks forever on a FIN-less corpse (nodeResponse.write -> false, then a drain event the corpse never produces) and the connection's whole recurring emission wedges behind it while HTTP stays healthy. Bound the fan-out instead: each connection emits through a bounded write queue with an offer timeout (half a watchdog window, so the engine heals itself before the SELF-STALL watchdog could declare the tick source silent). When the offer bound trips, the subscriber is dead: noteDrop names it in the Phase A telemetry registry (SSE-DROP line) and unregisters it, the write queue is shut down, and the raw response + socket are destroyed engine-side. A dead subscriber wedges nothing — every other connection keeps its heartbeats and events.
…ut (#1798 phase C) The corpse test simulates the frontdoor's #1757 FIN-less kill exactly: an external socket with a 4KB kernel receive buffer, held open, blocked in recv, never closed (bun's own sockets absorb multi-MB into userland buffers before kernel backpressure reaches the engine's write path, so the corpse must be external). It pins that a wedged corpse is dropped (SSE-DROP line with the write queue pinned full, registry teardown) while a healthy subscriber keeps its heartbeats and the SELF-STALL watchdog never fires. The propagation test pins the wedge window itself: with the corpse's tick source silent, the bus fan-out still delivers events to the healthy subscriber and its Stream.tick keeps cadence through the wedge and past the drop.
…ills the corpse class (#1798) The #1757 hazard, fixed at the source: a bare close() on a socket with a concurrent reader parked in recv (or a writer in sendall) sends no FIN on Linux — the parked syscall holds the description open, the peer never sees EOF, and the engine keeps writing into the corpse until its SSE fan-out wedges (the #1798 fleet-wide heartbeat-stall propagation; 19,391 FIN-less recycles in the observed window). The engine-side half cannot fix this at the wire (BUN's node:http sends no FIN), so the frontdoor's shutdown is the engine's only EOF signal. - _fin_close: shutdown(SHUT_RDWR) then close — FIN out immediately, parked syscall woken; every lifecycle close that can race a concurrent socket user goes through it - wired: the dead-man recycler's upstream kill (was a bare s.close()), the per-member waiter teardown (was a bare c.close()), and _drop refit onto the shared helper (behavior unchanged from its #1757 shutdown) - the #1757 comment in _drop now records the fix, not just the hazard Tests: the runner test drives the real module's _fin_close under the exact #1757 condition (blocked reader, EOF at the peer; RED pre-fix); the bare-close control pins the corpse as real under plain close; the wedge fixture now probes for the frontdoor's FIN end-to-end (recycle -> engine sees EOF); the member-teardown test proves the FIN is additive (normal drain keeps the group healthy). Suite: 51 tests, OK (skipped=1).
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1798.
The stall, root-caused from a live capture: the frontdoor recycler killed upstream sockets FIN-lessly (#1757's Linux close-with-concurrent-reader hazard); the engine's SSE fan-out blocked writing to the corpse; every SSE route on the engine wedged while HTTP served at 2ms; the dead-man fired and manufactured MORE corpses (19,391 recycles in one window) — causal, fleet-propagated.
Both ends, either breaks the loop, both make it structural:
Director gates (own runs): scoped corpse/propagation/telemetry 15/0; engine session suite 599/5 (exact known baseline); frontdoor suite 51 OK (+4 tests, the bare-close control among them); drift PASS.
Stays DRAFT pending: the deploy decision (HITL — both surfaces: engine binary via the provenance ritual + the live frontdoor), then the 72h soak gate (zero stall windows from the cadence logs) per the approved spec.