Repository navigation
fix(hub): forward the engine child's stdout to the shard log — the telemetry's eyes open in production (#1808) - #1809
Merged
aarontrowbridge merged 1 commit intoOct 16, 2026
Conversation
…CK/SELF-STALL reach production (#1808) The mute sink was never the telemetry module: the service runner spawns the engine with piped stdio and consumed both pipes into a private engineLog string surfaced only on boot failure, so the default console.log sink (engine stdout) never reached the runner's stdout — the surface the systemd unit appends to the shard log. Forward every chunk as it arrives (injectable engineOutput for tests; the diagnostics buffer stays), and add the AMICODE_SSE_TICK_STALL=1 drill override that parks the /event heartbeat tap mid-fiber so the out-of-tree SELF-STALL watchdog can be rehearsed against the production wiring end-to-end.
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1808.
Root cause, named: the service runner spawned the engine with stdio pipes consumed into a private in-memory buffer surfaced only on health-check failure or unexpected exit — EVERY engine console line (SSE-TICK, SSE-DROP, SELF-STALL, boot banners, self-shutdown) was swallowed in production. The Phase A telemetry was never broken; its transport was. The runner now forwards every engine-child stdout/stderr chunk to its own stdio as it arrives — the exact surface the systemd units append to the shard logs. Injectable engineOutput for tests; the diagnostics buffer kept.
Verified with real processes (scratch ports 9411-9415, scratch XDG, hub untouched): a branch-built engine's SSE-TICK lines land in the runner-captured stdout within one cadence window (quoted in the issue evidence), and the out-of-tree SELF-STALL watchdog's firing is greppable through the production wiring under the env-driven stall drill (AMICODE_SSE_TICK_STALL=1 — an ops-only rehearsal override, parks the tick mid-fiber, the #1798 wedge shape).
Director gates re-run: runner suite green (the forwarding test red→green), drift PASS, scoped engine trio 14/2 (the briefed environmental corpse pair), full-suite failure set byte-identical to HEAD.
Composes into the one idle-window deploy with P1a (PR #1807) — reaching the live shards requires the runner bundle rebuild + the engine build, both HITL ops.