test(e2e): explain a managed serve that exits 0 and never serves

[rominf]

Observability prerequisite for #260. It does not fix #260, which stays open — see "What this does not do" below.

The problem

rocm serve <model> --managed returns as soon as the supervisor is launched, so an engine that dies afterwards leaves a zero exit code. The E2E serve preconditions opened with run_rocm_ok plus a bare readiness poll, and that pair has nothing to say about this outcome: the poll panicked with only

endpoint http://127.0.0.1:11435/v1/models did not serve model Qwen3-0.6B within 600s

while the CLI's output, the engine's own log, and the device state were all dropped before the panic. The engine log is the one file that holds the answer, and it lives in the scenario's isolated TempDir — deleted with the scenario, while CI uploads only tests/e2e-cucumber/results/. So every run destroys its own evidence.

#260 is stuck exactly there: the failure reproduces only on the self-hosted Strix Halo Windows lane, and three runs have now produced nothing to reason about.

What changed

• A stalled serve now reports the invocation and exit code, the device state it started on, both CLI streams, the tail of the engine's log, and how stopping the service went — the diagnostics setup_gpu_model already collected, now shared with the four preconditions that lacked them (setup_lemonade_model, setup_lemonade_hf_checkpoint_model, setup_large_gpu_model, user_serves_default_engine) through one collector, so they cannot drift apart.
• The full engine log is copied into the results directory before the scenario's TempDir takes it. The 40-line tail cannot show the startup banner naming the backend and device that were selected, which is what an engine that never answers has to be read against. Oversized logs keep both ends with the gap marked, read by seeking rather than loading the file, so a crash-looping engine can neither inflate the artifact nor risk an allocation abort in the middle of writing the report.
• The headline says "exited 0" out loud, because nothing else about the failure distinguishes it from a healthy serve.

Reuses cli_failure_report's section formatting rather than adding a parallel one, so every bundle reads the same way and an empty stream stays explicitly marked (empty) — a silent engine and a harness that lost the output call for opposite next steps.

Non-obvious decisions

• Ordering: read the log, archive it, then stop the service. The tail then reflects what the engine wrote on its own, and — more importantly — rocm services stop has no timeout, so archiving after it would put the archive behind a call that can wedge on exactly the misbehaving engine this exists to explain. The cost is that shutdown output is not archived.
• commands.jsonl deliberately unchanged. Recording stdout/stderr for every invocation would inflate the coverage record on every passing command, and recording it only for non-zero exits would miss this bug entirely, since it exits 0. The evidence goes into the failure report and the archived log instead.
• user_serves_default_engine still does not fail on a non-zero exit code. It is a When; the CLI printed the plan line its scenarios are about, and failing here would pre-empt the Then step that names the actual disagreement.

Verification

Run locally on Linux: cargo fmt --check, cargo clippy --locked --workspace --all-targets -- -D warnings, cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings, cargo test -p e2e-cucumber --lib (81 pass), and cargo xtask e2e — 46 scenarios, 0 unexpected failures, identical to main.

New unit tests in src/serve_log.rs cover the parts that are testable without the hardware: that a stalled-serve report carries every piece of evidence and marks empty streams, that the full log lands under results/ byte for byte at the artifact-relative path it reports, the cap boundary (exactly at the cap is verbatim; one byte over elides exactly one byte and the halves never overlap), and that every unarchivable case explains itself instead of panicking inside a failure path.

No scenario is added: this changes no user-observable CLI behaviour, only what the harness reports when a serve stalls. The changed step definitions are all @requires-gpu, so they are exercised by the self-hosted lanes rather than the mock lane.

What this does not do

• It does not fix rocm serve owner/repo:variant on lemonade exits 0 but never serves the model on Strix Halo Windows #260. I have no Strix Halo Windows hardware, so I could not reproduce the underlying serve failure locally and did not attempt a fix for it. Anything I wrote would have been a guess.
• No xfail row was added. The point is to make the lane diagnosable, not to hide it — that lane stays red after this merges, and correctly so.

What the next red run should produce

For scenario serve-hf-checkpoint-inference, instead of the bare timeout line: the serve plan and log_path, the device state, and service-logs/<scenario>--<service-id>.log in the e2e-gpu-strix-windows-report artifact containing the packaged llama-server's complete stdout and stderr — the engine writes both there. That should distinguish the three candidates #260 names (direct llama-server launch, GGUF path resolution in the HF cache, port binding) without another round trip.

Risk: low — test harness only, no production code, no dependency or config changes. The one effect outside the harness is a bounded artifact-size increase, capped at 4 MiB per archived failed serve.

Related: #248 fixed the sibling problem of discarded stderr on a non-zero exit; this path exits 0, so it was not covered.

[rominf]

Moving back to draft — this regresses the lane it was meant to make legible.

On the Strix Halo Windows lane this branch produces 5 unexpected failures, against a
baseline of 1:

	scenarios	unexpected failures
#262 (run 32029548789, 12:22–13:21)	39 (35 passed)	1 — serve-hf-checkpoint-inference
this branch (run 32030232229, 13:01–13:13)	39 (31 passed)	5

The four extra are bench-load-real-serve, chat-tool-definitions-accepted,
chat-end-to-end-local-model and serve-lemonade-inference — all lemonade scenarios, all
reporting:

Installing backend: llamacpp:rocm
Error: request_failed: Lemonade backend install failed with status exit code: 1

Both runs are on the same self-hosted runner in the same window, so this is the branch, not
the environment. The GPU lane is unaffected (1 unexpected, same as baseline) — consistent
with its serve failing at exit-1 before the new stalled-serve path is reached.

Likely mechanism, not yet confirmed: serve_failure_evidence calls stop_scenario_services
while gathering evidence, and this lane deliberately shares one installed managed runtime
across serve/chat scenarios, so stopping services inside a failing scenario may disturb state
the later ones depend on.

Consequence for #260: the backend install failed line above is the first concrete error this
scenario has ever produced, but until the cascade is untangled I cannot tell whether it is
#260's actual cause or an artifact of this branch's own interference. It should not be quoted
as the root cause yet.

[rominf]

I re-ran the Windows lane and the regression does not reproduce. The lane is back to the baseline of exactly 1 unexpected failure, with no code change — the earlier run was an infrastructure flake on a different machine. Details, because the original comparison was not like-for-like:

The two runs were on different runners. Job metadata:

run	runner	started	unexpected
#262 32029548789	strix-halo-windows	12:53Z	1
this branch 32030232229 (1st)	strix-halo-windows-2	13:02Z	5
this branch 32030232229 (re-run)	strix-halo-windows	13:35Z	1

strix-halo-windows-2 appears exactly once in the last 40 workflow runs — that first attempt. Its e2e-prewarm cache was cold, so it had to install the llama.cpp backend, and the install failed on the network:

Installing Lemonade llamacpp:rocm backend...
Error: Failed to download llama-server.exe from:
https://github.com/lemonade-sdk/llama.cpp/releases/download/b9752/llama-b9752-bin-win-rocm-7.13-x64.zip
  - Download failed after 6 attempts.
  Last error: Download failed: SSL connect error (CURL code: 35)
Error: request_failed: Lemonade backend install failed with status exit code: 1

That truncated backend install failed line was the symptom; the TLS failure above it is the cause. All five failures on that attempt share it.

The stop_scenario_services hypothesis is disproven, on two independent grounds:

• Empirically: the first failure was bench-load-real-serve, the 4th scenario and the first lemonade serve of the run — before any stop could have run. Its own report says --- stop --- no services recorded in ...\rocm-e2e-c1Jx40\data\services, i.e. the stop was a no-op. It also reaches that step through setup_gpu_model, which already called stop_scenario_services on main; this PR adds no stop there.
• Structurally: stop_managed_services only reads <isolated root>/data/services/*.json and runs rocm services stop with ROCM_CLI_*_DIR pointed at that same isolated root. The engine state it can touch is data/engines/<engine>/state (AppPaths::service_engine_state_path). The shared managed runtime is data/runtimes, which is a separate subtree the stop never reads or writes. So it cannot reach another scenario's state, shared or otherwise.

Diff comparison of the re-run against main's own run on the same warm runner (32028129073) is now zero differing outcomes, same 39 scenarios, same single serve-hf-checkpoint-inference failure. The GPU lane is unchanged at its baseline (1 unexpected, #259).

Both remaining red lanes are red on main for reasons this PR does not touch: GPU/MI300X is #259, Strix Windows is #260 — which this PR deliberately does not fix.

Marking ready again. Posting the #260 evidence separately on that issue, since it is now trustworthy.

[r0x0r]

Approve — well-scoped, test-harness-only, drives #260 toward diagnosability

Reviewed the diff, the new serve_log.rs, the step-definition refactor, the tests, and the live CI. fmt, clippy, unit tests, and build-and-test are green; the only red is the self-hosted Strix Halo Windows lane, which this PR intentionally and correctly leaves red.

What it does well

• Correct root problem. A --managed serve returns once the supervisor launches, so an engine that dies afterward exits 0 and the old run_rocm_ok + readiness-poll pair panicked with a bare did not serve … within 600s, discarding CLI output, engine log, and device state. This captures all four and archives the full engine log under results/ (the only path CI uploads) before the scenario TempDir deletes it.
• Good deduplication. section() is lifted to pub(crate) and shared between cli_failure_report and the new serve_attempt_report, so every bundle reads identically and empty streams stay explicitly (empty). The five preconditions share one collector (serve_failure_evidence) so they cannot drift.
• Load-bearing ordering (read log → archive → then stop the service) is correct: services stop has no timeout and can wedge on exactly the misbehaving engine, so archiving after it would risk losing the artifact.
• read_clamped is careful. Bounding by seeking rather than read-then-trim avoids an allocation abort mid-report on a crash-looping engine; the head/tail-with-elision preserves both the startup banner (backend/device selection) and the failure tail. Tests cover the cap boundary precisely (at-cap verbatim; one-over drops exactly one region; halves never overlap).
• No cli_output regression. The replaced run_rocm_ok never set world.cli_output for these preconditions, and serve_and_wait doesn't either; user_serves_default_engine still carries cli_output/cli_stderr/cli_rc for its deferred Then.

Non-blocking observations

1. The diagnostics already paid off — and show the current red isn't #260. The failing run's newly-captured evidence is Failed to download llama-server.exe … SSL connect error (CURL code: 35) … Lemonade backend install failed, i.e. a transient GitHub-releases download flake, not the exit-0 stall #260 describes. Might be worth a one-liner in the PR noting the lane's present red is environmental (and that surfacing it is the improvement over the old bare timeout).
2. device state: no GPU tooling (mock/local run) on the Strix Halo Windows lane. In the same run, the device-state section rendered blank of any GPU info on the very lane where #260 reproduces. If ensure_serve_port_free's snapshot cannot read device state on Windows, the "device it started on" pillar may come back empty exactly where #260 needs it. Reasonable follow-up, out of scope here.
3. Minor: MODELS_URL/SERVE_BASE_URL hardcode 11435 while SERVE_PORT exists — a small drift risk; a doc cross-reference or const composition would close it.

Approving. The remaining Strix Halo Windows red is expected and correct per the PR's own "What this does not do".

[rominf]

Thanks for the review. On the three non-blocking observations:

1. Current lane red is environmental, not rocm serve owner/repo:variant on lemonade exits 0 but never serves the model on Strix Halo Windows #260 — confirmed and written up in detail in the two comments above (the strix-halo-windows-2 cold-cache TLS download failure vs. the re-run on strix-halo-windows landing back at baseline). Didn't fold it into the PR description since the investigation is time-boxed to that run; the comments capture it.
2. Blank device-state on the Windows lane where rocm serve owner/repo:variant on lemonade exits 0 but never serves the model on Strix Halo Windows #260 reproduces — agreed, tracked as a follow-up against rocm serve owner/repo:variant on lemonade exits 0 but never serves the model on Strix Halo Windows #260 rather than this PR.
3. serving_steps.rs hardcoding 11435 in SERVE_BASE_URL/MODELS_URL instead of reusing SERVE_PORT — fixed in 2d7cfd4. Both are now LazyLock<String> derived from SERVE_PORT, so the three can't drift again.