ci: preserve cache failure evidence and compile through daemon outages

[zackees]

Stacked: builds on #1365 (startup failure diagnostics and shared Xvfb lifetime fix). The additional commits preserve launcher-recovery evidence and compile uncached when the daemon is confirmed unavailable. Base remains master; once #1365 merges the diff reduces to the cache recovery change.

Summary

Refs #1286.

Master job 117523909439 failed with 210 launcher exits 125. Its existing recovery overwrote the original build log, leaving recovery-cooldown messages and hiding the initial cause.

Preserve the first build log beside the retry log, print its bounded error summary, and print bounded daemon status/stop diagnostics before the existing restart. Diagnostic command failures remain nonfatal; the second build still determines success. No extra retry, changed C tests or compiler-error suppression. If daemon status confirms unavailability, the existing single recovery build uses ZCCACHE_DISABLE=1 inside a subshell. Real compiler failures still fail that build; a healthy daemon retains the existing restart behavior.

RED / GREEN

• RED d3593cc15: executing the actual recovery block loses the first failure in both successful and failed retry fixtures.
• GREEN 725b15761: uv run --no-project python -m unittest ci.test_tier2_build_errors — 6 tests pass. The recovery test checks first-log preservation, original diagnostic, status/stop output, exactly two build calls and final exit status for both retry outcomes.
• bash -n rust/tools/linux-validate/validate.sh and git diff --check pass.

This improves evidence collection; the daemon failure root cause remains unknown. No workflow file added and no submodule change.

Live failure and availability fallback

The initial diagnostics CI run (job 117525500312) captured a lost daemon connection followed by connection refused and recovery cooldown. It also verified restoration of the new container cook namespace (542 files); the standalone C/C++ daemon failure is separate.

Follow-up RED 058974baa requires uncached compilation when status reports no daemon. GREEN 1f0f7090c plus fixture correction 19031378e passes all 6 tests, including the four combinations of healthy/unavailable status and successful/failed retry. The fixture uses the production pipefail setting. Exactly two build calls remain, and only the unavailable-daemon retry receives the bypass. This addresses cache availability without claiming the daemon-death cause is fixed.

Stack verification: all 11 focused recovery, timing-persistence and Xvfb-lifetime checks pass together. This source update incorporates the demonstrated fix for the startup failure that blocked the previous PR run; no failed job was blindly retried.

The stack now includes #1365's exported-interposer correction. The real CMake-built diagnostic test passed locally under a valid X display after reproducing the exact CI failure first; all 11 combined recovery/lifetime/timing checks pass. Cancellation was requested for the superseded run 39080141182.

Preserve the last healthy remote seed

RED 162ec1d96 exercises the real recovery block and requires an unhealthy output only for daemon-unavailable bypass, plus a workflow publication guard. GREEN 0aa23908e emits cache-healthy=false through the build step output and skips remote compiler-cache publication in that case, retaining the prior seed. Healthy saves retain existing behavior; the compiler result still decides job success. The fixture isolates GITHUB_OUTPUT in its temporary directory and covers both retry exit results and daemon-health states. All 12 combined focused checks pass; shell syntax and diff checks pass.

The PR was dequeued only after this guard was implemented/tested so its queued branch could be updated. Automatic queue entry is re-enabled for fresh CI. #1365 remains an independent prerequisite.

[zackees]

Additional real-binary verification for the fallback (not just shell stubs): downloaded standalone zccache 1.15.5 into ignored in-repo scratch and verified SHA-256 39906c92b0287e2aac51ae9169b368d36eabc3af363c3c7bd29dc3783975e579, matching the installer pin. In an isolated empty cache, zccache status returned exit 1 with daemon-not-running. With ZCCACHE_DISABLE=1, a valid C translation unit compiled successfully, while a deliberate #error current-source-must-fail returned exit 1 with that exact diagnostic. Thus the availability fallback invokes the actual compiler and preserves current compiler errors. No shared cache or daemon was stopped/deleted. Raw results remain in build_ci_audit/zccache-1.15.5-probe/.

Updated PR CI run 39077243497 is in progress. This does not claim the daemon-death root cause is fixed or the CI fallback has yet been exercised live.

[zackees]

Updated PR CI Tier 2 build 117526986641 succeeded: exact new container cook restore (542 files), 781/781 cold standalone compiler calls, 349-second build step / 439-second job. The daemon remained available, so recovery was not exercised live. This verifies the normal path; availability fallback behavior remains covered by shell execution tests and the checksum-verified real-binary compiler checks above. OFF/ON test jobs and the required gate still need to complete.

[zackees]

New live CI evidence from #1286 follow-up PR #1362, run 39081664635: both OFF job 117540316673 and ON job 117540316794 failed the unchanged test_recording_shutdown_wait at line 121 (out.active.load() is not false). OFF also failed line 142 (stopped is not false) in the following no-stop case. Each suite registered 295 tests; the new startup diagnostics passed. No retry, skip, or assertion change performed.

Observed OFF log: stopped 1 after 220 ms, posted timer fired 1 (while waiting 1), then line 121; next case no stop: returned 1 after 300 ms then line 142.

Source inspection identifies a concrete race candidate in the already-merged #1342 implementation: State::Stopped() returns deactivated.load() || !active(). The unchanged fixture signals deactivate before setting active=false, so the UI thread can observe the signal and return before the active flag clears. The following-case failure may be fallout from cmocka's failing assertion bypassing the stack fixture's normal destruction; that part is an inference requiring reproduction. These are state assertions, not merely tight elapsed-time bounds; scheduling the test serially would not establish a correctness fix.

Recording here under the existing fresh claim rather than starting competing source work. This is currently blocking #1362 CI; its build job succeeded. Raw job logs are retained locally under build_ci_audit/recovery-latest-{off,on}-.log.