fix(lemonade): retry interrupted backend setup

[volen-silo]

Summary

• Retry Lemonade's llama.cpp backend installation once when the backend subprocess fails during first serve.
• Keep the retry at the backend boundary so deterministic archive, disk, and backend-selection failures are not repeated and a healthy embeddable is not re-extracted.
• After the retry fails, recommend rocm engines install lemonade --reinstall before retrying rocm serve.
• Add deterministic unit coverage and a black-box Cucumber scenario for the visible retry count and recovery guidance.

Why: a transient connection loss during first-time Lemonade backend preparation previously aborted rocm serve with a generic error, forcing users to discover the manual recovery sequence themselves.

Risk: low. The change is limited to the backend-install phase, allows one additional attempt, and exposes its failure fixture only in E2E-enabled builds.

Test plan

• cargo xtask e2e -- -n 'Repeated Lemonade preparation failure'
• cargo test -p rocm-engine-lemonade backend_install_
• cargo test -p e2e-cucumber --lib
• cargo clippy --workspace --all-targets -- -D warnings
• cargo test --workspace --all-targets -- --skip therock::tests::extracting_the_sdk_archive_removes_it
• cargo fmt --all -- --check

The skipped archive test is a pre-existing parallel-suite flake on this host (tar temporarily missing from PATH) and passes in isolation.

[r0x0r]

Reviewed the diff and the tests. The change is well-scoped and the reasoning holds up.

Strengths

• Correct retry boundary. Putting the single retry around the backend install rather than the whole runtime prep is the right call, and the doc comment justifies it well: the embeddable download already has bounded transport retries, and repeating the outer op would redo deterministic archive/disk failures and risk re-extracting a healthy runtime. The transient this targets (backend subprocess losing its connection to lemond after a completed download) is exactly the case a second call can reuse the cache and recover from.
• Terminal error is the retry's failure, not the first attempt's, with actionable recovery guidance appended (rocm engines install lemonade --reinstall, then retry rocm serve). The stops_after_one_retry_with_reinstall_guidance test pins this precisely, including that the first error is not the one surfaced.
• Layered coverage. Three deterministic unit tests (success-without-retry, recover-on-second, stop-after-one-with-guidance) plus the black-box scenario 15 that asserts the visible retry count and the recovery command.
• No production behavior leak. The failure seam is #[cfg(feature = "e2e-test-hooks")] and the serve() precondition bypass is cfg!(feature) && env_var — both compile out of release builds. The only non-gated change is wrapping run_lemonade_backend_install in the retry helper, which is strictly additive robustness and can't turn a previously-passing install into a failure.

Minor / non-blocking

• The e2e assert in setup_lemonade_hf_checkpoint_model panics with {stdout} while the real serve error lands on stderr, so when that path fails the root cause isn't visible in the captured output. Pre-existing (not introduced here), but a follow-up to also surface stderr would make failures in that scenario much easier to triage.
• xtask now builds all e2e binaries with --features rocm/e2e-test-hooks, including the self-hosted GPU lanes. The seam is inert without the env var so this is fine, but it does mean those binaries are no longer feature-identical to release — worth a line in the PR description so it's a deliberate, understood choice.

Overall the implementation, layering, and tests look good.