fix(install): restore the engine's pinned deps after an SDK install

[volen-silo]

Summary

rocm install sdk could leave the managed runtime holding a torch build the installed vLLM engine does not accept, while every status surface still reported the runtime ready. The first symptom a user gets is a serve failure whose traceback names neither torch nor the SDK.

Root cause. The runtime venv has two owners. install sdk writes TheRock's torch stack into it, and the vLLM engine installs into the same interpreter, pinning a torch build from its own index. A second install sdk writes the SDK's torch back over that pin. The engine auto-install that runs immediately afterwards should have restored it, but its short-circuit tested whether vLLM resolved rather than whether vLLM's requirements were met — and since only torch had moved, vLLM still resolved, so the pin was never re-asserted.

The two torch builds cannot be reconciled by picking differently on the SDK side: the engine pins an exact local version (+git<sha>) that no TheRock build satisfies under PEP 440. The engine deliberately wins; the bug is that the SDK silently un-wins it.

What changed

• A dependency-consistency primitive in rocm-core over uv pip check: report the unsatisfied Requires-Dist entries in an environment, filtered to the distribution that declares them.
• The vLLM engine's install short-circuit is now gated on its own requirements being met, so a violated pin falls through to an install that restores it. This makes rocm engines install vllm self-heal too, and leaves reinstall: false meaning "don't redownload a healthy engine".
• install sdk states the outcome as dependency_check: satisfied | violated | not_verified and records it in the audit log — on every auto-install outcome, including a failed one, which is where a violated pin is most likely.

Non-obvious decisions

• The check runs with --color never. uv colourises its findings under FORCE_COLOR/CLICOLOR_FORCE even when stderr is a pipe. The escape prefix pushes every finding past the parser, which reads as "could not verify" and silently restores the old behaviour — i.e. the fix would quietly do nothing on such a host. There is a regression test for the colourised line specifically.
• The parser requires the full frame. uv pip check reports several other conditions under the same The package \x`opening — a missingWHEEL/METADATAfile, an unsatisfiedRequires-Python, multiple installed distributions. None is a Requires-Dist` that reinstalling the requirer can clear, so accepting them would force a futile multi-gigabyte reinstall and print a remedy that cannot help.
• The repair targets the interpreter that was assessed. The two resolvers disagree — one walks the candidates until it finds the one that actually has vLLM, the other takes the first unconditionally — and runtime ids match by prefix, so several candidates routinely qualify. Repairing a different interpreter than the one found broken would leave the broken one broken and report it fixed.
• install sdk stays exit-0 when a violation survives. The SDK itself installed correctly, and abandoning a multi-gigabyte install costs the user more than the warning, which names the one command that clears it. Say so if you'd rather it were fatal.
• Only the engine's own violations are reported. These environments carry chronic unrelated conflicts between third-party packages; surfacing them all would bury the one finding the user can act on.

Scope boundaries

Deliberately not included: teaching examine / runtimes list / engines list to stop reporting a runtime with a violated engine pin as ready. That is tracked separately; the primitive added here is what that work will build on.

Also known and not addressed here: the report is wired into the SDK auto-install path only. That path early-returns when there is no preferred engine, and vLLM is preferred only for -dcgpu families and gfx906/gfx908/gfx90a, never on native Windows — so none of this behaviour runs on consumer/RDNA hardware, including this project's own non-dcgpu E2E lanes. A host outside those families that installed vLLM by hand still gets the overwrite with no repair and no report. Happy to fold that in if reviewers would rather have it in one change.

Test plan

• Unit coverage for the parser (real multi-line uv pip check bodies, the clean body, a colourised line, the four other diagnostic shapes that share the opening frame), the repair predicate, the assessed-interpreter selection, and each rendered dependency_check outcome.
• New Gherkin scenario @id:runtime-sdk-reinstall-keeps-engine-consistent covering a second install sdk on a runtime that already has the engine. Tagged @requires-gpu @requires-engine:vllm @nightly: it needs a real SDK install, a real engine install and a second SDK install, so it cannot run in default CI — it runs on the nightly GPU lane. vLLM is gated because only it shares the runtime environment; Lemonade manages its own.
• cargo test --workspace, cargo clippy --workspace --all-targets, cargo fmt, prek all clean locally.
• The uv behaviours the change depends on were probed directly against uv 0.9.30 rather than assumed: the stdout/stderr split and exit codes, all five line shapes, that --color never defeats FORCE_COLOR, and that a plain uv pip install <pkg>==<pin> does restore a dependency that was downgraded underneath it (uv's satisfaction fast-path is recursive over the closure, so the repair is not a no-op).

Not verified here: the original reproduction needs a gfx94X host. The install sdk → install sdk → compare-against-Requires-Dist sequence, the follow-up serve, and the new scenario all still need a run on that hardware.

Risk

Medium. On a healthy host the short-circuit behaves exactly as before, so install sdk is unchanged; only an already-violated environment takes the new path. The real exposure is that a violated pin now always triggers a uv resolve where previously nothing ran — on a host where that resolve fails, a repeat install sdk that used to be quiet will now surface the engine-install failure. That failure is reported rather than swallowed, which is the intended direction, but it is a visible behaviour change.

[tomastola]

I reported this one, so treating it mostly as: does this close the failure I hit.

The root cause matches. The short-circuit testing "does vLLM resolve" rather than "are vLLM's requirements met" is exactly the gap — the second install sdk moved only torch, so vLLM kept resolving and the install pass that would have re-pinned it never ran.

I checked the part the fix hinges on, since it isn't obvious from reading: install_vllm_with_uv runs a plain uv pip install vllm==<pin> on the repair path (reinstall is false there), which looks like a no-op given vLLM is already present. It isn't — uv's satisfaction check is recursive over the dependency closure, so a violated transitive pin does get restored. Worth a word in the code comment; a future reader will very reasonably assume that call does nothing and "simplify" the repair into a real no-op.

A few things before I'd sign off.

1. The printed remedy skips the safeguard this PR just added. On a violation the tool prints action: rocm engines install {engine} --reinstall. But reinstall: true forces resolved = None in install_response, so assessed is None, and the install falls back to resolve_managed_runtime_python — candidates.into_iter().next(), first candidate unconditionally, under prefix matching in runtime_matches. That's the resolver the comment above assessed says must be avoided, for the reason given there: "installing into a different interpreter than the one found broken would leave the broken one broken and report it fixed." The printed command also carries no runtime scoping, so it resolves the default candidate.

Since install sdk deliberately exits 0 on a surviving violation, this line is the user's only recourse — on the multi-candidate hosts the comment says routinely occur, following it can repair a healthy environment and report success. Either thread assessed through the reinstall: true path too, or scope the printed remedy to the runtime that was checked. (The string predates this PR, but this is the first place the tool actively directs users to it in a situation where the distinction is known to matter.)

2. The coverage note undersells the gap. The description says the report is wired to the SDK auto-install path only, so a host where vLLM "was installed by hand" is unprotected. Accurate, but maybe_auto_install_sdk_preferred_engine early-returns whenever there is no preferred engine, and preferred_serve_engine_for_therock_family returns vllm only for families ending -dcgpu or in ["gfx906","gfx908","gfx90a"], never on native Windows. So none of the new behaviour runs on consumer/RDNA families — gfx1201 normalizes to gfx120X-all and matches neither condition, including on this project's own non-dcgpu E2E lanes. Fine as a scope boundary, but one sentence naming it would stop reviewers from reading "installed by hand" as a corner case.

3. The decision the PR changes has no coverage in default CI. The new unit tests all exercise pure helpers — rendering, parse_dependency_violations, repair_from_violations, assessed_python_for_repair. None drives the install_response match that is the actual fix, and check_dependencies' branching (exit-0 vs non-zero-with-findings vs non-zero-empty) is untested. With the scenario tagged @nightly @requires-gpu, nothing in the PR lane covers the gate. ROCM_CLI_UV_BINARY already exists — a stub emitting canned stderr and exit codes would cover all three branches and the gate itself, no GPU needed. Given the parser is acknowledged as brittle and uv_version() defaults to latest, that seems worth having.

4. assert_engine_still_ready can't fail. It asserts engines list contains runtime: ready — the signal the description identifies as staying green while the runtime was broken, and whose repair is explicitly deferred. It would have passed before this change. Not harmful, but it reads as corroboration the scenario doesn't provide; the dependency_check: satisfied assertion above it is doing all the work.

Two things that check out, since they're easy to doubt: the colorized path does degrade safely — a colorized finding parses empty, the exit is non-zero, so check_dependencies bails and it surfaces as not_verified, never a false satisfied. And requiring both halves of the frame to exclude the broken-WHEEL / Requires-Python / multiple-distribution shapes is right; none of those is fixable by reinstalling the requirer.

On CI: the red E2E tests (Strix Halo, Windows) looks pre-existing rather than yours — it fails on CURL code: 35 fetching llama-server.exe from the lemonade-sdk release, and the same lane fails identically on a same-day Dependabot PR that only bumps action SHAs. The two CANCELLED jobs stalled inside actions/checkout and hit the job time cap; windows-build-and-test did run cargo test --workspace --all-targets and smoke_local.py green, so the suite isn't actually unrun.

Not approving yet only because the original reproduction hasn't been run on gfx94X — you flag that yourself. Once the install sdk → install sdk → serve sequence has been through on that hardware, and (1) is resolved either way, this looks good to me.

[tomastola]

Ran the reproduction on a gfx942 (MI300X) host, which closes the gap noted as unverified in the description. Three passes against one runtime (release-wheel-gfx94x-dcgpu-7-13-0), the first two on the merge base and the third on this branch.

Baseline, first install sdk — engine auto-install selects vLLM (reason: detected ROCm GPU family prefers vllm (gfx94X-dcgpu)), environment lands on vLLM's pins with torch 2.11.0+gitd0c8b1f.

Baseline, second install sdk — the bug, verbatim:

torch  2.11.0+gitd0c8b1f  ->  2.11.0+rocm7.13.0

Found 3 incompatibilities
The package `vllm` requires `torch==2.11.0+gitd0c8b1f`, but `2.11.0+rocm7.13.0` is installed
The package `vllm` requires `torchvision==0.24.1+d801a34`, but `0.26.0+rocm7.13.0` is installed
The package `vllm` requires `torchaudio==2.9.0+eaa9e4e`, but `2.11.0+rocm7.13.0` is installed

reinstall: false, exit 0, no dependency warning. Silently unusable runtime.

This branch, install sdk against that same environment — the gate fires, the install runs, every pin is restored:

warning: the runtime environment did not satisfy vLLM's pinned dependencies; vLLM was reinstalled to restore them (...)
warning: if this recurs after `rocm install sdk`, the SDK torch stack is being written over vLLM's pinned torch
dependency_check: satisfied

torch 2.11.0+gitd0c8b1f · torchvision 0.24.1+d801a34 · torchaudio 2.9.0+eaa9e4e
All installed packages are compatible

Worth noting that pass covers both cases at once: install sdk overwrote the torch stack again on the way in, and the gate caught and repaired it — so it exercises prevention on a repeat SDK install as well as healing an already-broken environment. It also confirms the point I raised earlier about the plain uv pip install vllm==<pin> not being a no-op: the recursive satisfaction check restored all three distributions, not just the one.

Two things from the run:

It is three packages, not one. torchvision and torchaudio are violated alongside torch, and every test and example in the PR shows only torch. The repair handles all three, but the violated-case note concatenates them into a single ~380-character warning line, which is unpleasant in a terminal. The violated branch of render_engine_dependency_check already prints one violation: line per finding — worth doing the same for the engine-side note rather than joining on ; . Reasonable to widen a test fixture to a multi-violation body too, since that is what the real failure looks like.

Not covered by this run: I did not start a server, so this establishes dependency consistency, not that the runtime serves.

Approving on that basis. My earlier point about the printed remedy — rocm engines install {engine} --reinstall routing through resolve_managed_runtime_python rather than the assessed interpreter — still stands and I would not want it lost, but it is advisory output on the path where the automatic repair has already failed, and the string predates this PR, so it does not seem worth holding a verified fix for. Happy to see it as a follow-up.

[volen-silo]

Pushed 62f6d73 addressing the three points from your review. The approval survives (dismiss_stale_reviews is off), so flagging what changed since you looked.

1. The printed remedy no longer routes to the wrong interpreter. Taken the first of your two options — threading the assessed environment through the forced-reinstall path — rather than only rescoping the printed string, since that fixes the behaviour instead of the advice about it. resolve_vllm_runtime now runs regardless of reinstall; only the short-circuit is gated on it. So --reinstall targets the environment that actually holds vLLM, and resolve_managed_runtime_python stays the fallback for the genuinely-not-installed case. Side effect worth naming: a forced reinstall now also assesses, so it can surface the violation notes explaining why the environment was inconsistent, where before it said nothing.

2. One note per violated pin. Your run is the fixture — the new test uses the real three-package body (torch, torchvision, torchaudio) instead of the single-pin one, asserts each finding gets its own note, and guards the line length so a future change cannot quietly re-join them.

3. Dropped the engine is still ready to serve. You were right that it could not fail. Rather than replace it, I removed it: with engines list reporting ready through a violated pin — and teaching it otherwise explicitly out of scope here — there is no falsifiable surface for that claim yet. The precondition helper keeps its Given use and now carries a comment recording why it has no Then counterpart, so nobody re-adds one.

Also took your point about the description underselling the coverage gap: it now names the family gating explicitly rather than leaving it as "installed by hand".

Not done, deliberately: the ROCM_CLI_UV_BINARY stub covering check_dependencies' three branches. It is a fair point and I would rather not bundle it into a verified fix — happy to take it as a follow-up unless you would rather it landed here.

cargo test green across the workspace, clippy and fmt clean. CI re-running: the two required checks that blocked this were CANCELLED rather than failed — both stalled in actions/checkout and hit the job cap, exactly as you diagnosed.