feat(diagnose): add the vLLM out-of-memory failure mode to the catalog (EAI-8060)

[r0x0r]

Summary

Adds the vLLM startup out-of-memory failure mode to the rocm-core diagnosis
catalog, with a matching print-only remediation recipe.

The error string covers two distinct faults, and the remediation splits on which
one it is:

• a tenancy collision — vLLM's fixed ~90% VRAM reservation runs into memory
  already in use on a shared or busy GPU; lowering the reservation is the fix, or
  steer onto a less-busy device with --gpu <index>.
• a model that genuinely does not fit — lowering the reservation only trades
  an earlier OOM for a later one; the fix is a smaller/quantized model or
  sharding across GPUs with --tensor-parallel-size <n>.

So the wording stays conditional: it never presents --gpu-memory-utilization
as the unconditional answer, and the verify step avoids the tenancy knob so it
does not unconditionally prescribe the tenancy workaround.

What changed

• crates/rocm-core/src/diagnose.rs — KEYWORDS_VLLM_OOM keyword table and
  check_16_vllm_oom. The match is keyword-only: an Examination carries no
  per-GPU VRAM or tenancy fields, so nothing structural can corroborate it — the
  user passes the error via --symptom, or arrives from the serve failure hint.
  The checker is gated to Linux and WSL2 (vLLM is Linux/WSL-only). On
  WSL2 this means a qualifying keyword match now produces a normal
  diagnosis instead of the platform's previous unconditional "out of
  scope" routing -- an exit-code/output contract change worth calling out
  explicitly here, not just in code comments. A required vLLM anchor (the
  word vllm, or one of its distinctive flags/logs) gates the whole
  keyword table, so a bare framework OOM string (e.g. torch.OutOfMemoryError: CUDA out of memory from an unrelated PyTorch job) is not misattributed
  to vLLM.
• crates/rocm-core/src/fix.rs — print-only fix-16-vllm-oom recipe
  (auto_applicable: false, no runner). Catalog count assertion 15 → 16 and the
  AUTO-set assertion strengthened to pin fix-16 as print-only.
• engines/vllm/src/lib.rs — the serve OOM hint points users at
  rocm diagnose --symptom '…'.
• tests/e2e-cucumber — a conditional-remediation scenario
  (@id:diagnose-vllm-oom-is-conditional, @requires-os:linux), its step
  definitions, and the catalog fix-id contract updated.

Dependencies / stacking

• Stacked on feat(vllm): Tackle out of memory errors (EAI-8058) #251 (gpu-out-of-memory, EAI-8058) — this PR targets that
  branch, not main, and stays in draft until feat(vllm): Tackle out of memory errors (EAI-8058) #251 merges upstream.
• Sibling sub-task: feat(vllm): enhance OOM diagnostics and guidance in serve summary (eai-8059) #284 (EAI-8059) adds the serve-summary OOM guidance that
  routes users here.

Test plan

• cargo test -p rocm-core --lib — passes (includes the checker tests: a
  high-confidence anchored OOM match, a bare "out of memory" staying below
  threshold, a bare framework OOM without a vLLM anchor not matching at all,
  Linux/WSL gating, and a WSL sub-threshold hit staying in matched instead
  of being routed out of scope).
• cargo test -p rocm-engine-vllm --lib — passes.
• cargo clippy --workspace --all-targets -- -D warnings — clean.
• e2e-cucumber crate compiles; the new @requires-os:linux scenario runs on the
  self-hosted Linux/GPU lane rather than locally.

[volen-silo]

A few observations from a read-through:

• KEYWORDS_VLLM_OOM needs nothing vLLM-specific to hit high confidence: torch.OutOfMemoryError: CUDA out of memory scores 45+45=90 under the top-2 rule, so any ROCm PyTorch job's OOM is reported as "vLLM ran the GPU out of memory at startup" with rocm serve-only remediation. The one vLLM-specific token, gpu_memory_utilization, is weighted lowest and never required — and all three new tests use vLLM-shaped text.
• The description says Linux-gated, but the checker registers for ["linux", "wsl"] and rewrites the diagnose() WSL branch so WSL2 is no longer unconditionally out-of-scope. That's an exit-code contract change worth stating in the body.
• In that branch, sub-threshold hits are dropped from matched, which the DiagnoseReport::matched doc says should never happen.
• feat(vllm): enhance OOM diagnostics and guidance in serve summary (eai-8059) #284 deletes the local log_tail_shows_oom this PR still calls, and edits the same oom_utilization_hint — conflict for whichever lands second.
• Body says this stays in draft until feat(vllm): Tackle out of memory errors (EAI-8058) #251 merges; it's currently open.

[r0x0r]

Thanks for the review — addressed both issues in bac9ce2 / rebased and pushed.

• Generic PyTorch OOM misattributed to vLLM: check_16_vllm_oom now requires an explicit vLLM anchor (vllm, gpu_memory_utilization/gpu-memory-utilization, or tensor_parallel/tensor-parallel) before KEYWORDS_VLLM_OOM is even scored. torch.OutOfMemoryError: CUDA out of memory on its own no longer matches at all — added generic_pytorch_oom_without_a_vllm_signal_does_not_match as a regression test for exactly that case. The vLLM engine's serve-failure hint and the matching e2e-cucumber step were updated to include the anchor ('vllm: torch.OutOfMemoryError: ...') so the self-referential rocm serve → rocm diagnose --symptom flow keeps working.
• Sub-threshold hits dropped in the WSL branch: fixed — diagnose() now keeps whatever run_all_checks returns on WSL (empty only when no wsl-applicable checker fired at all), instead of discarding nonzero sub-threshold matches in favor of the out-of-scope message. This restores the DiagnoseReport::matched contract ("every checker that fired at all lands here"). Added wsl_sub_threshold_vllm_signal_is_preserved_not_routed_out_of_scope to cover it.
• Linux/WSL exit-code contract change: confirmed and worth calling out explicitly — yes, check_16_vllm_oom being registered for ["linux", "wsl"] means WSL2 is no longer unconditionally out of scope; a keyword match on WSL now returns a normal diagnosis rather than always routing to the WSL "out of scope" message. This is intentional (vLLM does run under WSL2), but the PR body undersold it by saying "gated to Linux" — I'll tighten that wording.
• Conflict with feat(vllm): enhance OOM diagnostics and guidance in serve summary (eai-8059) #284 (log_tail_shows_oom / oom_utilization_hint): noted. feat(vllm): enhance OOM diagnostics and guidance in serve summary (eai-8059) #284 is still open/unmerged, so nothing to resolve yet, but since this PR just touched oom_utilization_hint's suggested --symptom text again, whichever of feat(vllm): enhance OOM diagnostics and guidance in serve summary (eai-8059) #284/feat(diagnose): add the vLLM out-of-memory failure mode to the catalog (EAI-8060) #290 lands second will need to rebase past the other's edit to that function.
• Draft status: this PR is out of draft already (not something I changed); base gpu-out-of-memory (feat(vllm): Tackle out of memory errors (EAI-8058) #251) has since merged its own follow-up fix (fix(vllm): validate --gpu against sysfs GPU count and harden DRM fallback), and this branch is now rebased on top of that tip.

All of cargo test -p rocm-core --lib, cargo test -p rocm-engine-vllm --lib, cargo clippy -p rocm-core -p rocm-engine-vllm --all-targets -- -D warnings, and cargo check -p e2e-cucumber --tests pass locally after the rebase.