feat(vllm): enhance OOM diagnostics and guidance in serve summary (eai-8059)

[r0x0r]

This pull request enhances GPU out-of-memory (OOM) diagnostics and user guidance for vLLM model serving, especially on shared or busy GPUs. It introduces actionable hints in both pre-launch warnings and post-failure summaries, ensuring users are clearly informed about how to adjust memory settings to avoid OOM errors. The changes also unify the messaging across CLI, engine logs, and documentation, and add comprehensive tests for these new behaviors.

Improvements to OOM diagnostics and user guidance:

• Added a clear, actionable hint (VLLM_GPU_MEMORY_UTILIZATION_HINT) that recommends adjusting --gpu-memory-utilization and/or targeting a less-busy GPU with --gpu <index>, shown both before launch (when low VRAM is detected) and after OOM failures in vLLM serves. [1] [2] [3] [4]
• Enhanced the logic to detect OOM errors in vLLM/PyTorch logs, including case-insensitive and signature-based matching, and surfaced the relevant memory knob hints only when applicable. [1] [2] [3] [4]

Integration and codebase updates:

• Updated collect_serve_notes and related serve summary logic to include vLLM-specific memory guidance, both pre-launch and post-failure, and ensured these notes are only shown when relevant. [1] [2] [3] [4]
• Ensured managed serve records retain log and manifest paths for accurate diagnostics after serve failures.

Documentation and testing:

• Expanded documentation to explain vLLM's memory reservation behavior and the recommended workarounds for OOM on shared/busy GPUs.
• Added extensive unit and end-to-end tests to verify OOM detection, note rendering, and user guidance in both CLI and engine logs. [1] [2] [3] [4] [5] [6]

These changes collectively improve the user experience by making OOM errors easier to diagnose and resolve, especially in shared GPU environments.<!--
Copyright © Advanced Micro Devices, Inc., or its affiliates.

SPDX-License-Identifier: MIT
-->

• If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows.

[volen-silo]

A few observations, mostly around the signature list and the stacking:

• vllm_log_shows_oom treats engine core initialization failed as an OOM signature, but vLLM emits that as the terminal wrapper for any EngineCore startup crash (unsupported arch, shm size, TP misconfig, missing weights). Being the last line, it reliably lands in the 80-line tail — so unrelated failures get reported as "the serve attempt ran out of GPU memory".
• The e2e scenario plants exactly that string as its OOM log, and a unit test asserts it must produce a hint — so the only end-to-end coverage never exercises a real OOM message.
• This branch removes the DRM sysfs VRAM fallback, the --gpu auto count derivation, their tests and the docs bullet — all added by its base feat(vllm): Tackle out of memory errors (EAI-8058) #251, which is still open. Intentional? auto_select_gpu_indices' doc still refers to that fallback.
• collect_serve_notes and append_oom_serve_note each emit the full hint verbatim, so a low-VRAM serve that then OOMs prints it twice.
• Setting log_path/manifest_path on the already-running branch makes the OOM note reachable when no launch happened — re-running serve against a starting service attributes that process's OOM to the new invocation.
• feat(diagnose): add the vLLM out-of-memory failure mode to the catalog (EAI-8060) #290 edits the same oom_utilization_hint and still calls log_tail_shows_oom, which this PR deletes.

[r0x0r]

Thanks for the detailed review — addressed all of these in b9316f3 (rebased onto the current gpu-out-of-memory tip):

• Generic "engine core initialization failed" signature: removed it from vllm_log_shows_oom. It really is vLLM's terminal wrapper for any EngineCore startup crash, not just OOM, so treating it as an OOM signature was wrong. Detection now only fires on the allocator-level signatures (torch.OutOfMemoryError, hip out of memory, out of memory).
• E2E coverage only exercising the wrapper string: the e2e scenario now plants a real allocator OOM signature (torch.OutOfMemoryError: HIP out of memory. Tried to allocate 7.21 GiB.) instead of the generic wrapper phrase, and unit tests were updated the same way.
• DRM sysfs fallback / --gpu auto count derivation removed: that was purely a stale-base artifact — this branch was still on top of an older point in gpu-out-of-memory than your feat(vllm): Tackle out of memory errors (EAI-8058) #251 fix commit. Rebased onto the current tip (a97a39f) and restored the gpu_vram_usage/gpu_vram_usage_amd_smi/gpu_vram_usage_sysfs split that got flattened by the rebase's merge resolution — verified with cargo test/clippy that the dispatcher and DRM fallback tests are back and passing.
• Duplicate hint when a low-VRAM serve then OOMs: append_oom_serve_note now skips adding the note if the shared VLLM_GPU_MEMORY_UTILIZATION_HINT text is already present in the notes collected pre-launch, so it's never printed twice.
• log_path/manifest_path on the already-running branch misattributing OOM: append_oom_serve_note now also takes already_running and bails out immediately when true, so re-running serve against a live/starting service never blames that invocation for whatever the other process's log contains. Added a regression test for this (append_oom_serve_note_ignores_an_already_running_services_log) and repointed the e2e scenario at this exact case (serve-oom-memory-guidance), asserting the summary does not show the OOM note when reusing an already-running service.
• feat(diagnose): add the vLLM out-of-memory failure mode to the catalog (EAI-8060) #290 touching the same oom_utilization_hint/log_tail_shows_oom: noted for awareness — nothing in this repo currently defines log_tail_shows_oom (this PR's function is vllm_log_shows_oom), so there's no conflict today, but that PR will need to reconcile with whichever version of this lands first.

All changes covered by unit tests; cargo test -p rocm --bin rocm, cargo clippy -p rocm -p rocm-core -p rocm-engine-vllm --all-targets -- -D warnings, and cargo fmt --check all pass locally.