feat(vllm): Tackle out of memory errors (EAI-8058)

[r0x0r]

This pull request improves GPU selection and user guidance for ROCm and vLLM, especially in shared or containerized environments where the standard amd-smi tool may not be available. It adds a fallback for GPU VRAM telemetry, enhances user warnings and hints for out-of-memory (OOM) conditions, and ensures consistent messaging across CLI and engine surfaces. The changes also include comprehensive tests for the new logic.

GPU selection and VRAM telemetry improvements:

• Added a fallback to read per-GPU VRAM usage from the amdgpu DRM sysfs counters (/sys/class/drm/card*/device/mem_info_vram_{total,used}) when amd-smi is not available, so --gpu auto and low-VRAM warnings work reliably in stripped-down containers and shared nodes. The device count for selection now also derives from these sysfs rows if amd-smi is missing. [1] [2] [3] [4]
• Added tests to verify that auto-selection uses the sysfs fallback and that VRAM usage is parsed and assigned ordinals correctly. [1] [2]

User guidance and warnings for vLLM:

• Introduced a shared constant VLLM_GPU_MEMORY_UTILIZATION_HINT for the recommended workaround when running out of memory on a shared/busy GPU, ensuring CLI and engine logs use consistent wording.
• The serve summary now prints a note about the --gpu-memory-utilization workaround when vLLM is selected and the GPU is busy, both interactively and in the deployment summary. [1] [2] [3] [4] [5]
• vLLM engine startup logs now append the same utilization hint if an OOM is detected, so users receive actionable advice post-failure.
• Added tests to ensure the low-VRAM warning and vLLM utilization hint are paired correctly and only shown for vLLM engines. [1] [2]

Documentation:

• Updated docs/vllm.md to explain the behavior on shared/busy GPUs, the fallback telemetry, and the recommended OOM workaround.

These improvements make GPU selection more robust in diverse environments and provide clear, actionable guidance to users encountering memory issues with vLLM.

[volen-silo]

A few observations from a read of this change:

• No Gherkin scenario covers the new user-visible behavior (the extra serve note, --gpu auto picking from the sysfs fallback); AGENTS.md §3 asks for one or a stated reason. The unit tests cover helpers rather than behavior.
• validate_pinned_gpu_index still sees only the amd-smi count, so on the exact environment this targets (no amd-smi, sysfs works) an out-of-range --gpu <n> is accepted and fails later in the engine.
• read_sysfs_u64(...mem_info_vram_used).unwrap_or(0) makes a card with an unreadable used counter look 100% free — which is what --gpu auto prefers first.
• read_drm_vram_usage orders by ascending card<N>; an AMD APU passes the same vendor + mem_info_vram_total filter, so on APU + dGPU the ordinal can diverge from HIP's, and it feeds HIP_VISIBLE_DEVICES directly.
• docs/vllm.md: the fallback probes amd-smi only, never rocm-smi.

On the stack: #284's diff against this branch removes the sysfs fallback, its tests and the docs bullet — looks unintended.

[r0x0r]

Thanks for the detailed review — pushed a fix commit (a97a39f) addressing the concrete bugs:

• validate_pinned_gpu_index only saw the amd-smi count: resolve_gpu_indices now derives an effective_gpu_count that falls back to the DRM sysfs row count when amd-smi is unavailable, so an out-of-range --gpu <n> is rejected up front on exactly the environment this PR targets (no amd-smi, sysfs works), instead of failing later inside the engine.
• read_sysfs_u64(...mem_info_vram_used).unwrap_or(0): a card whose used counter can't be read is now skipped entirely rather than treated as 0 bytes used (100% free).
• read_drm_vram_usage ordinal divergence on APU + dGPU: ascending card<N> order only mirrors HIP's KFD-topology ordinal when there's exactly one AMD DRM card. The function now returns no rows at all when more than one AMD card is found, instead of guessing an ordinal that could feed the wrong device into HIP_VISIBLE_DEVICES.
• docs/vllm.md: fixed — the fallback only ever probes amd-smi, never rocm-smi; wording corrected.

Added unit tests for all four (effective_gpu_count_*, resolve_gpu_indices_rejects_out_of_range_index_from_sysfs_fallback_count, read_drm_vram_usage_skips_a_card_with_an_unreadable_used_counter, read_drm_vram_usage_withholds_telemetry_when_multiple_amd_cards_are_present).

On the Gherkin scenario: I didn't add one for the "no amd-smi, sysfs fallback" path — there's no runner in the current fleet shaped like that (real GPU hardware with amd-smi absent), so I couldn't author or verify a @requires-gpu scenario for it. The corrected behavior is covered at the unit level instead (pure functions, planted sysfs fixtures). Happy to add e2e coverage if/when a suitable lane exists.

On #284: worth double-checking before merging that stack — its diff against this branch appears to drop the sysfs fallback, its tests, and the docs bullet, which does look unintended given this PR is what introduces them.