feat(detector): align the detect path with the operator, split information from usage, and report workload exit status

[thxCode]

Aligns gpustack_runtime/detector with gpustack-operator's pkg/devicemanager/detector, splits the
detect path into an information query and a usage query, retires the physical-index switch, and makes
workloads report why they stopped.

Spec: specs/2026-08-14-detector-alignment-and-workload-exit-status.md (Status: Built).

Closes the runtime side of:

• Set CUDA_DEVICE_ORDER=PCI_BUS_ID by default for NVIDIA enviornments gpustack#6041 — CUDA_DEVICE_ORDER vs. the physical-index design.
• K8s cluster: no frontend error/notice when the inference backend image fails to pull gpustack#5869 — a K8s image-pull failure surfacing nothing to the user.
• Display the exited code when the model instance status changes to error gpustack#4217 — showing the exit code when an instance goes to error, on both Docker
  and Kubernetes.

What changed

Detect-path parity (nine vendors). Every vendor's binding calls were audited against the operator's
and the gaps closed. The defects that cost devices or reported them wrong:

• MetaX skipped physical functions with an inverted comparison, so a virtualization-enabled host
  reported zero devices.
• MThreads read a nonexistent mpcCap attribute, which raised and failed the whole detect pass.
• Ascend was undetectable on older drivers (four *_v2/*_v3 calls with no V1 fallback) and did not
  filter by unit type.
• Iluvatar had no v2→v1 memory fallback.
• AMD / Hygon resolved the device name from the driver only; they now follow the operator's full
  precedence — pci.ids → HSA ProductName → libdrm amdgpu_get_marketing_name → ASIC / ROCm-SMI
  name. pyamdgpu gained that libdrm binding in this PR.
• Cambricon was not a binding at all — a cnmon shell-out with a TODO where the sample output
  should be. It is now a hand-written pycndev ctypes binding derived from the operator's cndev.h,
  with its struct layouts pinned against an independently compiled probe.

Information / usage split. detect_info() and detect_usage() are both abstract, detect(usage=True)
composes them, and detect_devices(usage=False) skips every metric call. Usage joins by UUID, never by
index. get_topology no longer pays for metrics it never reads.

Whole-card reporting. The vGPU/virtual-card classification is gone for all nine vendors — the filters
dropped physical cards on MetaX and crashed the pass on MThreads. MIG-style instances stay in
appendix["mig_devices"], and no sample may resurrect the retired appendix keys.

Device numbering. Device.index is the detector's enumeration index; driver-physical numbering
(minor_number, card_id/physical_id) moved to the appendix, and the PHYSICAL_INDEX_PRIORITY switch
is retired. CUDA_DEVICE_ORDER=PCI_BUS_ID is injected by the deployer for NVIDIA workloads given all
devices or run privileged, never overriding a value the container already declares.

Workload exit status. WorkloadStatus.exits is populated by all three deployers, and a Kubernetes
workload whose image cannot be pulled reports Failed instead of sitting at Pending, with the Pod
Event appended. One new RBAC rule grants get/list on core events — downstream charts need the
same rule.

One deliberate divergence worth calling out. NVIDIA memory is what the driver reports, i.e. what the
card can allocate. The operator adds back the ~1/16 that ECC parity carves out of a GDDR part, but that
figure is display-only there and takes no part in allocation, while this one does: restoring it would make
memory - memory_used over-report free space and over-commit the card.

Review

Three axes ran over the whole diff — spec, standards, and two external cross-checks — plus this PR's own
review. The findings worth naming:

• Ascend could mount the wrong NPU. /dev/davinciN is numbered by the physical id; the path was
  built from the logic id when the physical id was unreadable. A device that cannot be addressed is now
  dropped, as the operator's ascend/device.go does. Pre-existing on the default path, not a regression
  of this branch.
• T-Head could address its neighbour. The node is named by the card ordinal while the recorded minor
  is the card's identity; the generator now compares the two and refuses the device when they disagree,
  which is the proof the operator's allocator performs.
• An ambiguous UUID broadcast one card's metrics onto every card. The merge now drops such an id, and
  cndevGetUUID no longer manufactures a bare "MLU-" that is one.
• A usage-query failure discarded an inventory already in hand, so a host with eight healthy GPUs
  reported none.
• A container that merely exited non-zero carried no reason, so nothing reached state_message —
  while Kubernetes reported Error for the same event. Docker's and Podman's exit parsers were 58 of 60
  lines identical and are now one function.
• Two tests could not fail. The usage-join test gave every card identical values, so a positional
  join, a reversed mapping and a broadcast-to-all all satisfied it; the memory-status test ran with health
  checks off, where both queries answer healthy without a driver call.

Verification

471 passed, 20 skipped, make lint clean, make docs builds.

Hardware validation ran on five environments — 2 × AMD RX 7800 XT, 1 × RTX 5090 D, 2 × RTX 4090 48G,
8 × Ascend 910B2, 16 × T-Head PPU-ZW810E — plus Hygon by proxy, every pass exiting 0. What no fixture
could establish:

1. T-Head's minor number is not its enumeration index (they run one apart on that host, as they do on
   the operator's). The node is named by the ordinal, so anything reading the minor instead addresses the
   neighbouring card.
2. Ascend reports physical_id for all 8 NPUs, so the invariant the device-node fix relies on holds.
3. AMD's per-card UUIDs are distinct, so the constant-id collision the merge now guards against is not
   happening in practice.
4. libdrm answers AMD Radeon RX 7800 XT for both cards, which is the new marketing-name step.

Failure reporting was exercised end to end rather than only in fixtures:

Scenario	Backend	Result
Unpullable image	k3s	Failed, ImagePullBackOff: ...; Error: ErrImagePull — waiting reason plus the Pod Event
Exit non-zero	Docker 29.6.1	exit_code=7, reason=Error, state_message=Error
Out of memory	Docker 29.6.1	exit_code=137, reason=OOMKilled, state_message=OOMKilled
Killed (SIGKILL)	Docker 29.6.1	exit_code=137, reason=Error, state_message=Error

Samples are refreshed from those hosts. Iluvatar, MetaX, MThreads and Cambricon have no environment, and
no MIG-enabled host was available, so those samples stay hand-aligned — stated in the samples README
rather than guessed at.

[Copilot]

Pull request overview

Aligns gpustack_runtime.detector behavior with the gpustack-operator detector path, separating inventory (“info”) from metrics (“usage”), removing virtual/physical-index switching, and improving workload termination diagnostics across deployers (Docker/Podman/Kubernetes).

Changes:

• Split device detection into detect_info() (inventory) and detect_usage() (metrics), with UUID-based merging and MIG instances kept under appendix["mig_devices"].
• Retire vGPU/virtual-card classification and physical-index priority; move driver-physical numbering into appendix (e.g., minor_number, physical_id).
• Add workload exit reporting (WorkloadStatus.exits), inject CUDA_DEVICE_ORDER=PCI_BUS_ID when containers can see all NVIDIA devices, and improve Kubernetes failure reporting by reading Pod Events (plus RBAC).

Reviewed changes

Copilot reviewed 67 out of 67 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/gpustack_runtime/detector/test_samples.py	Guards sample JSON against Device schema and retired appendix keys.
tests/gpustack_runtime/detector/test_mthreads.py	Adds fake driver + tests for MThreads info/usage split and UUID merge.
tests/gpustack_runtime/detector/test_mig_devices.py	Updates MIG indexing expectations and removes vGPU appendix usage.
tests/gpustack_runtime/detector/test_metax.py	Adds fake driver + tests for MetaX info/usage split and UUID merge.
tests/gpustack_runtime/detector/test_iluvatar.py	Adds fake binding tests for Iluvatar v2→v1 memory fallback + CDI minor-number addressing.
tests/gpustack_runtime/detector/test_detector_types.py	Adds tests for Detector ABC split + merge_devices_usage semantics and PCI name lookup.
tests/gpustack_runtime/detector/test_detector_cli.py	Adds CLI tests for --no-usage and JSON/table rendering rules.
tests/gpustack_runtime/detector/samples/topology_output_thead_ppu.json	Refreshes T-Head topology sample output.
tests/gpustack_runtime/detector/samples/topology_output_nvidia_rtx4090_48g.json	Adds NVIDIA RTX 4090 (48G) topology sample.
tests/gpustack_runtime/detector/samples/topology_output_amd_rx7800xt.json	Updates AMD RX7800XT topology sample for multi-device output.
tests/gpustack_runtime/detector/samples/README.md	Documents which samples are captured vs hand-aligned and device-node-relevant facts.
tests/gpustack_runtime/detector/samples/detect_output_nvidia_rtx5090d.json	Refreshes RTX 5090D detect output; replaces vgpu with mig and adds minor.
tests/gpustack_runtime/detector/samples/detect_output_nvidia_rtx4090d.json	Replaces vgpu appendix key with mig.
tests/gpustack_runtime/detector/samples/detect_output_nvidia_rtx4090_48g.json	Adds RTX 4090 (48G) detect output sample.
tests/gpustack_runtime/detector/samples/detect_output_nvidia_rtx4080super.json	Replaces vgpu appendix key with mig.
tests/gpustack_runtime/detector/samples/detect_output_nvidia_h200.json	Replaces vgpu appendix key with mig.
tests/gpustack_runtime/detector/samples/detect_output_nvidia_h100.json	Replaces vgpu appendix key with mig.
tests/gpustack_runtime/detector/samples/detect_output_nvidia_h100_mig.json	Converts MIG instances into appendix["mig_devices"] under their parent cards.
tests/gpustack_runtime/detector/samples/detect_output_nvidia_gb10.json	Replaces vgpu appendix key with mig.
tests/gpustack_runtime/detector/samples/detect_output_metax_c500.json	Removes vgpu appendix key from MetaX sample.
tests/gpustack_runtime/detector/samples/detect_output_hygon_k100ai.json	Removes vgpu appendix key from Hygon sample.
tests/gpustack_runtime/detector/samples/detect_output_ascend_910b2.json	Refreshes Ascend sample and adds physical_id appendix key.
tests/gpustack_runtime/detector/samples/detect_output_ascend_310p3.json	Removes vgpu appendix key from Ascend 310P3 sample.
tests/gpustack_runtime/detector/samples/detect_output_amd_rx7800xt.json	Refreshes AMD RX7800XT sample; updates names/versions and adds second device.
tests/gpustack_runtime/detector/samples/detect_output_amd_mi308x.json	Removes vgpu appendix key from AMD MI308X sample.
tests/gpustack_runtime/detector/samples/detect_output_amd_mi300x.json	Removes vgpu appendix key from AMD MI300X sample.
tests/gpustack_runtime/deployer/test_workload_status.py	Adds serialization/back-compat tests for WorkloadStatus.exits.
tests/gpustack_runtime/deployer/test_visible_devices_ordering.py	Adds tests for CUDA device ordering injection behavior across deployers.
tests/gpustack_runtime/deployer/test_podman_status.py	Adds Podman exit parsing/state_message tests using shared exit parser behavior.
tests/gpustack_runtime/deployer/test_docker_status.py	Adds Docker exit parsing/state_message tests including timestamp normalization.
ruff.toml	Adds pycndev binding directory to Ruff ignore list.
mkdocs.yml	Adds Detector Parity doc to MkDocs navigation.
gpustack_runtime/envs.py	Removes GPUSTACK_RUNTIME_DETECT_PHYSICAL_INDEX_PRIORITY.
gpustack_runtime/detector/pydcmi/init.py	Adds Ascend DCMI v1/v2 struct/call fallbacks used for older drivers.
gpustack_runtime/detector/mthreads.py	Implements detect_info/detect_usage split and removes virtRole/vGPU filtering.
gpustack_runtime/detector/metax.py	Implements detect_info/detect_usage split and removes virtualization-mode filtering.
gpustack_runtime/detector/iluvatar.py	Implements split, adds v2→v1 memory fallback, moves minor number into appendix.
gpustack_runtime/detector/hygon.py	Implements split; prefers PCI-ID-derived marketing name over driver chip name.
gpustack_runtime/detector/cambricon.py	Replaces cnmon shell-out with pycndev binding; implements split + Neuware version parsing.
gpustack_runtime/detector/amd.py	Implements split; adds PCI ID name lookup and factors UUID derivation helper.
gpustack_runtime/detector/utils.py	Adds pci.ids parsing + get_pci_device_name; removes physfn-based vGPU classification helper.
gpustack_runtime/detector/types.py	Makes info/usage abstract, adds merge_devices_usage, updates Device.index semantics.
gpustack_runtime/detector/init.py	Threads usage= through detect_devices; avoids usage query for topology.
gpustack_runtime/deployer/podman.py	Adds exit collection + CUDA device ordering injection for all-device/privileged requests.
gpustack_runtime/deployer/kuberentes.py	Adds exit collection, image-pull failure → Failed, and Pod Event message enrichment + ordering injection.
gpustack_runtime/deployer/docker.py	Adds exit collection + CUDA device ordering injection for all-device/privileged requests.
gpustack_runtime/deployer/cdi/thead.py	Clarifies T-Head device node naming is by enumeration index, not minor number.
gpustack_runtime/deployer/cdi/iluvatar.py	Uses appendix["minor_number"] for device node path, falling back to Device.index.
gpustack_runtime/deployer/cdi/ascend.py	Uses appendix["physical_id"] for /dev/davinciN, skipping devices without it.
gpustack_runtime/deployer/types.py	Adds WorkloadStatusExit + shared Docker/Podman exit parser + ordering mapping.
gpustack_runtime/cmds/detector.py	Adds --no-usage and omits/unrenders usage fields when metrics weren’t queried.
deploy/manifests/kubernetes.yaml	Adds RBAC get/list permission for core events to support Pod Event diagnostics.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[gemini-code-assist]

Code Review

This pull request aligns the runtime detector with the operator's device manager and introduces structured workload exit status reporting. Key changes include splitting the device detection process into inventory (detect_info) and metrics (detect_usage) queries, allowing callers to opt out of metrics queries via a new --no-usage CLI flag. It refactors all nine vendor detectors to implement this split, resolving various parity gaps (such as NVIDIA GDDR ECC capacity restore, a new pycndev binding for Cambricon, and pci.ids name resolution for AMD/Hygon). Additionally, the deployers (Docker, Podman, and Kubernetes) are updated to report structured container exit information (WorkloadStatusExit), handle Kubernetes image-pull failures as failed states enriched with Pod Event messages, and automatically pin CUDA_DEVICE_ORDER=PCI_BUS_ID for NVIDIA workloads seeing all host devices. Comprehensive unit tests and documentation have been added to support these changes. I have no feedback to provide as there are no review comments.