fix(deployer): resolve device injection from the target node, and pin ordering by device count

[thxCode]

Two fixes on the deployer's device-injection path, both found while deploying an exclusive (whole-card)
model onto a Kubernetes cluster running the GPUStack Operator: the workload bypassed the operator
entirely, so the card it occupied never appeared in the operator's accounting.

The KDP policy was decided from the wrong machine

get_resource_injection_policy() resolved auto by probing the local
/var/lib/kubelet/device-plugins/kubelet.sock. That reads correctly for the Docker and Podman
deployers, which deploy onto the host they run on. The Kubernetes deployer orchestrates remotely —
whether the deploying process can reach a kubelet socket says nothing about whether the target node
runs a device plugin.

Deployed from a plain Pod (a GPUStack worker), that socket is absent, so every accelerated workload fell
back to env injection. Observed on a live cluster, the Pod came out as:

resources: {}                      # no limits at all
securityContext: {privileged: true}
env:
  - NVIDIA_VISIBLE_DEVICES: GPU-bbc5…,GPU-1068…   # BOTH cards of the node
  - CUDA_VISIBLE_DEVICES: "1"

Privileged, seeing every device on the node, with the allocation absent from the device plugin's ledger —
so the operator's InstanceType status reported the card as free while a workload was using it.

auto now probes the target node's allocatable resources instead:

• only the suffixed families count (.shared / .sliced / .partitioned), mirroring the operator's own
  families in pkg/nodefeature. A bare CDI kind (nvidia.com/gpu) on its own is what a stock vendor
  plugin advertises, and requesting a device from that accounts for nothing;
• the probe goes through list_node with a field selector, so it needs no permission beyond the node
  list the deployer already requires to resolve a default node name;
• it resolves once per Pod, not once per container, and an explicit Env/KDP policy never probes at
  all;
• a probe that cannot answer — absent, failing, or unauthorized — falls back to KDP. Env injection
  hands the container every device of the host and leaves the allocation off the ledger, so it is the
  more damaging of the two guesses.

After the fix, the same workload comes out as nvidia.com/gpu.shared: "1" with the operator's
device.gpustack.ai/accelerator.allocated and …/accelerator.preferred-index annotations present, and
privileged dropped — _resolve_privileged already drops privilege for plugin-allocated devices, it
simply never saw the KDP policy before.

CUDA_DEVICE_ORDER was pinned by the wrong condition

The ordering pin fired on r_v == "all" or privileged. Requesting several specific devices produces a
container that numbers them itself exactly as one holding all of them does, and it went unpinned — the
case gpustack/gpustack#6041 is about. Conversely "all" on a single-device host has nothing to reorder
and got pinned anyway.

The condition is now privileged or <devices the request resolves to> > 1, with "all" measured
rather than special-cased. The Docker and Podman deployers carried the same condition, so the count moves
to Deployer.count_requested_devices and all three share it.

Tests

tests/gpustack_runtime/deployer/test_resource_injection_policy.py is new (14 cases: family detection,
explicit-policy precedence, probe failure fallback, and that an explicit policy spends no API call).
test_visible_devices_ordering.py gains the two cases the old condition got wrong, across all three
deployers. Suite: 491 passed, 20 skipped; make lint clean.

Verified on a live cluster

k3s + gpustack-operator v0.8.3, one RTX 4090 node. Exclusive request before/after as quoted above;
sliced requests (nvidia.com/gpu.sliced + percentage keys) unchanged, confirming the passthrough path
was not disturbed.

[gemini-code-assist]

Code Review

This pull request refactors device ordering pinning and resource injection policy resolution. It pins device ordering whenever a container sees more than one device or is privileged, rather than special-casing "all" devices. Additionally, under the "auto" policy, it now probes the target node's allocatable resources to detect the presence of a device plugin, resolving this policy once per Pod. Feedback suggests caching the resolved default node name in self._node_name during the node allocatable probe to avoid redundant API calls and ensure consistency across other deployer methods.

[Copilot]

Pull request overview

This PR fixes Kubernetes device injection behavior to avoid bypassing node device plugins (and thus operator accounting) by resolving the default resource-injection policy from the target node rather than the deploying process’s local kubelet socket. It also corrects when CUDA_DEVICE_ORDER is pinned by basing the decision on the resolved device count (or privilege) instead of special-casing "all".

Changes:

• Resolve Auto resource injection policy by probing target node allocatable resources for operator-style device-plugin families, with conservative fallback to KDP when probing can’t answer.
• Pin CUDA_DEVICE_ORDER=PCI_BUS_ID when a container will see >1 device (or is privileged), shared via Deployer.count_requested_devices.
• Add/extend tests for resource injection policy resolution, privilege dropping under KDP, and visible-device ordering across deployers.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/gpustack_runtime/deployer/test_visible_devices_ordering.py	Adds ordering-pin test cases for “several specific devices” and “all on single-device host”; adjusts monkeypatch for new policy resolver signature.
tests/gpustack_runtime/deployer/test_resource_injection_policy.py	New tests for node allocatable family detection and Auto/explicit policy behavior (including “explicit policy does not probe”).
tests/gpustack_runtime/deployer/test_privileged.py	Updates privileged-resolution test to pass the resolved KDP decision explicitly instead of monkeypatching.
gpustack_runtime/deployer/podman.py	Updates ordering pin condition to use privilege or resolved device count > 1.
gpustack_runtime/deployer/kuberentes.py	Probes target node allocatable for Auto policy, resolves KDP once per Pod, passes KDP decision into privilege resolution, and updates ordering pin condition.
gpustack_runtime/deployer/k8s/devicemanager/init.py	Adds target-node allocatable probing + family detection, and updates get_resource_injection_policy() API to accept a probe callback.
gpustack_runtime/deployer/docker.py	Updates ordering pin condition to use privilege or resolved device count > 1.
gpustack_runtime/deployer/types.py	Adds count_requested_devices() and updates ordering documentation to reflect “>1 device” semantics.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.