playbooks: run nvidia-smi tasks outside the ssh cgroup so they survive the login GPU guard

[100-JM]

Problem

With slurm_login_on_compute: true, roles/slurm/tasks/login-compute-setup.yml sets DeviceAllow=/dev/nvidiactl on the ssh service so users cannot touch GPUs outside a Slurm job. Ansible connects through that same ssh service, so its own processes land in the restricted cgroup (/system.slice/ssh.service) and become does not change that — the device filter is per cgroup, not per user. Every task that runs nvidia-smi after the guard is applied therefore sees no GPUs:

$ nvidia-smi; echo rc=$?                      # in an ssh session on the node
No devices were found
rc=6

This breaks the second Ansible run that docs/slurm-cluster/slurm-single-node.md asks for ("reboot manually when prompted and then run Ansible again"):

task	effect once the guard is active
playbooks/utilities/gpu-clocks.yml — nvidia-smi -acp UNRESTRICTED	rc=6 → play fails (when allow_user_set_gpu_clocks: yes)
playbooks/nvidia-software/nvidia-cuda.yml — test nvidia-smi	rc=6 → play fails (default slurm_cluster_install_cuda: yes)
playbooks/nvidia-software/nvidia-mig.yml — … | grep -v 'N/A'	rc=0 with No devices were found → treated as a MIG-capable node
roles/nvidia-mig-manager/tasks/main.yml — same probe, same filter	same misdetection inside the role
playbooks/utilities/nvidia-set-gpu-clocks.yml — nvidia-smi -lgc/-rgc	rc=6 → play fails

scripts/validation/validate_slurm.py already documents the phenomenon for the validator and uses srun as the authoritative test; the playbook tasks above were still running plainly over ssh.

Observed on a DGX B300, DGX OS 7.5.0, driver 580.126.20, DeepOps master.

Fix

Run every one of those nvidia-smi invocations through systemd-run --wait --pipe --collect --quiet. The command executes in a transient unit under system.slice, outside the ssh cgroup, so the check still runs for real instead of being skipped. Both MIG probes (playbook and role) now accept only Enabled/Disabled and get an explicit become: true (the rest of that play/role already requires root). Behaviour on nodes without the guard is unchanged.

nvidia-driver.yml carries the same test nvidia-smi; it is deliberately not touched here to avoid conflicting with #1396, and can take the same wrapper once that lands.

Side note: nvidia-smi -acp is deprecated in current drivers ("This option is deprecated … Treating as warning and moving on") and has no effect on Blackwell; a comment records that, the task is kept as-is otherwise.

Verification

On the affected node with the guard active, same ssh session:

$ nvidia-smi -L; echo rc=$?
No devices were found
rc=6
$ sudo systemd-run --wait --pipe --collect --quiet nvidia-smi -L; echo rc=$?
GPU 0: NVIDIA B300 SXM6 AC (UUID: GPU-364baba8-…)
…
GPU 7: NVIDIA B300 SXM6 AC (UUID: GPU-95bf3df7-…)
rc=0

ansible-playbook --syntax-check passes for the four playbooks; ansible-lint 26.1.1 with the project config reports 0 failures. systemd-run --wait/--pipe/--collect need systemd ≥ 236, which every supported Ubuntu/RHEL release ships.

[dholt]

The transient-service approach addresses the SSH device restriction, but the complete MIG path still needs attention:

• nvidia-mig-parted apply and assert still run directly in the restricted SSH context. Once the repaired probe succeeds, those GPU operations can still fail. Apply the same privileged isolation to them.
• Both probes suppress command failure and use it to skip configuration. Distinguish a successful unsupported-device result from a failed GPU query or transient service; an operational failure must not look like successful configuration.
• #1396 is now closed without merging, so its driver-check correction will not land there. Please include the remaining driver-check dependency here.

Please cover failed queries, unsupported devices, apply/assert failures and a repeated run with the SSH restriction already active. Keep the ordinary login restriction intact.

[100-JM]

@dholt thanks — all three points are addressed in 32b04b74.

MIG administration runs in the transient unit too. nvidia-mig-parted apply and assert drive the same GPUs as the probe, so they now run the same way, and the play itself runs privileged (which the standalone nvidia-mig-parted invocations needed regardless). Neither task suppresses its status, so a layout that could not be applied does not pass as applied.

A failed query no longer looks like a missing capability. With the query piped straight into the match, pipefail reports the match's status either way, so N/A (a successful answer) and "the query never ran" were indistinguishable — and the second was silently read as "no MIG here". The query is now kept apart from the match so its own status survives:

rc	meaning	action
0	a GPU reports a MIG mode	configure the node
1	queried fine, no GPU supports MIG	skip the node
2	the query itself failed	fail, with the reason

Deliberately stream-independent: nvidia-smi reports a missing device on stdout and a broken driver on stderr, so keying on either one would have been fragile. rc == 2 is only fatal where ansible_local['gpus']['count'] (lspci) actually found NVIDIA GPUs, so a node with no NVIDIA hardware still passes through untouched and the ordinary login restriction is left intact.

Two smaller things from the same reading: the match now accepts only a real MIG mode, since grep -v 'N/A' also matched warnings nvidia-smi prints next to the values; and the Red Hat install branch requires a successful capability probe, which the Debian branch already did.

The driver check from #1396. nvidia-driver.yml's test nvidia-smi now runs in the transient unit as well. I left nvidia_driver_test_enabled out — it only exists to let slurm-cluster.yml turn the check off for the DCGM exporter, which is a separate change with no home in this PR; say the word and I will add it here.

Coverage — scripts/validation/tests/test_gpu_task_contracts.py (27 tests, python3 -m unittest discover scripts/validation/tests green). It pulls the probe's shell out of the YAML and executes it as written against a mocked nvidia-smi and systemd-run, so it tests behaviour rather than the text of the task:

• failed query — device missing (stdout, rc 6), driver not answering (stderr, rc 9), binary absent (rc 127) → all rc 2
• unsupported device — N/A → rc 1
• successful but noisy query — N/A + a warning line → rc 1, which grep -v 'N/A' got wrong
• MIG capable → rc 0
• SSH restriction already active — the mock systemd-run consumes its options up to -- and clears the guard, so the test fails if the -- separator or the wrapper is dropped; asserted for a first run and a repeated run
• static contracts — every GPU command wrapped, apply/assert unsuppressed, install branches gated

I mutation-checked it: reverting to grep -v 'N/A', dropping the apply wrapper, dropping the Red Hat gate, collapsing the query back into one pipe, and deleting the guard task each fail the suite.

ansible-lint 26.1.1 / ansible 10.7.0 pass on the roles and the touched playbooks (profile min, and production also passes), and all five playbooks pass --syntax-check.

Noted on the merge commit rather than squash — the branch keeps its three original commits.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UiFVTVAyEMmn68Xvo9LSTf

[dholt]

The revision addresses the remaining driver/MIG review points: the administrative operations use the isolated service path and operational query failures no longer silently skip configuration. The 27 validation tests pass in a merge preview with current master, and public CI is green.

Approved. The original contributor commits will be preserved with a merge commit.