Skip to content

test(bindings): work around Orin NVML field-values bug - #3148

Open
rwgk wants to merge 2 commits into
NVIDIA:mainfrom
rwgk:work_around_orin_nvml_field_values_bug
Open

rwgk wants to merge 2 commits into
NVIDIA:mainfrom
rwgk:work_around_orin_nvml_field_values_bug

Conversation

@rwgk

@rwgk rwgk commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

Seed the field status with an invalid sentinel so a successful native no-op cannot accidentally pass on recycled memory. Xfail when Orin leaves the sentinel unchanged, preserving normal checks on fixed drivers and other devices.

Label the workaround explicitly in the source comment and xfail reason.

Interactive testing on Jetson Orin:

jetson:~/wrk/forked/work_around_orin_nvml_field_values_bug $ grep_pytest_summary `nlog`
/home/rgrossekunst/wrk/logs/cuda-python_qa_bindings_linux_2026-10-18+234118+0000_tests_log.txt
rootdir: /home/rgrossekunst/wrk/forked/work_around_orin_nvml_field_values_bug/cuda_pathfinder
======================= 1618 passed, 2 skipped in 13.78s =======================
rootdir: /home/rgrossekunst/wrk/forked/work_around_orin_nvml_field_values_bug/cuda_bindings
====== 633 passed, 80 skipped, 1 xfailed, 17 warnings in 70.06s (0:01:10) ======
rootdir: /home/rgrossekunst/wrk/forked/work_around_orin_nvml_field_values_bug/cuda_bindings
====== 633 passed, 80 skipped, 1 xfailed, 17 warnings in 66.12s (0:01:06) ======
rootdir: /home/rgrossekunst/wrk/forked/work_around_orin_nvml_field_values_bug/cuda_bindings
============================== 9 passed in 1.01s ===============================
rootdir: /home/rgrossekunst/wrk/forked/work_around_orin_nvml_field_values_bug/cuda_core
= 1 failed, 3741 passed, 535 skipped, 3 xfailed, 3 warnings, 1 error, 9 subtests passed in 127.94s (0:02:07) =
rootdir: /home/rgrossekunst/wrk/forked/work_around_orin_nvml_field_values_bug/cuda_core
============================== 1 passed in 0.38s ===============================

xrefs:

Seed the field status with an invalid sentinel so a successful native no-op
cannot accidentally pass on recycled memory. Xfail when Orin leaves the
sentinel unchanged, preserving normal checks on fixed drivers and other
devices.

Label the workaround explicitly in the source comment and xfail reason.
@copy-pr-bot

copy-pr-bot Bot commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the cuda.bindings Everything related to the cuda.bindings module label Oct 18, 2026
@rwgk rwgk added the bug Something isn't working label Oct 18, 2026
@rwgk rwgk self-assigned this Oct 18, 2026
@rwgk rwgk added this to the cuda.bindings next milestone Oct 18, 2026
@rwgk
rwgk marked this pull request as ready for review October 19, 2026 03:45
@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

📝 Summary

Summary by CodeRabbit

  • Tests
    • Updated NVLink checks to account for expected device-specific behavior on Orin systems. Successful queries and responses indicating that NVLink is unsupported continue to be validated as before. No user-facing functionality changed.

The NVLink test initializes the queried field's scope and return-value sentinel. When the sentinel remains and the device name contains "Orin", the test reports an expected failure. Other results continue to the existing assertion.

Changes: NVLink Test Handling

Layer / File(s) Summary
NVLink query result handling
cuda_bindings/tests/nvml/test_nvlink.py
The test initializes the field scope and return-value sentinel. It marks the specified Orin result as an expected failure and keeps the existing assertion for other results.

Suggested reviewers: mdboom

Priority: ⬇️ Low

Change: Bug fix

Merge Risk: 🔵 Low · up to 41265

The NVLink test may skip checks for other devices when it encounters an Orin. Deferring the expected failure would preserve those checks; the issue is limited to test coverage, so merge risk is low.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/cuda-python/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 85c60964-6f71-40be-90e0-cf913a2bf0c1
📥 Commits

Reviewing files that changed from the base of the PR and between 576ef68 and 412659b.

📒 Files selected for processing: 1
  • cuda_bindings/tests/nvml/test_nvlink.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

fields[0].nvml_return = -1
value = nvml.device_get_field_values(device, fields)[0]
if value.nvml_return == -1 and "Orin" in nvml.device_get_name(device):
pytest.xfail(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

set -eu
printf '%s\n' '--- reviewed head test ---'
git show 412659b856b8c1d7086bdef9351f66e3b4cecabd:cuda_bindings/tests/nvml/test_nvlink.py | nl -ba | sed -n '1,90p'
printf '%s\n' '--- base test ---'
git show 576ef68874f2fcbbb00f7f2302960c3b20597164:cuda_bindings/tests/nvml/test_nvlink.py | nl -ba | sed -n '1,90p'
printf '%s\n' '--- diff ---'
git diff --unified=30 576ef68874f2fcbbb00f7f2302960c3b20597164 412659b856b8c1d7086bdef9351f66e3b4cecabd -- cuda_bindings/tests/nvml/test_nvlink.py

Repository: NVIDIA/cuda-python

Length of output: 5440


important: Defer the Orin xfail only after bypassing its sentinel assertion.

When the Orin sentinel remains, moving pytest.xfail() after the loop still reaches the first return-code assertion, which rejects -1 before the alternate query and later devices run. Record the Orin condition, skip only that assertion, keep the alternate query active, and call pytest.xfail() after the loop.

Suggested fix
 def test_nvlink_get_link_count(all_devices):
     """
     Checks that the link count of the device is same.
     """
+    orin_nvml_bug = False
     for device in all_devices:
...
         value = nvml.device_get_field_values(device, fields)[0]
         if value.nvml_return == -1 and "Orin" in nvml.device_get_name(device):
-            pytest.xfail(
-                "Orin NVML nvbug 6952048: nvmlDeviceGetFieldValues returns success without populating the result"
-            )
-        assert value.nvml_return == nvml.Return.SUCCESS or value.nvml_return == nvml.Return.ERROR_NOT_SUPPORTED, (
-            f"Unexpected return {value.nvml_return} for link count field query"
-        )
+            orin_nvml_bug = True
+        else:
+            assert value.nvml_return == nvml.Return.SUCCESS or value.nvml_return == nvml.Return.ERROR_NOT_SUPPORTED, (
+                f"Unexpected return {value.nvml_return} for link count field query"
+            )
...
         assert value.nvml_return == nvml.Return.SUCCESS or value.nvml_return == nvml.Return.ERROR_NOT_SUPPORTED, (
             f"Unexpected return {value.nvml_return} for link count field query"
         )
+
+    if orin_nvml_bug:
+        pytest.xfail(
+            "Orin NVML nvbug 6952048: nvmlDeviceGetFieldValues returns success without populating the result"
+        )

@github-actions

Copy link
Copy Markdown
Contributor

@rwgk
rwgk requested a review from mdboom October 19, 2026 05:12

@mdboom mdboom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should just workaround this in a test. Seems like a bug that we should protect our users from. That is, we should set an error code inside of the device_get_field_values function and populate the error back out.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.bindings Everything related to the cuda.bindings module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants