Test using pytest-run-parallel and related fixups in the tests by seberg · Pull Request #2194 · NVIDIA/cuda-python

seberg · 2026-06-10T20:20:57Z

Description

Enable testing with pytest-run-parallel on CI for 3.14t and 3.15t as well some residual related fixes.

Checklist

New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2026-06-10T20:21:01Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

seberg · 2026-06-11T13:14:56Z

/ok to test ed40f60

github-actions · 2026-06-11T13:42:26Z

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-2194/
https://nvidia.github.io/cuda-python/pr-preview/pr-2194/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-2194/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-2194/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

seberg · 2026-06-11T14:09:07Z

Fun, the refactor made the cufile xfail-strict tests pass on CI, but I didn't set up the parallel run correctly... one more try:

/ok to test eb6a2ff

seberg · 2026-06-11T14:10:34Z

/ok to test eb6a2ff

seberg · 2026-06-11T14:57:38Z

/ok to test 7b59bff

copy-pr-bot · 2026-07-08T11:40:36Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

seberg · 2026-07-08T12:04:41Z

Still quite a few changes in this now (and let's see if CI passes) including gh-2321. But maybe it's still fine to review.

The most annoying one is probably the test_object_protocols.py changes. IIRC the problem was that the design with the .getfixturevalue() deferred things into threads, which might even have been fine with initialized.
But none of the tests actually test things deep enough to require cuda set up in the test itself.

Andy-Jost

This needs another review round after #2454 merges and it is rebased/merged with main

Andy-Jost · 2026-08-10T18:19:59Z

+if python -c 'import sys; assert not sys._is_gil_enabled()' 2> /dev/null; then
+  FREE_THREADING="-ft"
+  PYTEST_PARALLEL_ARGS=(--parallel-threads=4)
+  pip install pytest-run-parallel


Can you add this to the test groups in the normal places under cuda_{bindings,core,pathfinder}/pyproject.toml?

Makes sense, I added it as a test-ft but it currently only contains pytest-run-parallel. Happy to do it differently.

(Semi-related, removed the special logic for cupy not support free-threaded as it does now.)

Signed-off-by: Sebastian Berg <sebastianb@nvidia.com>

This also seems to fix the issues around memory resource finishing/cleanup. I.e. the main part of the issue was apparently issueing device syncs from all threads. (Clean-up in a follow-up.)

seberg

(Few comments for review 🤞 that it will run through successfully in CI with all the drift going on.)

seberg · 2026-08-28T18:05:42Z

-        # never enqueued because their owning object had not been collected yet.
-        gc.collect()
-        driver.cuCtxSynchronize()
-        _ = _device_unset_current()


This was causing issues for some graph tests (because a device sync disrupts graph capture).

But, actually, it also seems to fix the mr cleanup issue... I.e. the dead-lock I was seeing happens only if there is a device synchronization on each thread.

That should allow a bunch of cleanups, but I would prefer to follow-up for it.

seberg · 2026-08-28T18:06:39Z

 @pytest.mark.agent_authored(model="gpt-5.6")
-@pytest.mark.parametrize("fixture_name", CLOSEABLE_TYPES)
-def test_closeable_object_state_and_safe_inspection(fixture_name, request):
+@pytest.mark.thread_unsafe(reason="closes a fixture object shared between threads")


Just marked as unsafe here, but also changed to follow the same indirect fixture pattern as the rest of the file.

seberg · 2026-08-28T18:09:03Z

        info = cuda_stream_attr()
    else:
        info = cuda_stream_attr
-        warnings.simplefilter("once", DeprecationWarning)


This triggered a failure. Could also just mark the test as unsafe, but mutating global state here seems wrong (e.g. what if the user wants to raise DeprecationWarnings?).

seberg · 2026-08-28T18:25:54Z

+if python -c 'import sys; assert not sys._is_gil_enabled()' 2> /dev/null; then
+  FREE_THREADING="-ft"
+  PYTEST_PARALLEL_ARGS=(--parallel-threads=4)
+  pip install pytest-run-parallel


Makes sense, I added it as a test-ft but it currently only contains pytest-run-parallel. Happy to do it differently.

(Semi-related, removed the special logic for cupy not support free-threaded as it does now.)

(It may be that there are more tests that need this)

seberg · 2026-09-01T13:19:50Z

Ping @Andy-Jost for awareness. Not burning, but would love to move this in so that the test fixups don't keep diverging (hopefully the amount of new tests needing skipping isn't too bad; on the plus side, a lot of the fixups here would have been cargo-culted by agents already if it had been merged at the time).

github-actions Bot added cuda.bindings Everything related to the cuda.bindings module cuda.core Everything related to the cuda.core module cuda.pathfinder Everything related to the cuda.pathfinder module labels Jun 10, 2026

seberg changed the title ~~Ft testing~~ Test using pytest-run-parallel and related fixups in the tests Jun 10, 2026

seberg force-pushed the ft-testing branch from 84e10a4 to a9ef88d Compare June 10, 2026 20:34

leofang assigned seberg Jun 11, 2026

leofang self-requested a review June 11, 2026 02:55

seberg force-pushed the ft-testing branch from 065330f to ed40f60 Compare June 11, 2026 13:05

seberg force-pushed the ft-testing branch from ed40f60 to eb6a2ff Compare June 11, 2026 14:05

github-actions Bot added the CI/CD CI/CD infrastructure label Jun 11, 2026

seberg force-pushed the ft-testing branch from 42bc9cd to 7b59bff Compare June 11, 2026 14:56

This was referenced Jun 15, 2026

Add critical section guards to accessors #2215

Merged

Use setdefault for internal cache to ensure identity caching #2217

Merged

TST: Use tempdir for cufile tests (and mark some as thread unsafe) #2218

Merged

seberg mentioned this pull request Jun 23, 2026

Fully clean up buffers/memory resources in tests #2243

Merged

seberg force-pushed the ft-testing branch from 7b59bff to 0856ba8 Compare July 8, 2026 11:40

seberg force-pushed the ft-testing branch 2 times, most recently from 7d33149 to 4f2cc7a Compare July 8, 2026 11:52

seberg marked this pull request as ready for review July 8, 2026 11:53

seberg added the P1 Medium priority - Should do label Jul 8, 2026

seberg added this to the cuda.core next milestone Jul 14, 2026

seberg force-pushed the ft-testing branch from 1de0de0 to de71910 Compare July 15, 2026 11:39

Andy-Jost self-requested a review July 27, 2026 16:37

seberg force-pushed the ft-testing branch 2 times, most recently from d9d8553 to 6847e09 Compare July 29, 2026 18:23

This was referenced Jul 29, 2026

Add identity preserving pattern to critical_sections #2453

Merged

Ft testing test fixes #2454

Merged

seberg force-pushed the ft-testing branch from 6847e09 to 240d6c4 Compare July 29, 2026 19:41

LeSingh1 mentioned this pull request Aug 9, 2026

fix(core): honour DLTensor.byte_offset in StridedMemoryView.from_dlpack #2576

Open

Andy-Jost requested changes Aug 10, 2026

View reviewed changes

seberg added 4 commits August 28, 2026 15:56

Move pytest-run-parallel setup (and hopefully actually make it work)

e5f4167

Signed-off-by: Sebastian Berg <sebastianb@nvidia.com>

Do not mutate global warning state

c2cf0f5

Fix conftest to only gc.collect/synchronize after all threads finish

5e243f3

This also seems to fix the issues around memory resource finishing/cleanup. I.e. the main part of the issue was apparently issueing device syncs from all threads. (Clean-up in a follow-up.)

Misc test fixes/thread-unsafe markers

ec849fa

seberg force-pushed the ft-testing branch from 240d6c4 to ec849fa Compare August 28, 2026 17:43

seberg added 2 commits August 28, 2026 20:18

Use a test-ft (only includes pytest-run-parallel) groups and allow cupy

5cd6e49

Mark graphics test as unsafe (second one is maybe not strictly, but...)

0f41fe6

seberg commented Aug 28, 2026

View reviewed changes

No cupy yet for Python 3.15 (marker blocked ft and 3.15).

25966eb

seberg requested a review from Andy-Jost August 28, 2026 18:50

seberg added 4 commits August 31, 2026 18:44

Mark subtests as thread-unsafe on windows for now

4d54538

Mark new tests with CPU buffer access as thread-unsafe on windows

47c8376

(It may be that there are more tests that need this)

Merge branch 'main' into ft-testing

07c2459

Merge branch 'main' into ft-testing

dc4c812

Uh oh!

Conversation

seberg commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Jun 10, 2026

Uh oh!

seberg commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026

Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

seberg commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg commented Jun 11, 2026

Uh oh!

seberg commented Jun 11, 2026

Uh oh!

copy-pr-bot Bot commented Jul 8, 2026

Uh oh!

seberg commented Jul 8, 2026

Uh oh!

Andy-Jost left a comment

Choose a reason for hiding this comment

Uh oh!

Andy-Jost Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

seberg Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

seberg Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

seberg Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

seberg Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

seberg Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

seberg commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

seberg commented Jun 10, 2026 •

edited

Loading

seberg commented Jun 11, 2026 •

edited

Loading