Skip to content

test(core): consolidate multiprocessing child-process helpers - #3151

Draft
rwgk wants to merge 1 commit into
NVIDIA:mainfrom
rwgk:run_child_noop_consolidation
Draft

rwgk wants to merge 1 commit into
NVIDIA:mainfrom
rwgk:run_child_noop_consolidation

Conversation

@rwgk

@rwgk rwgk commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Description

The VMM IPC and descriptor-leak tests independently implemented child execution and multiprocessing warmup. The first spawned process opens a persistent resource-tracker pipe; warming up after a file-descriptor baseline can therefore produce an order-dependent false leak failure. Child cleanup must also release IPC handles before fixture teardown.

Follow-up to the consolidation discussion on #3146.

Consolidate the common behavior in helpers/child_processes.py:

  • run_child(target, *args) applies the sanitizer-aware timeout, kills surviving children when a join times out or is interrupted, checks the exit status, and closes the Process handles before returning.
  • noop(*args) supplies one picklable target for warmup and argument-serialization tests.
  • warmup_multiprocessing() starts the resource tracker before descriptor counting and caches only a successful warmup.

Reuse these helpers in the VMM IPC and descriptor-leak tests. The serialization-failure case still serializes the resource-owning object before encountering Irreducible; its child target never executes. Tests that communicate with a running child or inspect the Process object's lifetime keep explicit process management.

Add six spawn-based regression tests covering argument delivery, failed exits, hung-child termination, interrupted-join cleanup, warmup retry/caching, and descriptor stability. Add agent guidance explaining when to reuse the helpers and how to guard specialized process-management tests.

Validation

Fresh local build and validation on Linux with Python 3.12.3, CUDA 13.4, and an NVIDIA A10G:

QA suite Passed Skipped Expected failures Passing subtests
cuda-pathfinder 1,615 5 0 0
cuda-bindings, default stream mode 669 27 0 21
cuda-bindings, per-thread default stream mode 669 27 0 21
cuda-bindings Cython tests 9 0 0 0
cuda-core 3,998 281 3 223
cuda-core Cython tests 1 0 0 0
Total 6,961 340 3 265
  • The full build and QA runbook both exited successfully.
  • A separate fresh pytest process ran the VMM descriptor-lifetime regression first, followed by all six new helper tests: 7 passed. This checks the cold-start baseline directly.
  • pre-commit run --all-files: all applicable checks passed.
  • Skips cover unavailable hardware capabilities, multiple-GPU requirements, and optional dependencies.

Review And CI

Ready for human review. This PR remains a draft solely to retain manual control over triggering CI.

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Share the child runner, picklable no-op target, and cached spawn warmup between the VMM IPC and file-descriptor leak tests. Kill children when joining is interrupted and close Process handles before returning.

Document helper reuse and cover successful execution, failed exits, timeouts, interrupted joins, warmup retries, and descriptor stability. Follow up on NVIDIA#3146.

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
@rwgk rwgk added this to the cuda.core 1.3.0 milestone Oct 19, 2026
@rwgk rwgk added test Improvements or additions to tests cuda.core Everything related to the cuda.core module labels Oct 19, 2026
@rwgk rwgk self-assigned this Oct 19, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@rwgk
rwgk requested a review from brandon-b-miller October 19, 2026 15:51
@rwgk

rwgk commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

@brandon-b-miller, this is the follow-on to PR #3146. Could you please make a review pass before I run the full CI?

At #3146 (review) @brandon-b-miller wrote:

Since we have a growing number of tests that hit this safety issue with multiprocessing it might be time to centralize the logic and put a note in the agent skill to reuse it.

We don’t yet have a repository-level test-authoring skill, so I added the helper-reuse guidance to AGENTS.md for this PR. I suggest we establish the skills foundation as another follow-up and keep this PR focused on the multiprocessing test helpers consolidation.

@brandon-b-miller brandon-b-miller left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.core Everything related to the cuda.core module test Improvements or additions to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants