Skip to content

fix(cuda.core): pass numa_id to cuMemCreate for host_numa virtual memory - #3149

Open
somtri wants to merge 2 commits into
NVIDIA:mainfrom
somtri:fix/2694-vmm-host-numa
Open

somtri wants to merge 2 commits into
NVIDIA:mainfrom
somtri:fix/2694-vmm-host-numa

Conversation

@somtri

@somtri somtri commented Oct 19, 2026

Copy link
Copy Markdown

Description

closes #2694

VirtualMemoryResource with location_type="host_numa" passed -1 as the location id to cuMemCreate, and the driver rejects that with CUDA_ERROR_INVALID_VALUE. This follows the fix described in the issue comment:

  • VirtualMemoryResourceOptions has a new numa_id: int | None = None field, added last so the positional order does not change. _check_config requires it for "host_numa", rejects it for every other location type, and rejects negative values with the message PinnedMemoryResourceOptions uses. _fill_prop writes it to prop.location.id.
  • location_type="host_numa_current" raises ValueError at construction. The enum member stays.
  • One addition the comment did not ask for: in modify_allocation, _check_config also rejects a per-call config whose numa_id differs from the resource's, as it already does for location_type. _fill_prop takes the location from the resource, so without the check a different value would be ignored silently. I can drop it if you prefer.
  • test_vmm_host_location_types_report_host_accessible and test_is_ipc_enabled built host_numa without numa_id and built host_numa_current, so both needed updates. Three new tests cover the allocation, the host_numa_current rejection, and the numa_id checks. They carry agent_authored markers.
  • Release note in 1.3.0-notes.rst.

The allocation test writes and reads the buffer from the host. The resource grants access only to its own location, so cuMemsetD8 on the buffer fails with CUDA_ERROR_INVALID_VALUE unless the device is in peers. That is the host-resource peers gap from #2910, which this PR leaves alone.

Testing

Windows 11, RTX 3080 Laptop GPU (WDDM, driver 617.42), CUDA 13.4, cuda-bindings 13.4.4, Python 3.14. The device reports host_numa_id = 0 and host_numa_virtual_memory_management_supported = True.

The issue's reproducer, extended to host_numa_current and catching each error, before the change:

host allocated 2097152 bytes
host_numa FAILED: CUDAError CUDA_ERROR_INVALID_VALUE: ...
host_numa_current FAILED: CUDAError CUDA_ERROR_INVALID_VALUE: ...

After the change, numa_id=0 allocates 2097152 bytes, and host_numa_current raises ValueError at construction.

$ pytest tests/test_memory.py -k "vmm_host_numa or vmm_numa_id or host_location_types" -v
8 passed
$ pytest tests/test_memory.py tests/memory_ipc/test_vmm_ipc.py -k "vmm or Vmm or VirtualMemory or is_ipc_enabled" -q
50 passed, 29 skipped    (before the change: 45 passed, 30 skipped)

test_is_ipc_enabled skips on Windows ("Sharing virtual memory needs POSIX file descriptor handles"), so the test_vmm_ipc.py change has not run locally. pre-commit run passes on the five changed files.

Claude Code (claude-opus-5-5) wrote this change and ran the tests above on my machine. I reviewed the diff.

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

VirtualMemoryResource passed -1 as the location id for host_numa
allocations, and cuMemCreate rejects that with CUDA_ERROR_INVALID_VALUE.

Add VirtualMemoryResourceOptions.numa_id. It is required for host_numa,
rejected for other location types, and must match in a per-call
modify_allocation config. Reject host_numa_current at construction,
because the driver does not accept it for physical allocations.

Closes NVIDIA#2694

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Som Tripathi <somtri@iastate.edu>
@copy-pr-bot

copy-pr-bot Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Oct 19, 2026
@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/cuda-python/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 5b7d2bd6-8064-416c-93cc-2d4e5dc70018
📥 Commits

Reviewing files that changed from the base of the PR and between c33542b and 1e11c4a.

📒 Files selected for processing: 2
  • cuda_core/cuda/core/_memory/_virtual_memory_resource.pyi
  • cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx
🚧 Files skipped from review as they are similar to previous changes: 2
  • cuda_core/cuda/core/_memory/_virtual_memory_resource.pyi
  • cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

📝 Summary

Summary by CodeRabbit

  • New Features
    • Physical memory allocations can target a specific host NUMA node with location_type="host_numa" and a nonnegative numa_id.
    • Device properties identify the NUMA node closest to the device; if that value is unavailable, numa_id=0 is used as the fallback.
  • Bug Fixes
    • Invalid NUMA configurations are rejected, including missing or misplaced NUMA IDs, negative IDs, resource growth on a different NUMA node, and use of host_numa_current for physical allocations.

VirtualMemoryResourceOptions now accepts a NUMA node ID for host NUMA allocations. VirtualMemoryResource validates the node configuration and uses that ID for physical allocations. Tests cover successful allocation, host-side memory access, and invalid configurations.

Changes: Host NUMA allocations

Layer / File(s) Summary
NUMA option contract
cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx, cuda_core/cuda/core/_memory/_virtual_memory_resource.pyi
Adds the optional numa_id option and documents its requirement for host_numa, rejection for other location types, and nonnegative constraint.
NUMA validation and allocation
cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx, cuda_core/tests/test_memory.py, cuda_core/tests/memory_ipc/test_vmm_ipc.py, cuda_core/docs/source/release/1.3.0-notes.rst
Validates numa_id, rejects host_numa_current for physical allocations, and sets the host NUMA allocation location ID. Tests cover allocation, host access, and invalid configurations. Release notes describe the option.

Assessment against linked issues

Objective Addressed Explanation
Enable host_numa allocations by providing the required NUMA node ID [#2694] ✅
Reject host_numa_current for physical allocations [#2694] ✅

Suggested reviewers: andy-jost

Priority: ➖ Normal

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 1e11c

The NUMA allocation change has no identified issue that needs resolution before merging.

✨ Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/cuda-python/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: b60e59cb-b28e-419e-a1f0-4c8405fefed6
📥 Commits

Reviewing files that changed from the base of the PR and between 576ef68 and c33542b.

📒 Files selected for processing: 5
  • cuda_core/cuda/core/_memory/_virtual_memory_resource.pyi
  • cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx
  • cuda_core/docs/source/release/1.3.0-notes.rst
  • cuda_core/tests/memory_ipc/test_vmm_ipc.py
  • cuda_core/tests/test_memory.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Comment thread cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx Outdated
The numa_id docstring points to Device.properties.host_numa_id, which is
-1 on a system without NUMA, and numa_id rejects -1. cuMemCreate wants
location id 0 on such a system, so say that in the docstring.

Signed-off-by: Som Tripathi <somtri@iastate.edu>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.core Everything related to the cuda.core module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: VirtualMemoryResource host_numa allocations always fail

1 participant