Skip to content

arch: Match default device memory queries to MPI placement - #3066

Open
ggorman wants to merge 1 commit into
mainfrom
fix-default-device-id
Open

ggorman wants to merge 1 commit into
mainfrom
fix-default-device-id

Conversation

@ggorman

@ggorman ggorman commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Without a visible-device mask, the generated code selects device rank % ngpus, but the Python memory query used the bare rank. With more ranks than devices (e.g. two ranks on one GPU) it queried a nonexistent device and nbytes_avail_mapper raised TypeError: unsupported operand type(s) for -=: 'NoneType' and 'int'. The query now uses the same mapping.

Test: the default-case block of TestDeviceID::test_visible_devices_mpi asserts rank % ngpus and that the device memory can be queried (no mocks). Verified on an H200 node: 2 ranks on 1 GPU pass; main fails.

#3067 is stacked on this one.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.17%. Comparing base (4fbf157) to head (f8d6cbb).

Files with missing lines Patch % Lines
tests/test_gpu_common.py 33.33% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3066   +/-   ##
=======================================
  Coverage   84.16%   84.17%           
=======================================
  Files         258      258           
  Lines       57290    57292    +2     
  Branches     4893     4893           
=======================================
+ Hits        48220    48224    +4     
+ Misses       8244     8242    -2     
  Partials      826      826           
Flag Coverage Δ
pytest-gpu-aomp-amdgpuX 68.41% <60.00%> (+<0.01%) ⬆️
pytest-gpu-gcc- 78.98% <20.00%> (+0.01%) ⬆️
pytest-gpu-icx- 78.90% <20.00%> (+<0.01%) ⬆️
pytest-gpu-nvc-nvidiaX 68.99% <60.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JDBetteridge
JDBetteridge requested a review from EdCaunt October 19, 2026 10:14
@ggorman
ggorman marked this pull request as ready for review October 19, 2026 18:41
With no visible-device environment variable, the generated code selects
device rank % ngpus, but ArgumentsMap._physical_deviceid used the bare
rank. With more ranks than GPUs (e.g., two decoupler workers on a 1-GPU
node), the device memory query then fails (None) and nbytes_avail_mapper
raises TypeError: unsupported operand type(s) for -=: 'NoneType' and 'Integer'.
@ggorman
ggorman force-pushed the fix-default-device-id branch from cddcdcb to f8d6cbb Compare October 21, 2026 21:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arch jitting, archinfo, ... GPU

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants