Skip to content

fix(group_offloading): support Ascend NPU stream when available - #14785

Open
li-lizhe wants to merge 1 commit into
huggingface:mainfrom
li-lizhe:fix/group-offload-npu-stream
Open

li-lizhe wants to merge 1 commit into
huggingface:mainfrom
li-lizhe:fix/group-offload-npu-stream

Conversation

@li-lizhe

Copy link
Copy Markdown

apply_group_offloading with use_stream=True only created a stream for CUDA (via torch.cuda.Stream()) or Intel XPU (via torch.Stream()), and raised Using streams for data transfer requires a CUDA device, or an Intel XPU device. on every other accelerator.

On Ascend NPU, torch.npu.Stream() is available (exposed by torch_npu), so stream-based onload/offload should be supported there too.

Changes:

  • Add an npu branch consistent with the existing CUDA/XPU checks (hasattr(torch, "npu") and torch.npu.is_available() -> torch.npu.Stream())
  • Update the error message to mention the Ascend NPU device

Testing:
Verified on an Ascend 910B NPU (torch 2.14 + torch_npu):

  • Old code raises ValueError (confirmed torch.cuda.is_available()=False, torch.xpu.is_available()=False)
  • New code: torch.npu.Stream() constructs successfully
  • hasattr(torch, "npu")=True, torch.npu.is_available()=True

5 lines added — minimal, device-agnostic, consistent with existing accelerator dispatch pattern.

@github-actions github-actions Bot added hooks size/S PR with diff < 50 LOC labels Sep 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi @li-lizhe, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked. See the contribution guide for more details. If this PR intentionally does not fix a tracked issue, a maintainer can add the no-issue-needed label to silence this reminder.

Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice.

Once the PR links an issue (or gets the no-issue-needed label), you can ignore this message — it stays here as a comment, but it no longer applies.

enable_group_offloading only checked cuda.is_available() then xpu. On Ascend NPU (torch_npu) with use_stream=True, this falls through to raise ValueError("requires CUDA or XPU"). Add elif branch for torch.npu.is_available() with torch.npu.Stream().
@li-lizhe
li-lizhe force-pushed the fix/group-offload-npu-stream branch from da76e36 to 626174d Compare September 18, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hooks size/S PR with diff < 50 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant