Skip to content

feat(sc): support multi-reward GDPO rollouts - #3962

Open
tianyi-zhang-02 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
tianyi-zhang-02:codex/sc-multi-reward-gdpo
Open

feat(sc): support multi-reward GDPO rollouts#3962
tianyi-zhang-02 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
tianyi-zhang-02:codex/sc-multi-reward-gdpo

Conversation

@tianyi-zhang-02

@tianyi-zhang-02 tianyi-zhang-02 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What does this PR do ?

Adds end-to-end multi-reward support for GDPO in the Single Controller path. Named reward components now survive native async rollout, JSON payload transport, TQ replay, checkpoint restore, and reward penalties, so GDPO can consume reward/<name> tensors instead of only the summed scalar reward.

The rollout boundary validates component schemas and keeps reward as the canonical scalar sum. GRPO/PPO behavior is unchanged when components are absent.

Issues

Relates to #2625.

Usage

The new functional recipe is:

uv run --locked bash tests/functional/gdpo_single_controller.sh

Validation

Check Environment Result
Relevant unit suite current main; Python 3.13 273 passed
SC GDPO functional 2x H100 NVL 94 GB; Qwen3-0.6B; Megatron + async vLLM; GSM8K; 2 prompt groups x 4 generations passed one real optimizer step
Runtime assertions 8 completions; 5,422 valid tokens; named reward tensors required by GDPO; metric checks 7/7 passed

The functional run reported loss 4.43e-06, grad norm 0.00323, generation KL 0.000715, and zero masked sequences. It ran from upstream main at 9b25508a3340bffdd8e3a2245ada72279fbc15d6 using nvcr.io/nvidia/nemo-rl:v0.7.0 (driver 570.133.20, CUDA 12.8 compatibility runtime, PyTorch cu130).

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

This change is independent of other open contributor branches.

Signed-off-by: Tianyi Zhang <123608656+tianyi-zhang-02@users.noreply.github.com>
Signed-off-by: Tianyi Zhang <123608656+tianyi-zhang-02@users.noreply.github.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Sep 2, 2026
@tianyi-zhang-02
tianyi-zhang-02 marked this pull request as ready for review September 2, 2026 15:11
@tianyi-zhang-02
tianyi-zhang-02 requested review from a team as code owners September 2, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant