feat(sc): stream PPO actor and critic minibatches

[tianyi-zhang-02]

What does this PR do ?

Lets Single Controller PPO stream actor and Megatron critic minibatches from native TQ replay without materializing one merged training payload. The critic now has an explicit begin/train-chunk/finish/abort lifecycle: chunks accumulate gradients and correctly normalized diagnostics, then perform one optimizer step at the end of each critic epoch.

The controller retains replay rows through all critic and actor epochs, cleans up partial failures, skips invalid chunks, and rejects configurations whose drop budget cannot preserve data-parallel batch divisibility.

Issues

Relates to #2625.

Usage

The extended functional recipe exercises training, checkpointing, and resume:

uv run --locked bash tests/functional/ppo_async_single_controller.sh

Validation

Check	Environment	Result
Relevant PPO/value/SC unit suite	current main; Python 3.13	175 passed
Split critic parity	2x H100 NVL 94 GB; Megatron value model; one synchronous batch vs two streamed chunks	passed loss, grad norm, and every diagnostic metric
SC PPO functional + resume	Qwen2.5-0.5B; async vLLM; Megatron policy/value; 2 prompt groups x 4 generations; 2 actor epochs + 2 critic epochs	steps 0-2 and resume 2-4 passed
Checkpoint assertions	policy, value, replay metadata, native data plane	complete at steps 1, 2, 3, and 4
TensorBoard assertions	both pre-resume and resumed runs	10/10 passed

At the final step the run had 8 valid samples / 4,034 valid tokens, actor loss 1.29e-04, critic loss 2.6001, actor grad norm 5.3551, critic grad norm 99.6567, and no masked sequences. The resumed run's critic loss stayed in [2.6001, 4.0994]; clamped probability-ratio bounds and logprob-error checks also passed.

The run used upstream main at 9b25508a3340bffdd8e3a2245ada72279fbc15d6, nvcr.io/nvidia/nemo-rl:v0.7.0, driver 570.133.20, CUDA 12.8 compatibility runtime, and PyTorch cu130.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

The GPU parity test exposed a real metric-normalization bug during development; the fix and a CPU regression assertion are included. This change is independent of other open contributor branches.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.