feat(profiling): add pluggable policy and vLLM rollout hooks

[minitu]

Summary

• Add opt-in profiler plugin contracts for Megatron policy workers and vLLM rollout workers.
• Drive policy plugins around complete monolithic and split training updates, including finish, abort, and shutdown handling.
• Drive rollout plugins around vLLM engine initialization and complete GRPO rollout attempts in both the legacy and TransferQueue trainers.
• Run rollout plugins in the GPU-owning vLLM process for tensor-parallel and asynchronous engines, with lifecycle fanout from the owning NeMo RL actor.
• Document installation, configuration, supported topology, and lifecycle behavior.

Motivation

Policy training and rollout generation execute in isolated Ray worker environments, so external profiling tools need stable worker-local lifecycle hooks. These interfaces allow profiler packages to integrate without patching NeMo RL or introducing a hard dependency on a specific profiler.

For TP>1 and asynchronous vLLM engines, the outer NeMo RL actor coordinates generation but does not launch the GPU kernels. The profiler therefore has to be constructed in every internal vLLM GPU worker rather than only in the outer actor.

User impact

Profiling remains disabled by default. When the selector environment variables are unset, NeMo RL does not import a profiler package and existing training and generation behavior is unchanged.

• NRL_POLICY_PROFILER_CLASS selects a policy-training plugin.
• NRL_ROLLOUT_PROFILER_CLASS selects a rollout plugin.
• Rollout profiling supports tensor_parallel_size>=1, pipeline_parallel_size=1, and expert_parallel_size=1 with either synchronous or asynchronous vLLM engines.
• The built-in NIXL vLLM worker composes with rollout profiling. Other custom vllm_kwargs.worker_cls values are rejected before engine creation.

The selected package must be installed in the corresponding Megatron policy-worker or vLLM generation-worker environment. For TP>1 and asynchronous engines, it must also be importable in vLLM's internal GPU-worker processes.

Validation

Qwen3-235B end-to-end golden workload

The ntrace nemo_rl_qwen3_235b golden was captured successfully on Lyris as job 2821136 using this PR's pre-rebase feature head, cd2b0a4a7abf9b8c4cc242f79cf34a7a09ba214e:

• 17 nodes total: one dedicated CPU Ray head plus 16 GPU worker nodes.
• 64 policy ranks with TP2 / PP4 / CP2 / EP16 / DP4.
• TP8 / PP1 / EP1 asynchronous vLLM rollout with 8 rollout ranks.
• Three training iterations with both policy and rollout profiler plugins enabled.
• Capture postflight passed and emitted a schema-v2 manifest.
• Strict parity passed across all six report surfaces used by the golden regression gate.

That exact feature commit was subsequently rebased onto current main; the post-rebase checks below validate the conflict resolutions. A fresh GPU run of the rebased SHA is not claimed here.

Post-rebase checks

• Rebased onto main at 13a10647ebbf0f940d2b06ea41800b3f2fb46099 on 2026-09-02.
• Ruff check and format --check passed for all 15 changed Python files.
• Python syntax compilation and git diff --check passed.
• 20 policy and rollout plugin-loader tests passed locally.
• The Megatron policy lifecycle suites were collected but skipped because the local macOS environment does not provide megatron.bridge.
• Local syntax and focused tests used Python 3.13.12 because a macOS arm64 build of the repository-pinned Python 3.13.14 was unavailable; CI remains authoritative for the pinned environment and backend-dependent coverage.

Earlier Lyris worker-environment checks

• 27 rollout loader, initialization, rank-mapping, lifecycle, topology, sync/async dispatch, NIXL composition, and shutdown tests passed in the Lyris vLLM worker environment.
• 4 focused GRPO rollout-boundary and conflict-resolution tests passed.
• 10 policy plugin-loader tests passed in the Lyris Megatron worker environment; the dated image skipped the policy lifecycle module because it does not contain megatron.bridge from current main.

Fresh CI on the rebased SHA is the remaining qualification gate.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[minitu]

/ok to test bdbbb19

[minitu]

/ok to test 1e9ef4b

[minitu]

/ok to test dbead9f

[minitu]

/ok to test e24c31f

[minitu]

/ok to test cd2b0a4

[minitu]

/ok to test 26c84f9