feat(sc): telemetry for rollout checkpointing

[macandro96]

What does this PR do?

Summary

Adds observability for Single Controller rollout checkpointing and recovery.

This PR measures whether periodic checkpointing affects rollout throughput, where checkpoint and restore time is spent, how much work is blocked by checkpoint barriers, and how effectively completed rollout work is reused after restart.

Stacked on #3924.

What changed

Rollout throughput

Adds metrics for:

• Raw generation-backend token throughput
• Committed token throughput published for training
• Committed prompt-group throughput
• Group completion latency
• Group admission/queue latency

Raw generation throughput uses cumulative counters exposed through the generation interface. vLLM implements these counters today; other inference backends can add support through the same interface. With token capture, committed tokens are counted exactly from valid staged rows; without token capture, they are estimated from the per-sample mean and completion count.

Checkpoint latency and backpressure

Adds phase-level checkpoint timing:

• Total checkpoint duration
• TQ save duration
• Time waiting to acquire the checkpoint barrier
• Exclusive barrier hold time
• Controller-sidecar serialization/write time
• Snapshot commit/publication time

Also records:

• Mutations blocked by checkpointing
• Mutation wait latency
• Per-operation blocking counters
• Replay, staging, and recovery-ledger row/group counts
• Controller-sidecar bytes

controller_sidecar_bytes intentionally excludes the native TQ payload because the current TQ checkpoint API does not report its size. The implementation avoids recursively scanning the shared checkpoint directory because doing so could perturb checkpoint benchmarks.

Checkpoint outcomes

Every scheduled checkpoint attempt records:

• Completed, skipped, or failed
• A machine-readable reason
• Configured attempt interval
• Time since the previous successful checkpoint
• Effective checkpoint cadence

Checkpoint invariant violations continue to fail immediately. Retryable I/O and timeout failures retain bounded retry behavior controlled by rollout_checkpointing.max_consecutive_failures.

Restore and recovery

Adds restore timing for:

• Snapshot resolution
• Dataloader restoration
• TQ restoration
• Replay-metadata restoration
• Recovery-ledger preparation
• Total restore duration

Recovery-efficiency metrics report:

• Complete training-ready groups restored
• Unfinished groups found during restore
• Siblings reused
• Siblings rerun
• Redispatch scheduling duration

The restore path now rejects a sampler cursor that is older than restored rollout work instead of silently dropping colliding prompts. Coverage includes a real rollout-pump borrow and repayment round trip through checkpoint capture and restore.

Logging behavior

Independent checkpoint, recovery, and throughput events use absolute wall-clock time for visualization so telemetry continues advancing during a long trainer step and across process restart.

nemo_rl/step remains the trainer-correlated step axis. W&B's internal _step is a monotonically increasing event-row index.

Wall-clock sampling is disabled by default. Set rollout_checkpointing.telemetry_interval_s to a positive interval in seconds to enable it; this does not change checkpoint cadence.

Primary dashboard

The intended high-level dashboard contains:

Area	Metrics
Generation	Raw generation token throughput
Usable output	Committed token and group throughput
Checkpoint latency	Total, TQ, barrier, exclusive-hold, sidecar, and commit time
Backpressure	Blocked mutation count and latency
Reliability	Checkpoint outcome, reason, and effective cadence
Restore	Total restore time and phase breakdown
Recovery efficiency	Complete groups restored and siblings reused versus rerun
Snapshot size	Row counts and controller-sidecar bytes

The remaining controller, buffer, vLLM request, KV-cache, and per-mutation metrics are diagnostic drill-down signals.

Additional fixes

This PR also incorporates review follow-ups that:

• Avoid nested data-plane mutation barriers during eviction
• Validate mutation-category strings at runtime
• Strengthen test fixtures for telemetry-enabled controller paths
• Keep generation configuration access correctly typed
• Classify newly introduced RPC and finalized-group fields
• Document the W&B step-axis behavior
• Remove defensive fallbacks around required checkpoint fields

Validation

Completed locally:

• ruff check
• ruff format --check
• git diff --check
• Python syntax validation

Recommended unit tests:

uv run --no-sync pytest -q \
  tests/unit/single_controller/test_checkpointing.py \
  tests/unit/single_controller/test_checkpoint_borrow_restore.py \
  tests/unit/single_controller/test_checkpoint_dispatch_races.py \
  tests/unit/single_controller/test_setup.py \
  tests/unit/single_controller/test_rollout_pump.py \
  tests/unit/single_controller/test_tq_replay_buffer.py \
  tests/unit/experience/test_rollout_manager.py \
  tests/unit/models/generation/test_vllm_generation.py \
  tests/unit/utils/test_logger.py

Issues

Closes #4047

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[macandro96]

/ok to test 70940a9

[macandro96]

/ok to test 23d1295

[terrykong]

This PR adds telemetry for rollout checkpointing in the single controller: a telemetry_interval_s sampling pump, throughput/save/restore/outcome metric families, checkpoint-barrier mutation counters in the replay buffer, and a new metrics doc section. It also moves WandbLogger.log_plot and log_histogram onto the buffered-step path so plots and scalars land in the same W&B history row.

Worth calling out: your earlier self-review round holds up. Checked against the code at HEAD rather than the "addressed in ..." replies — the rebase rewrote the commits those replies cite — 10 of the 11 findings are genuinely fixed. The 11th has its own thread reply: the test-side half landed and the production-side half did not, and it is harmless, so it just needs the thread not to close on a stale premise.

Local evidence at a4c10a78:

• tests/unit/utils/test_logger.py + tests/unit/single_controller/: 1014 passed.
• tests/unit/experience/ + tests/unit/data_plane/: 648 passed, 15 skipped. All 15 skips are missing dependencies (4x nemo_gym, 11x mooncake), none GPU.
• ruff 0.9.9 clean on all three pinned hooks. Pyrefly reported no error in any file this PR touches, but the dependency set was incomplete, so treat that as a partial check rather than a clean bill.

Two housekeeping items. The PR is currently marked as conflicting with main — please rebase and resolve before the next round. And the review is pinned to a4c10a78; the head has since moved to 23d12951, which differs only in .github/workflows/cicd-main.yml and two functional-test scripts (13 lines), so every comment below still points at the code it describes. /ok to test 23d1295 is now running, which is what closes out the two pyrefly findings from the last round and the canonical_output_tokens arms that no local lane executed.

One coverage note: tests/unit/data_plane/test_rollout_reassembler.py is module-level skipped locally for missing nemo_gym, so the new canonical_output_tokens assertion only runs in CI's Nemo_Gym lane.

Nineteen inline comments: two getattr guards that cannot do what they look like they do, an ungated per-tick print, one docs/code mismatch in the recovery metrics table (the most substantive one — a round-trip test on the real restore path is attached, and it asks that the re-admission discard raise instead of dropping prompts, with a startup check that the restored sampler cursor is not behind the restored buffer), a naming ask for the canonical_* throughput keys, a note that the long-stalled recovery test closes #4047 and should say so, an undocumented on-switch, a hand-copied kind list, three test-coverage gaps, and six placement/duplication notes on the telemetry constants (including the mutation-kind list, which should be an enum so a wrong kind fails instead of vanishing from the series, and the hard-coded three-failure cap on the snapshot pump) — one of which is a wall-clock axis that restarts at 0 on every resume, so before- and after-restart series overlap.

Generated by Claude Code

[macandro96]

/ok to test a3dfa6c

[terrykong]

Follow-up on 195df04 (head a3dfa6c, rebased on main). Checked every thread from the first review against the code rather than the replies: 18 of 19 are fixed and resolved. Two things below, one of which blocks approval.

Verified locally at a3dfa6c: ruff 0.9.9 check + format clean on the 14 changed Python files; tests/unit/single_controller + tests/unit/utils/test_logger.py + tests/unit/experience/test_rollout_manager.py → 5 failed / 1084 passed, all five from the one regression below. (13 further failures in test_setup.py::TestOPDFullValidation are main's #3978 needing an env var; not this PR.)

CI on a3dfa6c agrees: L0_Unit_Tests_Other fails on exactly this — test_run_restores_native_tq_replay_metadata_without_payload_reput with the target_step_list AttributeError at single_controller.py:991 (the run stops at the first failure, so the other four show up once that one is fixed). L0_Unit_Tests_Models_1 is test_stream_weights_via_ipc_zmq_impl timing out, which nothing in these commits touches, and the quality gate is downstream of both.

Two inline comments: the _FakeTQBuffer regression from the new cursor check (blocking, one-line fix), and a low-severity symmetry note on the admission guard.

Generated by Claude Code

[macandro96]

/ok to test e48366d

[macandro96]

/ok to test 054bb6f