docs(sc): clarify checkpoint retention and metric constraints - #3784
Open
tianyi-zhang-02 wants to merge 6 commits into
Open
docs(sc): clarify checkpoint retention and metric constraints#3784tianyi-zhang-02 wants to merge 6 commits into
tianyi-zhang-02 wants to merge 6 commits into
Conversation
The Known Missing Features list said "Checkpointing and validation are not yet supported (setup raises if enabled)". Both halves are wrong now. Checkpointing landed in NVIDIA-NeMo#3429: SingleControllerActor holds a CheckpointManager, _save_checkpoint writes GRPOSaveState plus a replay_buffer.pt and the dataloader position, _maybe_restore_replay_buffer reads them back, and tests/unit/single_controller/test_sc_checkpointing.py covers it. Nothing raises on either feature. grep for a raise about checkpointing or validation in single_controller_utils/setup.py returns nothing. The only guard is narrower and points the other way: validate_single_controller_config rejects a non-"train:" checkpointing.metric_name precisely because checkpointing works while validation does not, so top-k retention would key off a metric that is never collected. Rewrites the bullet to cover validation only, states the metric_name constraint it implies, and adds a line to the migration section so a legacy user can tell that checkpointing carries over rather than reading its absence as an omission. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Tianyi Zhang <123608656+tianyi-zhang-02@users.noreply.github.com>
github-actions
Bot
added
the
Documentation
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
# Conflicts: # docs/guides/single-controller.md
4 tasks
# Conflicts: # docs/guides/single-controller.md
tianyi-zhang-02
changed the title
tianyi-zhang-02
changed the title
Signed-off-by: Tianyi Zhang <123608656+tianyi-zhang-02@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Clarifies two SingleController checkpointing details that were still missing from the guide:
checkpointing.metric_namemust benullor use atrain:metric, because SC does not run validation metrics yet;The guide already says checkpointing itself is supported. This PR only documents the retention constraint and resume behavior.
Validation
Final SHA:
5f5d0baf6eea159c426acfb1d865b72b5d7d3f93, refreshed onto upstreammainatccbcd4cc5.Docs only; the claims were checked against
validate_single_controller_config,_save_checkpoint, and_maybe_restore_replay_buffer.