Say why a bf16 gradient norm cannot be clipped from - #8614
Open
vineethsaivs wants to merge 1 commit into
Open
vineethsaivs wants to merge 1 commit into
vineethsaivs wants to merge 1 commit into
Conversation
Reject only a negative norm, with a message naming the sentinel, and let a zero norm clip to the no-op it is. Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
vineethsaivs
requested review from
loadams,
tjruwase and
tohtana
as code owners
September 20, 2026 19:42
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BF16_Optimizer.stepguards clipping with a bareassert all_groups_norm > 0.. It stops a step whose gradients are all exactly zero, which a fully masked micro-batch produces, and it explains nothing when it fires.The norm helpers do not raise on a non-finite norm;
mask_nan_or_inf_with_val_inplacewrites -1 andget_norm_with_moe_layersreturns -1. Handed that,clip_tensors_by_global_normcomputesmax_norm / (-1 + 1e-6), a negative clip coefficient, and negates every gradient rather than shrinking it: measured here, 2.0 becomes -2.000002. So the assert has real work to do, but it conflates that with a legitimate zero and disappears underpython -O. Reject only a negative norm, with a message naming the sentinel, and let a zero norm clip to the no-op it is.Test:
TORCHDYNAMO_DISABLE=1 DS_ACCELERATOR=cpu PYTHONPATH=.:tests python -m pytest tests/unit/runtime/half_precision/ -q. Two new cases fail before: an all-zero-gradient step raises, and a non-finite norm gives a bare AssertionError instead of a reason. Three more pin the premises, that the helpers report -1 and that clipping from -1 negates the gradients, and that an ordinary norm still clips 2.0 to 0.5. Nine pass and 32 skip after. yapf 0.43.0 and flake8 pass on the changed files. Executed on Apple M2 Pro CPU; no GPU or distributed run. The guard and the clip arithmetic are local to one rank.