Skip to content

fix(quantization): honor fp8 emulate for rowwise recipes - #4178

Open
MohammedAlkindi wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
MohammedAlkindi:fix/fp8-emulate-rowwise-recipes
Open

MohammedAlkindi wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
MohammedAlkindi:fix/fp8-emulate-rowwise-recipes

Conversation

@MohammedAlkindi

Copy link
Copy Markdown
Contributor

What does this PR do ?

For the rowwise and rowwise_with_gw_hp recipes, apply_fp8_to_model built Float8LinearConfig.from_recipe_name(...), which does not take emulate, so fp8.emulate: true was dropped. Below SM89 the call then raised "FP8 is only supported on SM89 or later GPUs ... pass emulate=True", even though emulate was set. The user's flag is now carried onto the recipe config.

Changelog

  • Set emulate=True on the recipe-built config (via dataclasses.replace) when fp8.emulate is true. On SM89+ this makes those recipes emulate when asked to, instead of running real FP8.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

  • Related to # (issue): none filed.

test_fp8.py: 25 passed, 3 skipped; the two new emulate cases raise that ValueError on main. AI-assisted; I ran the file on 2d0467e.

Signed-off-by: Mohammed Alkindi <alkndymhmd692@gmail.com>
@MohammedAlkindi
MohammedAlkindi requested a review from a team as a code owner October 7, 2026 01:18
@copy-pr-bot

copy-pr-bot Bot commented Oct 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Oct 9, 2026
@HuiyingLi

Copy link
Copy Markdown
Contributor

/review

@HuiyingLi

Copy link
Copy Markdown
Contributor

/ok to test 2d0467e

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No high-confidence findings. Reviewed the FP8 recipe configuration change and regression tests, including the directly affected caller. The emulation flag now reaches both the hardware check and conversion configuration; non-emulated recipes retain rejection below SM89.

Static review only; tests were not executed.

Review generated by codex

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-customer Waiting on the original author to respond and removed waiting-on-customer Waiting on the original author to respond labels Oct 12, 2026
HuiyingLi
HuiyingLi previously approved these changes Oct 13, 2026

@HuiyingLi HuiyingLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. replace(torchao_config, emulate=True) is the right way to carry the flag past from_recipe_name; unit tests pass locally (28 passed). The earlier CI failures were the install-test numpy/pyarrow issue fixed on main and an unrelated runtime-budget overrun in test_ddp_chunked_ce_two_rank_parity; re-triggered via proxy #4276.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-customer Waiting on the original author to respond waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-customer Waiting on the original author to respond labels Oct 13, 2026
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
@HuiyingLi
HuiyingLi dismissed stale reviews from svcnvidia-nemo-ci and themself via 18616c3 October 17, 2026 20:59
@HuiyingLi
HuiyingLi requested a review from a team as a code owner October 17, 2026 20:59
@HuiyingLi

Copy link
Copy Markdown
Contributor

/ok to test 18616c3

@HuiyingLi HuiyingLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete three-file change at 18616c333c87edb63800c57874ed1210fe535b19, its callers, the recipe/config contract, and the actual imported TorchAO implementation. No blocking correctness issues found. The emulation override reaches all three GEMM configurations while preserving each recipe's scaling and backward settings, parameter identity, filtering, config precedence, and default hardware checks.

Validation: the native FP8 suite passed 25 tests, with 3 CUDA-gated skips. Against the defective implementation, exactly the two rowwise emulation regressions failed; the other 23 tests passed. An independently designed supplemental CPU suite passed 35 tests, including all recipe fields and call paths, FP32/BF16 forward/backward and one SGD update on 2D/3D inputs. All 16 comparisons with direct TorchAO conversion were exact; that reference shares TorchAO's numerical implementation. Separately, the high-precision weight-gradient recipe exactly matched the mathematical weight/bias-gradient oracle, while deliberately selecting ordinary rowwise failed that oracle.

These executed CPU results are reused from 2d0467e2dc566afcc51a12358ae49f7ab4d41310: its production and test sources are byte-identical to this head, whose sole additional change is the CI constraint. The recorded runtime was PyTorch 2.14.0a0+4fdf77b940.nv26.08 and TorchAO 0.17.0+git02105d4, with CUDA hidden. No hardware-FP8, GPU, compiled, distributed, checkpoint-loading, full-model, or performance coverage is claimed.

The installation constraint is the exact trusted-main PyArrow cap; YAML/shell validation and a fresh NumPy 1.26.4 / PyArrow 25.0.1 import passed. CI was triggered for this exact head: Installation Test is in progress and CICD NeMo is waiting. Those runs are not yet green.

@HuiyingLi
HuiyingLi enabled auto-merge (squash) October 17, 2026 21:16
@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Oct 18, 2026

This branch was successfully deployed

3 active deployments
public — 18616c33 Deployed Oct 17, 2026 by copy-pr-bot[bot] via release / finalize / notify #5870
test — 18616c33 Deployed Oct 17, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #12021
nemo-ci — 18616c33 Deployed Oct 17, 2026 by copy-pr-bot[bot] via UV - Python - ARM64/Darwin - No CUDA #11177
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants