Repository navigation
fix(unroll): fold const-generic loop bounds - #1569
Open
praveenperera wants to merge 1 commit into
Open
praveenperera wants to merge 1 commit into
praveenperera wants to merge 1 commit into
Conversation
A loop bound written `N / 8` with a const generic `N` reaches the unroll pass as a division of two constants, so `#[unroll]` reported a runtime trip count and left the loop rolled. rustc's GVN also computes the shared bound once in an outer loop's header, which made that loop look like it computed a value its body reads. Run SCCP on annotated functions before analyzing their loops, read the `builtin.constant` it materializes as a constant bound, and hoist header constants into the preheader before the shape checks. Closes NVIDIA#1494. Signed-off-by: Praveen Perera <me@praveenperera.com>
praveenperera
marked this pull request as ready for review
October 20, 2026 14:58
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1494.
The problem
#[unroll]on a loop bounded by an expression of a const generic, such aswhile j < N / 8, warned that the trip count was "only known at runtime" and left the loop rolled, so a register array indexed byjmoved to local memory.After monomorphization the bound reaches the unroll pass as
mir.divof twomir.constants, not as one constant.induction::const_i128reads only a directmir.constant, so the trip count was unknown. rustc's GVN also computes a repeatedN / 8once, at the use that dominates the others. In a nest, that is the outer loop's header, and the inner loops read it from there. So even with the division folded, the outer loop was rejected with "the header computes a value the body reads".The fix
All changes are in
mir-transforms:unroll_annotated_loopsrunssccpon each annotated function once before analyzing its loops. It already ran after unrolling to clean up. The folders indialect-mir/src/const_fold.rsapply Rust semantics: wrappingadd/sub/mul, and no folding of division by zero orINT_MIN / -1.const_i128also reads thebuiltin.constantthatsccpmaterializes.hoist_header_constantsmoves constants defined in a loop header into its preheader before the shape checks. The preheader is the header's only entry from outside the loop, so it dominates every use. This handles a folded bound that the body or later code reads, and a constant thatsccpmaterializes for a constant header block argument.The folders cover integer arithmetic, bitwise operations, shifts and comparisons. Casts and overflow-checked arithmetic in a bound are not yet folded, and the docs say so.
Testing
On an RTX 5090 (sm_120), CUDA 13.1:
mir-transforms: two new end-to-end unroll tests on the GVN shape, with onemir.divin the outer header read by both exit tests and by the exit block. One test fully unrolls both loops and checks the folded result. The other unrolls only the outer loop and runs the result. Both fail without the fix, and the outer-only test also fails with just the hoist disabled.unroll_smoke: a newconst_generic_bound::<256>kernel. It isnested_lane_rangewith every bound writtenLANES / 16, and the host checks it against the same expected sums.verify-code-shape.shnow requires that it uses no local memory. Before the fix, its PTX had 34ld.local/st.local. After the fix it has none, and the entry is the same size asnested_lane_range(8249 vs 8250 lines). The PTX of the example's other 22 kernels is unchanged byte for byte.N = 128: no local memory and no branches, down from 18ld/st.localand 2 branches. It has 64ld.globaland 16st.global.just check, plusscripts/smoketest.shin GPU and--compile-onlymodes: 240/240 each.--full-debug: 234/240. The other six fail the same way on unmodifiedmain:llvm-dwarfdumpis missing on this machine, IKET does not support full debug, and libNVVM rejects four with "invalid input program". Full-debug builds skip the unroll pass.