intrinsic-test: sve support by davidtwco · Pull Request #2160 · rust-lang/stdarch

davidtwco · 2026-06-15T16:12:38Z

This patch contains all the changes necessary for intrinsic-test to generate tests for the SVE intrinsics:

~~Some of the commits are just general refactors to the tool, motivated by the commits that follow and that I felt were hard to justify on their own~~
- All of the refactorings prior to this PR have already been landed (intrinsic-test: sve type parsing and constraints #2121, intrinsic-test: run rustfmt on generated sources #2122, intrinsic-test: add lots of documentation #2124, intrinsic-test: refactor populate_random and related code #2126, intrinsic-test: remove arm::argument module #2127)
- All of the refactorings split out of this PR have landed (intrinsic-test: add filters for sve #2169, intrinsic-test: make types generic over SupportedArchitecture #2170, intrinsic-test: remove concatln! and redundant newlines #2171, intrinsic-test: simplify architecture constants #2172, intrinsic-test: simplify architecture constants #2172, intrinsic-test: simplify type printing + comparison abstraction #2173)
~~Some of the commits are small changes I made along the way (e.g. forwarding args from intrinsic-test.sh to cargo test)~~
- ~~I could split these out if we wanted~~
- Merged in intrinsic-test: fwd args in intrinsic-test.sh #2164
~~Some of the commits modify the definitions of intrinsics so that they pass tests - this only happened for a handful of intrinsics and the changes required to their definitions were very minor~~
- ~~I could split these out if we wanted~~
- Merged in core_arch: redefine svrev, svzip and svuzp #2163
Some of the commits update arm_intrinsics.json to add some additional constraints necessary to generate the right tests (we're making sure that these addl. constraints are reflected in the source of truth we use to generate that file internally)
- ~~I could split these out if we wanted~~
- Merged in update arm_intrinsics.json for svset and svget #2162
Any remaining commits are changes to make the tool generate the correct tests, primarily:
- Adding the arm_sve.h headers and appropriate target feature flags for C and Rust
- Using the svld1 intrinsic when loading an argument of scalable vector type
  - There is a special case for svbool_t arguments which do not have an svld1 intrinsic, we load a svint8_t and use svcmpne_n_s8 against zero to produce a svbool_t from it
- Generating test arrays for intrinsics that take boolean values and svbool_t values (though the earlier commits use the same svptrue_$ty predicate for these arguments initially)
- Supporting calls to enum-typed const generic arguments (such as svpattern), using simple const functions that map integers from intrinsic-test constraints to the appropriate variant
- Generating a local containing a predicate that is used by all intrinsics which require a predicate (just svptrue_$ty() to enable every lane)
- Using svcmpeq_$ty and svptest_any to compare the Rust and C results instead of assert_eq!
  - There is a special-case for tuples of vectors, where multiple svcmpeq_$ty and svptest_any blocks are generated for each vector in the tuple, with svgetN calls to extract the appropriate vector from the result tuples
  - There is also a special-case for intrinsics that return svbool_t, where there is no equivalent svcmpeq_$ty call, so sveor_b_z (exclusive OR) and !svptest_any is used
  - There is yet another special case for scalable vectors of floats, where the pre-existing NanEqF* type cannot be used, so a (rust == c) || (isnan(rust) && isnan(c)) check is performed instead, via a handful of SVE intrinsics
I haven't been been able to test this on x86 to check that I haven't broken the test generation, but I don't think I've made any changes to the generated output that weren't in the Arm-specific parts

sayantn · 2026-06-17T23:01:58Z

-        } = **self
-        {
-            let quad = if self.num_lanes() * bl > 64 { "q" } else { "" };
+    fn comparison_function(&self) -> String {


Is there some way to print the diff for SVE? I would imagine it is difficult due to SVE being non-const sized, but is there some way? That would massively improve debuggability. One approach I can suggest is have some small functions that convert from SVE vectors to &[T], e.g.

#[target_feature(enable = "sve")] pub fn svfloat32_to_slice(a: &svfloat32_t) -> &[NanEqF32] { unsafe { core::slice::from_raw_parts(core::ptr::from_ref(a).cast(), svcntw() as usize) } }

this might work, with significantly less complexity

View changes since the review

When I've needed to debug them, I've disabled the invocation of intrinsic-test in intrinsic-test.sh and added these snippets to the generated tests that I wanted to debug.

For non-bool vectors, just replace the x0_val and x1_val with the vectors that you want to inspect, e.g. __rust_return_value or __c_return_value:

{ let num_elems = svcnth() as usize; let mut x0_buf = Vec::with_capacity(num_elems); let mut x1_buf = Vec::with_capacity(num_elems); svst1_u16(__pred, x0_buf.as_mut_ptr(), x0_val); x0_buf.set_len(num_elems); svst1_u16(__pred, x1_buf.as_mut_ptr(), x1_val); x1_buf.set_len(num_elems); for i in 0..num_elems { let x0_val = x0_buf[i]; dbg!(i, x0_val); } for i in 0..num_elems { let x1_val = x1_buf[i]; dbg!(i, x1_val); } }

Similarly for bool vectors:

{ let num_elems = svcntb() as usize; let mut _op_val = Vec::with_capacity(num_elems); svst1_u8(op_val, _op_val.as_mut_ptr(), svdup_n_u8(1)); _op_val.set_len(num_elems); for i in 0..num_elems { let _op_val_el = if _op_val[i] == 1 { true } else { false }; dbg!(i, _op_val_el); } }

I'm not sure if what you've suggested will work, happy to try. I'm more than happy to make some improvements here for debuggability of the tests when they do fail, but could that be left to a follow-up?

Ended up doing this in the current patch because it was useful for working out what was going on with CI failures

Sorry, didn't understand what you meant by this and the current patch, the GH version shows the sveor version. Also, https://godbolt.org/z/a481YdW1P seems to work.

sayantn · 2026-06-17T23:04:02Z

-                "
+{intrinsics}
+"#,
+        intrinsics = intrinsics.iter().format_with("", |intrinsic, fmt| {


very minor nit, but you can use something like intrinsic.specializations().for_each(|imm_values| fmt(...))

View changes since the review

I wouldn't have a fmt to call if I used for_each

sayantn · 2026-06-17T23:05:41Z

also, it would be nice if this can be split into a few focused PRs (I can help if you want)

davidtwco · 2026-06-18T15:33:30Z

Submitted rust-lang/rust#158088 to fix the failure in debug mode - should pass CI after that

davidtwco · 2026-06-18T15:38:30Z

also, it would be nice if this can be split into a few focused PRs (I can help if you want)

I'll try and split some out tomorrow

@lqd

…o-no-opt, r=lqd codegen_ssa: no dbginfo for scalable vec local w/ `-O0` LLVM uses GlobalISel with `-O0` that doesn't support scalable vectors. It normally falls back to SDAG which does support scalable vectors, but there's a bug that means that isn't happening for debuginfo - so temporarily don't emit debuginfo for scalable vector locals when there are no optimisations until that bug is fixed. cc llvm/llvm-project#204585 cc rust-lang/stdarch#2160 r? @lqd

@lqd

Rollup merge of #158088 - davidtwco:scalable-vector-debuginfo-no-opt, r=lqd codegen_ssa: no dbginfo for scalable vec local w/ `-O0` LLVM uses GlobalISel with `-O0` that doesn't support scalable vectors. It normally falls back to SDAG which does support scalable vectors, but there's a bug that means that isn't happening for debuginfo - so temporarily don't emit debuginfo for scalable vector locals when there are no optimisations until that bug is fixed. cc llvm/llvm-project#204585 cc rust-lang/stdarch#2160 r? @lqd

@lqd

… r=lqd codegen_ssa: no dbginfo for scalable vec local w/ `-O0` LLVM uses GlobalISel with `-O0` that doesn't support scalable vectors. It normally falls back to SDAG which does support scalable vectors, but there's a bug that means that isn't happening for debuginfo - so temporarily don't emit debuginfo for scalable vector locals when there are no optimisations until that bug is fixed. cc llvm/llvm-project#204585 cc #2160 r? @lqd

davidtwco · 2026-06-19T10:59:13Z

Split out the first batch of changes in #2169

davidtwco · 2026-06-19T11:13:19Z

Split out the second batch of changes in #2170

davidtwco · 2026-06-19T11:21:26Z

Split out the third batch of changes in #2171

davidtwco · 2026-06-19T11:29:00Z

Split out fourth batch of changes in #2172

davidtwco · 2026-06-19T11:40:28Z

Split out final batch of changes in #2173. Shouldn't be much less other than the actual SVE testing implementation here after those are merged, but there will be a few merge conflicts between them as they land.

davidtwco · 2026-06-21T21:32:18Z

Rebased after all the prerequisites landed - should pass CI now, works in dev and release locally and the rustc patches are all in nightly - but I'll check back in tomorrow

This is just more helpful for knowing what all needs to be fixed when CI fails.

Replacing `iter_specializations` (which repeatedly invokes a callback) with an iterator implementation allows `Itertools::format_with` to be used more broadly, which in turn allows disparate string interpolation to be combined and hopefully provide greater context to the reader.

Updates `get_load_function` to return `svld{n}_{ty}` when loading a scalable vector type. Caller of `get_load_function` will still need updated to handle passing the predicate arguments to these load functions.

Updates the headers used by generated C code and the target feature flags passed to the C compiler to enable SVE.

Some SVE intrinsics take booleans as arguments, so there is a need to support generating a test value array for booleans.

Constraints that correspond to enum types - such as `svpattern` and `svprfop` - need to be converted to the enum type in order to be used in a generic instantiation - so introduce a const function for both types that provides this mapping.

Predicate arguments of type `svbool_t` do not need test value arrays to be generated as the same enable-all-lanes predicate will be passed to all invocations of the intrinsic under test. There is no `svld1` equivalent for `svbool_t` that could be used even if there were test values to use.

Support defining a local variable containing the predicate that will be used with all subsequent scalable vector intrinsics.

Implementation of `get_comparison_function` and `get_predicate_function` for SVE which uses the relevant SVE intrinsics.

Instead of assuming that any scalable boolean argument is a predicate, handle predicates specifically and generate test values for `svbool_t` values.

SVE isn't a baseline target feature for `aarch64-unknown-linux-gnu` but should be enabled when running tests.

Like with non-SVE test generation, comparison of float results in scalable vectors need special-handling of comparisons.

Enables generation of tests for SVE intrinsics leveraging the changes from the previous commits.

This makes it far easier to debug what's potentially gone wrong with an intrinsic test.

Same as previous redefinitions in stdarch#2163 - these were missed in that PR because the hardware being tested on was missing the hardware feature required for the instructions these use.

davidtwco · 2026-06-22T13:03:57Z

Pushed c89cacc to fix the failures with svtrn{1,2}, missed these locally as they require f32mm/f64mm and my local test machine didn't have those, but it's the same fix as #2163; and also pushed c04381b to print out the contents of scalable vectors when the tests fail.

Only remaining failures are svcreate and svset. These are curious because I already fixed them locally a while ago with a rustc patch that has long landed. They only fail with the second, third and fourth vectors in the tuple, and when I had investigated this previously, we were generating a memcpy that just didn't copy the second, third and fourth vectors. I fixed that in rust-lang/rust#157915 by correcting how rustc understood the size of these types and that fixed these tests locally for me - but they seem to still be failing in a very similar way here, it's very strange.

davidtwco · 2026-06-22T13:39:05Z

Only remaining failures are svcreate and svset. These are curious because I already fixed them locally a while ago with a rustc patch that has long landed. They only fail with the second, third and fourth vectors in the tuple, and when I had investigated this previously, we were generating a memcpy that just didn't copy the second, third and fourth vectors. I fixed that in rust-lang/rust#157915 by correcting how rustc understood the size of these types and that fixed these tests locally for me - but they seem to still be failing in a very similar way here, it's very strange.

I think it's just that rust-lang/rust#157915 is wrong - it works by chance for me locally because of the vector length my hardware has (and let me assume that maybe llvm.memcpy has some special handling when the type being copied was a scalable vector), but under QEMU w/ sve512, it doesn't copy enough.

davidtwco · 2026-06-22T14:43:40Z

This will be fixed once rust-lang/rust#158253 lands and is on nightly

davidtwco added the A-aarch64 label Jun 15, 2026

davidtwco force-pushed the intrinsic-test-sve branch 2 times, most recently from c29cf85 to 6561db8 Compare June 15, 2026 16:17

davidtwco mentioned this pull request Jun 15, 2026

Tracking Issue for Scalable Vectors rust-lang/rust#145052

Open

23 tasks

davidtwco force-pushed the intrinsic-test-sve branch 5 times, most recently from 9ff4174 to e817965 Compare June 16, 2026 10:25

This was referenced Jun 16, 2026

update arm_intrinsics.json for svset and svget #2162

Merged

core_arch: redefine svrev, svzip and svuzp #2163

Merged

intrinsic-test: fwd args in intrinsic-test.sh #2164

Merged

This comment was marked as resolved.

Sign in to view

davidtwco mentioned this pull request Jun 16, 2026

Sized Hierarchy and Scalable Vectors rust-lang/rust-project-goals#270

Open

27 tasks

davidtwco force-pushed the intrinsic-test-sve branch 2 times, most recently from 95f286a to ed132a7 Compare June 17, 2026 10:15

sayantn reviewed Jun 17, 2026

View reviewed changes

davidtwco mentioned this pull request Jun 18, 2026

codegen_ssa: no dbginfo for scalable vec local w/ -O0 rust-lang/rust#158088

Merged

davidtwco mentioned this pull request Jun 19, 2026

intrinsic-test: add filters for sve #2169

Merged

davidtwco force-pushed the intrinsic-test-sve branch from ed132a7 to ea73af3 Compare June 19, 2026 10:59

davidtwco mentioned this pull request Jun 19, 2026

intrinsic-test: make types generic over SupportedArchitecture #2170

Merged

davidtwco mentioned this pull request Jun 19, 2026

intrinsic-test: remove concatln! and redundant newlines #2171

Merged

davidtwco mentioned this pull request Jun 19, 2026

intrinsic-test: simplify architecture constants #2172

Merged

davidtwco mentioned this pull request Jun 19, 2026

intrinsic-test: simplify type printing + comparison abstraction #2173

Merged

This comment has been minimized.

Sign in to view

davidtwco force-pushed the intrinsic-test-sve branch from ea73af3 to 060d803 Compare June 21, 2026 21:30

davidtwco assigned sayantn Jun 21, 2026

davidtwco marked this pull request as ready for review June 21, 2026 21:31

davidtwco added 14 commits June 22, 2026 11:10

intrinsic-test: default to --no-fail-fast

708a463

This is just more helpful for knowing what all needs to be fixed when CI fails.

intrinsic-test: impl get_load_function for SVE

dbcba56

Updates `get_load_function` to return `svld{n}_{ty}` when loading a scalable vector type. Caller of `get_load_function` will still need updated to handle passing the predicate arguments to these load functions.

intrinsic-test: use arm_sve.h and target feats

27974a0

Updates the headers used by generated C code and the target feature flags passed to the C compiler to enable SVE.

intrinsic-test: bool test values

e5edbe7

Some SVE intrinsics take booleans as arguments, so there is a need to support generating a test value array for booleans.

intrinsic-test: introduce predicate_function

d6909a4

Support defining a local variable containing the predicate that will be used with all subsequent scalable vector intrinsics.

intrinsic-test: sve comparison and predicates

0f59585

Implementation of `get_comparison_function` and `get_predicate_function` for SVE which uses the relevant SVE intrinsics.

intrinsic-test: test values for svbool_t

be5cbea

Instead of assuming that any scalable boolean argument is a predicate, handle predicates specifically and generate test values for `svbool_t` values.

intrinsic-test: enable non-baseline target features

714507c

SVE isn't a baseline target feature for `aarch64-unknown-linux-gnu` but should be enabled when running tests.

intrinsic-test: SVE float comparison

4038db1

Like with non-SVE test generation, comparison of float results in scalable vectors need special-handling of comparisons.

intrinsic-test: stop skipping SVE intrinsics

7023f35

Enables generation of tests for SVE intrinsics leveraging the changes from the previous commits.

intrinsic-test: print scalable vectors on test fail

c04381b

This makes it far easier to debug what's potentially gone wrong with an intrinsic test.

davidtwco force-pushed the intrinsic-test-sve branch from 060d803 to c04381b Compare June 22, 2026 12:35

core_arch: redefine svtrn{1,2}

c89cacc

Same as previous redefinitions in stdarch#2163 - these were missed in that PR because the hardware being tested on was missing the hardware feature required for the instructions these use.

davidtwco mentioned this pull request Jun 22, 2026

codegen_ssa: multiply scalable vec size by vscale rust-lang/rust#158253

Open

Conversation

davidtwco commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

sayantn Jun 17, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidtwco Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidtwco Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

sayantn Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

sayantn Jun 17, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidtwco Jun 21, 2026

Choose a reason for hiding this comment

Uh oh!

sayantn commented Jun 17, 2026

Uh oh!

davidtwco commented Jun 18, 2026

Uh oh!

davidtwco commented Jun 18, 2026

Uh oh!

davidtwco commented Jun 19, 2026

Uh oh!

davidtwco commented Jun 19, 2026

Uh oh!

davidtwco commented Jun 19, 2026

Uh oh!

davidtwco commented Jun 19, 2026

Uh oh!

davidtwco commented Jun 19, 2026

Uh oh!

This comment has been minimized.

davidtwco commented Jun 21, 2026

Uh oh!

davidtwco commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidtwco commented Jun 22, 2026

Uh oh!

davidtwco commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

davidtwco commented Jun 15, 2026 •

edited

Loading

sayantn Jun 17, 2026 •

edited by rustbot

Loading

davidtwco Jun 18, 2026 •

edited

Loading

sayantn Jun 17, 2026 •

edited by rustbot

Loading

davidtwco commented Jun 22, 2026 •

edited

Loading