Stabilise walltime benchmarks across runs

[joseph-isaacs]

Summary

The walltime CodSpeed legs report the same code up to 2x apart from one run to the next. add_shapes_neon[(128, PerRowPerRow)] reads ~1.8 µs on most develop pushes and ~2.9–3.7 µs on some (e.g. cbabac3, a DuckDB export change), so a PR whose base and head landed in different runs is reported as a change it did not make. #9639 is the latest to be hit. Two separate things are going on; this fixes both.

1. A fixed per-sample cost that differs by host

Comparing a slow develop run against a fast one, every benchmark in the neon binary_ops binary is slower by the same ~1.2 µs whatever its length:

benchmark	fast run	slow run
add_shapes_neon[(128, PerRowPerRow)]	1.8 µs	2.9 µs
subtract_shapes_neon[(128, PerRowPerRow)]	1.9 µs	2.8 µs
add_u32_nonnull_neon	6.6 µs	7.7 µs
mul_u8_nonnull_neon	8.3 µs	9.6 µs
add_constant_shapes_neon[(16384, ConstantPerRow)]	9.3 µs	10.5 µs

Timer precision was 29 ns in both. The codspeed-divan-compat runner brackets each sample with its own timestamps and fifo marker hooks, and divan sizes samples by its first, cold run, so everything over a few microseconds was being measured one iteration per sample (iterPerRound: 1 in CodSpeed) with that per-sample overhead landing on it. Run without the runner, the 128-row case reads 1.55 µs flat.

#[cpu_features] now requires sample_size in #[divan::bench(..)], chosen so a sample lasts ~100 µs or more, and injects the 1,000-sample count itself. The workflow no longer sets DIVAN_SAMPLE_COUNT: an env or CLI value overrides the attribute in divan, so it would silently defeat this.

2. Process layout

arrow_checked_add_u32_neon[16384] reads 13 µs or 20 µs depending on where the process's memory lands. On one c7g.metal host with ASLR turned back on, it landed in either about half the time on the main thread, and held 13 µs through every run on a spawned thread. A sweep of the environment block's size (0–4096 bytes in 32-byte steps, both threads, both legs) moved nothing, which rules out the obvious "stack offset from env size" story.

#[vortex_bench_support::main] runs the benchmark binary's body on a spawned thread with an 8 MiB stack, and the walltime job refuses a tagged binary whose fn main lacks it.

Sweep run (throwaway workflow): https://github.com/vortex-data/vortex/actions/runs/33615899226

What this PR's CodSpeed runs showed

Four runs on four sets of hosts:

• add_shapes_neon[(128, PerRowPerRow)]: 1.61, 1.67, 1.67, 1.67 µs, standard deviation ~50 ns over 1,000 samples of 64 iterations (it was 290–400 ns over single-iteration samples).
• arrow_checked_add_u32_neon[16384]: 13.9 µs every run, its fast mode. On avx2 the spawned thread pins it to 21.3 µs, which the sweep showed is that leg's common mode (17.7 µs was the rare one), so CodSpeed reports it as a 17% regression against a base that happened to land in the rare mode. The flamegraphs of a slow and a fast base run show the whole difference inside arrow's per-element nullable loop with no allocation or kernel time, consistent with 4K aliasing between the output and an input buffer.
• Every small case reads 10–40% faster, which is the per-sample overhead no longer being charged to a single iteration.
• The first run had cases with large outputs up to 3x slower: divan holds a sample's outputs until it ends and builds its with_inputs up front, so with sample_size of them alive each iteration wrote to cold memory. The second commit drops a large output inside the closure and gives collect_bool one output buffer, after which those cases read at or below their base. The third commit keeps collect_bool on divan's input-slot loop, which brought words_gather_dispatch_avx2[1024] back to its base 17 ns.
• Two stable level shifts remain flagged: words_gather_dispatch_avx512[1024] 9 → 12 ns and words_gather_dispatch_avx2[65536] 1.3 → 1.53 µs. Their base buffer was a fresh zeroed vec per iteration, primed in L1 by its own memset, and the reused buffer is not. They read the same to the nanosecond across runs now, which is the property this PR is after.
• Suite run time per leg went from ~17 s to ~25–30 s.

Tests

• cargo +nightly fmt --all
• cargo clippy --all-targets --all-features
• cargo test --doc -p vortex-bench-support
• yamllint --strict -c .yamllint.yaml .github/workflows/codspeed.yml
• Built binary_ops, lane_kernels and collect_bool with cargo codspeed build -m walltime and ran them locally to check the sample sizes take effect.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YV1PdJJrd7r1Zcts63RrWa

[codspeed-hq]

Merging this PR will regress 3 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 65 improved benchmarks
❌ 3 regressed benchmarks
✅ 2068 untouched benchmarks
⏩ 164 skipped benchmarks¹
🗄️ 1 archived benchmark run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	words_gather_dispatch_avx512[1024]	9 ns	12 ns	-25%
❌	WallTime	arrow_checked_add_u32_avx2[16384]	17.7 µs	21.3 µs	-17.22%
❌	WallTime	words_gather_dispatch_avx2[65536]	1.3 µs	1.5 µs	-11.79%
⚡	WallTime	compare_int_constant_neon	4.2 µs	3 µs	+41.13%
⚡	WallTime	compare_u64_avx2	4 µs	3.1 µs	+32.04%
⚡	WallTime	compare_int_eq_neon	4.7 µs	3.6 µs	+30.83%
⚡	WallTime	infallible_bool_neon[i32, PerRowPerRow]	3.7 µs	2.8 µs	+30%
⚡	WallTime	compare_int_neon	4.6 µs	3.5 µs	+29.7%
⚡	WallTime	compare_int_constant_avx512	2.7 µs	2.1 µs	+29.35%
⚡	WallTime	compare_u64_neon	4.6 µs	3.6 µs	+29.27%
⚡	WallTime	compare_int_nullable_neon	5 µs	3.9 µs	+27.75%
⚡	WallTime	add_u32_nonnull_neon	6.6 µs	5.2 µs	+26%
⚡	WallTime	infallible_bool_constant_neon[PerRowConstant]	4.3 µs	3.5 µs	+24.23%
⚡	WallTime	mul_i32_nonnull_avx512	7.9 µs	6.4 µs	+23.72%
⚡	WallTime	lt_i64_nullable_neon	3.6 µs	3 µs	+22.97%
⚡	WallTime	deferred_bool_avx512[i64, PerRowPerRow]	4 µs	3.3 µs	+22.61%
⚡	WallTime	infallible_bool_constant_neon[ConstantPerRow]	4.3 µs	3.5 µs	+22.58%
⚡	WallTime	compare_int_avx512	3.2 µs	2.6 µs	+22.12%
⚡	WallTime	compare_int_eq_avx512	3.2 µs	2.6 µs	+21.35%
⚡	WallTime	compare_int_eq_avx2	3.6 µs	2.9 µs	+21.17%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ji/bench-main-thread (e5bb971) with develop (b4863a1)}

Footnotes

1. 164 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. 1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩