chore(ci): Stabilize Advisory Benchmark Comparisons

[refcell]

Summary

Runs base and head benchmarks in an interleaved ABBA sequence using prebuilt binaries pinned to one CPU, reducing drift without adding an external benchmarking service. Aggregates paired Criterion runs with geometric means and repeat-spread checks so noisy or inconsistent results are reported without producing performance claims. Keeps the existing advisory threshold and PR comment workflow.

[cb-heimdall]

🟡 Heimdall Review Status

Requirement	Status	More Info
Reviews	🟡 0/1	Denominator calculation Show calculation 1 if user is bot 0 1 if user is external 0 2 if repo is sensitive 0 From .codeflow.yml 1 Additional review requirements Show calculation Max 0 0 From CODEOWNERS 0 Global minimum 0 Max 1 1 1 if commit is unverified 0 Sum 1

[github-actions]

Nit: paired_deltas is only called after is_significant gates on has_all_repetitions, so in practice both tuples are the same length. But the method itself doesn't enforce that precondition — strict=False silently truncates to the shorter side. If a future caller invokes paired_deltas without the is_significant gate, the mismatch is swallowed and the len(deltas) == self.expected_repetitions check in is_regression/is_improvement silently returns False rather than surfacing the bug.

Consider either using strict=True (to fail loudly on mismatched repetitions) or adding an assert, so a broken call site can't produce a silently-wrong result:

for base_median, head_median in zip(
    base.repeat_medians, head.repeat_medians, strict=True
)

[github-actions]

The early-return path on line 304 checks regressed or improved or dropped or incomplete, but incomplete only contains benchmarks that failed has_all_repetitions. A benchmark with complete repetitions but unstable repeats (has_stable_repetitions returns False) would be flagged as is_notable (line 196) but doesn't appear in any of the four lists checked here. The early return fires and the unstable benchmark's row is never rendered — despite is_notable wanting to surface it.

Consider adding an unstable list (analogous to incomplete) and including it in this guard:

unstable = [
    b
    for b in bench_ids
    if b in base and b in head
    and self.has_all_repetitions(base[b], head[b])
    and not self.has_stable_repetitions(base[b], head[b])
]
...
if not (regressed or improved or dropped or incomplete or unstable):

[github-actions]

Review Summary

CI-only changes to stabilize advisory benchmarks (ABBA ordering, prebuilt binaries, CPU pinning, geometric-mean aggregation) plus a benchmark relocation from base-flashblocks-node to base-common-consensus. Not block-production-sensitive.

Findings

Two inline comments were posted on etc/scripts/ci/bench_compare.py:

1. paired_deltas uses strict=False on zip — silently truncates mismatched repeat_medians tuples instead of failing loudly. Current callers are gated by has_all_repetitions, but a future caller without that gate would get silently wrong results. strict=True is the safer default.

2. Unstable-repeat benchmarks fall through the early-return guard — is_notable returns True for benchmarks with complete but unstable repetitions, but the early-return in render() only checks regressed or improved or dropped or incomplete. A benchmark that is complete but has unstable A/A spread would be swallowed by the early return and never rendered, contradicting the intent of is_notable.

Notes

• The sender_recovery benchmark relocation is clean — dependencies (rayon, criterion, alloy-consensus[secp256k1]) follow the bench correctly, and the run_bench_subset.sh gracefully handles both old and new locations for cross-commit compatibility.
• The HEAD_SHA change from pull_request.head.sha to github.sha (merge commit) is well-motivated and consistently applied to both the env var and the checkout ref.
• The always() → success() condition change on the comment step means a bench_compare.py crash will leave a stale PR comment from a prior run rather than cleaning it up. This is a minor trade-off given that the script now always produces a comment body (even for "no significant changes").

[github-actions]

Base Std historical fork tests

Fork	Result	Passed	Failed	Skipped	base/base	base-anvil	base-std
Beryl	pass	616	0	13	3e5cd9ab	8d0f5b8a	4658f1b7
Cobalt	pass	721	0	14	3e5cd9ab	9df661bc	e30b3421

View run

[github-actions]

Caution

This PR may regress performance. 1 benchmark(s) slower by more than 10% beyond the noise band: execution/Open 1024 nodes - 4096 nodes (+50.2%).

Benchmark results (advisory)

Geometric-mean time from two base-head-head-base (ABBA) runs on one pinned CPU. A wall-clock change is only flagged when both paired comparisons clear the threshold in the same direction, the repeated-run confidence envelopes do not overlap, and each side's same-code A/A spread is below 10%. Only notable or incomplete results are listed. This check never blocks a merge.

Benchmark	Base	Head	Δ geometric mean	A/A spread (base / head)
batch_queue/drain_cached_span_batches	64.37 µs	48.73 µs	-24.3% · unstable repeats	70.7% / 0.1%
execution/Open 1024 nodes - 4096 nodes	36.82 µs	55.29 µs	+50.2% ⚠️ slower	5.1% / 1.1%
execution/Open 1024 nodes - 65,536 nodes	69.14 µs	65.35 µs	-5.5% · unstable repeats	13.8% / 25.7%

54 benchmark(s) within ±10% omitted.

View run · Re-run benchmarks