bench: one shared constant set, runtime zisk sharding, stage-scoped aiur measures - #582
Merged
Conversation
samuelburnham
force-pushed
the
ci-bencher-aiur
branch
from
August 20, 2026 22:24
5734d99 to
019787f
Compare
…iur measures The benchmark suite ran different constants per backend, curated through Benchmarks/Vectors.csv's tier/shard_target/primary columns plus a fixed aiur selection, and the aiur pipeline's stage measures collided with the plain Phase-1 names via a mode-scoping convention. This consolidates all of it. Shared constant set. Ix/BenchConstants.lean (a plain importable array of (name, env)) replaces Vectors.csv as the single selection: every per-constant backend (aiur, zisk, ooc, lean4lean) runs the same constants, now including Std.HashMap, Lean.Json, and Multiset.sort at the heavy end. The tier/shard_target/primary columns, fixedSelections, --full/--tier/--shard-only/--csv flags, and the BENCH_FULL/BENCH_SHARD tokens are gone with the CSV. benchExclusions shrinks to hard infeasibility only: bitblast on the zkVMs, whose ~18B-step atomic mutual block crashes the ZisK executor and cannot be split by the shard cutter. bitblast itself is commented out of the set with a re-add note — its Aiur prove exceeds any current host, so a scheduled run could only re-document the same OOM; --consts/BENCH_CONSTS still run it on demand. Runtime zisk sharding. With the tier column gone, every zisk constant goes through extract → profile → `ix shard --max-ram <ceiling>` at bench time: the planner's RAM budget sizes the partition from the closure's predicted cost, so a closure that fits gets a one-shard plan and runs as a single leaf. Artifacts still pre-cut in bench-main's compile job and ride the zkshards cache; a failed cut falls back to the whole closure under the watchdog. Formerly-light constants now measure from their extracted sub-env (host peak-rss tightens once; guest cycles unchanged). Stage-scoped aiur measures. The pipeline's measures rename from the plain/recursive- split to a uniform stage<N>- prefix (stage1-prove-time, stage2-fft-cost, …) so the KZG stages join as stage3-/stage4- without renames. stagePrefixOf/dropStagePrefix move to the registry and parse the prefix generically; formatting kinds, units, improvement direction, and stage-table labels all derive from the base name. The execute mode keeps plain Phase-1 names on its own testbed, so the modes no longer share measure names at all. Throughput returns as a column in every stage table plus the ledger (stage1-/stage2-throughput, and pipeline-throughput = constants / total-time), emitted by bench-typecheck; none carry thresholds — constants is pinned exactly, so each is the pure inverse of an already-bounded time. bench-recursive-verifier adopts the same stage1-/stage2- keys. Dashboard. New plots: Aiur Total Throughput, Aiur Total Peak RAM Usage (which stage sets the ceiling can shift as stages are added), and Aiur Stage 1 Proof Size (it sizes stage 2's in-circuit workload). The input-constants overlay renames to Kernel Input Constants and sources from the zisk testbed — sharded execution keeps every closure feasible, so its rows upload even where the aiur prove OOMs and the row is dropped. Zisk Shards becomes a regular full-set plot (runtime-planned counts, flat at 1 for closures that fit). The sync now warn-skips a measure bencher hasn't seen yet, like benchmarks and testbeds, so it can run at any point mid-migration. CI hardening. bench-main's upload step no longer fails on a cell whose every row OOM'd — the new single-constant aiur cells (Lean, Mathlib) may have that as their steady state — the empty bmf result skips the upload and the run step stays the failure authority. A bench-measures unit suite pins the stage-prefix parsing edge cases. Migration: the renamed measures start fresh bencher series (the old recursive-* measures orphan; other testbeds keep the plain names). After this lands on main, let bench-main finish, then dispatch the plots sync. A !benchmark aiur run before that first bench-main completes shows n/a base columns for the stage measures; no threshold reset is needed.
samuelburnham
force-pushed
the
ci-bencher-aiur
branch
from
August 20, 2026 22:35
019787f to
52cc8c6
Compare
samuelburnham
enabled auto-merge
August 21, 2026 00:21
samuelburnham
disabled auto-merge
August 21, 2026 00:21
…oy recursion harness
The pipeline's stage measures were prefixed ordinally (stage1-/stage2-),
which says nothing about what a stage proves and would leave stage3-/
stage4- equally opaque when the KZG stages land. They rename to the
stage's subject: ixvm-* for the IxVM-typecheck prove, fri-verifier-* for
the in-circuit FRI verifier over that proof (seven measures each:
execute-time, prove-time, throughput, peak-rss, proof-size, verify-time,
fft-cost). The ledger keeps total-time / pipeline-throughput /
pipeline-peak-rss. The `!benchmark` stage tables retitle to "IxVM on
FRI" / "FRI verifier on FRI"; their columns still strip the prefix under
the heading. Plot titles follow ("Aiur IxVM Time", "Aiur FRI Verifier
Time", …).
The prefix machinery simplifies from stage<digits>- parsing to a named
list (stagePrefixes = ixvm-/fri-verifier-/pipeline-) that the KZG stages
extend; formatting kind, units, improvement direction, and stage-table
labels all still derive from the stripped base name, and bench-measures
pins the parsing. Nothing with the short-lived stage<N>- slugs ever
uploaded, so bencher history is untouched.
bench-recursive-verifier is retired: since #567 the aiur benchmark IS
the recursion measurement — bench-typecheck --recursive drives the same
Ix.MultiStark verifier over real constants' proofs — and CI only ever
built and shipped the toy-factorial harness without measuring through it
(its correctness coverage lives in Tests/MultiStark.lean's aiur-prove /
aiur-hashes runners). The exe, its lakefile target, and the build /
package / base-restore plumbing in both bench workflows go with it.
bench-recursion-debug stays: the prove-once/replay debugging harness for
the recursive verifier (from #527), used by hand only. In its remaining
inner-vs-outer split, the local harness's outer metrics take
fri-verifier-* and the inner statement keeps plain names.
Docstring corrections along the way: the aiur registry comment counted
five stage columns (it is seven since throughput landed) and omitted
pipeline-throughput from the ledger; bench-recursion-debug claimed its
--queries default matches the pipeline (the pipeline runs 50 queries
with no query PoW; the tool defaults to 100 with 20 PoW bits — now
documented as such); plot-skip and docs prose name the actual slugs.
johnchandlerburnham
approved these changes
Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The benchmark suite ran different constants per backend, curated through Benchmarks/Vectors.csv's tier/shard_target/primary columns plus a fixed aiur selection, and the aiur pipeline's stage measures collided with the plain Phase-1 names via a mode-scoping convention. This consolidates all of it.
Shared constant set. Ix/BenchConstants.lean (a plain importable array of (name, env)) replaces Vectors.csv as the single selection: every per-constant backend (aiur, zisk, ooc, lean4lean) runs the same constants, now including Std.HashMap, Lean.Json, and Multiset.sort at the heavy end. The tier/shard_target/primary columns, fixedSelections, --full/--tier/--shard-only/--csv flags, and the BENCH_FULL/BENCH_SHARD tokens are gone with the CSV. benchExclusions shrinks to hard infeasibility only: bitblast on the zkVMs, whose ~18B-step atomic mutual block crashes the ZisK executor and cannot be split by the shard cutter. bitblast itself is commented out of the set with a re-add note — its Aiur prove exceeds any current host, so a scheduled run could only re-document the same OOM; --consts/BENCH_CONSTS still run it on demand.
Runtime zisk sharding. With the tier column gone, every zisk constant goes through extract → profile →
ix shard --max-ram <ceiling>at bench time: the planner's RAM budget sizes the partition from the closure's predicted cost, so a closure that fits gets a one-shard plan and runs as a single leaf. Artifacts still pre-cut in bench-main's compile job and ride the zkshards cache; a failed cut falls back to the whole closure under the watchdog. Formerly-light constants now measure from their extracted sub-env (host peak-rss tightens once; guest cycles unchanged).Stage-scoped aiur measures. The pipeline's measures rename from the plain/recursive- split to a uniform stage- prefix (stage1-prove-time, stage2-fft-cost, …) so the KZG stages join as stage3-/stage4- without renames. stagePrefixOf/dropStagePrefix move to the registry and parse the prefix generically; formatting kinds, units, improvement direction, and stage-table labels all derive from the base name. The execute mode keeps plain Phase-1 names on its own testbed, so the modes no longer share measure names at all. Throughput returns as a column in every stage table plus the ledger (stage1-/stage2-throughput, and pipeline-throughput = constants / total-time), emitted by bench-typecheck; none carry thresholds — constants is pinned exactly, so each is the pure inverse of an already-bounded time. bench-recursive-verifier adopts the same stage1-/stage2- keys.
Dashboard. New plots now that #567 merged: Aiur Total Throughput, Aiur Total Peak RAM Usage (which stage sets the ceiling can shift as stages are added), and Aiur Stage 1 Proof Size (it sizes stage 2's in-circuit workload). The input-constants overlay renames to Kernel Input Constants and sources from the zisk testbed — sharded execution keeps every closure feasible, so its rows upload even where the aiur prove OOMs and the row is dropped. Zisk Shards becomes a regular full-set plot (runtime-planned counts, flat at 1 for closures that fit). The sync now warn-skips a measure bencher hasn't seen yet, like benchmarks and testbeds, so it can run at any point mid-migration.
CI hardening. bench-main's upload step no longer fails on a cell whose every row OOM'd — the new single-constant aiur cells (Lean, Mathlib) may have that as their steady state — the empty bmf result skips the upload and the run step stays the failure authority. A bench-measures unit suite pins the stage-prefix parsing edge cases.
Migration: the renamed measures start fresh bencher series (the old recursive-* measures orphan; other testbeds keep the plain names). After this lands on main, let bench-main finish, then dispatch the plots sync. A !benchmark aiur run before that first bench-main completes shows n/a base columns for the stage measures; no threshold reset is needed.