Skip to content

bench: one shared constant set, runtime zisk sharding, stage-scoped aiur measures - #582

Merged
johnchandlerburnham merged 4 commits into
mainfrom
ci-bencher-aiur
Aug 21, 2026
Merged

bench: one shared constant set, runtime zisk sharding, stage-scoped aiur measures#582
johnchandlerburnham merged 4 commits into
mainfrom
ci-bencher-aiur

Conversation

@samuelburnham

@samuelburnham samuelburnham commented Aug 20, 2026

Copy link
Copy Markdown
Member

The benchmark suite ran different constants per backend, curated through Benchmarks/Vectors.csv's tier/shard_target/primary columns plus a fixed aiur selection, and the aiur pipeline's stage measures collided with the plain Phase-1 names via a mode-scoping convention. This consolidates all of it.

Shared constant set. Ix/BenchConstants.lean (a plain importable array of (name, env)) replaces Vectors.csv as the single selection: every per-constant backend (aiur, zisk, ooc, lean4lean) runs the same constants, now including Std.HashMap, Lean.Json, and Multiset.sort at the heavy end. The tier/shard_target/primary columns, fixedSelections, --full/--tier/--shard-only/--csv flags, and the BENCH_FULL/BENCH_SHARD tokens are gone with the CSV. benchExclusions shrinks to hard infeasibility only: bitblast on the zkVMs, whose ~18B-step atomic mutual block crashes the ZisK executor and cannot be split by the shard cutter. bitblast itself is commented out of the set with a re-add note — its Aiur prove exceeds any current host, so a scheduled run could only re-document the same OOM; --consts/BENCH_CONSTS still run it on demand.

Runtime zisk sharding. With the tier column gone, every zisk constant goes through extract → profile → ix shard --max-ram <ceiling> at bench time: the planner's RAM budget sizes the partition from the closure's predicted cost, so a closure that fits gets a one-shard plan and runs as a single leaf. Artifacts still pre-cut in bench-main's compile job and ride the zkshards cache; a failed cut falls back to the whole closure under the watchdog. Formerly-light constants now measure from their extracted sub-env (host peak-rss tightens once; guest cycles unchanged).

Stage-scoped aiur measures. The pipeline's measures rename from the plain/recursive- split to a uniform stage- prefix (stage1-prove-time, stage2-fft-cost, …) so the KZG stages join as stage3-/stage4- without renames. stagePrefixOf/dropStagePrefix move to the registry and parse the prefix generically; formatting kinds, units, improvement direction, and stage-table labels all derive from the base name. The execute mode keeps plain Phase-1 names on its own testbed, so the modes no longer share measure names at all. Throughput returns as a column in every stage table plus the ledger (stage1-/stage2-throughput, and pipeline-throughput = constants / total-time), emitted by bench-typecheck; none carry thresholds — constants is pinned exactly, so each is the pure inverse of an already-bounded time. bench-recursive-verifier adopts the same stage1-/stage2- keys.

Dashboard. New plots now that #567 merged: Aiur Total Throughput, Aiur Total Peak RAM Usage (which stage sets the ceiling can shift as stages are added), and Aiur Stage 1 Proof Size (it sizes stage 2's in-circuit workload). The input-constants overlay renames to Kernel Input Constants and sources from the zisk testbed — sharded execution keeps every closure feasible, so its rows upload even where the aiur prove OOMs and the row is dropped. Zisk Shards becomes a regular full-set plot (runtime-planned counts, flat at 1 for closures that fit). The sync now warn-skips a measure bencher hasn't seen yet, like benchmarks and testbeds, so it can run at any point mid-migration.

CI hardening. bench-main's upload step no longer fails on a cell whose every row OOM'd — the new single-constant aiur cells (Lean, Mathlib) may have that as their steady state — the empty bmf result skips the upload and the run step stays the failure authority. A bench-measures unit suite pins the stage-prefix parsing edge cases.

Migration: the renamed measures start fresh bencher series (the old recursive-* measures orphan; other testbeds keep the plain names). After this lands on main, let bench-main finish, then dispatch the plots sync. A !benchmark aiur run before that first bench-main completes shows n/a base columns for the stage measures; no threshold reset is needed.

…iur measures

The benchmark suite ran different constants per backend, curated through
Benchmarks/Vectors.csv's tier/shard_target/primary columns plus a fixed
aiur selection, and the aiur pipeline's stage measures collided with the
plain Phase-1 names via a mode-scoping convention. This consolidates all
of it.

Shared constant set. Ix/BenchConstants.lean (a plain importable array of
(name, env)) replaces Vectors.csv as the single selection: every
per-constant backend (aiur, zisk, ooc, lean4lean) runs the same
constants, now including Std.HashMap, Lean.Json, and Multiset.sort at
the heavy end. The tier/shard_target/primary columns, fixedSelections,
--full/--tier/--shard-only/--csv flags, and the BENCH_FULL/BENCH_SHARD
tokens are gone with the CSV. benchExclusions shrinks to hard
infeasibility only: bitblast on the zkVMs, whose ~18B-step atomic mutual
block crashes the ZisK executor and cannot be split by the shard cutter.
bitblast itself is commented out of the set with a re-add note — its
Aiur prove exceeds any current host, so a scheduled run could only
re-document the same OOM; --consts/BENCH_CONSTS still run it on demand.

Runtime zisk sharding. With the tier column gone, every zisk constant
goes through extract → profile → `ix shard --max-ram <ceiling>` at bench
time: the planner's RAM budget sizes the partition from the closure's
predicted cost, so a closure that fits gets a one-shard plan and runs as
a single leaf. Artifacts still pre-cut in bench-main's compile job and
ride the zkshards cache; a failed cut falls back to the whole closure
under the watchdog. Formerly-light constants now measure from their
extracted sub-env (host peak-rss tightens once; guest cycles unchanged).

Stage-scoped aiur measures. The pipeline's measures rename from the
plain/recursive- split to a uniform stage<N>- prefix
(stage1-prove-time, stage2-fft-cost, …) so the KZG stages join as
stage3-/stage4- without renames. stagePrefixOf/dropStagePrefix move to
the registry and parse the prefix generically; formatting kinds, units,
improvement direction, and stage-table labels all derive from the base
name. The execute mode keeps plain Phase-1 names on its own testbed, so
the modes no longer share measure names at all. Throughput returns as a
column in every stage table plus the ledger (stage1-/stage2-throughput,
and pipeline-throughput = constants / total-time), emitted by
bench-typecheck; none carry thresholds — constants is pinned exactly,
so each is the pure inverse of an already-bounded time.
bench-recursive-verifier adopts the same stage1-/stage2- keys.

Dashboard. New plots: Aiur Total Throughput, Aiur Total Peak RAM Usage
(which stage sets the ceiling can shift as stages are added), and Aiur
Stage 1 Proof Size (it sizes stage 2's in-circuit workload). The
input-constants overlay renames to Kernel Input Constants and sources
from the zisk testbed — sharded execution keeps every closure feasible,
so its rows upload even where the aiur prove OOMs and the row is
dropped. Zisk Shards becomes a regular full-set plot (runtime-planned
counts, flat at 1 for closures that fit). The sync now warn-skips a
measure bencher hasn't seen yet, like benchmarks and testbeds, so it
can run at any point mid-migration.

CI hardening. bench-main's upload step no longer fails on a cell whose
every row OOM'd — the new single-constant aiur cells (Lean, Mathlib) may
have that as their steady state — the empty bmf result skips the upload
and the run step stays the failure authority. A bench-measures unit
suite pins the stage-prefix parsing edge cases.

Migration: the renamed measures start fresh bencher series (the old
recursive-* measures orphan; other testbeds keep the plain names). After
this lands on main, let bench-main finish, then dispatch the plots sync.
A !benchmark aiur run before that first bench-main completes shows n/a
base columns for the stage measures; no threshold reset is needed.
…oy recursion harness

The pipeline's stage measures were prefixed ordinally (stage1-/stage2-),
which says nothing about what a stage proves and would leave stage3-/
stage4- equally opaque when the KZG stages land. They rename to the
stage's subject: ixvm-* for the IxVM-typecheck prove, fri-verifier-* for
the in-circuit FRI verifier over that proof (seven measures each:
execute-time, prove-time, throughput, peak-rss, proof-size, verify-time,
fft-cost). The ledger keeps total-time / pipeline-throughput /
pipeline-peak-rss. The `!benchmark` stage tables retitle to "IxVM on
FRI" / "FRI verifier on FRI"; their columns still strip the prefix under
the heading. Plot titles follow ("Aiur IxVM Time", "Aiur FRI Verifier
Time", …).

The prefix machinery simplifies from stage<digits>- parsing to a named
list (stagePrefixes = ixvm-/fri-verifier-/pipeline-) that the KZG stages
extend; formatting kind, units, improvement direction, and stage-table
labels all still derive from the stripped base name, and bench-measures
pins the parsing. Nothing with the short-lived stage<N>- slugs ever
uploaded, so bencher history is untouched.

bench-recursive-verifier is retired: since #567 the aiur benchmark IS
the recursion measurement — bench-typecheck --recursive drives the same
Ix.MultiStark verifier over real constants' proofs — and CI only ever
built and shipped the toy-factorial harness without measuring through it
(its correctness coverage lives in Tests/MultiStark.lean's aiur-prove /
aiur-hashes runners). The exe, its lakefile target, and the build /
package / base-restore plumbing in both bench workflows go with it.
bench-recursion-debug stays: the prove-once/replay debugging harness for
the recursive verifier (from #527), used by hand only. In its remaining
inner-vs-outer split, the local harness's outer metrics take
fri-verifier-* and the inner statement keeps plain names.

Docstring corrections along the way: the aiur registry comment counted
five stage columns (it is seven since throughput landed) and omitted
pipeline-throughput from the ledger; bench-recursion-debug claimed its
--queries default matches the pipeline (the pipeline runs 50 queries
with no query PoW; the tool defaults to 100 with 20 PoW bits — now
documented as such); plot-skip and docs prose name the actual slugs.
@johnchandlerburnham
johnchandlerburnham added this pull request to the merge queue Aug 21, 2026
Merged via the queue into main with commit c571fa3 Aug 21, 2026
13 checks passed
@johnchandlerburnham
johnchandlerburnham deleted the ci-bencher-aiur branch August 21, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants