Ingest benchmark parameters and named metric values

[epompeii]

Benchmark parameters and named metric values become rows. The schema underneath is settled; this layer is behavior: ingest, detection, billing, and the report response.

Fourth of the benchmark parameters stack. Targets benchmark-parameters/bmf-v1 and stays a draft while the layers below it are drafts.

The one decision this layer cannot make for itself

Flagged rather than decided, and it gates merge: what the legacy metric-count meter does with a measure that names no point estimate. That is under billing below. The two other open decisions of earlier rounds, a missing ceiling on parameter creation and the nullability of the deprecated metric field, are both closed.

An optional metric field and a value-less measure

A BMF v1 measure may name only p99 and never mention value at all, which the design states in as many words, and ingest is best effort by rule, so such a measure is accepted rather than refused. It has no value row for the deprecated metric to be reconstructed from.

JsonReportMeasure.metric is optional, and it is absent exactly when the measure carries no value name. Such a measure appears in the report response like any other, carrying its named values in metrics with no deprecated triple beside them. The field is skipped rather than serialized as null, matching the other optional fields on JsonReport. threshold and boundary are already optional and come back null there, as they do for any ungated measure: a bare threshold gates only the value name, so a value-less measure has no boundary to report. That is asserted rather than reasoned.

The alternative was leaving such a measure out of the results, which an earlier round of this branch shipped. It made data that is stored, billed as an active series, and queryable invisible in the response that created it. Filling metric from another named scalar was never on the table: it asserts a value the user never reported, which is exactly the invention the fold ruling refuses.

Why this is compatible. The field can only ever be absent on a measure with no value name, which is a shape only a BMF v1 payload can create. Every legacy response, meaning anything reachable from a BMF v0 payload on any CLI age, keeps the field exactly as before. A pinned older CLI handed a v1 file would see the absent field and fail to deserialize the response after ingest succeeds; that is a new-format adoption error rather than a legacy break, and the v1 reference docs will state that BMF v1 requires a current CLI. v0_measure_response_is_unchanged pins that claim: it captures the whole measure object of a BMF v0 report, with only the minted UUIDs and the measure entity normalized, and asserts it byte for byte, the populated metric and the null threshold and boundary included. The regenerated spec has JsonReportMeasure.required as ["measure", "metrics"] with metric nullable, and metrics and measure both still required.

The counts follow the results, which is what they always had to do. counts.measures is computed two ways, from the loaded results on the report endpoint and by aggregate query on the list endpoint, and into_json_full_and_collapsed_agree exists to keep the two saying the same thing. A value-less measure is returned, so it is counted: the aggregate counts every measure with a metric row, which is precisely the set the results carry. No BMF v0 project's counts move, because a v0 measure always names a value.

The parameter creation ceiling

Parameter rows are minted straight from report content, one per grid point, and until this round nothing bounded them. QueryBenchmark, QueryMeasure, QueryBranch, QueryTestbed, QueryThreshold, and QueryReport all carry a per project ceiling; the report-level limit bounds reports per window, not entities minted inside a single payload, which is why each of those needs its own. A harness that interpolates a commit sha or a timestamp into parameters would otherwise mint a fresh parameter row per report without limit, and every new grid point is also a new billable active series.

parameter has no project_id, so fn_rate_limit! does not apply as written. The ceiling is hand written instead and counts the window through the join that already exists: a parameter set belongs to its benchmark, and the benchmark belongs to the project. The limits and the error are the ones every other resource a report mints already uses, so a project that trips it gets exactly the message it gets today for a flood of benchmark names.

Two consequences worth stating.

No existing project is touched. A BMF v0 payload only ever resolves the empty parameter set, which every benchmark is born with and which is returned before any creation is attempted, so the ceiling is only ever reached by a payload that names parameters.

The count includes the empty parameter set each benchmark is born with, which makes the ceiling slightly conservative for a project creating benchmarks and grid points in the same window. That is the safe direction, and it costs a project nothing the benchmark ceiling was not already going to cost it.

The alternatives were denormalizing project_id onto parameter, which reopens a settled lower layer's schema for a count, and recording the gap as accepted, which leaves the same unbounded path fn_rate_limit! exists to close.

Ingest

Each BMF v1 entry resolves to its parameter row during the existing phase 1 preparation, alongside QueryBenchmark::get_or_create and QueryMeasure::get_or_create, never inside the phase 2 write transaction. That placement is forced: write_transaction! does not nest, and QueryParameter::get_or_create opens one of its own to insert a new parameter set. An entry with no parameters resolves to the benchmark's empty parameter set, which already exists by the birth invariant and is never created lazily; a missing empty set is data corruption and is reported as such rather than minted. A resolved parameter row that is archived is unarchived, exactly as QueryBenchmark::get_or_create does.

The parameter cache keys on (benchmark, parameter set) rather than on the benchmark, since one benchmark has as many grid points as it has parameter sets.

Named metric rows are one metric row per name per measure per report_benchmark. The value row is written first so last_insert_rowid still names the row a boundary attaches to.

Duplicate grid points. Two v1 entries that canonicalize to the same parameter set and carry the same measure now union at the metric-name level, last wins per name. Previously the later entry replaced the whole AdapterMetric, so [{measures:{latency:{p50:1.0}}}, {measures:{latency:{p99:2.0}}}] yielded p99 alone with a drop count of zero. Nothing is dropped now, so nothing needs counting. The comment at the merge site was narrowed from measure granularity to name granularity.

Detection

Baselines key on (benchmark, parameter, measure): the historical query in detector/data.rs gains report_benchmark.parameter_id, so one grid point's sample is never the benchmark's grid pooled.

Bare thresholds gate only the conventional value series, of every parameter set under their measure. That is exactly what a measure-level threshold over flat benchmarks has always done, so no project's alert volume changes. Threshold matching itself is untouched: still (branch, testbed, measure), and threshold gains no columns.

Alert-parity evidence. alert_volume_is_identical_for_flat_benchmarks_and_grid_points ingests the same six reports of the same two measurement series twice: once as two flat benchmarks (bench_16, bench_32) under one measure-level threshold, and once as one benchmark's two parameter sets under the same threshold. Both projects raise exactly one alert, and the grid project's alert is on the grid point that regressed.

Per grid point baselines, proved separately. The parity fixture cannot tell a per grid point baseline from a pooled one: its two grid points are a decade apart and the regression is tenfold, so it is an outlier either way. baselines_separate_by_parameter therefore has its own numbers, chosen so that pooling hides the regression: two tight histories two decades apart, [10, 11, 12, 13, 14] beside [1000, 1001, 1002, 1003, 1004], with the small grid point finishing at 50 and the large one steady at 1004. Fifty is a large outlier against its own history and sits well inside the pooled spread. Delete the parameter_id filter from detector/data.rs and the test goes from one alert to none, which is recorded below.

Billing

series_last_seen is recreated with parameter_id in its primary key, per the table-recreate convention, with every existing row backfilled to its benchmark's empty parameter set and all four indexes re-created (plus a fifth on the new parameter_id foreign key). The backfill joins parameter on the empty set, which is an inner join because every benchmark has one: it is born with it, and the parameter migration backfilled every benchmark that predates the birth invariant. last_seen carries over untouched, so no series is resurrected and none is lost. down.sql collapses grid points back into one series per (testbed, benchmark, measure), keeping the greatest last_seen of the rows that merge.

Each grid point bills as its own series, which preserves today's economics exactly: a project whose grid points are currently flat benchmarks bills the same after migrating to parameters. Named values collapse into their measure's series and are not billed; the cap of eight is the guardrail on that.

The four series columns now travel together as a SeriesKey, because they are one identity and they are what the primary key is.

The backfill itself is pinned against a legacy database by migration_backfill_preserves_every_series_row, described under the tests below.

What happened to metric-count billing

The legacy metric-count meter (QueryMetric::usage, used for Team, metered Enterprise, and licensed entitlements) already filters on name = 'value', so named values do not raise a v1 project's metric count. A measure carrying value, lower_value, upper_value, and p99 counts one, which is exactly what the row-per-measure table counted. That filter predates this layer.

The reachable change runs the other way. A BMF v1 measure may name only p99 and never name value. Such a measure counts zero on QueryMetric::usage, zero in metric_count_by_report, and zero in the in-request usage the plan check and the metrics.create counter see, while still billing as an active series on Pro. That is a meter hole that no payload could reach before this layer, and it is not obviously the intended meter.

Nothing was adjusted for it. QueryMetric::usage is untouched. The only alignment made was to iteration_metric_count, which now counts measures that named a value rather than measures outright, so the in-request meter and the billing read cannot drift from each other. named_values_do_not_change_the_metric_count pins both halves of what happens today. This is flagged for a decision, not decided here.

The report response

JsonReportMeasure gains metrics: Vec<JsonReportMetric>, one entry per named scalar in lexicographic order, each carrying boundaries: Vec<JsonReportBoundary> that pairs every threshold that gated it with the boundary it produced. The list is plural now so the wire never breaks again when threshold predicates ship; it is length 0 or 1 until then.

It retains metric, threshold, and boundary as deprecated fields reconstructed from the value row and its lower_value/upper_value siblings. Nothing in the repository uses deny_unknown_fields, so the additive fields are invisible to existing clients. For a measure that named no value, metric is absent and threshold and boundary are null, per the section above; all three are present for everything an older client can produce.

The results are assembled in a pending shape and finished at the end, because the deprecated triple is only in hand once every row of a measure has been read.

The grouping fix this layer owed regardless of response shape. into_report_results_json grouped by consecutive rows sharing a benchmark uuid, so two parameter sets under one benchmark silently merged into one JsonReportResult. The ordering is now (iteration, benchmark.name, parameter.id, measure.name, metric.name) and the grouping keys on the grid point, so two grid points are two results. JsonReportResult carries parameter: JsonReportParameter { uuid, parameters }, so the response is never ambiguous about which grid point a result belongs to.

Both halves of that fix are pinned. report_response_echoes_named_values_and_separates_grid_points runs two measures across two parameter sets, which is the smallest fixture that can see the ordering: with one measure per grid point the rows arrive grouped whether or not the parameter is in the ORDER BY. With two, dropping the parameter lets the measure name outrank it, the rows interleave by grid point, and the consecutive row grouping emits four results of one measure each. The test asserts two results each carrying both measures.

The counts follow: JsonReportIterationCounts.benchmarks is now a count of distinct report_benchmark.id, which is exactly one per grid point, so the aggregate-query count and the count taken from the loaded results still agree. into_json_full_and_collapsed_agree covers that, and full_and_collapsed_counts_agree_for_a_value_less_measure covers the measure count when a measure carries no value name.

The alerts query still reads through the metric_boundary view, since alerts only ever attach to value rows, which is what the view keys on.

Fold

Per the ruling, --fold stays a BMF v0 operation. A v1 payload with fold requested warns and ingests unfolded: each iteration lands as its own report_benchmark row, exactly as it would without the flag. Never an error, and never a silently dropped named value. The fold machinery is not removed and v0 fold works exactly as before; fold over v0 keys on the benchmark and its empty parameter set by construction, since a v0 payload only ever carries the empty set. v0_fold_still_folds is the end-to-end guard on that promise, described under the tests below. The CLI-side deprecation warning is a following layer.

The cap's log and counter

The adapter truncates and reports a drop count. This layer emits the log line and increments a new metrics.names.dropped counter, because this is where the logger and the meter are in scope. Never an ingest error: named_value_cap_does_not_fail_the_report posts a measure with ten names and gets a report back with eight.

Tests, red before green

Commit 8fce999 is tests only, and every hunk in it is inside a #[cfg(test)] module or a test binary. How each area failed there:

• The ten end-to-end ingest tests failed with 400 against 201, from the "not yet supported on ingest" refusal this layer removes.
• The two adapter union tests failed on assertions: p50 was missing from the unioned names.
• grid_points_count_as_distinct_series and named_values_collapse_into_their_measure_series failed on assertions, 1 against 3 and 1 against 2, because the cache and its oracle both keyed without the parameter.
• backfill_maps_existing_rows_to_the_empty_parameter_set failed at runtime with no such column: s.parameter_id, deliberately written as raw SQL so the red failure is a runtime one rather than a compile error across the crate.

named_values_do_not_change_the_metric_count was added after the implementation, as a probe that records what the legacy meter does rather than as a red-first invariant.

The two hardened tests, proved by mutation rather than by red

Commit 4aaa122 changes no production code. It rewrites two tests that named an invariant they could not observe, so they cannot be red against a correct implementation; the gap was in the tests, and the proof is that each now fails when the code it describes is removed. Both mutations were applied to a clean tree, built, run, and reverted.

Mutation	Test	Before	After
Delete .filter(schema::report_benchmark::parameter_id.eq(parameter_id)) from detector/data.rs	baselines_separate_by_parameter	passed	fails, left: [] against right: [({"size_mb": 16}, "value")]
Delete schema::parameter::id from the results ORDER BY in report/mod.rs	report_response_echoes_named_values_and_separates_grid_points	passed	fails, left: 4 against right: 2 results

Unmutated, every test in the file passes.

This round: the migration backfill, and the ceiling

Two more commits, in that order.

Rate limit parameter creation per project follows the red-first rule literally. The test commit before it posts one report over the ceiling and expects the refusal a benchmark name flood already gets; it failed there with left: 201 against right: 429. The test server grew a creation limit knob in the same commit, because every existing test server has its limits wide open and that is the only way to reach a ceiling without throttling the requests that get there.

Pin the series migration backfill against a legacy database cannot be committed red, because it pins behavior that is already correct. The gap it closes is that the existing backfill_maps_existing_rows_to_the_empty_parameter_set runs against a database where every migration has already been applied, so its rows are inserted afterwards and the backfill statement never executes against one: deleting the join leaves it green. The new test reverts to the series migration, seeds rows in the shape the old schema had, keyed on testbed, benchmark, and measure with no parameter at all, and re-applies. The empty parameter sets those rows have to land on are the ones Diesel minted rather than ones spelled with jsonb('{}') in SQL, which is exactly where the two encodings have to agree. Two mutations stand in for the red commit:

Mutation of the backfill	Result
AND 1 = 0 on the join, so it matches nothing	fails: every legacy row is silently dropped
Drop p.benchmark_id = s.benchmark_id from the join	fails: every row lands on every benchmark's empty set, four rows where there should be two

What a dropped row costs in production is worth naming: it is not an error. An organization's active-series count for the current period reads zero until every series reports again, and the upgrade looks clean throughout.

An earlier round: v0 fold, and a first pass at the value-less measure

Commit 697883d is tests only. value_less_measure_is_stored_but_not_echoed failed there on an assertion, left: [Some("latency"), Some("throughput")] against right: [Some("latency")]: the measure that named no point estimate was echoed without a metric.

v0_fold_still_folds rides along in that commit as a probe rather than a red-first invariant, because it pins behavior that is already correct. It is the test the fold ruling was owed: --fold is deprecated, not deleted, so a BMF v0 report must fold exactly as it always has. Two BMF v0 iterations with fold: min land as one report_benchmark row on the benchmark's empty parameter set, carrying the smaller iteration's whole triple, and metering one metric, which is the half of the ruling that says no bill moves. The conversion it covers, From<FoldableResults> for AdapterResults, is new in this layer and every folded v0 report passes through it. Two mutations stand in for the red commit:

Mutation of the fold conversion	Result
Yield an empty results map, so a folded report lands no rows at all	fails: left: [] against right: [({}, 1)]
Key the folded results on a parameter set other than the empty one	fails: the row lands on a minted grid point, left carries two parameter sets where there should be one

Before this test, the first of those mutations left the entire workspace green: a pipeline running bencher run --fold min would have landed reports with no metrics, no boundaries, no alerts, and a metric count of zero, at 201 with a green build.

Commit d515105 is then tests only again, for the counts. full_and_collapsed_counts_agree_for_a_value_less_measure failed there on the two counts disagreeing, measures: 1 from the loaded results against measures: 2 from the aggregate query, which is the defect the following commit fixes.

Those four commits took the response call in the other direction, leaving a value-less measure out of the results and out of the counts. The round below reverses it. v0_fold_still_folds and its mutation evidence stand unchanged.

This round: a value-less measure is returned

Commit 71e07c5 is tests only, and every hunk in it is inside a #[cfg(test)] module or a test binary. How each failed there:

Test	Red failure
value_less_measure_is_stored_billed_and_echoed	assertion, left: [Some("latency")] against right: [Some("latency"), Some("throughput")]
full_and_collapsed_counts_agree_for_a_value_less_measure	assertion, left: 1 against right: 2 measures returned
report_table_value_less_measure	missing field metric

The comment test is red on the wire contract itself rather than on an assertion: it builds its report results from JSON rather than from constructors, precisely because the absent deprecated field is what is under test, so a required metric fails it at deserialization.

v0_measure_response_is_unchanged rides along in that commit as a probe rather than a red-first invariant, because it pins the compatibility claim, which is already true and has to stay true.

The counts expectation flips deliberately, and it is the one behavior of the earlier round that this reverses rather than extends: a value-less measure is counted because it is returned, so get_report_counts drops the name = 'value' filter that round added.

The three views a report has of the same measure are asserted together, since the whole point of the change is that they agree: value_less_measure_is_stored_billed_and_echoed reads the metric rows that ingest wrote, the series_last_seen row that bills them, and the measure the response carries. The legacy metric-count meter is pinned in the same test at one, unchanged and still the open question under billing above.

bencher_comment renders such a report without panicking. There is no point estimate to draw, so the measure takes no column in the results table and the report's missing-threshold warning still names it, which is what the code did before the field was ever required.

Changes to a lower layer's tests

Two are worth a reviewer's attention.

The single-valued metric equivalence test no longer captures the report response. metric_migration.rs seeds a database in the pre-migration shape, captures the raw bytes of every response that reads a metric, migrates, and captures again. The report response can no longer participate: it is built from the named metric rows, and those rows do not exist before the migration, so it has no pre-migration form to compare. It is also not stable across a down-and-up round trip, because the migration mints a fresh uuid for every bound row it recreates and the response now echoes those uuids. Neither is a regression: a server runs its migrations before it serves, and the value row, which every reader of the old shape saw, keeps its identity. The perf, metrics, and alerts responses still capture on both sides and still pin the equivalence, since they all read through the metric_boundary view whose column list that migration holds unchanged.

revert_migration now reverts down to the metric migration rather than reverting the last one. The metric migration is no longer last, so reverting only the last one would revert this layer's instead. Same pattern the parameter layer already uses.

Gates

Gate	Outcome
cargo nextest run	2080 passed, 3 skipped
cargo nextest run -p bencher_schema --features plus	198 passed
cargo nextest run -p api_projects --features plus	205 passed
cargo test --doc	passed
cargo fmt --check	clean
cargo clippy --no-deps --all-targets --all-features -- -Dwarnings	clean
cargo check --no-default-features	clean
cargo gen-types	ran; the regenerated services/api/openapi.json and services/console/src/types/bencher.ts are committed
cargo deny check	not run; no dependency changed
Console typecheck	tsc --noEmit, 435 errors with this branch's bencher.ts and 435 with the previous commit's, an identical set once line numbers are normalized, so no new error. The console has a pre-existing backlog that this is measured against rather than into

bencher_client and bencher_cli build against the regenerated spec, so progenitor accepts the new JsonParameters schema (an object whose additional properties are a string, number, or bool).

One lint suppression

ApiCounter::description crossed clippy::too_many_lines when the new counter's arm was added, and carries #[expect(clippy::too_many_lines, reason = "exhaustive match over every counter variant")], matching how exhaustive matches are already handled elsewhere in the workspace.