chore(bench): regenerate metrics on the reference host - #753
Merged
Conversation
First regeneration since #749 moved measurement off CI. Every number here was produced on the reference workstation; the series was previously measured on GitHub runners, so this is a machine change, not a code change. prev Intel Xeon 6973P-C (GHA, ci: true) next Intel i9-7980XE (reference host, idle 0.08%) The whole suite reads 39.56% slower at the median. Nothing regressed -- the Xeon is a current-generation server part and the reference box is a 2017 desktop. This is exactly the case the env fingerprint exists to make visible, and the case no scale factor can bridge. Because of that, the per-version history was re-measured with --force rather than topped up: leaving 1.0.0 and 1.0.1 on Xeon numbers while 1.0.2 came from the reference host would have put a 40% step in the middle of every sparkline. All three points now carry the same fingerprint. 1.0.2 also had no snapshot before this. Four of sixteen feature badges demote on identical code, purely because tiers are absolute thresholds and the reference machine is slower: createRegistry blazing -> fast useProxyRegistry blazing -> fast createDataGrid fast -> good createFilter fast -> good metrics.json now carries _env instead of _apparatus.
The first pass recorded governor: powersave in every artifact, which host-guard.ts warns about — the published numbers were taken in a state the apparatus itself flags as untrustworthy. Fixed by setting the governor for the run. Two of the four tier demotions were governor artefacts and are now gone: createRegistry and useProxyRegistry are back to blazing, both having sat within a few percent of their edges. 14 of 16 badges are unchanged from the GHA-measured series. The two that remain are real. createDataGrid (Compute spans, 10,000 items: 39.5ms) and createFilter (Update query 10 times, 10,000 items: 39.3ms per op) both exceed 33.4ms — two frames at 60fps — on this host. That threshold is anchored to something a user can feel, unlike the per-item budgets, so these are the demotions worth keeping.
The Performance Tiers section documented a scheme replaced in #539 on 7 July: four ops/s buckets split by O(1)/O(n)/O(n2), a complexity-detection table keyed on benchmark names, and a claim that group tiers are the average of their benchmarks. None of that is true. getTier grades cost, not throughput: collection operations on per-item microseconds AND single-operation latency against frame budgets, worst axis wins; one-shot calls on call latency alone against their own budgets. getGroupTier takes the worst, never the average -- averaging is precisely what it exists to avoid, since one flattering microbench would mask a slow path. Replaces the four tier cards and the complexity table with the two budget tables the code implements, and says which thresholds are anchored to something real: 16.7ms is a frame at 60fps and 100ms is the perceptibility threshold, while the per-item and per-call budgets are conventions. Also drops 'pnpm metrics' from Running Benchmarks. It regenerates published artifacts, which now happens only on the reference machine; a reader following that instruction produces numbers CI will reject.
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One PR, replacing four. #734, #748 and #754 are closed and folded here or made obsolete by #749.
First regeneration since #749 moved benchmark measurement off CI onto the fixed reference workstation. Every number moved because the machine changed, not the code.
ci: trueperformanceThe suite reads ~40% slower. That is a real single-thread gap, verified independently on the 13 pure-JS calibration anchors from the old artifact — no v0 code involved — which show the same ~30% median deficit, worst on allocation/hash-heavy work (
map build−51.9%,string build−52.2%) and mildest on tight loops (array sum−19.4%). Skylake-X against Granite Rapids, plus Skylake-X's mesh interconnect on cache-sensitive workloads.Tier badges: 14 of 16 unchanged
An earlier pass showed four demotions; two were an artefact of running under the
powersavegovernor, whichhost-guard.tswarns about and which the first artifacts recorded. Re-measured with the governor set:createDataGridcreateFiltercreateRegistryuseProxyRegistryBoth remaining demotions cross 33.4ms — two frames at 60fps. That threshold is anchored to something a user can feel, unlike the per-item budgets which are engineering conventions. On this host those operations genuinely do miss it, so the badges are telling the truth and I'd keep them. Retuning thresholds to recover a badge is the goalpost move.
Why the whole history was re-measured
metrics:history --force, not a top-up.1.0.0and1.0.1were Xeon numbers; adding a reference-host1.0.2beside them would have put a 40% step in the middle of every sparkline. All three points now share one fingerprint.1.0.2had no snapshot at all before this.Also included: the docs page describing tiers was wrong
guide/fundamentals/benchmarks.mddocumented a scheme replaced in #539 back on 7 July — ops/s tier cards bucketed by O(1)/O(n)/O(n²), a complexity-detection table, and a claim that group tiers are the average of their benchmarks. The code grades cost (per-item µs and latency against frame budgets, worst axis wins) and takes the worst group, never the average. Replaced with what the code does, −64/+31 lines. Also dropspnpm metricsfrom that page, since following it now produces artifacts CI rejects.Verification
pnpm metrics:check— ok.pnpm metrics:verifyon two fresh median-of-3 samples: feature median 0.79%, worst 2.42%.-diffin.gitattributes; compare withpnpm metrics:delta, not raw JSON.