docs: describe the tier system that actually exists - #754
Closed
johnleider wants to merge 1 commit into
Closed
Conversation
The Performance Tiers section documented a scheme replaced in #539 on 7 July: four ops/s buckets split by O(1)/O(n)/O(n2), a complexity-detection table keyed on benchmark names, and a claim that group tiers are the average of their benchmarks. None of that is true. getTier grades cost, not throughput: collection operations on per-item microseconds AND single-operation latency against frame budgets, worst axis wins; one-shot calls on call latency alone against their own budgets. getGroupTier takes the worst, never the average -- averaging is precisely what it exists to avoid, since one flattering microbench would mask a slow path. Replaces the four tier cards and the complexity table with the two budget tables the code implements, and says which thresholds are anchored to something real: 16.7ms is a frame at 60fps and 100ms is the perceptibility threshold, while the per-item and per-call budgets are conventions. Also drops 'pnpm metrics' from Running Benchmarks. It regenerates published artifacts, which now happens only on the reference machine; a reader following that instruction produces numbers CI will reject.
Member
Author
|
Folded into #753 so there is one PR to review rather than two. Same commit, cherry-picked — the docs fix and the numbers it describes belong together. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Performance Tiers section of
guide/fundamentals/benchmarks.mddocuments a scheme that was replaced in #539 on 7 July. Three specific claims on that page are false against current code:What
getTieractually does (scripts/lib/benchmarks.ts:118-151) is grade cost, not throughput — deliberately, because raw ops/s mostly measures how much work a benchmark was handed. Collection operations are scored on per-item microseconds and single-operation latency against frame budgets, with the worse axis winning. One-shot calls have no workload to amortise and are scored on call latency alone.getGroupTiertakes the worst, never the average — averaging is exactly what it exists to prevent, since one flattering microbenchmark would otherwise mask a slow path.This replaces the stale cards and table with the two budget tables the code implements, and adds the distinction that was never written down:
16.7msis a frame at 60fps and100msis the perceptibility threshold, so those are anchored to something real, while the per-item and per-call budgets are engineering conventions. That asymmetry matters when someone proposes moving a threshold.Net −64/+31 lines.
Also drops
pnpm metricsfrom Running Benchmarks. It regenerates the published artifacts, which since #749 happens only on the project's reference machine — a reader following that instruction produces numbersmetrics:checkwill reject on their PR.pnpm test:benchstays, with a note that local numbers are for spotting your own regressions and are not comparable to the published ones.