Skip to content

docs: describe the tier system that actually exists - #754

Closed
johnleider wants to merge 1 commit into
masterfrom
docs/benchmark-tiers
Closed

docs: describe the tier system that actually exists#754
johnleider wants to merge 1 commit into
masterfrom
docs/benchmark-tiers

Conversation

@johnleider

Copy link
Copy Markdown
Member

The Performance Tiers section of guide/fundamentals/benchmarks.md documents a scheme that was replaced in #539 on 7 July. Three specific claims on that page are false against current code:

  • Four tier cards keyed on ops/s, bucketed by O(1) / O(n) / O(n²)
  • A Complexity Detection table that infers complexity from patterns in the benchmark name
  • "Group tiers are the average of their individual benchmark tiers"

What getTier actually does (scripts/lib/benchmarks.ts:118-151) is grade cost, not throughput — deliberately, because raw ops/s mostly measures how much work a benchmark was handed. Collection operations are scored on per-item microseconds and single-operation latency against frame budgets, with the worse axis winning. One-shot calls have no workload to amortise and are scored on call latency alone. getGroupTier takes the worst, never the average — averaging is exactly what it exists to prevent, since one flattering microbenchmark would otherwise mask a slow path.

This replaces the stale cards and table with the two budget tables the code implements, and adds the distinction that was never written down: 16.7ms is a frame at 60fps and 100ms is the perceptibility threshold, so those are anchored to something real, while the per-item and per-call budgets are engineering conventions. That asymmetry matters when someone proposes moving a threshold.

Net −64/+31 lines.

Also drops pnpm metrics from Running Benchmarks. It regenerates the published artifacts, which since #749 happens only on the project's reference machine — a reader following that instruction produces numbers metrics:check will reject on their PR. pnpm test:bench stays, with a note that local numbers are for spotting your own regressions and are not comparable to the published ones.

The Performance Tiers section documented a scheme replaced in #539 on 7 July:
four ops/s buckets split by O(1)/O(n)/O(n2), a complexity-detection table
keyed on benchmark names, and a claim that group tiers are the average of
their benchmarks.

None of that is true. getTier grades cost, not throughput: collection
operations on per-item microseconds AND single-operation latency against
frame budgets, worst axis wins; one-shot calls on call latency alone against
their own budgets. getGroupTier takes the worst, never the average --
averaging is precisely what it exists to avoid, since one flattering
microbench would mask a slow path.

Replaces the four tier cards and the complexity table with the two budget
tables the code implements, and says which thresholds are anchored to
something real: 16.7ms is a frame at 60fps and 100ms is the perceptibility
threshold, while the per-item and per-call budgets are conventions.

Also drops 'pnpm metrics' from Running Benchmarks. It regenerates published
artifacts, which now happens only on the reference machine; a reader
following that instruction produces numbers CI will reject.
@johnleider johnleider added this to the v1.0.x milestone Jul 30, 2026
@johnleider johnleider added the documentation Improvements or additions to documentation label Jul 30, 2026
@johnleider johnleider self-assigned this Jul 30, 2026
@johnleider

Copy link
Copy Markdown
Member Author

Folded into #753 so there is one PR to review rather than two. Same commit, cherry-picked — the docs fix and the numbers it describes belong together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant