Short, opinionated notes on benchmark methodology — one failure mode per note, each with a claim, a failure example, evidence, and the counter-case where the advice inverts.
Written after watching our own numbers lie to us, in both directions.
| Note | The failure it prevents |
|---|---|
| median-over-mean | a single 40 ms hiccup becoming your headline number |
| warmup-and-jit | measuring the JIT compiler and calling it your kernel |
| checksum-anti-dce | timing an empty loop the optimizer left behind |
| ffi-boundary-crossover | "WASM is faster" at exactly one workload size |
| publish-the-negative | a wins-only table nobody can believe |
| ci-is-a-detector-not-a-ruler | publishing CI-runner noise as a performance claim |
| repeats-and-confidence | "we ran it 7 times" as a substitute for a precision budget |
| pin-the-model-not-the-alias | benchmarking a pointer that moves |
| thermal-state-is-a-variable | the afternoon re-run that "regressed" 12% |
| cache-aware-cost-accounting | a per-task price that only exists while the cache is warm |
| agent-benchmarks-measure-the-stack | quoting a scaffold score as a model score |
| the-environment-block-is-part-of-the-result | a number with no machine attached |
| paired-comparisons | batch A then batch B, and measure the cooling curve |
| the-benchmark-code-is-the-result | a cited "10x faster" chart whose harness was never published |
| where-in-the-distribution | a "10% cheaper" headline whose median run got dearer |
| measure-outflow-not-inflow | "active community" inferred from open PRs |
- Every note carries a failure example — advice without a way it goes wrong is a slogan.
- Every note carries a counter-note — the condition under which the advice inverts. Methodology without inversion conditions is dogma.
- Numbers quoted from real projects link to the public artifact they came from, with machine, runtime and profile in the caption.
Orvii — Open, Research, Vision, Innovation & Ideas. Contributions welcome: a new note needs a claim, a failure example, evidence and a counter-note. Anything less is a tweet.
Part of the Orvii research set: harness-atlas · convention-map · equivalence-notes · provider-reliability · context-file-evidence · retractions · svg-instruments · ts-lto-research.