[lessthanjacob]

Summary

Adds a more robust benchmark suite to get more accurate readings on how changes may affect performance in a variety of situations.

Written by the author.

Changes

Added

• Render benchmark: rake perf times rendering across 20 one-factor sweeps and six composite profiles (107 timed scenarios), varying one factor at a time from a baseline of plain objects. Each scenario is rendered once before timing. If that render raises or runs a database query, the scenario is left out and listed with the reason; otherwise its output is fingerprinted so a later run can tell whether a change altered it. Entry point spec/benchmarks/perf/run.rb.
• Run comparison: BASELINE=<file|latest> and rake perf:compare report each scenario's change in wall time, CPU time, and allocations per rendered object, mark time changes inside either run's IQR as ~, and lead with a Changed table.
• Benchmarking doc: docs/benchmarking.md explains what is measured and how far to trust a difference.

Changed

• Performance gate: AGENTS.md, docs/testing.md, and docs/making-changes.md ask a render-path change for rake perf before-and-after numbers in place of two rake benchmarks figures.
• Development gems: rainbow and ruby-progressbar are declared in the Gemfile for the progress bar and colored summary; both were already in the bundle as rubocop dependencies.

Fixed

• Specs on Ruby 4.0: ostruct is declared in the Gemfile. oj 3.17.8 (2026-11-03) dropped its runtime dependency on it, and it isn't a default gem as of Ruby 4.0, so a fresh bundle couldn't load the specs that require 'ostruct'.

🎯 Review Focus

1. Compare::Row#beyond_noise? in spec/benchmarks/perf/compare.rb ~L38

   A time change counts as real when it exceeds the larger of the two runs' IQR/median, with no fixed floor, so a scenario with very steady timings gets a small cutoff. On two runs of identical code, 7 of 104 rows were flagged, the largest at 7.5%. Should a fixed floor (say 5%) sit under the IQR, given it would also hide a real 4% regression?

2. PerfBench::Fixtures.widgets in spec/benchmarks/perf/fixtures.rb

   Every sweep varies from plain objects with attr_readers; Hash input appears only in the input_kind row and the Hash export profile. Is that enough coverage for a change confined to HashExtractor, or should the core sweeps (size, width, depth) also run on Hash input?

Context

Behavior: new rake perf and rake perf:compare tasks; run locally, not by CI

git switch main && bundle exec rake perf         # full run, ~85 s; results in benchmarks/results/
git switch my-branch                             # then, on the branch:
BASELINE=latest bundle exec rake perf            # compare against the newest results file
BASELINE=a.json CURRENT=b.json bundle exec rake perf:compare
QUICK=1 bundle exec rake perf                    # smoke run: every scenario builds and renders

Comparing 29069ec's :if/:unless memoization fix (reverted as the baseline) with the fixed code, at da2f380:

104 scenarios compared, 100 changed: 100 faster and 0 slower beyond noise, 0 with a different allocation count, 0 with different output.

Warning: Load differs: baseline 29.0, current 7.8 (mean 1-minute load average over each run).

scenario	ms	time	CPU	alloc/obj	alloc	output
size / 100	2.77 -> 1.75	−37%	−37%	2.81 -> 2.81	0%	same
width / 50	4.75 -> 3.05	−36%	−36%	2.81 -> 2.81	0%	same
conditions / Proc if, 0% skipped	2.52 -> 1.95	−23%	−23%	2.81 -> 2.81	0%	same
input_kind / ActiveRecord	6.26 -> 5.10	−18%	−18%	11.3 -> 11.3	0%	same

Why This Approach: a separate local task beside rake benchmarks; paired A/B runs left for a follow-up

• Kept rake benchmarks: it stays the automated floor in CI. Its tests are pass/fail floors (an iterations-per-second minimum or a linearity fit), which say whether a render got slower but not where; rake perf reports per-scenario changes.
• Not in CI: wall time on a shared runner varies more than the changes worth catching. Allocation counts and output digests don't depend on load, so a CI job reporting only those columns is a possible follow-up (docs/benchmarking.md).
• Plain-object baseline: applications usually render model objects, not Hashes, and Hash input takes its own extractor path (HashExtractor) that a render-path change may not touch.
• Follow-up, paired runs: baseline and current are separate runs, so a load change between them reads as a speed change, and the comparison only warns about it. Alternating baseline and current batches per scenario, each in its own subprocess, would let load hit both sides equally.

Testing: CI green on de85267 (Ruby 3.1–4.0); full runs on Ruby 4.0.2, smoke run on 3.1.4

• ✅ CI on de85267: test on Ruby 3.1, 3.2, 3.3, and 4.0, CodeQL, and DCO pass. The first attempt failed on Ruby 4.0 in ActiveRecordBigOTest#bench_render_active_record (linearity 0.9839 against 0.99) with all 410 specs passing; re-running the failed jobs passed. That timing test is unchanged here, and main saw the same kind of failure at 22e824b (BigOTest#bench_render_basic, 0.9706 against 0.98).
• ✅ bundle exec rake on de85267 with oj 3.17.8 and ostruct 0.6.3: 410 examples, 0 failures; RuboCop, 34 files, no offenses. Without the ostruct line, the same bundle fails to load spec/integrations/base_spec.rb and spec/units/extensions_spec.rb, as test (ubuntu-latest, 4.0) did on ff39fcc.
• ✅ PerfBench::Markdown.escape gives the same output before and after ff39fcc on a|b, a\b, a\|b, and plain; the rewrite answers CodeQL rb/incomplete-sanitization, which didn't recognize the chained gsub as escaping backslashes.
• ✅ Two full rake perf runs of identical code at da2f380 (Ruby 4.0.2): 7 of 104 rows flagged, none beyond 7.5%; 0 allocation or output changes.
• ✅ The 29069ec revert pair above: 0 allocation or output changes on any row.
• ✅ QUICK=1 rake perf on Ruby 3.1.4, the oldest in the CI matrix, at 7fb674d: 102 scenarios, exit 0. The one skipped scenario, output / JSON (MultiJson), is skipped on Ruby 4.0.2 too, because multi_json isn't in the bundle.
• ✅ No file under spec/benchmarks/perf/ matches rake benchmarks' spec/benchmarks/**/*_test.rb or the rspec *_spec.rb pattern (git ls-files), so CI runs the same tests as on main; the 4 existing _test.rb files still match.
• ✅ Output fingerprint follows rendered field order: the same blueprint under sort_fields_by = :name_asc twice gives one digest, and under :definition a different one. From the authoring session on 2026-10-29.

Checklist

• I have updated the necessary documentation
• I have signed off all my commits as required by DCO
• My build is green

---

_{current as of de85267}

🤖 Generated with Claude Code

[jhollinger]

This looks great! So the idea is?

1. Run rake perf pre-change
2. Run BASELINE=... rake perf:compare after

[lessthanjacob]

@jhollinger That’s the idea! I’ll likely wrap this up in a skill as a follow up here.