Skip to content

chore(benchmark): compare branches against the noise of the same code - #888

Open
nigrosimone wants to merge 4 commits into
fastify:mainfrom
nigrosimone:bench-noise
Open

nigrosimone wants to merge 4 commits into
fastify:mainfrom
nigrosimone:bench-noise

Conversation

@nigrosimone

@nigrosimone nigrosimone commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

npm run bench:cmp benchmarks one branch and then the other, and marks the rows that moved more than 5%. On a laptop the same code moves more than that between two runs (here a fixed CPU loop varied up to 2x), so which branch wins is a lottery.

Now each benchmark runs in four workers, two on the other revision (A) and two on the working tree (B), that take turns in batches of about 10ms. A/A and B/B are the same code against itself and give the noise band of A/B: a row is faster or slower only outside the band, and by at least what is left of A/B. This is #886 against main on Node 26:

short array of numbers...................................... A/A ±2.8%  B/B ±1.9%  A/B +4.5% ±3.1%  faster by at least 1.4%
short array of integers..................................... A/A ±2.4%  B/B ±1.5%  A/B +19.0% ±2.5%  faster by at least 16.1%
short array of short strings................................ A/A ±2.1%  B/B ±2.2%  A/B +235.7% ±2.7%  faster by at least 226.8%
short array of long strings................................. A/A ±3.3%  B/B ±4.5%  A/B -46.5% ±5.0%  slower by at least 43.8%
short array of objects with properties of different types... A/A ±3.0%  B/B ±2.2%  A/B +29.6% ±3.3%  faster by at least 25.5%
simple object............................................... A/A ±1.9%  B/B ±1.3%  A/B +0.7% ±2.1%  noise

Four of the five arrays #886 rewrites are faster and the long strings are slower: each call writes 39MB, so how much depends on the GC, 0.76x in separate processes with the default heap. 23 of the 24 rows where #886 writes the same code read noise, and on Node 24, where it writes the same code everywhere, all 30 do. The long strings row used the short strings, fixed here. bench:cmp:ci, cli-select and simple-git are removed, npm run bench:cmp -- --against <ref> covers them.

@nigrosimone
nigrosimone marked this pull request as ready for review September 26, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant