Skip to content

Improve benchmark reliability - #14

Open
RowDaBoat wants to merge 3 commits into
mainfrom
benchmark-update
Open

RowDaBoat wants to merge 3 commits into
mainfrom
benchmark-update

Conversation

@RowDaBoat

@RowDaBoat RowDaBoat commented Sep 7, 2026 •

Copy link
Copy Markdown
Owner

Finding

The previous comparison labeled any median change above a fixed 5% as significant, while its CSV retained only one median per benchmark. A same-source CI comparison reproduced false 5.1-5.5% regressions, and a local same-binary run produced false 13-18% improvements in churn rows.

The first revision preserved inner samples, but an optimization trial exposed a second issue: samples from one process are correlated and cannot stand in for independent runs. Two process medians moved by several percent while the pooled samples still produced directional verdicts.

Changes

  • Batch fast read and iteration workloads to roughly 1 ms per timing sample while reporting normalized operation time.
  • Preserve raw timing and memory samples for diagnostics and preserve one median per independent process run.
  • Compare 15 matched process pairs, alternating candidate/baseline execution order to balance drift.
  • Base verdicts on conservative 99.9% paired-run median intervals plus a 1% minimum directional effect.
  • Report uncertain results as inconclusive and retain compatibility with the existing four-file comparison command.
  • Add focused regression tests and a standalone comparison command.

Validation

  • Complete tests suite passes, including the benchmark regression tests.
  • tests/manycomponents.nim passes with -d:ArchetypeWords=2.
  • A 15-pair, 30-process identical-binary comparison classified every benchmark as unchanged.
  • The previously suspicious two-process data is now correctly inconclusive.
  • Public vecs source and API are unchanged.

This branch is the required baseline for subsequent optimization work.

@RowDaBoat
RowDaBoat marked this pull request as ready for review September 7, 2026 22:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant