Repository navigation
docs: re-measure the fork against upstream on 2026-11-01 - #614
Merged
Merged
Conversation
Full seven-corpus Node comparison at fork 8313e9a against upstream cd04b21, timed end to end with Hyperfine 2.0.0. The runner change means every arm was re-run; the 2026-10-31 figures are not mixed in. The fork indexes faster on five of seven (1% pretix to 1.79x CPython); gin and supabase are inside their spreads. Supabase is no longer an 11% loss: 1.9% inside a 2.1% fork spread on 18.5% more nodes. Sync is faster on six of seven (1.22x to 2.63x); discourse is 3% slower. README chart and tables updated, new benchmark record added; MCP left at its 2026-10-31 measurement and labelled as not re-run.
![bompus-ai-review[bot]](https://avatars.githubusercontent.com/u/253635?s=60&v=4){kind=small}
The MCP table used the existing process-tree harness (mcp-bench.mjs), not Hyperfine: warm explore 142 ms against 244 ms, eight explores 1.89 s against 2.40 s, initialize 62 ms against 120 ms, idle memory 2.08 GiB against 4.31 GiB. The watcher sync is 12% slower (754 ms against 676 ms). The table stays separate from the Hyperfine index/sync figures.
Add the Hyperfine whole-command runner, the per-run append-probe helper and the chart generator; the benchmark record now gives the exact commands, the corpus list and how each repeated sync receives a fresh edit (unique marker appended by Hyperfine's --prepare, file restored after each arm). The chart is regenerated from the Hyperfine JSON, and run-index-sync.sh is labelled as the pre-Hyperfine /usr/bin/time harness. Align the README's sync percentages to elapsed-time reductions (18% gin to 62% n8n, with 1.22x-2.63x reciprocal speed) and fix the stale claim that CLI sync wins on all seven corpora; discourse is 3% slower.
Runner: read the list file it advertises; reject a dirty probe and keep its original bytes in a temp file restored after every arm and in an EXIT/INT/TERM trap; reject a non-fresh BENCH_OUT; apply the BENCH_WRAP prefix to every timed command; require the probe in the fresh index and verify the last sync stored the last appended bytes by comparing the file SHA-256 with files.content_hash. Chart: accept --arms base,candidate; match corpus names case-insensitively and require the expected corpora; reject a sample count that does not match one campaign (2 index / 4 sync by default); derive the qualification gap from the fork/base elapsed ratio for both metrics. Record: list each corpus commit and edited file, and describe the new guards. The chart is regenerated from the Hyperfine JSON and the README image tag is bumped to v16.
A full results filesystem can leave mktemp's file empty or partial, and the EXIT handler would then copy it over the probe. Copy, compare, and only then register the backup; keep it when a restore fails.
The runner deleted the corpus's .cg-bench before every arm. A corpus that uses that name for a real index (or viewer trails) lost it. Use .cg-bench-<UTC>-<pid> as CODEGRAPH_DIR and delete only that directory.
Both restore paths now verify the probe's diff contains only the runner's appended marker blocks. An external edit keeps the current file and the backup, so the edit is not silently lost.
An added source line beginning with ++ is +++ in a diff and was skipped as a file header. Verify the current file is the backup bytes plus only the runner's marker blocks instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The README's measured-results tables and chart were from 2026-10-31 at fork
c63e3d7c, before #605-#613. This refreshes them at8313e9a0and records the run.Method
Seven corpora (gin, Alamofire, pretix, CPython, discourse, supabase, n8n), upstream
cd04b21aagainst fork8313e9a0, Node 24.22.2, Hyperfine 2.0.0 for whole commands. Arms in mirrored order; index median of 2 runs, sync median of 4. A difference counts only when the gap exceeds the larger arm's(max-min)/medianspread. Admission passed under the host lock before each corpus; raw Hyperfine JSON, graph counts and admission receipts are in~/cg-scratch/bench-2026-11-01/.The runner changed from the hand-written
/usr/bin/timeloop to Hyperfine, so every arm was re-run; the 2026-10-31 numbers are not mixed in. The Hyperfine runner, append-probe helper and chart generator are added underscripts/benchmarks/runtime/, and the benchmark record's Reproduce section contains the exact commands, corpus list and per-run edit preparation.Headline
Checks
__tests__/readme-sync.test.ts: 6 passed.assets/fork-vs-upstream-performance.svgis regenerated and the image tag bumped to?v=14.docs/benchmarks/fork-vs-upstream-2026-11-01.md.The fork's graph counts in the table are the current ones (e.g. supabase 135,658 nodes / 311,496 edges); the upstream counts are unchanged.