Repository navigation
perf(resolve): resolve 500 refs per pool chunk - #610
Merged
Merged
Conversation
Re-measured on the current work-stealing pool: 500 beats 250 on n8n (51.80 -> 50.75 s), supabase (16.39 -> 16.15 s) and discourse (14.56 -> 14.38 s); CPython is flat and pretix shows no clear change. The 250-era vendor-JS cluster tail did not reproduce with the current queue. 125 bought ~1% on pretix for +300MB RSS; 250 stays the finer fallback.
![bompus-ai-review[bot]](https://avatars.githubusercontent.com/u/253635?s=60&v=4){kind=small}
Review of #610: with CHUNK_SIZE=500, a page configured to 500 refs enqueued one worker message, using one worker instead of the pool. Size the chunk per page: 500 for large pages, and enough chunks (up to four, capped by the pool size) that a small page still fans out. A unit test pins the boundary.
Review of #610: capping target chunks at four left a 3,000-ref prerequisite page at six 500-ref chunks on a 12-worker pool, half the workers idle. Size toward the 250-ref floor when 500s cannot fill the pool: a 3,000-ref page gets twelve 250s; below 4x250 pages still split finer. 40k pages keep 500.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
CHUNK_SIZE250 → 500 in the resolver pool, with the stale 250-era rationale replaced by the current measurements.Measured
On the current head (
fbda999f, includes #607–#609), 2 runs per arm, admission passed:An earlier probe at
723df837measured n8n −3.7%, supabase −2.7%, discourse −1.2%; CPython was flat, and pretix showed no clear change across four runs per arm (8.69/8.68/8.86/7.97 vs 9.53/8.45/8.54/8.52 — the spreads overlap and the 9.53 outlier was a vendor-cluster straggler, not a repeatable regression). The 250 value was originally chosen for that pretix cluster; the current work-stealing queue no longer reproduces the tail.Tests
npx vitest run: 958 files, 11,345 passed, 46 skipped, 0 failed.README impact
The README measured-results table predates this change; n8n/supabase index times will move. Re-measuring the corpora is heavy and is recorded as a follow-up rather than hand-editing one row.