Repository navigation
perf(resolve): pool the prerequisite phase again (fix the 2026-10-31 index regression) - #605
Merged
Merged
Conversation
#591 moved the prerequisite (imports/extends/implements) batches to the main-thread kernel so the workers could open one post-prerequisite snapshot. That serialized the phase on one core: at matched graph sizes n8n's full index went 54.5-57.2 s -> 70.3-70.7 s and supabase 17.3-17.5 s -> 21.8-22.5 s, with index CPU falling from 630% to 460%. Restore the two-phase pool: open on the pre-prerequisite snapshot so the prerequisite refs resolve across the workers, then take a second snapshot after their supertype edges persist and recycle the workers onto it. Keep #591's early spawn, single worker lookup, "loaded" message and synthesis-boundary open; a sweep with no calls batch refreshes before synthesis instead of dropping the pool. A/B on the 24-core host: supabase 21.60/21.88 -> 17.11/17.27 s, n8n 70.72/73.17 -> 53.58/53.91 s. Peak index RSS on n8n rises ~0.6 GB from the second snapshot generation.
Second 2026-10-31 Node campaign at the fix head (c63e3d7) against upstream cd04b21. Full index: CPython 27.06 s against 46.34 (1.71x), n8n 53.95 against 71.36 (1.32x), pretix 11%, discourse 5% and Alamofire 4% faster; gin 8% and supabase 11% slower (supabase on 33% more nodes). One-file sync is faster on all seven, 1.08x to 2.69x; MCP warm explore 1.75x, idle memory 2.19 against 4.46 GiB. The new doc records the bisect that names #591, the matched-host A/B, the spreads and the admission receipt. The first 2026-10-31 run and the 2026-10-30 doc keep their dates and now point at the attribution; the unit conversion for the MCP memory rows is MiB/1024, correcting the /1000 values in the first run's table and list.
This was referenced Nov 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
c63e3d7crestores the two-phase resolver pool. perf(resolve): spawn pool workers early and snapshot once after prerequisites #591 had moved the prerequisite (imports/extends/implements) batches to the main-thread kernel so the workers could open one post-prerequisite snapshot; that serialized the whole phase on one core. The pool now opens on the pre-prerequisite snapshot so those refs resolve across the workers, takes a second snapshot once their supertype edges persist, and recycles the workers onto it. It keeps perf(resolve): spawn pool workers early and snapshot once after prerequisites #591's early spawn, single worker-script lookup,loadedmessage and synthesis-boundary open, and refreshes before synthesis when a sweep had no calls batch.cd04b21a, indocs/benchmarks/fork-vs-upstream-2026-10-31b.md, with the bisect and matched-host A/B.?v=13, the first 2026-10-31 run and the 2026-10-30 doc keep their dates and now point at the attribution. The MCP memory rows use MiB/1024 (the first run's table and idle list had divided by 1000); the 2026-10-30 3.58 GiB row is labeled 2026-10-27 as its source says.The regression and its cause
Bisect, same host and corpus state, fresh index per run:
216093dfabc462ea05077514#590alone is inside the pre-#590 spread; its rows ran at 1-minute load 9–11, and both corpora sit above the 25k-refs-per-worker cap.#591is the regression.Interleaved A/B at
0d4c48dcvs the fix:Headline numbers (fixed head vs upstream, spreads in the doc)
MCP n8n: warm explore 240 → 138 ms (1.75×), wave 2.38 → 1.88 s, init 115 → 64 ms, idle 4.46 → 2.19 GiB, busy 3.81 → 3.02 GiB; watcher sync 702 → 766 ms (9% slower).
The fix trades ~0.3 GiB of peak index RSS on n8n for the speed (8.8–8.9 GiB against 8.2–8.5 before; upstream 11.9 GiB), from the second snapshot generation.
Recommendation impact
The 2026-11-02 “continue” recommendation was conditional on this regression being bisected and fixed. It is: index health is restored, and the fork is faster than upstream on five of seven index cells (1.04× to 1.71×) plus all seven sync cells and MCP, with supabase's 11% loss carrying 33% more nodes.
Tests
wal-deferral,resolver-pool-sizing,resolver-pool-engage,kernel-resolve-parity— 51 passed, 1 skipped.npx vitest run: 11,340 passed, 46 skipped, 4 failed in__tests__/gitignore-pattern-parity.test.ts. Those four also fail at the base commit0d4c48dc(checked in the primary checkout); they are not touched by this PR.check:kernel-rulesis red at base too:codegraph-kernel/src/resolve/member_fn_ref.rs:1082(from feat(kernel): port the remaining upstream resolution drift #587) uses a name-parameterized regex. This PR is TypeScript-only and does not touch it.readme-sync: 6/6.~/cg-scratch/bench-2026-10-31d/admission.json.README rows checked
Upstream merge point (now
cd04b21a), measured-results intro, headline chart (?v=13), index table and the graph-size sentence, the new regression-closure paragraph, sync table, MCP table and the idle reconciliation, “What it costs”, and the fork-adds/feature tables (no numbers there moved).