Skip to content

perf(resolve): pool the prerequisite phase again (fix the 2026-10-31 index regression) - #605

Merged
bompus merged 2 commits into
fork/consolidatedfrom
perf/restore-prereq-pool
Oct 31, 2026
Merged

bompus merged 2 commits into
fork/consolidatedfrom
perf/restore-prereq-pool

Conversation

@bompus

@bompus bompus commented Oct 31, 2026

Copy link
Copy Markdown
Owner

What changed

  • Fix: c63e3d7c restores the two-phase resolver pool. perf(resolve): spawn pool workers early and snapshot once after prerequisites #591 had moved the prerequisite (imports/extends/implements) batches to the main-thread kernel so the workers could open one post-prerequisite snapshot; that serialized the whole phase on one core. The pool now opens on the pre-prerequisite snapshot so those refs resolve across the workers, takes a second snapshot once their supertype edges persist, and recycles the workers onto it. It keeps perf(resolve): spawn pool workers early and snapshot once after prerequisites #591's early spawn, single worker-script lookup, loaded message and synthesis-boundary open, and refreshes before synthesis when a sweep had no calls batch.
  • Measurement: second 2026-10-31 Node campaign at the fix head against upstream cd04b21a, in docs/benchmarks/fork-vs-upstream-2026-10-31b.md, with the bisect and matched-host A/B.
  • Docs: README measured-results rewritten to the fixed run, chart ?v=13, the first 2026-10-31 run and the 2026-10-30 doc keep their dates and now point at the attribution. The MCP memory rows use MiB/1024 (the first run's table and idle list had divided by 1000); the 2026-10-30 3.58 GiB row is labeled 2026-10-27 as its source says.

The regression and its cause

Bisect, same host and corpus state, fresh index per run:

Commit What it is supabase n8n n8n index CPU
216093df parent of #590 17.27 / 17.46 s 54.46 / 57.22 s 630–636%
abc462ea #590 (pool sizing) 17.98 / 18.29 s 55.89 / 58.48 s 626–627%
05077514 #591 (early spawn, one snapshot) 21.84 / 22.51 s 70.34 / 70.70 s 458–462%

#590 alone is inside the pre-#590 spread; its rows ran at 1-minute load 9–11, and both corpora sit above the 25k-refs-per-worker cap. #591 is the regression.

Interleaved A/B at 0d4c48dc vs the fix:

Corpus Before After
supabase 21.60 / 21.88 s 17.11 / 17.27 s
n8n 70.72 / 73.17 s 53.58 / 53.91 s

Headline numbers (fixed head vs upstream, spreads in the doc)

Corpus Index fork/upstream Sync upstream/fork
gin 1.08× (8% slower, 0.07 s) 1.23×
Alamofire 0.96× (4% faster) 1.39×
pretix 0.89× (11% faster, fork spread 9%) 1.62×
CPython 0.58× (1.71× faster) 2.05×
discourse 0.95× (5% faster) 1.08×
supabase 1.11× (11% slower, 33% more nodes) 2.17×
n8n 0.76× (1.32× faster) 2.69×

MCP n8n: warm explore 240 → 138 ms (1.75×), wave 2.38 → 1.88 s, init 115 → 64 ms, idle 4.46 → 2.19 GiB, busy 3.81 → 3.02 GiB; watcher sync 702 → 766 ms (9% slower).

The fix trades ~0.3 GiB of peak index RSS on n8n for the speed (8.8–8.9 GiB against 8.2–8.5 before; upstream 11.9 GiB), from the second snapshot generation.

Recommendation impact

The 2026-11-02 “continue” recommendation was conditional on this regression being bisected and fixed. It is: index health is restored, and the fork is faster than upstream on five of seven index cells (1.04× to 1.71×) plus all seven sync cells and MCP, with supabase's 11% loss carrying 33% more nodes.

Tests

  • Targeted: wal-deferral, resolver-pool-sizing, resolver-pool-engage, kernel-resolve-parity — 51 passed, 1 skipped.
  • Full npx vitest run: 11,340 passed, 46 skipped, 4 failed in __tests__/gitignore-pattern-parity.test.ts. Those four also fail at the base commit 0d4c48dc (checked in the primary checkout); they are not touched by this PR.
  • check:kernel-rules is red at base too: codegraph-kernel/src/resolve/member_fn_ref.rs:1082 (from feat(kernel): port the remaining upstream resolution drift #587) uses a name-parameterized regex. This PR is TypeScript-only and does not touch it.
  • readme-sync: 6/6.
  • Campaign admission passed under the host lock (CPU 0.8/4 cores, 98.9 GiB free, memory-full 0/10 ms, I/O-full 9.4/100 ms). Receipt: ~/cg-scratch/bench-2026-10-31d/admission.json.

README rows checked

Upstream merge point (now cd04b21a), measured-results intro, headline chart (?v=13), index table and the graph-size sentence, the new regression-closure paragraph, sync table, MCP table and the idle reconciliation, “What it costs”, and the fork-adds/feature tables (no numbers there moved).

#591 moved the prerequisite (imports/extends/implements) batches to the
main-thread kernel so the workers could open one post-prerequisite
snapshot. That serialized the phase on one core: at matched graph sizes
n8n's full index went 54.5-57.2 s -> 70.3-70.7 s and supabase 17.3-17.5 s
-> 21.8-22.5 s, with index CPU falling from 630% to 460%.

Restore the two-phase pool: open on the pre-prerequisite snapshot so the
prerequisite refs resolve across the workers, then take a second
snapshot after their supertype edges persist and recycle the workers
onto it. Keep #591's early spawn, single worker lookup, "loaded" message
and synthesis-boundary open; a sweep with no calls batch refreshes
before synthesis instead of dropping the pool.

A/B on the 24-core host: supabase 21.60/21.88 -> 17.11/17.27 s, n8n
70.72/73.17 -> 53.58/53.91 s. Peak index RSS on n8n rises ~0.6 GB from
the second snapshot generation.
Second 2026-10-31 Node campaign at the fix head (c63e3d7) against
upstream cd04b21. Full index: CPython 27.06 s against 46.34 (1.71x),
n8n 53.95 against 71.36 (1.32x), pretix 11%, discourse 5% and Alamofire
4% faster; gin 8% and supabase 11% slower (supabase on 33% more nodes).
One-file sync is faster on all seven, 1.08x to 2.69x; MCP warm explore
1.75x, idle memory 2.19 against 4.46 GiB.

The new doc records the bisect that names #591, the matched-host A/B,
the spreads and the admission receipt. The first 2026-10-31 run and the
2026-10-30 doc keep their dates and now point at the attribution; the
unit conversion for the MCP memory rows is MiB/1024, correcting the
/1000 values in the first run's table and list.
@bompus bompus added the author:deepseek Pull request written by a deepseek model label Oct 31, 2026

@bompus-ai-review bompus-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bompus AI Review (Claude Fable 5.1, max reasoning): 0 finding(s).

@bompus
bompus merged commit 73bac2e into fork/consolidated Oct 31, 2026
@bompus
bompus deleted the perf/restore-prereq-pool branch October 31, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author:deepseek Pull request written by a deepseek model

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant