perf(graph): stop targeted reads re-auditing the whole store (#190) - #196
Merged
theyashasvipandey merged 10 commits intoSep 17, 2026
Merged
theyashasvipandey merged 10 commits into
theyashasvipandey merged 10 commits into
Conversation
…us audit "LSH bucket(s) without a node" and "LSH bucket(s) without a fingerprint" re-derived faults that other checks already report: - a bucket with no fingerprint is counted as a malformed owner by the ordered merge walk in inspectFingerprintInvariants; - a bucket whose fingerprint has no node implies that fingerprint has no node, which "fingerprint(s) without a node" reports. On the 288 MB store in mex-memory#190 those two joins were 31.6 s of a 48.7 s audit. Both corruption kinds are still classified corrupt; only the redundant lines leave the diagnostic message. Adds a regression test for each. Refs mex-memory#190
A targeted read (graph query, graph get, impact) inspects freshness once before answering and again in revalidateFreshness before committing its buffered output. That second inspection must stay: output is committed under the class it was labelled with. But it re-ran PRAGMA quick_check and the full persisted-invariant audit against a database that the observation comparison rejects anyway unless its identity [dev, ino, size, mtimeMs, ctimeMs] is unchanged, and that SQLite holds open immutable. loadFreshGraphReadSession now passes the bound observation's canonical path and database identity into the revalidation inspection as auditedDatabase. The inspection skips quick_check and inspectCoreInvariants only when the database it opened has that exact canonical path and the identity statted both before and after the immutable open equals the audited one. Sources, Git, manifest, semantic inputs, snapshot, schema and sidecar checks all still run. The value is passed explicitly per read: nothing is cached across reads and a read writes nothing. graph status and every other caller pass nothing and keep the full audit. Tests: the skip is gated on exact path and identity; revalidation hands back only the bound identity; a database rewritten in place before final validation is re-audited and classified corrupt, and the read is discarded. Refs mex-memory#190
…lver Every live-source pass resolves each supported source path before and after reading it, and a targeted read runs four such passes. Node's JavaScript realpathSync walks one lstat per path component. The mex-memory#190 profile on a 3,259-file corpus measured it at 1,489 us per file against 294 us for realpathSync.native, with identical results. Add resolveRealPath (realpathSync.native) to src/paths.ts and use it at every realpath call site in graph status and the read session, including the project root and database path. The native resolver returns on-disk casing where the JavaScript one keeps the casing it was handed, so a root and a file resolved by different implementations could disagree; routing all of them through one function keeps containment comparisons consistent. Both the before-read and after-read checks are kept. Build-time resolution in engine-impl is unchanged: none of its real paths are compared against these. Tests: a source symlink inside the project is still hashed; a source swapped for a symlink out of the project mid-read is refused; a .mex directory junction out of the project is refused, and one that stays inside is read; a case-variant project root on a case-insensitive volume resolves to the same contained root for both status and a targeted read. Refs mex-memory#190
Collaborator
|
Thanks, this is a careful PR. I reviewed it end to end on six real repositories and it holds up. please add the follow-ups below to this same branch, and we'll merge once they're in. With all of them in, What I verified
Step 3 helps more here than on your machine. Here JS Why reads stay slower than
|
| hono | TS, 658 files | TS monorepo | |
|---|---|---|---|
| process startup | 2.0 s | 2.0 s | 1.9 s |
| structural audit (fingerprint/LSH walk) | 3.1 s (2.7) | 7.7 s (6.5) | 13.3 s (11.3) |
| 4 live-source passes | 2.6 s | 4.7 s | 22.1 s |
| answering the query | 0.1 s | 0.3 s | 0.2 s |
Inside a live pass, SHA-256 is under 2%. The cost is the per-file syscalls: realpath, lstat, open, fstat, read, fstat, realpath, lstat.
Next steps: add to this branch, in this order.
Make one commit per step, and keep the rules from steps 1-3: no weakened check, no module-level cache, and nothing written by a read. Line numbers are on cc1e5f8.
4. Run each live pass's per-file checks with bounded concurrency.
- Change: in
inspectLiveSources(status.ts:1843), add an async twin ofreadStableContainedUtf8Filethat performs the same checks in the same order. Run it through a worker pool capped at 8. Usefs.promises.realpath, which is the same libuv call as.native, and put it insrc/paths.ts. - Determinism: collect the results by index, then build the hashes, diagnostics and
maxSourceBytestotal in sorted path order, so the output is identical to today. Once a limit trips, stop scheduling new reads. - Test hooks:
afterSourceReadmust still fire after each file's read. The race tests instatus.test.tsandgraph-cli-freshness.test.tsmust pass. - Measured on one pass: hono 0.58 -> 0.17 s, 658-file repo 1.0 -> 0.31 s, monorepo 5.0 -> 1.4 s. Hashes were identical.
5. Stop loading unused dependencies at startup.
- Why they load: with
splitting: falsein the CLItsupconfig, esbuild hoists every external import intodist/cli.js. Soink/react(~0.9 s) andtypescript(~0.5 s) load on every command. - Change: set
splitting: truefor the CLI entry. - Verify: check the code that resolves files from
import.meta.url:--version, the wasm grammars,graph rebuild(candidate process),mex hubassets and the skills installer. Then runnpm run test:hub:packageand the release benchmark. - Keep
typescripton reads: the tsconfig hash needs it. - Measured:
--version2.0 -> 0.44 s,scopeabout 1 s faster,queryup to 1.2 s faster. Output was identical.
6. Make the fingerprint walk cheaper.
- Today:
inspectFingerprintInvariants(status.ts:2267) sorts everylsh_bucketsrow byref(the primary key is(band, band_hash, ref)), parses a BigInt per row, and callscreateHashper band. - Change:
- Load the fingerprints into a
ref-> index map, with the expected band hashes in aBigInt64Array. - Scan the buckets unsorted, counting per slot in a
Uint16Array. - Compute band hashes with one-shot
crypto.hash(check the"buffer"output on Node 22.5).
- Load the fingerprints into a
- Tests: every failure kind must produce the same counts as today: orphan ref, TEXT ref, missing band, duplicate band, out-of-range band, wrong hash, and orphaned buckets.
- Measured: about 21% faster (monorepo 11.3 -> 8.9 s). Counts were identical on the clean store and all six injected corruptions.
7. Scoped Step 4: skip the fingerprint/LSH walk for reads that never use it.
- Change: add an internal option next to
auditedDatabase, e.g.structuralAudit: "full" | "graph"."graph"keepsquickCheckand every node/edge/FTS invariant, and drops only the fingerprint/LSH walk. - Who gets which:
query,getandimpactpass"graph", unless--fingerprintis set.graph status,mex check(grounding) and maintenance stay"full".
- Before coding: grep the
query/get/impactpaths for reads ofnode_fingerprintsandlsh_buckets, and list the results in the PR. - Tests: with LSH corrupted,
graph statusiscorrupt,mex checkrefuses,querystill answers, andget --fingerprintrefuses. A dangling edge still makesqueryrefuse. - Saves: about 2.7 s, 6.5 s and 11.3 s on the three repos above.
Rough projection, not yet measured end to end: after 4-6, about 5.5 s on hono, 10 s on the 658-file repo and 23 s on the monorepo. With 7 as well, about 3.0 s, 3.6 s and 12 s.
Please update the PR description with before/after numbers per commit, measured the same way as before. Thanks again, happy to help on any of these.
A targeted read runs four live-source passes, and on a large repository they dominate its cost: 22 s of a 39 s query on a 3,253-file corpus. The time is per-file syscalls (realpath, lstat, open, fstat, read, fstat, realpath, lstat), not hashing, so a sequential walk mostly waits on I/O. inspectLiveSources now reads up to 8 sources at once through an asynchronous twin of readStableContainedUtf8File that performs the same containment, identity and size checks in the same order. Results are kept by position and accounted in sorted path order afterwards, so hashes, per-file skips, the cumulative maxSourceBytes limit and diagnostic order are exactly what the sequential walk produced. Paths start strictly in order and scheduling stops once completed reads alone exceed maxSourceBytes, which keeps the work bounded and guarantees the in-order accounting breaches the limit before it needs a read that never started. resolveRealPathAsync (fs.promises.realpath, libuv's uv_fs_realpath) joins resolveRealPath in src/paths.ts so both passes resolve with the same resolver. One pass, same checks, identical hashes: 0.58 -> 0.17 s on hono, 1.0 -> 0.31 s on a 658-file repository, 5.0 -> 1.4 s on the 3,253-file corpus. Refs mex-memory#190
With splitting disabled, esbuild inlines every module into dist/cli.js and
hoists each module's external imports to the top of the file. The dynamic
import("./tui.js") therefore saved nothing: ink and react (~0.9 s) and the
TypeScript compiler (~0.5 s) loaded before any command ran, including
mex --version.
Enable splitting for the CLI and graph-candidate entries. Modules reached only
through import() now live in their own chunks and load when a command needs
them. Graph reads still load TypeScript, because the manifest hashes tsconfig
through ts.parseConfigFileTextToJson and that hash must match the build.
Chunks are emitted flat in dist/, next to cli.js, so every module that locates
files from import.meta.url (graph assets, the candidate process, Hub assets,
packaged skills, setup templates, version) resolves exactly as before. The
library build keeps splitting disabled.
mex --version: 2.0 s -> 0.45 s. graph scope ~1 s faster; graph query up to
1.2 s faster. Output of rebuild, status, query, get, impact and scope is
unchanged, and the packed-package smoke test passes.
Refs mex-memory#190
The fingerprint audit recomputes all 32 band hashes of every stored fingerprint on each inspection, and builds compute them for every node. Each band created a Hash object for a two-number JSON input, which made hashing the largest single term of the walk: 5.0 s of an 11.3 s audit on a 3,253-file corpus. bandHashInts and bandHashes now use one-shot crypto.hash with its default hex output. The int64 form reads the first 16 hex digits as a signed 64-bit integer, the same value readBigInt64BE(0) produced, so persisted stores stay valid. Measured on 40,000 fingerprints: 4.6 s -> 2.7 s. The default hex encoding exists since crypto.hash was introduced, well below the Node 22.5 engines floor. The audit keeps its two ordered cursors. Walking lsh_buckets unsorted into typed arrays measured a little faster, but it holds the whole fingerprint and LSH corpus in memory, which inspectFingerprintInvariants deliberately avoids and which maxIndexBytes (2 GiB) would not bound usefully. A new test pins both forms against the Hash-object derivation, including all-zero, all-max, alternating and negative int64 bands. Refs mex-memory#190
After the earlier steps, the structural audit a targeted read runs once is mostly the fingerprint/LSH walk: 2.7 of 3.1 s on hono, 6.5 of 7.7 s on a 658-file repository, 11.3 of 13.3 s on a 3,253-file corpus. graph query, graph get and impact never read lsh_buckets, and read node_fingerprints only when their fingerprint option asks for serialized fingerprints. Inspections take an internal structuralAudit option. "full", the default, is unchanged. "graph" still runs SQLite's quick-check and every relational and full-text invariant, and skips only inspectFingerprintInvariants. The targeted readers pass "graph" unless the fingerprint option is set; the read session forwards it to both freshness inspections. The CLI exposes --fingerprint only on graph scope, which runs no audit, so every CLI query, get and impact takes the "graph" audit; programmatic callers that request fingerprints keep the full one. graph status, grounding (mex check), the Hub, maintenance and scope keep their current audit. The consequence is deliberate: a store whose only damage is in fingerprint or LSH rows answers a plain query while graph status and mex check still report it corrupt, because nothing in that answer reads the damaged rows. Anything the answer does depend on, such as a dangling edge or a fingerprint without a node, still refuses the read. Tests: with an orphan LSH bucket, status is corrupt; query, get and impact answer; the same reads with the fingerprint option and a default read session refuse. A dangling edge still refuses get and who-calls. Forcing either audit mode for every read fails one of these tests. The existing rewrite re-audit test now injects a dangling edge instead of an orphan LSH bucket, so it keeps proving that a database rewritten mid-read is audited again. Refs mex-memory#190
Reading live sources with bounded concurrency made each pass 3.3-4x faster on Windows, where every per-file syscall is expensive. On Linux the same change was a regression: page-cached syscalls cost microseconds there, and routing each file's nine filesystem calls through the libuv thread pool cost more than overlapping them saved. The release-performance gate measured large-fixture graph refresh at a 2,060-2,130 ms median against main's 1,961 ms. Under WSL2 with Node 22.22, graph refresh on the same 48-file fixture went from 3.58 s to 3.80 s, and graph rebuild from 3.63 s to 3.76 s. Choose the concurrency by platform: 8 on Windows, 1 elsewhere. With 1, the pass is the original sequential synchronous walk, reading each file only as it is accounted, so non-Windows behaviour is exactly what it was before concurrency. An internal liveSourceReadConcurrency override lets the ordering test cover both walks on every platform. Refs mex-memory#190
The shipped agent guidance described graph query and graph get as "exact and cheap". They are exact, but every call first proves the whole index is still fresh, so its cost grows with the repository: about 3 s on a 381-file project and 14 s on a 3,253-file one after the mex-memory#190 work, roughly twice graph scope. An agent that takes "cheap" literally expands symbols with one graph get call per id and pays that proof each time. Keep "exact", say what a call costs and why, and tell agents to pass several ids to one graph get, which proves freshness once for all of them. The same sentence is updated in every shipped and maintained copy (AGENTS.md, CLAUDE.md, copilot-instructions.md, .cursorrules, .windsurfrules) so they stay aligned. Refs mex-memory#190
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #190.
Targeted reads (
graph query,graph get,impact) ran the whole store audit twice for every read, so the audit cost more than the answer. This PR follows the three steps from the maintainer comment, in order, with one commit per step.revalidateFreshnessand every containment check are still in place and none is weakened, and nothing is cached across reads.1. Drop the two redundant LSH invariant joins —
9730c7aLSH bucket(s) without a nodeandLSH bucket(s) without a fingerprintonly re-found faults that other checks already report:inspectFingerprintInvariantscounts it as a malformed owner.fingerprint(s) without a nodereports it.Tests:
status.test.tsnow expectsmalformed LSH bucket owner row(s)in place of the removed lines. New regression tests cover a fingerprint deleted with its buckets left behind, and a bucketed fingerprint whose node is missing. Both still come outcorrupt.2. Revalidation reuses the audit of its bound database —
4b9b59aloadFreshGraphReadSessionnow passes the bound observation'scanonicalDbPathanddatabaseIdentityinto the revalidation inspection, asauditedDatabase.quickCheckandinspectCoreInvariantsare skipped, and only if all of these hold:graph statusand every other caller pass nothing, so they keep the full audit.Tests:
corrupt, and the read is discarded.3. Native realpath for containment —
cc1e5f8resolveRealPath(realpathSync.native) lives insrc/paths.ts.status.tsandread-session.ts, including the project root and the database path, so every containment comparison uses one implementation. The before-read and after-read checks are both kept.engine-impl(the build path) still uses JS realpath. None of its real paths are compared against these.Tests:
.mexdirectory junction leading out of the project is refused, and one that stays inside is read, on both the status path and the read path.Measurements
How I measured
main(4bc8419), plus this branch at steps 1+2 (4b9b59a) and at steps 1+2+3 (cc1e5f8)..mex/, indexed once withmex graph rebuildonmain. All three builds read the same store (schema v4, same extractor), so only the read path differs.errorrecord. All runs answered, and everygraph statusreportedfresh.impactused a symbol that resolves to one node, so no run stopped early onTARGET_AMBIGUOUS.Environment: Node v24.8.0 on Windows 11 (26200), i7-12650H, 16 GB RAM, NVMe, Defender real-time protection on.
Corpora
aa50e96H3/ its class idH3Eventedd138eHono/class:4e5dafd3…compose7a9009dDataSource/ its class idDataSourceMedians, seconds
graph querygraph getimpactgraph statusgraph scopegraph querygraph getimpactgraph statusgraph scopegraph querygraph getimpactgraph statusgraph scopeThe process floor (
mex --version) was 0.67 s on every corpus.Every run, in seconds
What the numbers show
realpathSyncis cheap on this machine. An isolated microbenchmark over ~4,000 files, about 10 path components deep, measured 43–45 µs per call for JS and 44–48 µs for native, against 1,489 µs vs 294 µs in the maintainer comment. The switch was not slower for any targeted read orgraph statushere, so the larger gain on that machine should carry over, but I could only measure this one.Verification
All checks below ran locally on Windows 11 with Node v24.8.0.
npm run typecheck: passes.npm run build: passes.npm test: I ran the full suite on this branch and onmain(4bc8419), each with its own fresh build, and compared the failing sets.main3,877 passed / 27 failed.scripts/release-benchmarkCI topology;cli-agent.test.tshitsEPERMremoving its temp dir inafterAll, after all 34 of its tests pass.main. The branch-only one passed 3 of 3 runs when re-run on its own. None of these import the graph code orsrc/paths.ts.Not verified: Linux, macOS, and a store as large as the 288 MB one in the issue. CI covers the first two.