feat: Add opt-in module state persistence for warm starts (scan 416 ms -> load 36 ms)#4
Draft
jeswr wants to merge 1 commit into
Draft
feat: Add opt-in module state persistence for warm starts (scan 416 ms -> load 36 ms)#4jeswr wants to merge 1 commit into
jeswr wants to merge 1 commit into
Conversation
ComponentsManager runs component discovery (a scan of the entire node_modules tree) on every build: ~416 ms for a Community Solid Server install (1,381 modules), even though the result only changes when the dependency tree changes. This adds an opt-in `moduleStateCachePath` option to ComponentsManagerBuilder. When set (and no explicit `moduleState` is provided), the builder tries to load the persisted module state from that file, and persists a freshly built state to it (atomically, via write-then-rename) after a scan. Staleness handling: the cache stores a fingerprint of the cache format version, the componentsjs version, the main module path, and the modification time and size of the main module's package.json, lock files (package-lock.json, yarn.lock, pnpm-lock.yaml, npm-shrinkwrap.json) and node_modules directory. Any mismatch or read failure falls back to a fresh scan. This is a heuristic: package installs/removals touch a lock file and node_modules and are detected, but in-place edits deep inside node_modules are not - which is why the feature is opt-in and documented accordingly. A/B on the real Community Solid Server tree (2-core shared box, warm file cache, medians): full discovery scan 416 ms vs cache load 36 ms (~11.5x, -380 ms per warm start; 2.4 MB cache file), with identical module state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
moduleStateCachePathoption toComponentsManagerBuilderthat persists the component-discovery result (IModuleState) to a file and reuses it on subsequent invocations — the follow-up flagged in #3. Discovery currently rescans the entirenode_modulestree on every build (~416 ms for a Community Solid Server install with 1,381 modules), even though the result only changes when the dependency tree changes.Behaviour
moduleStateCachePathis set and no explicitmoduleStateis provided: the builder loads the persisted state if it is fresh, otherwise runs a normal discovery scan and persists its result (atomically, write-then-rename, so concurrent processes never observe a partial file).moduleStateis provided, the cache is not consulted (unchanged precedence).Staleness story (explicit)
The cache entry embeds a fingerprint of:
IModuleState's shape changes),package.json,package-lock.json,yarn.lock,pnpm-lock.yaml,npm-shrinkwrap.json, and thenode_modulesdirectory itself.Any mismatch → the cache is ignored (with an info log) and rebuilt. This detects package installs/removals and lockfile changes (they touch a lockfile and/or
node_modules's mtime). It deliberately does not detect in-place edits deep insidenode_modules(e.g. hand-editing an installed package'scomponents/files): scanning for that would cost as much as discovery itself. That limitation is why the feature is opt-in, and it is documented on the option and the class. Escape hatch: delete the cache file, or drop the option.Measurements
Real Community Solid Server tree, warm file cache, medians, alternating in one gated run (
pss-solid-test, 2-core shared,nice -n 18):~11.5×, −380 ms per warm start, with identical module state (verified key-by-key). Composes with #3 (which speeds up the cold scan itself) and with #1 (which removes the JSON-LD parse cost after discovery).
Tests
13 new unit tests for
ModuleStateCache(fingerprint sensitivity, staleness, corrupt/missing files, round-trip, atomic write, failure logging) + 3 builder wiring tests (cache hit, miss + persist, explicit-moduleStateprecedence). Full suite green, 100 % coverage retained.Generated by an agent (Claude Fable 5) on @jeswr's behalf — staged on the fork for his review before upstreaming.
Review timing: This draft was prepared with Claude; I (@jeswr) will personally review it before it progresses. I'm currently batching a lot of work in flight, so expect active review Wednesday–Friday (8–10 July).