perf: Reduce and parallelize file system operations in the module state scan (~1.35x)#3
Draft
jeswr wants to merge 1 commit into
Draft
perf: Reduce and parallelize file system operations in the module state scan (~1.35x)#3jeswr wants to merge 1 commit into
jeswr wants to merge 1 commit into
Conversation
…te scan
ModuleStateBuilder.buildModuleState walks the entire node_modules tree
on every ComponentsManager build (1,381 modules for a Community Solid
Server install). Three refinements, none of which change the resulting
module state:
1. buildNodeModulePathsInner awaited the recursion for each non-scoped
dependency sequentially, while scoped dependencies were already
handled with Promise.all. Both now recurse in parallel. (The
ignorePaths check-and-set is synchronous after the realpath await,
so deduplication is unaffected.)
2. buildPackageJsons performed a stat (fileExists) followed by a
readFile for every module; it now reads directly and treats a read
failure as "no package.json", halving the file system operations of
the hottest phase (1,381 -> 0 extra stats). Malformed JSON still
throws exactly as before.
3. preprocessPackageJson probed components/components.jsonld,
components/context.jsonld, components/ and config/ with four
sequential stats per lsd:module package; the probes now run in
parallel.
A/B against the real Community Solid Server node_modules tree
(1,381 modules, warm file system cache, 9 reps, 2-core shared box,
alternating arms in the same run):
before: median 355 ms (min 324)
after: median 259-267 ms (min 236) ~1.35x, -90 ms per boot
The resulting module state is identical in both arms
(1,381 nodeModulePaths / 1,381 packageJsons / 168 componentModules /
169 contexts / 175 importPaths). The gain grows with cold file caches,
where the parallel recursion overlaps disk latency instead of paying it
serially.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ModuleStateBuilder.buildModuleStatewalks the entirenode_modulestree on everyComponentsManagerbuild — 1,381 modules for a Community Solid Server install. Three refinements that reduce and parallelize its file system work, with no change to the resulting module state:buildNodeModulePathsInnerawaited the recursion for each non-scoped dependency sequentially, while scoped (@org/...) dependencies were already handled withPromise.all. Both now recurse in parallel. TheignorePathscheck-and-set is synchronous after therealpathawait, so deduplication semantics are unaffected.buildPackageJsons. It performed astat(fileExists) followed by areadFilefor every module; it now reads directly and treats a read failure as "no package.json" (1,381 stats saved on the CSS tree). Malformed JSON still throws exactly as before — only the existence probe is folded into the read.preprocessPackageJson. The four sequentialstatprobes perlsd:modulepackage (components/components.jsonld,components/context.jsonld,components/,config/) now run concurrently.Measurements
A/B against the real Community Solid Server
node_modulestree (1,381 modules, warm file system cache, 9 reps per arm, alternating arms in the same gated run;pss-solid-test2-core shared box,nice -n 18):~1.35×, −90 ms per boot, with byte-identical module state in both arms (1,381 nodeModulePaths / 1,381 packageJsons / 168 componentModules / 169 contexts / 175 importPaths). The gain should grow with cold file caches, where parallel recursion overlaps disk latency instead of paying it serially.
Notes
nodeModulePaths(and object key) ordering can differ under parallel completion — as it already could for scoped modules and the top-level import-path scan, which usedPromise.allbefore this change. Version-conflict resolution is semver-based and unaffected; the full test suite (including allModuleStateBuildertests) passes unchanged with 100 % coverage.ComponentsManagerBuilder.buildalready skips the scan entirely when amoduleStateis supplied — persisting the (JSON-serializable) module state across warm boots would remove the remaining ~260 ms; that needs an invalidation story, so it is left for discussion.Generated by an agent (Claude Fable 5) on @jeswr's behalf — staged on the fork for his review before upstreaming.
Review timing: This draft was prepared with Claude; I (@jeswr) will personally review it before it progresses. I'm currently batching a lot of work in flight, so expect active review Wednesday–Friday (8–10 July).