Skip to content

perf: Reduce and parallelize file system operations in the module state scan (~1.35x)#3

Draft
jeswr wants to merge 1 commit into
masterfrom
perf/module-state-scan
Draft

perf: Reduce and parallelize file system operations in the module state scan (~1.35x)#3
jeswr wants to merge 1 commit into
masterfrom
perf/module-state-scan

Conversation

@jeswr

@jeswr jeswr commented Jul 5, 2026

Copy link
Copy Markdown
Owner

Summary

ModuleStateBuilder.buildModuleState walks the entire node_modules tree on every ComponentsManager build — 1,381 modules for a Community Solid Server install. Three refinements that reduce and parallelize its file system work, with no change to the resulting module state:

  1. Parallel recursion for non-scoped dependencies. buildNodeModulePathsInner awaited the recursion for each non-scoped dependency sequentially, while scoped (@org/...) dependencies were already handled with Promise.all. Both now recurse in parallel. The ignorePaths check-and-set is synchronous after the realpath await, so deduplication semantics are unaffected.
  2. Half the fs ops in buildPackageJsons. It performed a stat (fileExists) followed by a readFile for every module; it now reads directly and treats a read failure as "no package.json" (1,381 stats saved on the CSS tree). Malformed JSON still throws exactly as before — only the existence probe is folded into the read.
  3. Parallel probes in preprocessPackageJson. The four sequential stat probes per lsd:module package (components/components.jsonld, components/context.jsonld, components/, config/) now run concurrently.

Measurements

A/B against the real Community Solid Server node_modules tree (1,381 modules, warm file system cache, 9 reps per arm, alternating arms in the same gated run; pss-solid-test 2-core shared box, nice -n 18):

arm median min
before 355 ms 324 ms
after 259–267 ms 236 ms

~1.35×, −90 ms per boot, with byte-identical module state in both arms (1,381 nodeModulePaths / 1,381 packageJsons / 168 componentModules / 169 contexts / 175 importPaths). The gain should grow with cold file caches, where parallel recursion overlaps disk latency instead of paying it serially.

Notes

  • nodeModulePaths (and object key) ordering can differ under parallel completion — as it already could for scoped modules and the top-level import-path scan, which used Promise.all before this change. Version-conflict resolution is semver-based and unaffected; the full test suite (including all ModuleStateBuilder tests) passes unchanged with 100 % coverage.
  • Follow-up idea (not in this PR): ComponentsManagerBuilder.build already skips the scan entirely when a moduleState is supplied — persisting the (JSON-serializable) module state across warm boots would remove the remaining ~260 ms; that needs an invalidation story, so it is left for discussion.

Generated by an agent (Claude Fable 5) on @jeswr's behalf — staged on the fork for his review before upstreaming.

Review timing: This draft was prepared with Claude; I (@jeswr) will personally review it before it progresses. I'm currently batching a lot of work in flight, so expect active review Wednesday–Friday (8–10 July).

…te scan

ModuleStateBuilder.buildModuleState walks the entire node_modules tree
on every ComponentsManager build (1,381 modules for a Community Solid
Server install). Three refinements, none of which change the resulting
module state:

1. buildNodeModulePathsInner awaited the recursion for each non-scoped
   dependency sequentially, while scoped dependencies were already
   handled with Promise.all. Both now recurse in parallel. (The
   ignorePaths check-and-set is synchronous after the realpath await,
   so deduplication is unaffected.)
2. buildPackageJsons performed a stat (fileExists) followed by a
   readFile for every module; it now reads directly and treats a read
   failure as "no package.json", halving the file system operations of
   the hottest phase (1,381 -> 0 extra stats). Malformed JSON still
   throws exactly as before.
3. preprocessPackageJson probed components/components.jsonld,
   components/context.jsonld, components/ and config/ with four
   sequential stats per lsd:module package; the probes now run in
   parallel.

A/B against the real Community Solid Server node_modules tree
(1,381 modules, warm file system cache, 9 reps, 2-core shared box,
alternating arms in the same run):

    before: median 355 ms (min 324)
    after:  median 259-267 ms (min 236)   ~1.35x, -90 ms per boot

The resulting module state is identical in both arms
(1,381 nodeModulePaths / 1,381 packageJsons / 168 componentModules /
169 contexts / 175 importPaths). The gain grows with cold file caches,
where the parallel recursion overlaps disk latency instead of paying it
serially.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant