perf: Reuse one shared context parser + document loader across all files#170
Draft
jeswr wants to merge 1 commit into
Draft
perf: Reuse one shared context parser + document loader across all files#170jeswr wants to merge 1 commit into
jeswr wants to merge 1 commit into
Conversation
Components.js constructs a fresh PrefetchedDocumentLoader and (indirectly, via jsonld-streaming-parser) a fresh ContextParser for every .jsonld file. As a result the well-known @contexts are re-loaded and re-normalized once per file: profiling a Community Solid Server boot showed >50% of active CPU inside jsonld-context-parser, with the componentsjs ^5.0.0 context alone normalized ~979 times. This threads a single, optionally cache-bearing ContextParser (carrying one shared PrefetchedDocumentLoader and raw-document cache) through the whole component/config load: - RdfParser.createSharedContextParser(...) builds the shared parser, attaching a normalized-context cache when the installed jsonld-context-parser exposes one (feature-detected, so older versions keep working). - RdfParserOptions gains an optional `contextParser`, forwarded to the JSON-LD parser as the `contextParser` option (honoured by a jsonld-streaming-parser that supports injecting a context parser; ignored otherwise). A per-file document loader is still provided as a fallback, preserving prior behaviour. - ComponentsManagerBuilder creates the shared parser once and hands it to both the ComponentRegistry and ConfigRegistry (and, transitively, to imported files via RdfStreamIncluder). The change is backwards compatible and fully covered; the existing suite stays green (865 tests, 100% coverage). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes: n/a (no prior PR for this change).
Motivation
CPU-profiling a Community Solid Server boot (
bin/server.js -c config/default.json) showed that JSON-LD context parsing dominates start-up:.jsonld) took ~9.2 s of a ~17.5 s cold boot.jsonld-context-parser(ContextParser.parse+ its term-expansion/IRI-validation callees), plus ~10 % GC driven by the same per-file allocations.PrefetchedDocumentLoaderper file (RdfParser), andjsonld-streaming-parserconstructs a newContextParserperParsingContext, so the caches never survive across files. During one boot the sharedcomponentsjs^5.0.0context was normalized ~979 times and the@solid/community-server^7.0.0context ~594 times — once per file that references them.What this PR does
It threads one shared, (optionally) cache-bearing
ContextParser— carrying a singlePrefetchedDocumentLoaderand a single raw-document cache — through the whole component/config load instead of building new ones per file:RdfParser.createSharedContextParser(...)builds the shared parser. It attaches a normalized-context cache when the installedjsonld-context-parserexposes one (feature-detected, so it stays compatible with versions that predate the cache).RdfParserOptionsgains an optionalcontextParser, forwarded to the JSON-LD parser via thecontextParserparser option (honoured by ajsonld-streaming-parserthat supports injecting a context parser; ignored by older versions, in which case the per-filedocumentLoaderfallback preserves today's behaviour exactly).ComponentsManagerBuildercreates the shared parser once and hands it to both theComponentRegistryandConfigRegistry(and, transitively, to imported files viaRdfStreamIncluder).The change is backwards compatible and fully covered — the existing suite stays green: 865 tests, 34 suites, 100 % coverage (
yarn test), andeslintis clean on the touched files.Sibling refreshes this integrates with
The full win depends on two upstream refreshes (both driven by @jeswr); this PR is the Components.js-side integration for them:
contextParseroption): branchfeat/context-parser-parameter-refreshon @jeswr's fork (PR pending).With those installed,
createSharedContextParserautomatically attaches the cache and the streaming parser reuses the shared parser. Without them, this PR is a safe no-op-to-small-win refactor.Benchmark (before/after) and an important finding
Measured end-to-end through the real
RdfParser(rdf-parse → comunica → jsonld-streaming-parser → jsonld-context-parser) with the two refreshes installed, parsing 447 documents that each reference the CSS^7.0.0context (= [componentsjs ^5.0.0 ctx, 525-key object]), with a distinctbaseIRIper file as in a real boot (best of 6 passes):Output quads are identical before/after.
Finding — the shared cache does not yet help a real boot, and this is the actionable result: over 447 files the normalized-context cache recorded 0 hits / 447 misses.
jsonld-streaming-parsercallscontextParser.parse(context, { baseIRI, parentContext: { '@base': baseIRI, … } })with the per-filebaseIRI, and #85 keys its cache on the full(context, options, parentContext)— so every file gets a distinct key. The expensive work (term expansion, IRI validation, container hashing) on the external well-known contexts is actuallybaseIRI-independent (external contexts drop@base), but #85 only memoises the top-levelparse()result after the per-file base is applied, and routes the recursive external-context normalization around the cache.For scale, an isolated micro-benchmark of that exact
parsecall shows the prize if the cache could hit: with a fixedbaseIRIthe shared cache is ~313× faster per file (3.4 ms → 0.011 ms); with distinctbaseIRIs it is ~1× (0 hits).Recommendation for #85: cache the normalized external context by URL (i.e., memoise the
baseIRI-independentload()+normalizeof a remote/prefetched context, beforeapplyBaseEntry), rather than only the base-applied top-level result. That is what turns this Components.js plumbing into the ~50 %-of-boot-CPU win the profile points at. (I also verified that pre-normalizing and serving the contexts from the loader is not a safe shortcut: it bakes onebaseIRIinto the ~523 nested scoped-context@basemarkers, which are legitimately per-file.)Notes for the reviewer
master(v6.0.1) because the CI token in the agent's environment lacks theworkflowscope needed to sync the fork up to upstreammaster. It merges cleanly intomaster; the only extra lines in the diff are pre-existing 6.0.1→master modernizations on the touched files. @jeswr can rebase onto upstreammasterto remove that noise before readying.package.jsonto actually enable the win.🤖 Generated with Claude Code