Optionally store source maps as VLQ encoded (2/2): Transformer output, unstable_compactSourceMaps (#1743)

[robhogan]

Summary:

This stack

Decoded tuple arrays are the single largest contributor to Metro's dev-server heap on large bundles (~10 million retained small arrays on FBiOS entry bundle, for example). Storing the same data as a compact VLQ string instead removes most of that footprint.

This reduces source map memory by ~51% on the heap and ~48% RSS for that ~16K module bundle.

The emitted whole-bundle source map is unchanged. When a module's map is stored as VLQ, fromRawMappings decodes it back to tuples just-in-time, with request-scoped caching. The trade-off is therefore decode + re-encode CPU when a .map is actually requested or /symbolicate request is made.

A plain string is used for mappings for now, since VLQ is ASCII by design. A UInt8Array would be marginally more efficient and potentially transferrable to/from worker threads, but would require more invasive changes to cache (de)serialisation. I did some benchmarking with this and it doesn't justify the complexity right now.

This diff

Adds unstable_compactSourceMaps (default false). When enabled, the transform
worker stores each module's source map as a compact VLQ string (VlqMap)
instead of a decoded Array<MetroSourceMapSegmentTuple>.

Each module's map originates from one of three sources, so we encode the VLQ the
cheapest way available in each case (all byte-identical to the decoded-tuple
output):

• transformJS, not minifying (the dominant path — Hermes targets don't minify):
  encode the VlqMap straight from result.decodedMap, which babel/generator
  computes eagerly while generating, via vlqMapFromBabelDecodedMap — never
  materialising tuples.
• transformJS, minifying: the minifier returns its own map (not Babel's), so we
  re-encode the resulting tuples with vlqMapFromTuples.
• transformJSON: builds tuples directly (no Babel generate), so it likewise
  re-encodes with vlqMapFromTuples.

countLines is split out of countLinesAndTerminateMap so the decoded-map fast
path can compute the terminating mapping without building and terminating a
tuple array first.

Benchmarks

Cold cache (n=3, means)

| Metric | base | compact |
|---|---|---|---|
| Heap used | 1653.7 MB | 809.7 MB (−51.0%) |
| RSS | 1854.2 MB | 955.2 MB (−48.5%) |
| Heap growth (build) | 1606.5 MB | 761.2 MB (−52.6%) |
| Build CPU (.bundle) | 23.05 s | 22.42 s (n.s.) |
| Serialize CPU (.map) | 11.99 s | 14.19 s (+18.4%) |

Warm cache (n=3, means)

| Metric | base | compact |
|---|---|---|---|
| Heap used | 1552 MB | 731 MB (−52.9%) |
| RSS | 1775 MB | 923 MB (−48.0%) |
| Build CPU (.bundle) | 10.92 s | 8.86 s (−18.9%) |
| Serialize CPU (.map) | 11.87 s | 13.89 s (+17.0%) |

Why behind a flag?

1. The map structure is exposed to custom serialisers, so changing it is semver-breaking. Landing this as experimental opt-in in a non-breaking release allows integrators to experiment with it.
2. This is a trade-off of retained memory vs CPU required to emit a flat source map or symbolicate errors. The trade-off largely goes away with indexed maps (coming next) - but that is a semver-breaking change to output.

Changelog:

 - **[Experimental]**: Add `unstable_compactSourceMaps` to use a more memory-efficient source map format.

Differential Revision: D109216060

[meta-codesync]

@robhogan has exported this pull request. If you are a Meta employee, you can view the originating Diff in D109216060.