Skip to content

Read entry names and classic mask styles in linear time - #77

Merged
ww-mw merged 2 commits into
mainfrom
fix-entry-splice-and-mask-regexes
Oct 12, 2026
Merged

ww-mw merged 2 commits into
mainfrom
fix-entry-splice-and-mask-regexes

Conversation

@ww-mw

@ww-mw ww-mw commented Oct 12, 2026

Copy link
Copy Markdown
Member

Two more file-reading regexes that were quadratic on a backtracking engine, found by the sweep after #74 and fixed the way #74 fixed its four: each is replaced by code that gives the regex's exact result on every input in linear time, the equivalence is shown exhaustively on a small alphabet and on real files, and a regression test beside the module's existing tests fails with the old pattern and passes with the new one. One commit per site, each with its regenerated dist/.

The sites

Site How it is reached Before (this machine) After
xmlEntrySplice entryNameOf / entryUuidOf: /<P Name="Name"[^>]*>([^<]*)<\/P>/ and its UUID twin findEntryObjectSpan / findEntryElementSpan, on a binary .sldd's data/chunk0.xml as the file holds it. Nothing in this package calls them. data-explorer-vscode calls them on every cell edit and rename, every delete of an entry or a child, every add-child and every cut-and-paste or move. A plain paste only finds the insertion point. A fragment with many <P Name="Name" opens that no > follows soon makes the match retry at each one and walk to the next >. A 4,246 B dictionary opens in 5 ms with no warning, then 5.9 s per lookup; at 6,143 B, 65.8 s. The UUID read is the same. 0.3 ms and 0.9 ms; a 30,672 B file takes 7.6 ms
MdlParser classicMask: (styles[slot] ?? 'edit').replace(/\(.*$/, '') Opening any classic .mdl, once for each MaskVariables declaration that names the slot. With no m flag, . stops at a line terminator, so the cut fails at every ( that has one after it, and each failure walks to that terminator. The scanner unescapes \n and \r inside a quoted value, which puts a real line break into the style. The cost also multiplies by the number of declarations that name the slot. 6.1 s for a 227 KB file; 3.1 s for 47.5 KB with fifty declarations naming the style; 5.5 s for 39 KB with a thousand 11 ms, 3.9 ms, 3.9 ms. With only the cut fixed, many declarations still multiply: 4.4 s at 347 KB and 53 s at 1.1 MB. Computing each slot's answer once brings those to 16 and 73 ms

Why the bare \(.*$ looked flat. In the file the line break is spelled as two characters, a backslash and an n. On that text the pattern takes 0.0–0.2 ms at 10,000–100,000 (. The scanner then turns those two characters into a real \n, and on that text it takes 62 ms, 556 ms and 5.9 s. In the 30,000 case the value that reaches the regex, captured in-process, is 60,001 characters with the \n at index 30,000, and a CPU profile of that parse puts 552 of its 559 ms in RegExp: \(.*$ under classicMask.

The fixes

  • pNodeText(fragment, open) finds the open with indexOf, then the first > after it, then the first < after that, and checks for </P> there. At any one start the pattern can match only one way. The open starts with its only < and contains no >, so every later start before that > meets the same > and < and fails the same way. The next start worth trying is therefore the next open at or after that <.
  • styleWithoutOptions(style) finds the last line terminator (\n, \r, U+2028, U+2029) and cuts at the first ( after it. That is exactly the first ( with no line terminator after it, which is where the regex cut. classicMask also computes each slot's answer once (a Map from slot to boolean), so declarations that name the same slot no longer multiply the cost.

Equivalence

Strings Mismatches
Name read, every string of up to 12 tokens over < > / P space Name="Name" x newline 78,536,544,841 0
UUID read, the same alphabet with Name="UUID", up to 11 9,817,068,105 0
Name read, those tokens plus <P Name="Name" and </P>, up to 10 11,111,111,111 0
Mask cut, every string of up to 12 characters over ( ) x \n \r U+2028 U+2029 space 78,536,544,841 0
parseMdl, old against new, on random masks (declarations naming slots repeatedly, styles with options, breaks and stray parentheses) 1,000,000 masks, 277,609 rows 0

The harness catches wrong rewrites.

  • Splice: three wrong rewrites disagree on 1,454 to 6,060 of 11,111,111 strings: trying only the first open, retrying from one past the <, and checking </P instead of </P>.
  • Mask: four wrong rewrites disagree on 1,151,041 to 10,235,620 of 19,173,961 strings: forgetting U+2029, checking only \n, cutting at the first ( in the string, and starting the search one character late.
  • Memo: keying every slot's answer to slot 0 changes the parseMdl result for 13,158 of 200,000 random masks.
  • Two rewrites survive, and they are equivalent by the arguments above. One retries from just past the > instead of from the <: no open can start between them. The other searches from the line break itself rather than one past it: a line break is never a (.

Real data. The roots are test/fixtures, test/parity/artifacts and the local corpus.

  • Splice: 223 .sldd files, 57 of them binary. Both reads agree on all 436,185 distinct entry fragments: the 242,089 the lookup slices out of the files, plus the 242,086 the package writes back for the entries it models. 5,516 lookups, by selector and by bare name, find the same spans through the old and new module.
  • Mask: there are 8 .mdl files, all in test/parity/artifacts, 5 of them classic. I added 28 more .mdl files found elsewhere on this machine, 16 of them classic. parseMdl gives the same whole result on all 36. In those files the cut runs only 15 times, on 3 distinct strings, so I also compared on wider sets: every MaskStyleString piece (4 distinct), every quoted value the scanner reads (896 distinct), and all 367,186 distinct text nodes, attribute values and lines of the 151 .slx and 8 .mdl files in the three roots. 0 mismatches.

Tests

  • test/xmlEntrySplice.test.ts
    • Keeps both patterns as the specification and checks pNodeText against them on 20,000 seeded random strings.
    • Bounds two cases to 1 s: a lookup past an entry whose comment holds 25,000 Name opens, and the UUID read past 25,000 UUID opens.
    • With the old patterns restored, together or one at a time, each case fails at 4.1 s (Name) and 4.1–5.5 s (UUID). They take at most 0.4 and 1.1 ms here over two full runs.
  • test/mdlParser.test.ts
    • Keeps the regex as the specification on 20,000 seeded random strings.
    • Bounds two cases to 1 s: a style of 80,000 ( before a \n, and 30,000 declarations naming one style of 200,000 characters.
    • The first fails at 3.9 s with only the regex restored. The second fails at 5.8 s with only the per-slot memo removed. With the old classicMask restored whole, both fail, at 3.9 and 3.7 s. They take at most 5 and 21 ms here over two full runs.
  • Every bound is past five times CI's time, taking CI as 3.3 times this machine (fe238c7). No existing test changed.

Verification

  • npm run verify at the head passes: typecheck, build, smoke, 6,637 tests passed and 26 skipped, check:pack, check:leak, check:browser.
  • At each commit, a fresh npm run build leaves git status --porcelain empty.

ww-mw added 2 commits October 11, 2026 22:11
findEntryObjectSpan reads each entry fragment's Name, and the UUID that breaks a tie between
two entries of one name, with /<P Name="Name"[^>]*>([^<]*)<\/P>/ and its UUID twin. The match
retried at every `<P Name="Name"` in the fragment and walked from each to the next `>`, so a
fragment holding many of them that no `>` follows soon cost their count times that walk.
Nothing in this package calls it. data-explorer-vscode does, on a binary dictionary's
data/chunk0.xml as the file holds it: on every cell edit and rename, every delete of an entry
or a child, every add-child and every cut-and-paste or move (BinarySlddEditorProvider, and
xmlStructuralEdit's spliceEntry, deleteEntryXml and deleteEntriesByNameXml). A plain paste only
finds the insertion point. Measured on this machine: a 4,246 B binary dictionary whose first
entry holds a comment of 30,000 Name opens opens in 5 ms with no warning, and then each lookup
of the entry after it took 5.9 s; at 100,000 opens (6,143 B) it took 65.8 s. The UUID read
took the same, 0.66 and 6.0 s at 10,000 and 30,000, and the bare pattern on 30,000 opens 5.9 s.

pNodeText returns the match's group, or null, with forward indexOf's. At one start the pattern
can match only one way: `[^>]*` cannot pass a `>` and `[^<]*` cannot pass a `<`, so it holds or
fails on the first `>` after the open and the first `<` after that. The open starts with its
only `<` and holds no `>`, so every later start before that `>` meets the same two and fails
the same way, and the next start worth trying is the next open from that `<`. The same lookups
take 0.1 to 0.9 ms, and a million opens (a 30,672 B file) 7.6 ms.

The old and new read agree on all 78,536,544,841 strings of up to twelve tokens over `<`, `>`,
`/`, `P`, the space, `Name="Name"`, `x` and the newline; for the UUID read, on all 9,817,068,105
of up to eleven with `Name="UUID"` in its place; and on all 11,111,111,111 of up to ten tokens
over those and `<P Name="Name"` and `</P>`, which reach several opens and closes in one string.
Three wrong rewrites each disagree on 1,454 to 6,060 of the 11,111,111 of those of up to seven
(trying only the first open, trying the next from one past the `<`, and `</P` for `</P>`). A
fourth, trying the next from just past the `>`, agrees on all of them, as the argument says it
must: no `<`, so no open, lies between that `>` and that `<`. Over the 223 .sldd of
test/fixtures, test/parity/artifacts and the local corpus, 57 of them binary, the two agree on
both reads of all 436,185 distinct fragments: the 242,089 the lookup slices out of the files
and the 242,086 the package writes back for the entries it models. 5,516 lookups through the
old and the new module, by selector and by bare name, on the 50 of those files with at most
4,000 entries, find the same spans.

test/xmlEntrySplice.test.ts keeps both patterns as the specification and holds pNodeText to
them on 20,000 seeded random strings. It holds a lookup past an entry with 25,000 Name opens,
and the UUID read past 25,000 UUID opens, to 1 s. With the old patterns put back, together or
each alone, its own case fails at 4.1 s (Name) and 4.1 to 5.5 s (UUID), and passes with the
new. The two take 0.4 and 1.1 ms here at most over two full runs, so 1 s is past five times
CI's time, at 3.3 times this machine's (fe238c7).
…e regex cut them

classicMask decides whether a classic .mdl mask parameter is an expression from its style,
the MaskStyleString piece its slot names, cut with `.replace(/\(.*$/, '')`. Without the `m`
flag `$` is the end of the text and `.` passes no line terminator, so the cut fails at every
`(` that one follows, and each failure walks to it: the square of a piece of many `(` ahead of
a line break. The `(` raise the depth splitMaskStyles splits at, so no comma splits them off,
and `)` after the break can bring it back down for the styles after them. The scanner
unescapes `\n` and `\r` in a quoted value and keeps a raw line break between its quotes, so the
file puts the break where it likes. In the file's own spelling, a backslash and an `n`, the
pattern is flat, which is why it looked linear on the text as written. Opening the file runs
it, once for each declaration in MaskVariables that names the slot. Measured on this machine,
mdlmask_R2011b.mdl with its first style made 10,000, 30,000 and 100,000 `(`, a `\n` and as
many `)` took 65 ms, 550 ms and 6.1 s to parse; a CPU profile of the second puts 552 of its
559 ms in the regex under classicMask. Fifty declarations naming the style of 10,000 took 3.1 s
(a 47,549 B file), and a thousand naming one of 3,000 took 5.5 s (39,249 B).

styleWithoutOptions finds the last line terminator (`\n`, `\r`, U+2028, U+2029) and cuts at
the first `(` after it, which is the first `(` that no line terminator follows: where the regex
cut. The same files take 3.8, 5.2 and 11 ms, and 3.9 ms for both of the others. Each slot's
answer is also kept for the declarations after the first that name it. Reading a style is
linear in its length, so with the cut alone those cost their product: 4.4 s for 20,000
declarations naming the style of 100,000 `(` (347,249 B), and 53 s for 100,000 naming one of
250,000 (1,127,249 B). Those take 16 and 73 ms now.

The old and new cut agree on all 78,536,544,841 strings of up to twelve characters over `(`,
`)`, `x`, `\n`, `\r`, U+2028, U+2029 and the space. Four wrong rewrites each disagree on
1,151,041 to 10,235,620 of the 19,173,961 of up to eight (U+2029 forgotten, `\n` alone, the
first `(` from the start, and the search begun one character late); a fifth, searching from the
break itself rather than one past it, agrees on all of them, as it must, since a break is not
a `(`. Old and new parseMdl give the same result for a million random masks (declarations
naming slots any number of times, styles holding options, breaks and stray parentheses) that
hold 277,609 rows, and keeping every answer under slot 0 instead differs on 13,158 of 200,000.
Over the 8 .mdl of test/parity/artifacts (test/fixtures and the local corpus hold none), 5 of
them classic, and 28 more on this machine, 16 of them classic, the whole result agrees, and so
does the cut on every string it is applied to (15 times, 3 distinct), on every MaskStyleString
piece (4 distinct) and on every quoted value the scanner reads (896 distinct). As a wider
population, it agrees on all 367,186 distinct text nodes, attribute values and lines of the
151 .slx and 8 .mdl of test/fixtures, test/parity/artifacts and the local corpus.

test/mdlParser.test.ts keeps the regex as the specification and holds styleWithoutOptions to
it on 20,000 seeded random strings. It holds a style of 80,000 `(` ahead of a `\n`, and 30,000
declarations naming a style of 200,000 characters, each to 1 s. The first fails at 3.9 s with
the regex put back alone, the second at 5.8 s with the per-slot answer removed alone, both at
3.9 and 3.7 s with the old classicMask put back whole, and both pass with the new. They take 5
and 21 ms here at most over two full runs, so 1 s is past five times CI's time, at 3.3 times
this machine's (fe238c7).
@ww-mw
ww-mw merged commit 15831c2 into main Oct 12, 2026
4 checks passed
@ww-mw
ww-mw deleted the fix-entry-splice-and-mask-regexes branch October 12, 2026 02:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant