Repository navigation
Read entry names and classic mask styles in linear time - #77
Merged
Merged
Conversation
findEntryObjectSpan reads each entry fragment's Name, and the UUID that breaks a tie between two entries of one name, with /<P Name="Name"[^>]*>([^<]*)<\/P>/ and its UUID twin. The match retried at every `<P Name="Name"` in the fragment and walked from each to the next `>`, so a fragment holding many of them that no `>` follows soon cost their count times that walk. Nothing in this package calls it. data-explorer-vscode does, on a binary dictionary's data/chunk0.xml as the file holds it: on every cell edit and rename, every delete of an entry or a child, every add-child and every cut-and-paste or move (BinarySlddEditorProvider, and xmlStructuralEdit's spliceEntry, deleteEntryXml and deleteEntriesByNameXml). A plain paste only finds the insertion point. Measured on this machine: a 4,246 B binary dictionary whose first entry holds a comment of 30,000 Name opens opens in 5 ms with no warning, and then each lookup of the entry after it took 5.9 s; at 100,000 opens (6,143 B) it took 65.8 s. The UUID read took the same, 0.66 and 6.0 s at 10,000 and 30,000, and the bare pattern on 30,000 opens 5.9 s. pNodeText returns the match's group, or null, with forward indexOf's. At one start the pattern can match only one way: `[^>]*` cannot pass a `>` and `[^<]*` cannot pass a `<`, so it holds or fails on the first `>` after the open and the first `<` after that. The open starts with its only `<` and holds no `>`, so every later start before that `>` meets the same two and fails the same way, and the next start worth trying is the next open from that `<`. The same lookups take 0.1 to 0.9 ms, and a million opens (a 30,672 B file) 7.6 ms. The old and new read agree on all 78,536,544,841 strings of up to twelve tokens over `<`, `>`, `/`, `P`, the space, `Name="Name"`, `x` and the newline; for the UUID read, on all 9,817,068,105 of up to eleven with `Name="UUID"` in its place; and on all 11,111,111,111 of up to ten tokens over those and `<P Name="Name"` and `</P>`, which reach several opens and closes in one string. Three wrong rewrites each disagree on 1,454 to 6,060 of the 11,111,111 of those of up to seven (trying only the first open, trying the next from one past the `<`, and `</P` for `</P>`). A fourth, trying the next from just past the `>`, agrees on all of them, as the argument says it must: no `<`, so no open, lies between that `>` and that `<`. Over the 223 .sldd of test/fixtures, test/parity/artifacts and the local corpus, 57 of them binary, the two agree on both reads of all 436,185 distinct fragments: the 242,089 the lookup slices out of the files and the 242,086 the package writes back for the entries it models. 5,516 lookups through the old and the new module, by selector and by bare name, on the 50 of those files with at most 4,000 entries, find the same spans. test/xmlEntrySplice.test.ts keeps both patterns as the specification and holds pNodeText to them on 20,000 seeded random strings. It holds a lookup past an entry with 25,000 Name opens, and the UUID read past 25,000 UUID opens, to 1 s. With the old patterns put back, together or each alone, its own case fails at 4.1 s (Name) and 4.1 to 5.5 s (UUID), and passes with the new. The two take 0.4 and 1.1 ms here at most over two full runs, so 1 s is past five times CI's time, at 3.3 times this machine's (fe238c7).
…e regex cut them classicMask decides whether a classic .mdl mask parameter is an expression from its style, the MaskStyleString piece its slot names, cut with `.replace(/\(.*$/, '')`. Without the `m` flag `$` is the end of the text and `.` passes no line terminator, so the cut fails at every `(` that one follows, and each failure walks to it: the square of a piece of many `(` ahead of a line break. The `(` raise the depth splitMaskStyles splits at, so no comma splits them off, and `)` after the break can bring it back down for the styles after them. The scanner unescapes `\n` and `\r` in a quoted value and keeps a raw line break between its quotes, so the file puts the break where it likes. In the file's own spelling, a backslash and an `n`, the pattern is flat, which is why it looked linear on the text as written. Opening the file runs it, once for each declaration in MaskVariables that names the slot. Measured on this machine, mdlmask_R2011b.mdl with its first style made 10,000, 30,000 and 100,000 `(`, a `\n` and as many `)` took 65 ms, 550 ms and 6.1 s to parse; a CPU profile of the second puts 552 of its 559 ms in the regex under classicMask. Fifty declarations naming the style of 10,000 took 3.1 s (a 47,549 B file), and a thousand naming one of 3,000 took 5.5 s (39,249 B). styleWithoutOptions finds the last line terminator (`\n`, `\r`, U+2028, U+2029) and cuts at the first `(` after it, which is the first `(` that no line terminator follows: where the regex cut. The same files take 3.8, 5.2 and 11 ms, and 3.9 ms for both of the others. Each slot's answer is also kept for the declarations after the first that name it. Reading a style is linear in its length, so with the cut alone those cost their product: 4.4 s for 20,000 declarations naming the style of 100,000 `(` (347,249 B), and 53 s for 100,000 naming one of 250,000 (1,127,249 B). Those take 16 and 73 ms now. The old and new cut agree on all 78,536,544,841 strings of up to twelve characters over `(`, `)`, `x`, `\n`, `\r`, U+2028, U+2029 and the space. Four wrong rewrites each disagree on 1,151,041 to 10,235,620 of the 19,173,961 of up to eight (U+2029 forgotten, `\n` alone, the first `(` from the start, and the search begun one character late); a fifth, searching from the break itself rather than one past it, agrees on all of them, as it must, since a break is not a `(`. Old and new parseMdl give the same result for a million random masks (declarations naming slots any number of times, styles holding options, breaks and stray parentheses) that hold 277,609 rows, and keeping every answer under slot 0 instead differs on 13,158 of 200,000. Over the 8 .mdl of test/parity/artifacts (test/fixtures and the local corpus hold none), 5 of them classic, and 28 more on this machine, 16 of them classic, the whole result agrees, and so does the cut on every string it is applied to (15 times, 3 distinct), on every MaskStyleString piece (4 distinct) and on every quoted value the scanner reads (896 distinct). As a wider population, it agrees on all 367,186 distinct text nodes, attribute values and lines of the 151 .slx and 8 .mdl of test/fixtures, test/parity/artifacts and the local corpus. test/mdlParser.test.ts keeps the regex as the specification and holds styleWithoutOptions to it on 20,000 seeded random strings. It holds a style of 80,000 `(` ahead of a `\n`, and 30,000 declarations naming a style of 200,000 characters, each to 1 s. The first fails at 3.9 s with the regex put back alone, the second at 5.8 s with the per-slot answer removed alone, both at 3.9 and 3.7 s with the old classicMask put back whole, and both pass with the new. They take 5 and 21 ms here at most over two full runs, so 1 s is past five times CI's time, at 3.3 times this machine's (fe238c7).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two more file-reading regexes that were quadratic on a backtracking engine, found by the sweep after #74 and fixed the way #74 fixed its four: each is replaced by code that gives the regex's exact result on every input in linear time, the equivalence is shown exhaustively on a small alphabet and on real files, and a regression test beside the module's existing tests fails with the old pattern and passes with the new one. One commit per site, each with its regenerated
dist/.The sites
xmlEntrySpliceentryNameOf/entryUuidOf:/<P Name="Name"[^>]*>([^<]*)<\/P>/and its UUID twinfindEntryObjectSpan/findEntryElementSpan, on a binary.sldd'sdata/chunk0.xmlas the file holds it. Nothing in this package calls them. data-explorer-vscode calls them on every cell edit and rename, every delete of an entry or a child, every add-child and every cut-and-paste or move. A plain paste only finds the insertion point. A fragment with many<P Name="Name"opens that no>follows soon makes the match retry at each one and walk to the next>.MdlParserclassicMask:(styles[slot] ?? 'edit').replace(/\(.*$/, '').mdl, once for eachMaskVariablesdeclaration that names the slot. With nomflag,.stops at a line terminator, so the cut fails at every(that has one after it, and each failure walks to that terminator. The scanner unescapes\nand\rinside a quoted value, which puts a real line break into the style. The cost also multiplies by the number of declarations that name the slot.Why the bare
\(.*$looked flat. In the file the line break is spelled as two characters, a backslash and ann. On that text the pattern takes 0.0–0.2 ms at 10,000–100,000(. The scanner then turns those two characters into a real\n, and on that text it takes 62 ms, 556 ms and 5.9 s. In the 30,000 case the value that reaches the regex, captured in-process, is 60,001 characters with the\nat index 30,000, and a CPU profile of that parse puts 552 of its 559 ms inRegExp: \(.*$underclassicMask.The fixes
pNodeText(fragment, open)finds the open withindexOf, then the first>after it, then the first<after that, and checks for</P>there. At any one start the pattern can match only one way. The open starts with its only<and contains no>, so every later start before that>meets the same>and<and fails the same way. The next start worth trying is therefore the next open at or after that<.styleWithoutOptions(style)finds the last line terminator (\n,\r, U+2028, U+2029) and cuts at the first(after it. That is exactly the first(with no line terminator after it, which is where the regex cut.classicMaskalso computes each slot's answer once (aMapfrom slot to boolean), so declarations that name the same slot no longer multiply the cost.Equivalence
<>/PspaceName="Name"xnewlineName="UUID", up to 11<P Name="Name"and</P>, up to 10()x\n\rU+2028 U+2029 spaceparseMdl, old against new, on random masks (declarations naming slots repeatedly, styles with options, breaks and stray parentheses)The harness catches wrong rewrites.
<, and checking</Pinstead of</P>.\n, cutting at the first(in the string, and starting the search one character late.parseMdlresult for 13,158 of 200,000 random masks.>instead of from the<: no open can start between them. The other searches from the line break itself rather than one past it: a line break is never a(.Real data. The roots are test/fixtures, test/parity/artifacts and the local corpus.
.slddfiles, 57 of them binary. Both reads agree on all 436,185 distinct entry fragments: the 242,089 the lookup slices out of the files, plus the 242,086 the package writes back for the entries it models. 5,516 lookups, by selector and by bare name, find the same spans through the old and new module..mdlfiles, all in test/parity/artifacts, 5 of them classic. I added 28 more.mdlfiles found elsewhere on this machine, 16 of them classic.parseMdlgives the same whole result on all 36. In those files the cut runs only 15 times, on 3 distinct strings, so I also compared on wider sets: everyMaskStyleStringpiece (4 distinct), every quoted value the scanner reads (896 distinct), and all 367,186 distinct text nodes, attribute values and lines of the 151.slxand 8.mdlfiles in the three roots. 0 mismatches.Tests
test/xmlEntrySplice.test.tspNodeTextagainst them on 20,000 seeded random strings.test/mdlParser.test.ts(before a\n, and 30,000 declarations naming one style of 200,000 characters.classicMaskrestored whole, both fail, at 3.9 and 3.7 s. They take at most 5 and 21 ms here over two full runs.Verification
npm run verifyat the head passes: typecheck, build, smoke, 6,637 tests passed and 26 skipped, check:pack, check:leak, check:browser.npm run buildleavesgit status --porcelainempty.