Repository navigation
Scan the catalog's tags, a decimal's digits and cdata's .0 in linear time - #74
Merged
Merged
Conversation
…nd them scanScXml read each tag of a binary dictionary's interface-dictionary part with /<(\/?)([A-Za-z_][\w.-]*)((?:[^>"]|"[^"]*")*?)(\/?)>/g, which CodeQL reports as js/polynomial-redos (code-scanning alert 2). On a backtracking engine it is quadratic on the wrong bytes, and the bytes are the file's. Measured on this machine: `<A` and 100,000 `-` took 12.4 s, and `<A` 100,000 times 24.7 s. The part deflates about 1000:1, so a 1,187 B binary dictionary held readSlddContent for 11.8 s. Past ~8.39M backtrack steps V8 throws a RangeError instead, so one stray `"` ahead of a long attribute value failed the whole read. XmlTagScanner returns exactly the regex's match sequence, the tag's index, end and four groups, with forward indexOf's. It remembers an attribute walk that failed, so the retry at each later `<` costs a count of quotes rather than a walk to the end. The same dictionary opens in 3 ms, and the test's five shapes, at ten times the size, take 7 to 83 ms. test/scCatalog.test.ts keeps the regex as the specification and holds the scanner to it tag for tag, on the real catalog and on 20,000 seeded random strings. Over every XML part of the 243 zip-packaged files in test/fixtures, test/parity/artifacts and the local corpus, 4,026 parts and 14,698,222 tags, the two agree on every tag. The linear-time case is held to 2 s, five times CI's time for its slowest shape (83 ms here, over two full runs, at 3.3 times this machine's; fe238c7).
… in linear time
Five patterns read a decimal as `\d+\.?\d*`. With no dot that is `\d+\d*`, and a backtracking
engine tries every split of a run of digits between the two before it gives up, so text that
is not a number cost the square of its length, and the cube in a pattern that holds two
decimals. A sweep for the shape after code-scanning alert 2 found them; CodeQL does not report
them. Measured on this machine:
- XmlUtils.parseComplexNum (COMPLEX_PART, two in each element) reads a text dictionary's
own non-finite complex text on open (MatlabVariableNode._isOwnNonFiniteText), and a
complex value's text in the Property Inspector (piOther.formatCdata). 5,000 digits, `+`,
5,000 more and an `x` took 55 s, and a 5,439 B text dictionary holding half as many took
6.9 s to open.
- SlxParser's NUMERIC_RE judges every block <P> value of a model (valueReferencesData).
100,000 digits and an `x` took 19.6 s, and so did opening a 744 B .slx that holds them.
- MatlabValueParser's parseMatlabNumber and parseComplex read what is typed into a cell:
5.6 s for 100,000 digits and an `x`, 6.9 s for 2,500, `+`, 2,500 more and an `x`.
`\d+(?:\.\d*)?` matches the same strings and leaves every group where it was, and its digits
split only one way. The same inputs take 0.1 to 0.5 ms, and a million digits 1 to 4 ms.
The old and new pattern at each site agree on test() and on every exec() group for all
4,581,298,449 strings of up to eight characters over `019.eE+-iInfNax` and the space, and for
5,229,043 strings of up to six tokens from those and `Inf`, `NaN` and `Infinity`. Opening the
447 .sldd, .slx, .mdl and .mat files of test/fixtures, test/parity/artifacts and the local
corpus through the built package, and building the Property Inspector object of each of their
3,288,107 nodes, applies NUMERIC_RE to 129,486 distinct strings and parseComplexNum's pattern
to 9. Opening a file never reaches MatlabValueParser. Old and new agree on all of them, on all
129,584 distinct strings that run saw put through every pattern, and on the 1,676,663 distinct
string leaves and whitespace tokens of the 222 dictionaries' content as read. (11 of the 447
do not open, for reasons outside this change: ten v7.3 MAT-files, and a text dictionary past
V8's longest string.)
Each new test fails with its old pattern put back alone, at 1.1 to 7.7 s against bounds of
100 ms and 1 s, and passes with the new. Each bound is past five times its case's CI time; the
slowest case is 11 ms here over two full runs (blockParamUsages), at 3.3 times this machine's
(fe238c7).
… replace made MatlabVariableNode._parseCdataText and piOther.formatCdata drop MATLAB's `.0` from each part of complex cdata text with `.replace(/(\d+)\.0(?=[+\-i]|$)/g, '$1')`. The scan tries a match at every digit of a run that no `.0` follows, and each walks the rest of the run, so a run of digits cost its square. The text is a dictionary's cdata as read, a binary one's complex value or a text one's, on open and in the Property Inspector. Measured on this machine: 100,000 digits took 4.3 s to read and 4.3 s to show, and a 100,419 B text dictionary holding them took 4.4 s to open. CodeQL does not report it. `(?<!\d)` lets a match start only at the first digit of a run, and changes no match. `\.0` has to follow `\d+`, so from any digit of a run `\d+` takes the rest of it, and the match holds or fails on what follows the run: the same for every start in it. The scan reaches a run's first digit before the rest, because it never resumes inside a run (a match is followed by `+`, `-`, `i` or the end). So a start inside a run only ever failed. 100,000 digits now take 0.5 ms, and ten million 49 ms. The old and new replace give the same output and the same matches, index and group, on all 4,581,298,449 strings of up to eight characters over `019.eE+-iInfNax` and the space, and on 5,229,043 strings of up to six tokens from those and `Inf`, `NaN` and `Infinity`. Opening the 447 files of test/fixtures, test/parity/artifacts and the local corpus through the built package, with every node's Property Inspector object built, runs it 320 times on 98 distinct strings. Old and new agree on those, on all 129,584 distinct strings that run saw at any of the six patterns of this commit and the last, and on the 1,676,663 distinct string leaves and whitespace tokens of the 222 dictionaries' content as read. test/matlabVariableNode.test.ts and test/piOther.test.ts hold 100,000 digits to 1 s. Each fails at 4.3 s with its own site's old replace put back alone, and passes with the new. Both take 0.6 ms here over two full runs, so 1 s is past five times CI's time, at 3.3 times this machine's (fe238c7). The slowest of every new case across these two runs and the last commit's was blockParamUsages' at 25 ms, a fortieth of its bound. The `.0` drop itself is pinned: only where a part ends, and after a part of many digits as after one of one.
This was referenced Oct 11, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes code-scanning alert 2 (
js/polynomial-redos,TAG_REinsrc/datamodel/parser/ScCatalog.ts). After it, a sweep for the same shape found three more patterns that a file's bytes reach, and one that only typed input reaches. CodeQL does not flag those. There are three commits, and each one carries its rebuiltdist/:85d65e7Find the interface dictionary's tags in linear time, as the regex found them. This is alert 2.bebec60Read a decimal's digits one way, so text that is no number is refused in linear time.\d+\.?\d*becomes\d+(?:\.\d*)?in five patterns.552b8fcDrop MATLAB's .0 from cdata text in linear time, with the matches the replace made.(?<!\d)is added to the cdata.0drop, at both of its sites.No change alters a result the old code finished computing, only how long it took. The one difference is commit 1's: where the old regex threw a
RangeError(below), the dictionary now opens. On a backtracking engine each old pattern was quadratic, or cubic, in the length of the wrong bytes. Those bytes are the file's own, and a deflated part shrinks them about 1000:1.The sites
Measured on this machine (Node 24.14.1). "Before" is the base,
origin/main8facc33, or the commit's parent. "After" is this branch.ScCatalog.tsTAG_RE(scanScXml). Alert 2, quadratic.sldd: the interface-dictionary part<Aand 100,000-: 12.4 s.<A100,000 times: 24.7 s. A 1,187 B dictionary heldreadSlddContentfor 11.8 s (all three are commit 1's figures)XmlUtils.tsCOMPLEX_PART(parseComplexNum). Two per element, so cubic.sldd: complex cdata text marked as this package's own non-finite text (MatlabVariableNode._isOwnNonFiniteText). The Property Inspector's Other rows (piOther.formatCdata)+, 5,000 more and anx: 55 s (1,000 characters: 80 ms; 3,000: 1.5 s). A 5,439 B text dictionary holding half that took 6.9 s to openSlxParser.tsNUMERIC_RE(valueReferencesData). Quadratic.slxor.mdl: every block<P>valuex: 19.6 s. Opening a 744 B.slxholding that: 19.6 s.slxholding them opens in 47 ms.0drop,MatlabVariableNode._parseCdataTextandpiOther.formatCdata. Quadratic.slddof either kind: complex cdata text. The Property Inspector's Other rowsMatlabValueParser.ts:parseMatlabNumber, andparseComplex's two patternsx: 5.6 s. 2,500 digits,+, 2,500 more and anx: 6.9 sWhy each result is the same
Commit 1.
XmlTagScannerreturns exactly the regex's match sequence: each tag's index, end and four groups. The regex stays intest/scCatalog.test.tsas the specification, and the scanner is held to it tag for tag, on the real catalog and on 20,000 seeded random strings. Over every XML part of the 243 zip-packaged files intest/fixtures,test/parity/artifactsand the local corpus (4,026 parts, 14,698,222 tags), the two agree on every tag. The scanner differs from the regex only where the regex never finished: past ~8.39M backtrack steps V8 threw aRangeError, so one stray"ahead of a long attribute value failed the whole read, and now the dictionary opens.Commit 2.
\d+(?:\.\d*)?matches the same strings as\d+\.?\d*and leaves every capture group where it was; its digits split only one way.Commit 3.
(?<!\d)lets a match start only at the first digit of a run, and changes no match.\.0has to follow\d+, so from any digit of a run\d+takes the rest of it, and the match holds or fails on what follows the run, the same for every start in it. The global scan reaches a run's first digit before the rest and never resumes inside a run (a match is followed by+,-,ior the end), so a start inside a run only ever failed.Checked, for commits 2 and 3. The old and new patterns were read out of the source files rather than retyped, and compared on
test(), the fullexec()result and, for the.0drop, thereplaceoutput and the whole match sequence. A deliberately wrong rewrite at each of the six patterns produced thousands of mismatches, so the harness can tell them apart. The real ones gave 0 mismatches on:019.eE+-iInfNaxand the space: 4,581,298,449 strings;Inf,NaNandInfinity;.sldd,.slx,.mdland.matfiles oftest/fixtures,test/parity/artifactsand the local corpus through the built package, with the Property Inspector object of each of their 3,288,107 nodes built:NUMERIC_RE241,638 applications (129,486 distinct strings),COMPLEX_ELEMENT16 (9), the.0drop 320 (98). Opening a file never reachesMatlabValueParser;11 of the 447 files do not open, for reasons outside this change: ten v7.3 MAT-files, and one 866 MB text dictionary past V8's longest string.
Regression tests
Each site has a test beside its module's existing tests that runs the shape it was slow on. With that site's old pattern put back alone, its test fails and the new pattern passes:
COMPLEX_PARTNUMERIC_RE(test/blockParamUsages.test.ts)MatlabValueParser:53 / :169 / :176 (lines at 8facc33).0drop,MatlabVariableNode/piOtherCommit 1's linear-time test in
test/scCatalog.test.tsruns five shapes of about a million characters each (1.4 million for the last). Only the first is ten times a size in the table above;<Arepeated runs at five times, and the other three are not in the table. They take 7 to 83 ms here, against a 2 s bound. (Commit 1's message says the five shapes run "at ten times the size"; that holds for the first only.)The bounds follow fe238c7: each is past five times its case's CI time, estimated at 3.3 times this machine's slowest over two full runs. Of the cases commits 2 and 3 add, the slowest across their four full runs (two at each commit) was blockParamUsages' at 25 ms, against a 1 s bound; commit 1's slowest, above, was 83 ms against 2 s. The
.0drop itself is pinned too: it applies only where a part ends, after a part of many digits as after one of one.Verification
npm run buildleavesgit status --porcelainempty, and the full suite passes: 6,621, 6,628 and 6,631 tests, each with 26 skipped.npm run verifyat the head passes: typecheck, build, smoke, 6,631 tests passed and 26 skipped,check:pack,check:leakandcheck:browser.describeblock totest/scCatalog.test.ts(both kept), anddist/datamodel/parser/ScCatalog.*was regenerated bynpm run buildrather than hand-merged.Shipping
Core commits its prebuilt
dist/, and each commit here carries its own. Users are protected only once mathworks/data-explorer-vscode pins the core tag that contains this change and ships a release built from it: today's v1.36.5 VSIX bundles the oldTAG_RE.Not in this PR
src/webview/components/dex-matrix-grid.ts:57has the same\d+\.?\d*fragment. It is not typed input only: the grid's cells come from core'sdisplayElements(), and on the core side an element read from a file was confirmed to reach that regex with the slow behaviour once the grid opens. The webview itself was not run, so the vscode side is a prediction. It needs its own change in that repo.crates/dex-model/src/catalog.rs,next_tag/tag_at) has no regex, so CodeQL cannot see it, but it keeps the many-<retry quadratic that commit 1 removes here.XmlUtils.formatComplexXmlstill contains\d+\.?\d*, deliberately unchanged: it is unanchored and never fails once a match starts, so it stays linear (a million characters in about 1 ms).