Skip to content

Scan the catalog's tags, a decimal's digits and cdata's .0 in linear time - #74

Merged
ww-mw merged 3 commits into
mainfrom
fix-redos-file-scanners
Oct 11, 2026
Merged

ww-mw merged 3 commits into
mainfrom
fix-redos-file-scanners

Conversation

@ww-mw

@ww-mw ww-mw commented Oct 11, 2026

Copy link
Copy Markdown
Member

Fixes code-scanning alert 2 (js/polynomial-redos, TAG_RE in src/datamodel/parser/ScCatalog.ts). After it, a sweep for the same shape found three more patterns that a file's bytes reach, and one that only typed input reaches. CodeQL does not flag those. There are three commits, and each one carries its rebuilt dist/:

  1. 85d65e7 Find the interface dictionary's tags in linear time, as the regex found them. This is alert 2.
  2. bebec60 Read a decimal's digits one way, so text that is no number is refused in linear time. \d+\.?\d* becomes \d+(?:\.\d*)? in five patterns.
  3. 552b8fc Drop MATLAB's .0 from cdata text in linear time, with the matches the replace made. (?<!\d) is added to the cdata .0 drop, at both of its sites.

No change alters a result the old code finished computing, only how long it took. The one difference is commit 1's: where the old regex threw a RangeError (below), the dictionary now opens. On a backtracking engine each old pattern was quadratic, or cubic, in the length of the wrong bytes. Those bytes are the file's own, and a deflated part shrinks them about 1000:1.

The sites

Measured on this machine (Node 24.14.1). "Before" is the base, origin/main 8facc33, or the commit's parent. "After" is this branch.

Site How a file reaches it Before After
ScCatalog.ts TAG_RE (scanScXml). Alert 2, quadratic Opening a binary .sldd: the interface-dictionary part <A and 100,000 -: 12.4 s. <A 100,000 times: 24.7 s. A 1,187 B dictionary held readSlddContent for 11.8 s (all three are commit 1's figures) 1.0 ms and 3.2 ms. At ten times the size: 3.5 ms and 24 ms
XmlUtils.ts COMPLEX_PART (parseComplexNum). Two per element, so cubic Opening a text .sldd: complex cdata text marked as this package's own non-finite text (MatlabVariableNode._isOwnNonFiniteText). The Property Inspector's Other rows (piOther.formatCdata) 5,000 digits, +, 5,000 more and an x: 55 s (1,000 characters: 80 ms; 3,000: 1.5 s). A 5,439 B text dictionary holding half that took 6.9 s to open 0.25 ms. A million digits: 1.1 ms. The dictionary, at twice the size: 1.5 ms
SlxParser.ts NUMERIC_RE (valueReferencesData). Quadratic Opening a .slx or .mdl: every block <P> value 100,000 digits and an x: 19.6 s. Opening a 744 B .slx holding that: 19.6 s 0.5 ms. A million digits: 4.1 ms, and a 1,615 B .slx holding them opens in 47 ms
cdata .0 drop, MatlabVariableNode._parseCdataText and piOther.formatCdata. Quadratic Opening a .sldd of either kind: complex cdata text. The Property Inspector's Other rows 100,000 digits: 4.3 s to read and 4.3 s to show. A 100,419 B text dictionary holding them took 4.4 s to open, and a 6,076 B binary one 4.3 s 0.5 ms. Ten million digits: 49 ms. The binary dictionary, with a million digits: 0.02 s
MatlabValueParser.ts: parseMatlabNumber, and parseComplex's two patterns Not from a file. These read what is typed into a cell, and opening all 447 files below never reached them 100,000 digits and an x: 5.6 s. 2,500 digits, +, 2,500 more and an x: 6.9 s 0.3 ms or less

Why each result is the same

Commit 1. XmlTagScanner returns exactly the regex's match sequence: each tag's index, end and four groups. The regex stays in test/scCatalog.test.ts as the specification, and the scanner is held to it tag for tag, on the real catalog and on 20,000 seeded random strings. Over every XML part of the 243 zip-packaged files in test/fixtures, test/parity/artifacts and the local corpus (4,026 parts, 14,698,222 tags), the two agree on every tag. The scanner differs from the regex only where the regex never finished: past ~8.39M backtrack steps V8 threw a RangeError, so one stray " ahead of a long attribute value failed the whole read, and now the dictionary opens.

Commit 2. \d+(?:\.\d*)? matches the same strings as \d+\.?\d* and leaves every capture group where it was; its digits split only one way.

Commit 3. (?<!\d) lets a match start only at the first digit of a run, and changes no match. \.0 has to follow \d+, so from any digit of a run \d+ takes the rest of it, and the match holds or fails on what follows the run, the same for every start in it. The global scan reaches a run's first digit before the rest and never resumes inside a run (a match is followed by +, -, i or the end), so a start inside a run only ever failed.

Checked, for commits 2 and 3. The old and new patterns were read out of the source files rather than retyped, and compared on test(), the full exec() result and, for the .0 drop, the replace output and the whole match sequence. A deliberately wrong rewrite at each of the six patterns produced thousands of mismatches, so the harness can tell them apart. The real ones gave 0 mismatches on:

  • every string of length 0 to 8 over 019.eE+-iInfNax and the space: 4,581,298,449 strings;
  • 5,229,043 strings of up to six tokens drawn from those and Inf, NaN and Infinity;
  • every string the patterns were applied to while opening the 447 .sldd, .slx, .mdl and .mat files of test/fixtures, test/parity/artifacts and the local corpus through the built package, with the Property Inspector object of each of their 3,288,107 nodes built: NUMERIC_RE 241,638 applications (129,486 distinct strings), COMPLEX_ELEMENT 16 (9), the .0 drop 320 (98). Opening a file never reaches MatlabValueParser;
  • all 129,584 distinct strings that run saw, put through every one of the six patterns, and the 1,676,663 distinct string leaves and whitespace tokens of the 222 dictionaries' content as read.

11 of the 447 files do not open, for reasons outside this change: ten v7.3 MAT-files, and one 866 MB text dictionary past V8's longest string.

Regression tests

Each site has a test beside its module's existing tests that runs the shape it was slow on. With that site's old pattern put back alone, its test fails and the new pattern passes:

Site put back Old pattern Bound
COMPLEX_PART 7.4 to 7.7 s (and commit 3's piOther case at 4.3 s, which reaches it too) 100 ms / 1 s
NUMERIC_RE (test/blockParamUsages.test.ts) 4.9 s 1 s
MatlabValueParser :53 / :169 / :176 (lines at 8facc33) 1.4 / 1.1 / 1.1 s 100 ms
.0 drop, MatlabVariableNode / piOther 4.3 s each 1 s

Commit 1's linear-time test in test/scCatalog.test.ts runs five shapes of about a million characters each (1.4 million for the last). Only the first is ten times a size in the table above; <A repeated runs at five times, and the other three are not in the table. They take 7 to 83 ms here, against a 2 s bound. (Commit 1's message says the five shapes run "at ten times the size"; that holds for the first only.)

The bounds follow fe238c7: each is past five times its case's CI time, estimated at 3.3 times this machine's slowest over two full runs. Of the cases commits 2 and 3 add, the slowest across their four full runs (two at each commit) was blockParamUsages' at 25 ms, against a 1 s bound; commit 1's slowest, above, was 83 ms against 2 s. The .0 drop itself is pinned too: it applies only where a part ends, after a part of many digits as after one of one.

Verification

Shipping

Core commits its prebuilt dist/, and each commit here carries its own. Users are protected only once mathworks/data-explorer-vscode pins the core tag that contains this change and ships a release built from it: today's v1.36.5 VSIX bundles the old TAG_RE.

Not in this PR

  • mathworks/data-explorer-vscode src/webview/components/dex-matrix-grid.ts:57 has the same \d+\.?\d* fragment. It is not typed input only: the grid's cells come from core's displayElements(), and on the core side an element read from a file was confirmed to reach that regex with the slow behaviour once the grid opens. The webview itself was not run, so the vscode side is a prediction. It needs its own change in that repo.
  • dex-core's Rust port (crates/dex-model/src/catalog.rs, next_tag / tag_at) has no regex, so CodeQL cannot see it, but it keeps the many-< retry quadratic that commit 1 removes here.
  • XmlUtils.formatComplexXml still contains \d+\.?\d*, deliberately unchanged: it is unanchored and never fails once a match starts, so it stays linear (a million characters in about 1 ms).

ww-mw added 3 commits October 11, 2026 19:16
…nd them

scanScXml read each tag of a binary dictionary's interface-dictionary part with
/<(\/?)([A-Za-z_][\w.-]*)((?:[^>"]|"[^"]*")*?)(\/?)>/g, which CodeQL reports as
js/polynomial-redos (code-scanning alert 2). On a backtracking engine it is quadratic on the
wrong bytes, and the bytes are the file's. Measured on this machine: `<A` and 100,000 `-`
took 12.4 s, and `<A` 100,000 times 24.7 s. The part deflates about 1000:1, so a 1,187 B
binary dictionary held readSlddContent for 11.8 s. Past ~8.39M backtrack steps V8 throws a
RangeError instead, so one stray `"` ahead of a long attribute value failed the whole read.

XmlTagScanner returns exactly the regex's match sequence, the tag's index, end and four
groups, with forward indexOf's. It remembers an attribute walk that failed, so the retry at
each later `<` costs a count of quotes rather than a walk to the end. The same dictionary
opens in 3 ms, and the test's five shapes, at ten times the size, take 7 to 83 ms.

test/scCatalog.test.ts keeps the regex as the specification and holds the scanner to it
tag for tag, on the real catalog and on 20,000 seeded random strings. Over every XML part of
the 243 zip-packaged files in test/fixtures, test/parity/artifacts and the local corpus,
4,026 parts and 14,698,222 tags, the two agree on every tag. The linear-time case is held
to 2 s, five times CI's time for its slowest shape (83 ms here, over two full runs, at 3.3
times this machine's; fe238c7).
… in linear time

Five patterns read a decimal as `\d+\.?\d*`. With no dot that is `\d+\d*`, and a backtracking
engine tries every split of a run of digits between the two before it gives up, so text that
is not a number cost the square of its length, and the cube in a pattern that holds two
decimals. A sweep for the shape after code-scanning alert 2 found them; CodeQL does not report
them. Measured on this machine:

  - XmlUtils.parseComplexNum (COMPLEX_PART, two in each element) reads a text dictionary's
    own non-finite complex text on open (MatlabVariableNode._isOwnNonFiniteText), and a
    complex value's text in the Property Inspector (piOther.formatCdata). 5,000 digits, `+`,
    5,000 more and an `x` took 55 s, and a 5,439 B text dictionary holding half as many took
    6.9 s to open.
  - SlxParser's NUMERIC_RE judges every block <P> value of a model (valueReferencesData).
    100,000 digits and an `x` took 19.6 s, and so did opening a 744 B .slx that holds them.
  - MatlabValueParser's parseMatlabNumber and parseComplex read what is typed into a cell:
    5.6 s for 100,000 digits and an `x`, 6.9 s for 2,500, `+`, 2,500 more and an `x`.

`\d+(?:\.\d*)?` matches the same strings and leaves every group where it was, and its digits
split only one way. The same inputs take 0.1 to 0.5 ms, and a million digits 1 to 4 ms.

The old and new pattern at each site agree on test() and on every exec() group for all
4,581,298,449 strings of up to eight characters over `019.eE+-iInfNax` and the space, and for
5,229,043 strings of up to six tokens from those and `Inf`, `NaN` and `Infinity`. Opening the
447 .sldd, .slx, .mdl and .mat files of test/fixtures, test/parity/artifacts and the local
corpus through the built package, and building the Property Inspector object of each of their
3,288,107 nodes, applies NUMERIC_RE to 129,486 distinct strings and parseComplexNum's pattern
to 9. Opening a file never reaches MatlabValueParser. Old and new agree on all of them, on all
129,584 distinct strings that run saw put through every pattern, and on the 1,676,663 distinct
string leaves and whitespace tokens of the 222 dictionaries' content as read. (11 of the 447
do not open, for reasons outside this change: ten v7.3 MAT-files, and a text dictionary past
V8's longest string.)

Each new test fails with its old pattern put back alone, at 1.1 to 7.7 s against bounds of
100 ms and 1 s, and passes with the new. Each bound is past five times its case's CI time; the
slowest case is 11 ms here over two full runs (blockParamUsages), at 3.3 times this machine's
(fe238c7).
… replace made

MatlabVariableNode._parseCdataText and piOther.formatCdata drop MATLAB's `.0` from each part
of complex cdata text with `.replace(/(\d+)\.0(?=[+\-i]|$)/g, '$1')`. The scan tries a match
at every digit of a run that no `.0` follows, and each walks the rest of the run, so a run of
digits cost its square. The text is a dictionary's cdata as read, a binary one's complex value
or a text one's, on open and in the Property Inspector. Measured on this machine: 100,000
digits took 4.3 s to read and 4.3 s to show, and a 100,419 B text dictionary holding them took
4.4 s to open. CodeQL does not report it.

`(?<!\d)` lets a match start only at the first digit of a run, and changes no match. `\.0` has
to follow `\d+`, so from any digit of a run `\d+` takes the rest of it, and the match holds or
fails on what follows the run: the same for every start in it. The scan reaches a run's first
digit before the rest, because it never resumes inside a run (a match is followed by `+`, `-`,
`i` or the end). So a start inside a run only ever failed. 100,000 digits now take 0.5 ms, and
ten million 49 ms.

The old and new replace give the same output and the same matches, index and group, on all
4,581,298,449 strings of up to eight characters over `019.eE+-iInfNax` and the space, and on
5,229,043 strings of up to six tokens from those and `Inf`, `NaN` and `Infinity`. Opening the
447 files of test/fixtures, test/parity/artifacts and the local corpus through the built
package, with every node's Property Inspector object built, runs it 320 times on 98 distinct
strings. Old and new agree on those, on all 129,584 distinct strings that run saw at any of
the six patterns of this commit and the last, and on the 1,676,663 distinct string leaves and
whitespace tokens of the 222 dictionaries' content as read.

test/matlabVariableNode.test.ts and test/piOther.test.ts hold 100,000 digits to 1 s. Each
fails at 4.3 s with its own site's old replace put back alone, and passes with the new. Both
take 0.6 ms here over two full runs, so 1 s is past five times CI's time, at 3.3 times this
machine's (fe238c7). The slowest of every new case across these two runs and the last
commit's was blockParamUsages' at 25 ms, a fortieth of its bound. The `.0` drop itself is
pinned: only where a part ends, and after a part of many digits as after one of one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant