Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@ site: src/states/PARSED_TEXT_CONTENT.ts › PARSED_TEXT_CONTENT

# Treat a backslash-escaped quote as text in parsed-text bodies and parsed strings

Inside a `TagType.text` body the only backslash handling is `STATE.checkForPlaceholder`, which declines unless the run of backslashes leads to `${`, so a `\` falls through to the eager text run and the `"`/`'` after it still enters `STATE.PARSED_STRING`. `src/states/PARSED_STRING.ts` › `PARSED_STRING` has the mirror hole, where the quote of `\"` matches `str.quoteCharCode` and closes the string. An odd number of escaped quotes therefore runs to EOF and emits `INVALID_TEMPLATE_STRING` "EOF reached while parsing string expression"; marko maps `<style>`/`<script>` to `TagType.text`, so it surfaces as a code frame on the line after the tag. Escaped quotes are legal in CSS selectors and `content:` strings, so `<style>.a\" { color: red }</style>` and `content: 'it\'s'` are both unparseable today. The defect is quote-specific rather than backslash-general: `\2014` parses fine, and `src/__tests__/fixtures/parsed-text-style-tag` passes only because its escaped quotes come in pairs. Fix symmetrically in the two files: when `checkForPlaceholder` declines a `CODE.BACK_SLASH` and the next char is a quote (the active `quoteCharCode` in `PARSED_STRING`), consume both chars as text instead of letting the quote change state. Lock it in with a new `src/__tests__/fixtures/` dir plus `pnpm test:update`.
A `TagType.text` body (`<style>`, `<script>`) is lexed as JavaScript strings, templates and comments, and a backslash counts only when `STATE.checkForPlaceholder` finds an escaped `${`. Otherwise it falls through as text and the character after it still switches state; `src/states/PARSED_STRING.ts` › `PARSED_STRING` has the mirror hole, where the quote of `\"` closes the string. So `<style>.a\" {}` and `content: 'it\'s'` run to the end of the file with "EOF reached while parsing string expression", as does a `<script>` regex with an escaped quote (`s.replace(/\"/g, "&quot;")`). A `<script>` regex with `\//` (`/^https?:\/\//`) opens a line comment instead, which silently drops every placeholder after it on that line. Fix both states: inside a parsed string a backslash escapes the next character; in the body it escapes a quote, backtick, slash or backslash, but not `<` or a newline, so `\</style>` still closes the tag and delimited blocks still end at the line. `src/scanner.c` › `scan_frame_content` in marko-js/tree-sitter mirrors both paths and needs the same change. The unescaped quote in `/["']/` is a separate gap: see "Lex regular expression literals in `<script>` bodies".

Check: `node --input-type=module -e 'import{createParser,TagType}from"./src/index.ts";const p=createParser({onError:e=>console.log("ERR",e.message),onText:r=>console.log("text",JSON.stringify(p.read(r))),onOpenTagName:()=>TagType.text});p.parse("<style>.a\\\" { color: red }</style>")'` prints `ERR EOF reached while parsing string expression` today and should print a single `text` range.
Check: `node --input-type=module -e 'import{createParser,TagType}from"./src/index.ts";const p=createParser({onError:e=>console.log("ERR",e.message),onPlaceholder:r=>console.log("placeholder",p.read(r.value)),onOpenTagName:()=>TagType.text});p.parse(process.argv[1])' '<script>if (/^https?:\/\//.test(u)) go(${x})</script>'` prints nothing (no `placeholder x`); with `'<style>.a\" {}</style>'` it prints `ERR EOF reached while parsing string expression`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
type: bug
impact: high
effort: med
site: src/states/PARSED_TEXT_CONTENT.ts › PARSED_TEXT_CONTENT
---

# Add a text body mode that does not lex JavaScript, for prose and CSS bodies

`TagType.text` is the only text-only body mode, and `PARSED_TEXT_CONTENT.parse` lexes it as JavaScript: a quote enters `PARSED_STRING`, a backtick `TEMPLATE_STRING`, and `//` or `/*` enter `JS_COMMENT_LINE`/`JS_COMMENT_BLOCK`. That fits `<script>`, but the README's `onOpenTagName` example also returns `TagType.text` for `textarea` and `html-comment`, and marko does the same for `<title>`, `<textarea>` (compiler `marko-html.json` `parse-options.text`) and `<html-comment>`. So an apostrophe in prose (`<title>It's</title>`, `<textarea>Don't</textarea>`) fails with "EOF reached while parsing string expression", `<title>a /* b</title>` swallows the closing tag, and a `${x}` inside backticks stays literal text. `<style>` hits the `//` case: in `url(http://x) ${c}` the `//` opens a line comment, so the placeholder reaches the CSS as a literal `${c}`. Add an opt-in TagType where only placeholders and the matching closing tag are special (HTML's own RAWTEXT/RCDATA rule), keeping existing `TagType.text` events unchanged, and have marko's `packages/compiler/src/babel-plugin/parser.js › onOpenTagName` select it for every `parseOptions.text` tag except `<script>`/`<html-script>`. That mode also settles the `<style>` cases of "Treat a backslash-escaped quote as text in parsed-text bodies and parsed strings", whose `PARSED_STRING` fix `<script>` still needs.

Check: `node --input-type=module -e 'import{createParser,TagType}from"./src/index.ts";const p=createParser({onError:e=>console.log("ERR",e.message),onText:r=>console.log("text",JSON.stringify(p.read(r))),onPlaceholder:r=>console.log("placeholder",JSON.stringify(p.read(r.value))),onOpenTagName:()=>TagType.text});p.parse(process.argv[1])' "<title>It's</title>"` prints `ERR EOF reached while parsing string expression`; the same command with `'<style>.a { background: url(http://x) ${c} }</style>'` prints a single `text` range containing `${c}` and no `placeholder`. In marko, `pnpm run compile -- -o html -d` on a template containing `<title>It's</title>` fails with the same error.
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
type: bug
impact: low
effort: low
site: src/states/EXPRESSION.ts › lookBehindForOperator
---

# Treat a spaced or `}`-trailing TypeScript `!` as postfix when it ends an unenclosed value

`lookBehindForOperator` reads a trailing `!` (or run of `!`) as postfix only when the character directly before it ends an operand: a word that is not a keyword operator, `)`, `]`, a closing quote or a backtick. A non-null assertion after whitespace (`x !`, which TypeScript accepts) or after an object literal (`{a:1}!`) is still read as a prefix operator waiting for an operand, so an unenclosed attribute, tag-variable or statement value ending in one swallows what follows (`<div a=x ! b/>` gives the one value `x ! b`). Both shapes are rare (prettier prints `x!`), so decide whether they are worth the extra look-behind; if so, walk back over whitespace before testing for an operand. A `}` needs care, since in a statement it can also close a block that a prefix `!` follows. Add a fixture per shape.

Check: `node --input-type=module -e 'import{createParser}from"./src/index.ts";const p=createParser({onAttrValue:r=>console.log("value",JSON.stringify(p.read(r.value)))});p.parse("<div a=x ! b/>")'` prints `value "x ! b"`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
type: bug
impact: med
effort: med
site: src/states/PARSED_TEXT_CONTENT.ts › PARSED_TEXT_CONTENT
---

# Lex regular expression literals in `<script>` bodies

`PARSED_TEXT_CONTENT` lexes a `TagType.text` body as JavaScript strings, templates and comments, but checks a `/` only for `//` and `/*`: it never enters `STATE.REGULAR_EXPRESSION` the way `EXPRESSION` does. A quote inside a regex literal therefore opens a string, so `<script>s.replace(/["']/g, "")</script>` runs to the end of the file with "EOF reached while parsing string expression". (An escaped quote or slash has its own item, "Treat a backslash-escaped quote as text in parsed-text bodies and parsed strings".) Regex lexing cannot simply be added, because the same body mode serves `<style>`, where `url(/favicon.ico)` would read as an unterminated regex. Once `<style>` and the other prose bodies move to a mode that does not lex JavaScript (see "Add a text body mode that does not lex JavaScript"), enter `REGULAR_EXPRESSION` here when the previous significant character cannot end an operand, as `EXPRESSION` does with `canFollowDivision`, and add a fixture.

Check: `node --input-type=module -e 'import{createParser,TagType}from"./src/index.ts";const p=createParser({onError:e=>console.log("ERR",e.message),onOpenTagName:()=>TagType.text});p.parse("<script>s.replace(/[\"\x27]/g, \"\")</script>")'` prints `ERR EOF reached while parsing string expression`.
Loading