Skip to content

is_ident_start reads the wrong byte for a hyphen before a backslash #440

Description

@hey-jj

When a - is followed by a backslash that begins an escape, the check that decides whether an identifier starts here reads the wrong byte. A backslash right before a newline is an invalid escape and cannot start an identifier, so -\<newline> is mis-tokenized.

Reproducer

cssparser 0.37.0, default features.

use cssparser::{Parser, ParserInput};

fn main() {
    let mut pi = ParserInput::new("1-\\\n");
    let mut p = Parser::new(&mut pi);
    let a = p.next().unwrap().clone();
    // cssparser: Dimension { value: 1.0, int_value: Some(1), unit: "-" }
    println!("{:?}", a);
}

Observed vs expected

Observed for 1-\<newline>: the tokens are Dimension { value: 1.0, unit: "-" } then Delim('\\'). For -\<newline> the first token is Ident("-"). For #-\<newline> the first token is IDHash("-").

Expected per CSS Syntax Level 3, matched by tinycss2: 1-\<newline> gives Number(1), Delim('-'), Delim('\\'). -\<newline> gives Delim('-') then Delim('\\'). #-\<newline> gives a hash token Hash("-") with the unrestricted type flag, not an id hash.

Root cause

src/tokenizer.rs:895. The would-start-an-identifier test reads the byte after the backslash to decide if -\ begins an identifier. A backslash directly before a newline is not a valid escape, but the offset used skips the backslash and reads the newline as though it were escaped.

Scope

A hyphen before an invalid escape produces a unit, an identifier, or an id hash that the input never held. A consumer that keys on token type, such as a linter or a minifier, then acts on tokens the source did not contain.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions