When a - is followed by a backslash that begins an escape, the check that decides whether an identifier starts here reads the wrong byte. A backslash right before a newline is an invalid escape and cannot start an identifier, so -\<newline> is mis-tokenized.
Reproducer
cssparser 0.37.0, default features.
use cssparser::{Parser, ParserInput};
fn main() {
let mut pi = ParserInput::new("1-\\\n");
let mut p = Parser::new(&mut pi);
let a = p.next().unwrap().clone();
// cssparser: Dimension { value: 1.0, int_value: Some(1), unit: "-" }
println!("{:?}", a);
}
Observed vs expected
Observed for 1-\<newline>: the tokens are Dimension { value: 1.0, unit: "-" } then Delim('\\'). For -\<newline> the first token is Ident("-"). For #-\<newline> the first token is IDHash("-").
Expected per CSS Syntax Level 3, matched by tinycss2: 1-\<newline> gives Number(1), Delim('-'), Delim('\\'). -\<newline> gives Delim('-') then Delim('\\'). #-\<newline> gives a hash token Hash("-") with the unrestricted type flag, not an id hash.
Root cause
src/tokenizer.rs:895. The would-start-an-identifier test reads the byte after the backslash to decide if -\ begins an identifier. A backslash directly before a newline is not a valid escape, but the offset used skips the backslash and reads the newline as though it were escaped.
Scope
A hyphen before an invalid escape produces a unit, an identifier, or an id hash that the input never held. A consumer that keys on token type, such as a linter or a minifier, then acts on tokens the source did not contain.
When a
-is followed by a backslash that begins an escape, the check that decides whether an identifier starts here reads the wrong byte. A backslash right before a newline is an invalid escape and cannot start an identifier, so-\<newline>is mis-tokenized.Reproducer
cssparser 0.37.0, default features.
Observed vs expected
Observed for
1-\<newline>: the tokens areDimension { value: 1.0, unit: "-" }thenDelim('\\'). For-\<newline>the first token isIdent("-"). For#-\<newline>the first token isIDHash("-").Expected per CSS Syntax Level 3, matched by tinycss2:
1-\<newline>givesNumber(1),Delim('-'),Delim('\\').-\<newline>givesDelim('-')thenDelim('\\').#-\<newline>gives a hash tokenHash("-")with the unrestricted type flag, not an id hash.Root cause
src/tokenizer.rs:895. The would-start-an-identifier test reads the byte after the backslash to decide if
-\begins an identifier. A backslash directly before a newline is not a valid escape, but the offset used skips the backslash and reads the newline as though it were escaped.Scope
A hyphen before an invalid escape produces a unit, an identifier, or an id hash that the input never held. A consumer that keys on token type, such as a linter or a minifier, then acts on tokens the source did not contain.