HTML API: Review locale-dependent ASCII comparisons

[sirreal]

Makes ASCII case-insensitive matching locale-independent across the HTML API and WP_Token_Map.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

Problem

Every case-insensitive comparison in the HTML API routed through locale-sensitive PHP functions, in two classes:

• substr_compare() with its case-insensitivity flag folds bytes via the C library's tolower() under the current process locale on every PHP version, including current master — the PHP 8.2 locale-independent case RFC did not touch it.
• strtolower(), strtoupper(), strcasecmp(), and stripos() are locale-dependent before PHP 8.2 (core supports 7.4+), and ctype_upper() is locale-dependent on every version.

Any plugin can change the process locale via setlocale(), and some platform defaults already fold non-ASCII bytes. Reproduced on PHP 8.5.8 under macOS' default C.UTF-8 locale with no setlocale() call:

substr_compare( "\xEC\xB8", "\xCC\xB8", 0, 2, true ); // 0 — "matches"

// U+0338 is "\xCC\xB8" in UTF-8. Same decoded value, opposite answers:
WP_HTML_Decoder::attribute_starts_with( '̸',  "\xEC\xB8", 'ascii-case-insensitive' ); // true  (wrong)
WP_HTML_Decoder::attribute_starts_with( "\xCC\xB8", "\xEC\xB8", 'ascii-case-insensitive' ); // false (correct)

Consequences:

1. Non-ASCII bytes spuriously match as case variants of each other: tag names, quirks-mode class names, attribute prefixes, DOCTYPE identifiers, and token-map words differing only in non-ASCII bytes are treated as the same name.
2. Under locales which fold across the ASCII boundary (e.g. Turkish dotless-i on glibc), pure-ASCII needles fail to match: PUBLIC does not match public in a DOCTYPE, </SCRIPT> is not escaped when writing JavaScript content into a <script> element, and application/xhtml+xml is not recognized as an HTML integration point encoding.
3. The same value gives different answers raw vs. encoded, and behavior varies across PHP versions and process locales.

Found via differential fuzzing of WP_HTML_Decoder::attribute_starts_with() against a decode_attribute() + str_starts_with() oracle; the audit then covered every case-fold site in src/wp-includes/html-api/ plus WP_Token_Map.

Changes

One commit per class, tests first:

1. WP_HTML_Decoder — adds locale-independent primitives (matches_ascii_case_insensitively(), ascii_lowercase(), ascii_uppercase(); @since 7.1.0, @access private) and uses them for both comparison paths of attribute_starts_with(), so the plain-character and character-reference paths can no longer diverge.
2. WP_HTML_Tag_Processor — migrates all 21 sites: sought tag names, quirks-mode class_list/has_class/add_class/remove_class, attribute-name normalization, get_tag(), RAWTEXT closer scanning, script type detection, and script-content escaping plus its rejection guard.
3. WP_HTML_Doctype_Info — <!DOCTYPE/PUBLIC/SYSTEM keyword matching and name/identifier folding for quirks-mode detection. Adds a WP_HTML_Decoder dependency, matching the Tag Processor's existing one.
4. WP_HTML_Processor + WP_HTML_Open_Elements — tag queries, breadcrumbs, foreign-content end tags, annotation-xml integration-point encodings, meta Content-Type, hidden input type, encoding labels, is_special()/is_void(), and ctype_upper() → strspn() in current_node_is().
5. WP_Token_Map — the same bug class in its documented ascii-case-insensitive mode (stripos, substr_compare, strtoupper). Kept dependency-free with private copies of the helpers. This also fixes a pre-existing misplaced parenthesis, strtoupper( $a !== $b ), which uppercased a boolean: ignore-case lookups of small words never folded case beyond the first byte, so read_token( 'AB', 0, $l, 'ascii-case-insensitive' ) failed to find 'ab' even though contains( 'AB', 'ascii-case-insensitive' ) reported it present — inconsistent on every PHP version and locale.

The fold helpers are strtr()-based (C-level, locale-free, identical to strtolower()/strtoupper() output on PHP 8.2+ under the C locale); the comparison helper is a zero-allocation byte loop with the same semantics as the substr_compare() calls it replaces, including needle-past-end-of-haystack behavior.

Testing notes

• Red-first locally (macOS, PHP 8.5.8, default C.UTF-8): decoder character-reference rows, next_tag() non-ASCII tag names, quirks-mode has_class and add_class/remove_class dedup, and both WP_Token_Map tests all failed before the fix.
• Canary pins: the DOCTYPE, script-escaping, strcasecmp, and strtolower/strtoupper failure modes require pre-8.2 PHP or a glibc Turkish locale, neither reachable on the development machine (macOS' tr_TR charmap does not implement the dotless-i fold). Those tests are born green but pin the contract with byte pairs chosen to fail loudly if a locale-sensitive comparison ever returns (0xCC/0xEC, 0xC4/0xE4 = Ì/ì, Ä/ä in ISO-8859-1; 0xDD = İ in ISO-8859-9).
• A dedicated decoder test probes for a locale whose tolower() folds 0xC4 onto 0xE4, runs the assertions under it, and restores the locale in finally (skips where no such locale exists).
• Existing tests already covered many fold-direction behaviors (</ScRiPt> escaping, mixed-case PublIC, both Content-Type spellings), so providers were extended rather than duplicated.
• Suites run clean: html-api + html-api-token-map (6037 tests), kses, interactivity-api; PHPCS clean on all changed files.

Scope

This covers src/wp-includes/html-api/ and WP_Token_Map (the HTML API's companion class, whose API documents the same 'ascii-case-insensitive' contract). Locale-sensitive case folding elsewhere in core (kses, shortcodes, …) is a wider, separate question.

Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Model(s): Claude Fable 5
Used for: The original decoder bug was found during an AI-assisted differential-fuzzing review. The audit, tests, implementation, and commit messages in this PR were authored by Claude Code working from a written handoff and task specification; directed and reviewed by @sirreal.

---

This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.