feat(gfql/cypher): Earley → LALR(1) parser (~70× faster parse) with machine-checked grammar invariants

[lmeyerov]

What

Replaces the GFQL Cypher Earley parser with a single LALR(1) parser (contextual lexer): ~70× faster parse (~0.25ms vs ~17ms raw per query), full WHERE language preserved (OR/XOR/NOT, parens, arithmetic, IN, pattern predicates — no subset restriction), byte-identical ASTs, and filter_dict/where_rows execution routing unchanged. The parser is now backed by a grammar that is provably unambiguous LALR(1) (zero conflicts, builds under Lark strict=True) with the parse-equivalence machine-checked in CI.

This is a mashup: it re-cuts @mj3cheun's #1653 as its foundation, then adds a grammar-purification pass and a machine-checked invariant suite on top. Provenance below is explicit so the review can separate "same as #1653" from "new here."

Provenance — what's kept, added, dropped, deferred

Kept from #1653 (@mj3cheun's 3 commits, first on the branch, essentially verbatim):

• The core switch: Earley → LALR(1), unified where_clause: "WHERE" expr, and the post-parse lift that recovers structured filter_dict predicates by re-parsing the WHERE body with a LALR sub-parser (start="where_predicate_chain").
• The predicate span-offset fix (his 036278ce).
• The flat-AND-chain restriction on the lift (his b7b43248): only a flat A AND B AND … chain lifts to filter_dict; parens / OR / XOR / NOT stay on where_rows. This is a correctness boundary, not just routing (see Edge cases).

Added here (8 commits on top):

• Grammar purification → 8 shift/reduce conflicts to 0, provably unambiguous (details in Design).
• A machine-checked invariant suite (test_grammar_invariants.py): zero-conflict + strict-build, semantic-ambiguity-zero, rule-coverage gate, LALR≡Earley differentials, pinned language fixes.
• A full-repo differential wired into existing CI (test_differential_grammar_parity.py).

Dropped: test_where_metamorphic.py (#1653's 225-line characterization test). It characterizes an execution-engine <>-over-null divergence between filter_dict and where_rows, which is downstream of the AST and orthogonal to the parser. Refiled as #1683 (see Deferred), so the finding stays loud without riding the parser PR as a permanent xfail.

Deferred: the flatten seam removal, the TCK-corpus differential, and the #1683 execution-semantics decision (see the Deferred section at the bottom).

Design

• Unified WHERE grammar: where_clause: "WHERE" expr. The old where_predicates | expr dual rule was a genuine reduce/reduce ambiguity (the structured predicate chain is a syntactic subset of expr) — the thing that forced Earley. Structured flat-AND predicates are recovered by a post-parse lift through a LALR sub-parser (start="where_predicate_chain"). filter_dict vs where_rows routing downstream is unchanged.
• Grammar purification (new in this PR):
  • WHERE binds to its preceding clause in the grammar (bundled into match_clause; with_stage owns WHERE-after-WITH). The WITH..WHERE attachment ambiguity is eliminated, not tie-broken by priorities.
  • Name-rooted dot chains derive only via qualified_name; property_access applies only to composite roots (f(x).y, (e).y, xs[0].y). Kills the dotted-name derivation redundancy.
  • Redundant label_predicate_expr dropped — (n:Admin) parses via grouped_expr over bare_label_predicate_expr (identical AST).
  • List elements exclude a top-level IN — [x IN xs … is comprehension syntax; allowing a bare IN list element overlapped it. Restricting list elements to a no-top-level-IN expr chain also rejects the invalid "list of IN-booleans" uniformly ([1 IN a, 2], [n.x IN a, y] too — these used to parse then fail downstream; [(x IN xs), y] still works for a genuine bool list).
  • Net: 8 shift/reduce conflicts → ZERO. The grammar is now provably unambiguous LALR(1) and builds under Lark strict=True (a build-time proof; also checks lexer collisions). Every input has a single derivation.
  • The clause-sequence assembler is unchanged: bundled MATCH..WHERE is re-flattened to the item sequence the assembler was written against (that's what kept this AST-byte-identical). Removing that seam is a scoped follow-up.

Machine-checked invariants (test_grammar_invariants.py + test_differential_grammar_parity.py)

1. Zero LALR conflicts / strict-clean — test_grammar_has_zero_lalr_conflicts asserts the build emits no conflicts (dependency-free, always-on in CI), and test_grammar_builds_under_strict_mode confirms Lark(strict=True) builds (a build-time proof; needs the optional interegular dep, skips without it). Any grammar edit that introduces an ambiguity produces a conflict and fails CI.
2. Semantic ambiguity = 0 — every corpus query has exactly one Earley derivation and it transforms to a single AST; an independent AST-level confirmation of (1), no residual exceptions.
3. Parser-choice neutrality — the production pipeline under LALR vs Earley yields byte-identical ASTs + identical rejections, on a constructed all-constructs corpus AND a full-repo scrape (1,800+ queries, runs in the existing cypher-frontend-differential-parity CI job via its test_differential*.py glob).
4. Pinned deliberate language fixes — the handful of accept-by-accident shapes that LALR (correctly) rejects are enumerated in DELIBERATE_LANGUAGE_FIXES and asserted both as production rejections and as the only allowed divergences in the full-repo differential (details in Edge cases below). Everything else is accept/reject- and AST-identical to the old parser.

Edge cases & how they're handled

• The lift is an internal optimization, never observable — WHERE n.x = 1 AND n.y = 2 lifts to filter_dict; anything with parens / OR / XOR / NOT / arithmetic stays on where_rows. This is a correctness boundary (from earley to lalr parsing for cypher #1653's b7b43248), not just routing: where_rows treats an absent property as null (openCypher semantics), while filter_dict requires the column to exist and would raise — so e.g. a IS NULL AND (b = 1) with b absent must stay on where_rows. The lift only fires when both paths agree.
• Deliberate language fixes (accept-by-accident shapes, now honest syntax errors, each pinned in DELIBERATE_LANGUAGE_FIXES so the differential stays honest):
  • RETURN DISTINCT (no items) — Earley's dynamic lexer re-lexed DISTINCT as a variable name (returned a var named DISTINCT, distinct=False); LALR's contextual lexer keeps it a keyword.
  • Double WHERE MATCH (a) WHERE p WHERE q — the old flat-clause assembler accepted it and kept both predicates in different AST fields (a latent wrong-answer); WHERE now binds to its clause in the grammar.
  • Double post-WITH WHERE, WHERE after UNWIND, WHERE before MATCH in a graph constructor — same class, all now grammar-level rejections.
  • "List of IN-booleans" [x IN xs, y] / [1 IN a, 2] / [n.x IN a, y] — [x IN xs … is comprehension syntax; a bare IN list element overlapped it, so it's rejected uniformly now (some of these used to parse and then fail downstream with GFQLTypeError). Parenthesize for a genuine bool list: [(x IN xs), y].
• Span fidelity across the lift — bundled MATCH..WHERE reconstructs the MatchClause span to end at the last pre-WHERE character, and lifted predicate spans are retargeted from atom-relative to absolute, so downstream error positions still point at the real source location.
• String/comment masking — pattern-existence and unsupported-syntax pre-scans mask quoted/backticked/commented regions, so lexemes inside string literals aren't misread as syntax.

Extending the grammar is safe-by-construction

The whole point of moving correctness into the grammar is that syntax edits catch their own mistakes. The flow is documented at the top of _GRAMMAR in parser.py (where an editor lands first):

1. Edit/add a rule.
2. Add a query exercising it to DIFFERENTIAL_CORPUS — forced: test_every_grammar_rule_is_exercised_by_the_corpus fails with the new rule's name until you do (every tree-producing rule must appear in some corpus parse). This is the gate that pulls new syntax through all the other checks.
3. The machine invariants then catch the rest automatically:

The edit…	…is caught by
adds a rule with no test	rule-coverage gate names the rule
introduces ANY ambiguity / conflict	zero-conflict + strict-build checks fail
introduces an AST-affecting ambiguity	semantic-ambiguity-zero (derivations disagree)
makes LALR ≠ Earley	the two differentials (constructed + full-repo scrape)
deliberately changes accept/reject	must be pinned in DELIBERATE_LANGUAGE_FIXES, else the differential fails

A new feature's own tests also enter the full-repo scrape automatically, so its queries get swept into the differential even without explicit registration. Net: you cannot extend the grammar and silently skip the safety net — the build fails loudly and tells you which knob to turn.

Performance

Raw parse time, LALR vs Earley over the same grammar (warm, mean of N iterations):

query	LALR	Earley	speedup
MATCH (n) WHERE n.x = 1 RETURN n	0.157 ms	10.6 ms	67×
flat-AND + ORDER BY/LIMIT	0.415 ms	38.7 ms	93×
MATCH (a)-[r:R*1..3]->(b) WITH … WHERE … OR … IN […]	0.612 ms	50.9 ms	83×

~70–90× on the parse step. (Parse memoization already in master via #1652/#1657 caches repeat identical queries and is orthogonal — this speeds up the first/uncached parse of every query.) The grammar-purification pass is parse-neutral (raw parse 0.565 → 0.573 ms; the 8→0 conflict reduction is about analyzability, not speed).

Verification (dgx-spark)

• Full cypher suite: 1,681 passed / 0 failed (6 skipped, 15 xfailed)
• Grammar builds with zero LALR conflicts and under strict=True (provably unambiguous)
• LALR-vs-Earley full-repo differential (1,850+ queries): 0 AST divergences, 0 error-class differences
• Old-vs-new production differential (for the purification pass): 0 AST diffs, exactly the pinned language fixes
• Raw parse speed unchanged by purification (0.57ms)

CI note

The only red check is test-networkx-scipy-policy (nx-upper-scipy, py3.12) — a pre-existing, unrelated flake in the networkx HITS test (a degenerate cycle graph interacting with a newer scipy; this branch touches zero networkx code, and master's last run was green only because its fresh lockfile pulled an older scipy). Fixed independently in #1686; once that lands, rebasing this PR onto master turns CI green.

Deferred / follow-ups

Tracked in the branch plan (plans/gfql-cypher-lalr-parser/plan.md, local — plans/ is gitignored):

• <>/null 3VL execution divergence → filed as GFQL Cypher WHERE: <> over null diverges between filter_dict (table semantics) and where_rows (Cypher 3VL) #1683 (cc @mj3cheun). filter_dict (table semantics) treats null <> 'ab' as TRUE; where_rows (openCypher 3VL) treats it as null/drop — so parenthesization alone can change row membership. This is the finding earley to lalr parsing for cypher #1653's metamorphic test characterized; it is pre-existing execution-engine behavior, orthogonal to this parser PR, and needs a semantics-intent decision (the issue lays out three options). Not blocking here.
• Remove the flatten seam → GFQL Cypher parser: remove the WHERE flatten seam (assembler should consume MatchClause.where directly) #1687 — bundled MATCH..WHERE is currently re-flattened so the legacy clause-sequence assembler runs untouched (that's what kept this change AST-byte-identical). Follow-up: have the assembler consume MatchClause.where directly and delete the positional inference, re-proven with the same differentials. Enabled by the "AST is an internal contract" framing.
• TCK-corpus differential → GFQL Cypher: run the TCK query corpus through the LALR≡Earley parser differential (parser axis) #1688 — the tck-gfql conformance suite already runs green on this branch (semantic axis). Follow-up: also feed the TCK query corpus through the LALR≡Earley differential harness (parser axis) in tck-gfql CI. The harness is corpus-parameterizable; no parser-side work needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W5vkD2ZCyv3bmecBYoYYQy