earley to lalr parsing for cypher

[mj3cheun]

Switch GFQL Cypher WHERE parser from Earley to LALR(1) (#1031)

Summary

Unifies the Cypher WHERE grammar and replaces the Earley parser with a single LALR(1) parser. The dual where_clause: where_predicates | expr rule had a FIRST-set ambiguity that forced Earley; collapsing it to one generic boolean expr makes the grammar LALR(1)-parseable. The structured-vs-row routing the old grammar did in-grammar is now recovered by a post-parse lift, so behavior is preserved (and slightly extended) while parsing gets dramatically cheaper.

What changed

Grammar

• where_clause unified to "WHERE"i expr -> generic_where_clause (removed the where_predicates alternative). where_predicate is retained only as a start symbol for the lift parser.

Parsers (parser.py)

• _parser() (Earley) → _parser_lalr() (parser="lalr"), now the sole whole-query parser.
• _pattern_parser() switched Earley → LALR(1).
• New _where_predicate_parser() (start="where_predicate", LALR) backing the lift.

Post-parse routing lift (new, parser.py)

• _lift_atom_as_where_predicate() — re-parses one WHERE atom into a structured WherePredicate, or None.
• _lift_and_spine_predicates() — flattens the top-level AND spine and lifts every conjunct; all-or-nothing. Wired into generic_where_clause: a fully-liftable clause becomes structured predicates (→ filter_dict), otherwise the whole clause stays on expr_tree (→ where_rows).

Removed

• The now-unreachable where_predicates and where_clause transformer methods.

Docs / tests

• New parser tests: single-LALR corpus parse, Earley-removal assertions, and flat/nested/OR lift-routing parity.
• CHANGELOG entry; ai/gfql_where_routing_optimization.md documenting deferred optimizations (OR→is_in, partial pushdown) for backend-aware reimplementation.

Behavior impact

• No query-syntax changes. Every previously-supported query still parses, including OR/XOR/NOT-in-WHERE and post-WITH WHERE.
• Query results unchanged.
• One internal routing change (result-identical): nested/parenthesized ANDs of simple predicates — e.g. WHERE a = 1 AND (b = 2 AND c = 3) — now route to the structured filter_dict fast path. The old flat where_predicate ("AND" where_predicate)* rule couldn't see through parentheses and sent these to where_rows. The reroute is safe by AND-associativity (a AND (b AND c) ≡ a AND b AND c) and one-directional (only where_rows → filter_dict, never the reverse); flat-AND routing is byte-for-byte unchanged.

Performance impact

• Parse: LALR(1) replaces Earley on every supported query (~80× faster parse path per the former Earley cost).

• Execution (nested-AND reroute, cuDF, 1M rows, measured):

  nested-AND workload	before (where_rows)	after (filter_dict)	speedup
  equality a=1 AND (b=2 AND c=3)	364 ms	80 ms	4.5×
  string CONTAINS (3 terms)	102 ms	69 ms	1.5×
  low-selectivity <> (each keeps ~90%)	350 ms	77 ms	4.5×

  The win scales with the number of AND terms (each term is a full-frame eager-3VL pass under where_rows), so multi-term nested ANDs are exactly the favorable case. No new pessimization class is introduced — flat ANDs already used filter_dict; this just makes the parenthesized form consistent.

Tests

• graphistry/tests/compute/gfql/cypher/test_parser.py: 165 passing
• Full graphistry/tests/compute/gfql/cypher/: 1655 passing, 15 xfailed (pre-existing)

[lmeyerov]

I found a behavior regression with a focused base-vs-PR pressure test on dgx-spark, using the RAPIDS/cuDF image graphistry/test-rapids-official:26.02-gfql with cuDF 26.02.01. This was not just parser-shape testing: I ran the same Cypher queries against base f08c4588 and current PR head 036278ce, in both pandas and cuDF.

The failing case is a missing-property null check inside a parenthesized AND:

MATCH (n)
WHERE n.missing IS NULL AND (n.x = 1)
RETURN n.id AS id
ORDER BY id

Base returns rows where x = 1, treating the absent property as null:

[{"id": "a"}, {"id": "c"}, {"id": "d"}]

PR head raises in both pandas and cuDF:

GFQLSchemaError: Column "missing" does not exist in node dataframe

The same problem happens for:

MATCH (n)
WHERE n.missing IS NOT NULL AND (n.x = 1)
RETURN n.id AS id
ORDER BY id

Base returns no rows. PR head raises the same missing-column error in both pandas and cuDF.

A fixed-length nested-AND control case still passes on both base and PR, so the issue is not every nested AND. The unsafe change is lifting IS NULL / IS NOT NULL checks on possibly absent properties into the early property-filter path. The old row-filter path preserves missing-property-as-null behavior; the new early property filter requires the dataframe column to exist.

Suggested fix: do not lift missing-property-sensitive null checks into early property filters, or make the early property filter match row-filter behavior for absent properties.

Remote repro artifacts from my run:

• Script: /tmp/lmeyerov-codex-pr1653-20260626-233411/focused_where_pressure.py
• Report: /tmp/lmeyerov-codex-pr1653-20260626-233411/focused-where-pressure-report-r2.md
• Raw outputs: /tmp/lmeyerov-codex-pr1653-20260626-233411/focused-base-r2.json and /tmp/lmeyerov-codex-pr1653-20260626-233411/focused-head-r2.json

[mj3cheun]

lift/unroll parenthesised is removed, this is parser switch + a parity lift that reconstructs base's routing only now @lmeyerov

validated using test amplification (1,040 zero-divergence differential queries against base)

[mj3cheun]

graphistry/tck-gfql#191

failing tck-gfql test issue made here, looks like it shouldnt be related to PR?

[lmeyerov]

Superseded by #1682 ("Earley → LALR(1) parser", merged into master 2026-07-05) — the core Earley→LALR work here landed there, and this PR is now CONFLICTING/stale (last updated 2026-06-30). Recommend closing. One thing to salvage first: the test_where_metamorphic.py here is not in master — the <>-over-null 3VL divergence it surfaces is tracked in the still-open #1683, but the metamorphic test itself would be worth landing on top of the merged parser if it isn't elsewhere.

[lmeyerov]

Superseded by #1682 (merged), which landed the Earley→LALR(1) switch + the post-parse routing lift from this PR, then went further with a provably-unambiguous, zero-conflict grammar (strict-build) and a machine-checked LALR≡Earley full-repo differential. Closing to avoid a stale/conflicting duplicate.

Salvage candidates worth forking out as their own PRs once we have benchmarks to justify them (this PR is the reference for the design intent):

1. WHERE routing optimizations — from ai/gfql_where_routing_optimization.md in this branch: OR→is_in rewrite and partial predicate pushdown for backend-aware lowering. These are the classic 'route more of the WHERE to the structured filter_dict fast path vs where_rows' wins, but the payoff is engine- and shape-dependent — gate on benchmarks before landing. (Note: perf(gfql/cypher): single-alias predicate pushdown in the row pipeline (L4) #1702 is already forking out single-alias predicate pushdown in the row pipeline, so coordinate with it.)

2. Nested-AND-through-parentheses → filter_dict fast path — WHERE a=1 AND (b=2 AND c=3) routing to the structured path (result-identical by AND-associativity). feat(gfql/cypher): Earley → LALR(1) parser (~70× faster parse) with machine-checked grammar invariants #1682 kept the AND-spine lift; if any paren-nesting case still lands on where_rows, benchmark whether forcing it structured is worth it.

3. Metamorphic WHERE differential test (test_where_metamorphic.py, 225 lines) — dropped from feat(gfql/cypher): Earley → LALR(1) parser (~70× faster parse) with machine-checked grammar invariants #1682 to keep that PR tight. Worth re-adding as standalone test infra (independent of benchmarks): a metamorphic/differential harness over the WHERE surface is a strong regression net for exactly this kind of routing change.

cc @mj3cheun — grammar work is fully in via #1682; the above are the pieces from here still worth harvesting.