fix(parser)!: move JSON extraction operators to Postgres's binary-operator precedence tier - #8063
fix(parser)!: move JSON extraction operators to Postgres's binary-operator precedence tier#8063fivetran-kwoodbeck wants to merge 4 commits into
Conversation
georgesittas
left a comment
There was a problem hiding this comment.
There are some operators that also aren't parsed correctly in main today: ?&, ?|, @>, <@ and &&. The problem is that they're in RANGE_PARSERS, i.e., in the "comparison tier". We can fix them as a follow-up.
SELECT '{"a":[1,2]}'::jsonb @> '{"a":[1]}'::jsonb -> 'a';
SELECT '{"a":[1]}'::jsonb <@ '{"a":[1,2]}'::jsonb -> 'a';
SELECT '{"a":1}'::jsonb ?& ARRAY['a'] -> 'b';
SELECT '{"a":1}'::jsonb ?| ARRAY['a'] -> 'b';
SELECT ARRAY[1] && ARRAY[1] -> 'a';
-- ERROR: operator does not exist: boolean -> unknown|
|
||
| # Binary-operator-tier JSON extraction ops (Postgres's "any other operator" tier, | ||
| # below +/-, level with ||). Same value signature as COLUMN_OPERATORS: (self, this, rhs). | ||
| JSON_EXTRACT_OPERATORS: t.ClassVar[dict[TokenType, t.Callable]] = {} |
There was a problem hiding this comment.
Not sure if this is a good name, e.g., Postgres adds JSONBContains in it which isn't an "extraction. What about BITWISE_OPERATORS?
There was a problem hiding this comment.
I'm not sure about BITWISE_OPERATORS, they don't have AND/OR/XOR/shift semantics. How about just JSON_OPERATORS?
| if k not in (TokenType.ARROW, TokenType.DARROW) | ||
| } | ||
|
|
||
| JSON_EXTRACT_OPERATORS = { |
There was a problem hiding this comment.
These lambdas and 3 lambdas in the Postgres parser are identical to the base ones. Should we clean these up?
4cdffb5 to
64de050
Compare
I'll add it as a ticket, I may have found another duckdb operator precedence issue too. |
SQLGlot Integration Test Results✅ All tests passedComparing:
Overallmain: 182937 total, 160861 passed (pass rate: 87.9%) sqlglot:parser/fix-json-arrow-operator-precedence: 176566 total, 155532 passed (pass rate: 88.1%) Transitions: Dialect pair changes: 0 previous results not found, 2 current results not found ✅ All tests passed |
Fixes #8035 where
->,->>,#>,#>>, and?were parsed as tight-binding column accessors (like.and::) viaCOLUMN_OPERATORS, instead of Postgres's actual "any other operator" precedence tier, below+/-, level with||and the bitwise operators, left-associative.This caused casts, subscripts, and arithmetic adjacent to these operators to bind to the wrong operand. To fix it:
JSON_EXTRACT_OPERATORS, a new operator tier parsed inside_parse_bitwiseinparser.py, alongside the existing||handling.COLUMN_OPERATORSand intoJSON_EXTRACT_OPERATORSforPostgresParserandDuckDBParser. Other dialects still treat->as lambda syntax.Sample input:
Previous: cast attaches to the whole extract:
Fixed: cast attaches to the path operand, matching Postgres: