feat(databricks): support and, or function syntax - #8065
feat(databricks): support and, or function syntax#8065fivetran-amrutabhimsenayachit wants to merge 2 commits into
and, or function syntax#8065Conversation
e6db6ec to
9652c7e
Compare
georgesittas
changed the title
and, or function syntax
georgesittas
left a comment
georgesittas
left a comment
There was a problem hiding this comment.
Is this only relevant to Databricks? What about other dialects in the Hive hierarchy?
| "AND": lambda args: exp.And(this=seq_get(args, 0), expression=seq_get(args, 1)), | ||
| "OR": lambda args: exp.Or(this=seq_get(args, 0), expression=seq_get(args, 1)), |
There was a problem hiding this comment.
What happens if you mix and() and or() w.r.t. operator precedence? Do we respect Databricks' semantics? Try the following:
and(or(true, false), false)
and(false, or (true, true))You should also see how this affects other expressions that involve these function calls, such as:
and(false, false) = false
not and(true, false)There was a problem hiding this comment.
The and() or() function calls is supported by Spark as well apart from Databricks, but it is not supported by HIVE.py. So moving the fix to SparkParser.
| COLON_IS_VARIANT_EXTRACT = True | ||
| COLON_CHAIN_IS_SINGLE_EXTRACT = False | ||
|
|
||
| FUNC_TOKENS = parser.Parser.FUNC_TOKENS | {TokenType.AND, TokenType.OR} |
There was a problem hiding this comment.
Why is this unioning the base parser's FUNC_TOKENS instead of Spark's?
There was a problem hiding this comment.
Moved this to SparkParser
| def _parse_interval_span( | ||
| self, this: exp.Expr, parse_function_unit: bool = True | ||
| ) -> exp.Interval: | ||
| # AND/OR in FUNC_TOKENS would be consumed as interval units; they never are. |
There was a problem hiding this comment.
| # AND/OR in FUNC_TOKENS would be consumed as interval units; they never are. |
| self, this: exp.Expr, parse_function_unit: bool = True | ||
| ) -> exp.Interval: | ||
| # AND/OR in FUNC_TOKENS would be consumed as interval units; they never are. | ||
| if self._curr and self._curr.token_type in (TokenType.AND, TokenType.OR): |
There was a problem hiding this comment.
Are these the only tokens that can follow an INTERVAL value and be incorrectly consumed as units? Are there any others? If yes, are the others handled correctly in main today?
There was a problem hiding this comment.
Looks like it. I tried running the fusion schema tests and caught this failure.
SQLGlot Integration Test Results✅ All tests passedComparing:
By Dialect
Overallmain: 182937 total, 160861 passed (pass rate: 87.9%) sqlglot:fix_parser_databricks_AND_function: 170743 total, 149711 passed (pass rate: 87.7%) Transitions: Dialect pair changes: 0 previous results not found, 3 current results not found ✅ All tests passed |
ee53599 to
b65725f
Compare
Databricks allows and(a, b) as a function, but SQLGlot's parser only knew AND as a keyword. Adding AND to the function-name whitelist fixed the parsing, and a small override prevented a side-effect where the interval parser mistook AND ( for a function call.