Repository navigation
fix: keep correlated args intact in table function sources - #151
DeanKamali wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
P1 (session_state.rs:2024-2038): This does not establish the stated lateral FLATTEN behavior. In datafusion/sql/src/relation/mod.rs:286-317 the default TableFactor::Function path turns the provider into an independent TableScan; forwarding an Expr::Column does not provide outer rows at scan time. Rustice registers LateralFlattenPlanner, which handles LATERAL FLATTEN in create_extension_relation before get_table_function_source is called (rustice/crates/functions/src/table/flatten/lateral.rs:208-269). Please add an end-to-end correlated table-function test that exercises this exact branch and verifies distinct output for distinct outer rows, or narrow the PR rationale to its actual contract. The PR currently adds no tests.
P2 (session_state.rs:2024-2037): The whole-argument skip also disables type coercion and simplification inside mixed expressions containing a column (for example ARRAY[outer_int, 0.1, 1]). This changes the TableFunctionArgs::exprs() contract established by apache#19914; the existing test_udtf_type_coercion only covers a column-free array. Please cover a column-bearing mixed expression and ensure its provider receives a usable typed expression, or explicitly document that such arguments must be handled by custom relation planners.
|
Thanks for the proposal. We reviewed this revision and will not merge it as-is: forwarding a column expression does not make the default table-function scan lateral, while Rustice handles LATERAL FLATTEN in a separate relation planner. Skipping coercion for the whole argument also changes validation and mixed-expression typing. If you have a generic DataFusion reproducer, a follow-up would need an end-to-end test with distinct outer rows plus coercion/invalid-column coverage; that generic fix would be a good candidate for Apache DataFusion upstream. A Rustice-specific fix can stay in this fork. |
What
get_table_function_sourcecoerces and simplifies table-function arguments againstDFSchema::empty(). That breaks any lateral table function whose argument references an outer column (e.g. Snowflake-styleLATERAL FLATTEN(INPUT => v:arr)): the coercion fails withSchema error: No field named ....Change
Skip
simplifier.coerce/simplifywhen the argument expression carries column references (!arg.column_refs().is_empty()). Column-free arguments keep the existing constant folding. The correlated expression is passed to the table function provider intact, so providers can evaluate it against the outer relation themselves (embucket's flatten provider already does).