Skip to content

fix: keep correlated args intact in table function sources - #151

Closed
DeanKamali wants to merge 1 commit into
Embucket:embucket-sync-df55.0.0from
DeanKamali:embucket-sync-df55.0.0
Closed

DeanKamali wants to merge 1 commit into
Embucket:embucket-sync-df55.0.0from
DeanKamali:embucket-sync-df55.0.0

Conversation

@DeanKamali

@DeanKamali DeanKamali commented Oct 16, 2026 •

Copy link
Copy Markdown

What

get_table_function_source coerces and simplifies table-function arguments against DFSchema::empty(). That breaks any lateral table function whose argument references an outer column (e.g. Snowflake-style LATERAL FLATTEN(INPUT => v:arr)): the coercion fails with Schema error: No field named ....

Change

Skip simplifier.coerce/simplify when the argument expression carries column references (!arg.column_refs().is_empty()). Column-free arguments keep the existing constant folding. The correlated expression is passed to the table function provider intact, so providers can evaluate it against the outer relation themselves (embucket's flatten provider already does).

@github-actions github-actions Bot added the core label Oct 16, 2026

@osipovartem osipovartem left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 (session_state.rs:2024-2038): This does not establish the stated lateral FLATTEN behavior. In datafusion/sql/src/relation/mod.rs:286-317 the default TableFactor::Function path turns the provider into an independent TableScan; forwarding an Expr::Column does not provide outer rows at scan time. Rustice registers LateralFlattenPlanner, which handles LATERAL FLATTEN in create_extension_relation before get_table_function_source is called (rustice/crates/functions/src/table/flatten/lateral.rs:208-269). Please add an end-to-end correlated table-function test that exercises this exact branch and verifies distinct output for distinct outer rows, or narrow the PR rationale to its actual contract. The PR currently adds no tests.

P2 (session_state.rs:2024-2037): The whole-argument skip also disables type coercion and simplification inside mixed expressions containing a column (for example ARRAY[outer_int, 0.1, 1]). This changes the TableFunctionArgs::exprs() contract established by apache#19914; the existing test_udtf_type_coercion only covers a column-free array. Please cover a column-bearing mixed expression and ensure its provider receives a usable typed expression, or explicitly document that such arguments must be handled by custom relation planners.

@DeanKamali DeanKamali closed this Oct 16, 2026
@osipovartem

Copy link
Copy Markdown
Collaborator

Thanks for the proposal. We reviewed this revision and will not merge it as-is: forwarding a column expression does not make the default table-function scan lateral, while Rustice handles LATERAL FLATTEN in a separate relation planner. Skipping coercion for the whole argument also changes validation and mixed-expression typing. If you have a generic DataFusion reproducer, a follow-up would need an end-to-end test with distinct outer rows plus coercion/invalid-column coverage; that generic fix would be a good candidate for Apache DataFusion upstream. A Rustice-specific fix can stay in this fork.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants