Describe the bug
Enabling join dynamic filter pushdown can silently discard a matching row when the filter passes through a FilterExec that carries an embedded projection over an input with duplicate column names (here a.id and b.id from a nested join).
With the same data and SQL, disabling datafusion.optimizer.enable_join_dynamic_filter_pushdown returns one row; enabling it returns zero rows, without an error.
To Reproduce
Run the following SQL in a fresh datafusion-cli session. The two /tmp/ Parquet paths must not already exist.
SET datafusion.optimizer.join_reordering = false;
COPY (SELECT 'a1' AS id, 'x1' AS ty)
TO '/tmp/df_dynamic_filter_filterprojection_a.parquet'
STORED AS PARQUET;
COPY (SELECT 'x1' AS id)
TO '/tmp/df_dynamic_filter_filterprojection_b.parquet'
STORED AS PARQUET;
CREATE EXTERNAL TABLE ta
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_filterprojection_a.parquet';
CREATE EXTERNAL TABLE tb
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_filterprojection_b.parquet';
-- Returns one row: (x1, a1, x1).
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = false;
SELECT s.id AS sid, a.id AS aid, b.id AS bid
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
WHERE (a.ty || '!') IS DISTINCT FROM b.id
)
ON s.id = b.id;
-- Incorrectly returns zero rows.
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;
SELECT s.id AS sid, a.id AS aid, b.id AS bid
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
WHERE (a.ty || '!') IS DISTINCT FROM b.id
)
ON s.id = b.id;
Expected behavior
No response
Additional context
FilterExec resolves pushed-down filter columns by name instead of through its embedded projection, so the filter on b.id is applied to a.id. Found while working on #25244; the fix is included in #25259.
Describe the bug
Enabling join dynamic filter pushdown can silently discard a matching row when the filter passes through a
FilterExecthat carries an embedded projection over an input with duplicate column names (herea.idandb.idfrom a nested join).With the same data and SQL, disabling
datafusion.optimizer.enable_join_dynamic_filter_pushdownreturns one row; enabling it returns zero rows, without an error.To Reproduce
Run the following SQL in a fresh
datafusion-clisession. The two/tmp/Parquet paths must not already exist.Expected behavior
No response
Additional context
FilterExecresolves pushed-down filter columns by name instead of through its embedded projection, so the filter onb.idis applied toa.id. Found while working on #25244; the fix is included in #25259.