Skip to content

Join dynamic filter pushdown through ProjectionExec can map duplicate output aliases to the wrong expression #25263

Description

@haohuaijin

Describe the bug

Enabling join dynamic filter pushdown can silently discard a matching row when the filter passes through a ProjectionExec whose output has two columns with the same alias (here a.id and b.id, both named id).

With the same data and SQL, disabling datafusion.optimizer.enable_join_dynamic_filter_pushdown returns one row; enabling it returns zero rows, without an error.

To Reproduce

Run the following SQL in a fresh datafusion-cli session. The two /tmp/ Parquet paths must not already exist.

SET datafusion.optimizer.join_reordering = false;

COPY (SELECT 'a1' AS id, 'x1' AS ty)
TO '/tmp/df_dynamic_filter_projection_a.parquet'
STORED AS PARQUET;

COPY (SELECT 'x1' AS id)
TO '/tmp/df_dynamic_filter_projection_b.parquet'
STORED AS PARQUET;

CREATE EXTERNAL TABLE ta
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_projection_a.parquet';

CREATE EXTERNAL TABLE tb
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_projection_b.parquet';

-- Returns one row: (a1, a1, x1, x1!).
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = false;

SELECT s.id, a.id, b.id, tag
FROM ta s
JOIN (
    SELECT a.id, b.id, a.ty || '!' AS tag
    FROM ta a JOIN tb b ON a.ty = b.id
)
ON s.id = a.id;

-- Incorrectly returns zero rows.
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;

SELECT s.id, a.id, b.id, tag
FROM ta s
JOIN (
    SELECT a.id, b.id, a.ty || '!' AS tag
    FROM ta a JOIN tb b ON a.ty = b.id
)
ON s.id = a.id;

Expected behavior

No response

Additional context

ProjectionExec resolves pushed-down filter columns by output alias instead of by position, so with two outputs named id the filter on b.id is rewritten to a.id. Found while working on #25244; the fix is included in #25259.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions