Skip to content

[SPARK-59597][SQL] Avoid quadratic CTE attribute restoration - #58873

Open
bhollis-dbx wants to merge 1 commit into
apache:masterfrom
bhollis-dbx:optimize-cte-restore-attrs
Open

bhollis-dbx wants to merge 1 commit into
apache:masterfrom
bhollis-dbx:optimize-cte-restore-attrs

Conversation

@bhollis-dbx

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Use AttributeMap when restoring attributes from CTE references to CTE definitions in PushdownPredicatesAndPruneColumnsForCTEDef. This replaces a linear semantic search per attribute with expression-ID lookup.

Why are the changes needed?

The current implementation linearly searches the CTE attribute mapping for every restored attribute, making planning quadratic in CTE width. The cost reaches tens of milliseconds per rule application for wide CTEs and is repeated for CTE references and optimizer iterations.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Ran PushdownPredicatesForCTEDefStalenessSuite. All 7 tests passed.

Local benchmark output for the full rule:

Columns Before After Change
1 2.38 us 2.48 us 4% slower
2 2.85 us 2.92 us 2% slower
5 5.02 us 4.72 us 6% faster
10 8.15 us 7.21 us 12% faster
20 14.78 us 12.41 us 16% faster
50 41.36 us 28.96 us 30% faster
100 100.11 us 54.71 us 45% faster
500 1.65 ms 0.27 ms 6.1x faster
1,000 6.53 ms 0.58 ms 11.2x faster
2,000 26.54 ms 1.33 ms 19.9x faster

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant