Skip to content

[SPARK-31760][SQL] Simplify contained integral range predicates - #58940

Open
xiaoh1024 wants to merge 1 commit into
apache:masterfrom
xiaoh1024:fix/spark-31760-integral-ranges
Open

xiaoh1024 wants to merge 1 commit into
apache:masterfrom
xiaoh1024:fix/spark-31760-integral-ranges

Conversation

@xiaoh1024

Copy link
Copy Markdown

What changes were proposed in this pull request?

Extend BooleanSimplification to remove redundant integral range comparisons:

a > 5 AND a > 0             --> a > 5
a > 1 OR a > 2              --> a > 1
a > 1 OR (a > 2 AND a < 4)  --> a > 1

Support <, <=, >, and >= between direct Byte/Short/Int/Long attributes and non-null literals of the same type, including reversed operands and one level of nested absorption. Bounds are compared without arithmetic to avoid overflow. Casts and arbitrary expressions are excluded.

Why are the changes needed?

The existing rule retains these redundant comparisons. Removing them reduces per-row evaluation. This implements the integral range subset of SPARK-31760.

Does this PR introduce any user-facing change?

Yes: eligible queries have simpler optimized plans and may execute faster. Query results, including SQL NULL semantics, are preserved. No public API or configuration changes.

How was this patch tested?

Added Catalyst and SQL tests covering operand order, inclusive/exclusive bounds, nested absorption, NULLs, integer extremes, unsupported expressions, and generated evaluation. The new contained-range regression test fails with the original rule.

On Spark 5.0.0-SNAPSHOT with Scala 2.13.18 and JDK 17.0.18:

  • 37 Catalyst tests passed across the new suite and the existing BooleanSimplification and BinaryComparisonSimplification suites.
  • The SQL suite passed all 16 filter/projection and AQE on/off combinations.
  • Catalyst source/test and SQL test scalastyle checks passed.

Local synthetic-data benchmarks used 30 million cached rows and local[2], covering filter/projection queries with 0% and 10% nulls. AND placed the wider condition first, OR placed the narrower condition first, and nested absorption placed the complex branch first. Four JVMs ran in baseline/candidate/candidate/baseline order. Median query time decreased by 3–5% for AND, 30–34% for OR, and 35–43% for nested absorption. A separate rule microbenchmark measured up to 3.2 microseconds additional time per 32-expression projection (about 8%).

Was this patch authored or co-authored using generative AI tooling?

Yes.

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a nice change @xiaoh1024! Adding @stevomitric to also review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants