diff --git a/docs/source/library-user-guide/upgrading/54.0.0.md b/docs/source/library-user-guide/upgrading/54.0.0.md index f8e7ac93c08d8..ce4d7c22a23de 100644 --- a/docs/source/library-user-guide/upgrading/54.0.0.md +++ b/docs/source/library-user-guide/upgrading/54.0.0.md @@ -335,44 +335,6 @@ This produces two user-visible changes: `ScalarSubqueryExpr` expression. Code that walks or transforms `LogicalPlan` / `ExecutionPlan` trees, as well as `EXPLAIN` output, may need updating. -### Filter predicate evaluation order may differ from query text - -The logical optimizer now reorders filters so that cheap predicates (most binary -comparisons, `IS NULL`, `Between`, `InList`, etc.) evaluate before expensive -ones (`LIKE`, regex, scalar function calls, subqueries). For example, -`WHERE col LIKE '%foo%' AND col2 = 5` may evaluate `col2 = 5` before -`col LIKE '%foo%'`. - -**Evaluation order has never been guaranteed to match the order written in the -query.** The SQL standard explicitly allows implementations to evaluate operands -in any order; major engines (PostgreSQL, SQL Server, Oracle, MySQL) document the -same. Queries should not rely on left-to-right evaluation or short-circuit -semantics for `AND` or `OR`. Previous versions of DataFusion already reordered -predicates (e.g., as part of expression simplification or predicate pushdown); -the new reordering pass just increases the scenarios where the optimizer will -change predicate evaluation order. - -**Fallible-predicate patterns are particularly affected.** For example: - -```sql -WHERE s ~ '^[0-9]+$' AND CAST(s AS INT) > 0 -``` - -The intent is likely to filter non-numeric strings before the `CAST` runs, -but this depends on evaluation-order behavior the SQL standard does not -guaranteed. The new reorder makes this kind of pattern more likely to fail -at runtime if the optimizer moves the `CAST` ahead of the regex. To force -conditional evaluation, rewrite using `CASE`, which has standardized -short-circuit semantics: - -```sql -WHERE CASE WHEN s ~ '^[0-9]+$' THEN CAST(s AS INT) > 0 ELSE false END -``` - -Volatile expressions (`random()`, `now()`, etc.) are exempt — their position -in the conjunct list is preserved so the number of times they evaluate per -query does not change. - ### `datafusion-proto`: expression deserialization now takes a `TaskContext` `Serializeable::from_bytes_with_registry` is renamed to `from_bytes_with_ctx` @@ -928,8 +890,3 @@ match register_function { RegisterFunction::Table(name, table) => {}, } ``` - -### New `Dialect::Spark` variant - -The `Dialect` enum in `datafusion_common::config` now includes a `Spark` variant. -If you match exhaustively on `Dialect`, add a `Dialect::Spark` arm. diff --git a/docs/source/library-user-guide/upgrading/55.0.0.md b/docs/source/library-user-guide/upgrading/55.0.0.md index f8b66d5f2cb92..abd4d0020e502 100644 --- a/docs/source/library-user-guide/upgrading/55.0.0.md +++ b/docs/source/library-user-guide/upgrading/55.0.0.md @@ -21,6 +21,44 @@ ## DataFusion 55.0.0 +### Filter predicate evaluation order may differ from query text + +The logical optimizer now reorders filters so that cheap predicates (most binary +comparisons, `IS NULL`, `Between`, `InList`, etc.) evaluate before expensive +ones (`LIKE`, regex, scalar function calls, subqueries). For example, +`WHERE col LIKE '%foo%' AND col2 = 5` may evaluate `col2 = 5` before +`col LIKE '%foo%'`. + +**Evaluation order has never been guaranteed to match the order written in the +query.** The SQL standard explicitly allows implementations to evaluate operands +in any order; major engines (PostgreSQL, SQL Server, Oracle, MySQL) document the +same. Queries should not rely on left-to-right evaluation or short-circuit +semantics for `AND` or `OR`. Previous versions of DataFusion already reordered +predicates (e.g., as part of expression simplification or predicate pushdown); +the new reordering pass just increases the scenarios where the optimizer will +change predicate evaluation order. + +**Fallible-predicate patterns are particularly affected.** For example: + +```sql +WHERE s ~ '^[0-9]+$' AND CAST(s AS INT) > 0 +``` + +The intent is likely to filter non-numeric strings before the `CAST` runs, +but this depends on evaluation-order behavior the SQL standard does not +guaranteed. The new reorder makes this kind of pattern more likely to fail +at runtime if the optimizer moves the `CAST` ahead of the regex. To force +conditional evaluation, rewrite using `CASE`, which has standardized +short-circuit semantics: + +```sql +WHERE CASE WHEN s ~ '^[0-9]+$' THEN CAST(s AS INT) > 0 ELSE false END +``` + +Volatile expressions (`random()`, `now()`, etc.) are exempt — their position +in the conjunct list is preserved so the number of times they evaluate per +query does not change. + ### Map casts with nested Structs use schema-evolution rules Casts of a `Map` whose key or value recursively contains a `Struct` now adapt @@ -211,6 +249,11 @@ Use `temp_file.size()` instead of [`RefCountedTempFile::current_disk_usage`]. [`refcountedtempfile::current_disk_usage`]: https://docs.rs/datafusion-execution/latest/datafusion_execution/disk_manager/struct.RefCountedTempFile.html#method.current_disk_usage [`refcountedtempfile::update_disk_usage`]: https://docs.rs/datafusion-execution/latest/datafusion_execution/disk_manager/struct.RefCountedTempFile.html#method.update_disk_usage +### New `Dialect::Spark` variant + +The `Dialect` enum in `datafusion_common::config` now includes a `Spark` variant. +If you match exhaustively on `Dialect`, add a `Dialect::Spark` arm. + ### `Dialect::AVAILABLE` replaced by `Dialect::available()` `datafusion_common::config::Dialect::AVAILABLE` has been removed. Use