Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 0 additions & 43 deletions docs/source/library-user-guide/upgrading/54.0.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,44 +335,6 @@ This produces two user-visible changes:
`ScalarSubqueryExpr` expression. Code that walks or transforms `LogicalPlan` /
`ExecutionPlan` trees, as well as `EXPLAIN` output, may need updating.

### Filter predicate evaluation order may differ from query text

The logical optimizer now reorders filters so that cheap predicates (most binary
comparisons, `IS NULL`, `Between`, `InList`, etc.) evaluate before expensive
ones (`LIKE`, regex, scalar function calls, subqueries). For example,
`WHERE col LIKE '%foo%' AND col2 = 5` may evaluate `col2 = 5` before
`col LIKE '%foo%'`.

**Evaluation order has never been guaranteed to match the order written in the
query.** The SQL standard explicitly allows implementations to evaluate operands
in any order; major engines (PostgreSQL, SQL Server, Oracle, MySQL) document the
same. Queries should not rely on left-to-right evaluation or short-circuit
semantics for `AND` or `OR`. Previous versions of DataFusion already reordered
predicates (e.g., as part of expression simplification or predicate pushdown);
the new reordering pass just increases the scenarios where the optimizer will
change predicate evaluation order.

**Fallible-predicate patterns are particularly affected.** For example:

```sql
WHERE s ~ '^[0-9]+$' AND CAST(s AS INT) > 0
```

The intent is likely to filter non-numeric strings before the `CAST` runs,
but this depends on evaluation-order behavior the SQL standard does not
guaranteed. The new reorder makes this kind of pattern more likely to fail
at runtime if the optimizer moves the `CAST` ahead of the regex. To force
conditional evaluation, rewrite using `CASE`, which has standardized
short-circuit semantics:

```sql
WHERE CASE WHEN s ~ '^[0-9]+$' THEN CAST(s AS INT) > 0 ELSE false END
```

Volatile expressions (`random()`, `now()`, etc.) are exempt — their position
in the conjunct list is preserved so the number of times they evaluate per
query does not change.

### `datafusion-proto`: expression deserialization now takes a `TaskContext`

`Serializeable::from_bytes_with_registry` is renamed to `from_bytes_with_ctx`
Expand Down Expand Up @@ -928,8 +890,3 @@ match register_function {
RegisterFunction::Table(name, table) => {},
}
```

### New `Dialect::Spark` variant

The `Dialect` enum in `datafusion_common::config` now includes a `Spark` variant.
If you match exhaustively on `Dialect`, add a `Dialect::Spark` arm.
43 changes: 43 additions & 0 deletions docs/source/library-user-guide/upgrading/55.0.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,44 @@

## DataFusion 55.0.0

### Filter predicate evaluation order may differ from query text

The logical optimizer now reorders filters so that cheap predicates (most binary
comparisons, `IS NULL`, `Between`, `InList`, etc.) evaluate before expensive
ones (`LIKE`, regex, scalar function calls, subqueries). For example,
`WHERE col LIKE '%foo%' AND col2 = 5` may evaluate `col2 = 5` before
`col LIKE '%foo%'`.

**Evaluation order has never been guaranteed to match the order written in the
query.** The SQL standard explicitly allows implementations to evaluate operands
in any order; major engines (PostgreSQL, SQL Server, Oracle, MySQL) document the
same. Queries should not rely on left-to-right evaluation or short-circuit
semantics for `AND` or `OR`. Previous versions of DataFusion already reordered
predicates (e.g., as part of expression simplification or predicate pushdown);
the new reordering pass just increases the scenarios where the optimizer will
change predicate evaluation order.

**Fallible-predicate patterns are particularly affected.** For example:

```sql
WHERE s ~ '^[0-9]+$' AND CAST(s AS INT) > 0
```

The intent is likely to filter non-numeric strings before the `CAST` runs,
but this depends on evaluation-order behavior the SQL standard does not
guaranteed. The new reorder makes this kind of pattern more likely to fail
at runtime if the optimizer moves the `CAST` ahead of the regex. To force
conditional evaluation, rewrite using `CASE`, which has standardized
short-circuit semantics:

```sql
WHERE CASE WHEN s ~ '^[0-9]+$' THEN CAST(s AS INT) > 0 ELSE false END
```

Volatile expressions (`random()`, `now()`, etc.) are exempt — their position
in the conjunct list is preserved so the number of times they evaluate per
query does not change.

### Map casts with nested Structs use schema-evolution rules

Casts of a `Map` whose key or value recursively contains a `Struct` now adapt
Expand Down Expand Up @@ -211,6 +249,11 @@ Use `temp_file.size()` instead of [`RefCountedTempFile::current_disk_usage`].
[`refcountedtempfile::current_disk_usage`]: https://docs.rs/datafusion-execution/latest/datafusion_execution/disk_manager/struct.RefCountedTempFile.html#method.current_disk_usage
[`refcountedtempfile::update_disk_usage`]: https://docs.rs/datafusion-execution/latest/datafusion_execution/disk_manager/struct.RefCountedTempFile.html#method.update_disk_usage

### New `Dialect::Spark` variant

The `Dialect` enum in `datafusion_common::config` now includes a `Spark` variant.
If you match exhaustively on `Dialect`, add a `Dialect::Spark` arm.

### `Dialect::AVAILABLE` replaced by `Dialect::available()`

`datafusion_common::config::Dialect::AVAILABLE` has been removed. Use
Expand Down