Add expression partitioning enum variant by gene-bordegaray · Pull Request #22207 · apache/datafusion

gene-bordegaray · 2026-05-15T16:57:19Z

Which issue does this PR close?

First mechanical PR for ExprPartitioning as described in thread: [DISCUSSION] Extending Partitioning to Support More Variants #21992.

Rationale for this change

DataFusion currently cannot represent some partitioning schemes truthfully. For example, range-partitioned data currently advertises itself as Partitioning::Hash only to avoid repartitioning, which makes later optimizer decisions brittle.

This PR introduces expression-based physical partitioning metadata so sources can eventually describe partition membership with predicates. This intentionally leaves optimizer and execution semantics unimplemented for follow-up PRs and to plan the shape of the partitioning API carefully.

What changes are included in this PR?

Adds Partitioning::Expr(ExprPartitioning) to the physical partitioning enum.
Adds ExprPartitioning, representing one partition predicate expression per output partition.
Documents the contract: each emitted row must match exactly one partition expression and be emitted by that partition. This is expected to be upheld by the source declaring this partitioning for correct results.
Adds conservative projection behavior:
- preserve ExprPartitioning only when all partition expressions can be remapped
- otherwise degrade to UnknownPartitioning
Adds not_impl_err! at call-sites where expression partitioning semantics are not implemented yet.
Adds proto serialization/deserialization.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes, additive only. This adds a public physical partitioning variant and public type:

Partitioning::Expr
ExprPartitioning

Dandandan · 2026-05-15T17:06:56Z

+/// NOTE: Optimizer and execution behavior for this partitioning is intentionally
+/// not implemented and will be introduced incrementally.
+#[derive(Debug, Clone)]
+pub struct ExprPartitioning {


Isn't this the same as Range Partitioning https://www.waitingforcode.com/apache-spark-sql/range-partitioning-apache-spark-sql/read#range_partitioning

Wouldn't it be better to use that naming?

https://dev.mysql.com/doc/refman/8.4/en/partitioning-range.html
https://www.dremio.com/wiki/range-partitioning/

I.e. this is a commonly used term.

Oh I see the issue already refers to it as range partititioning. Any reason of why not using the terminology here?

The reason is that we aim to be more flexible here. This can support Range partitioning but also extens beyond that to any physical expr the source wants to provide. I just gave range in the description as one concrete example of how this could be used.

Someone could partition using this scheme on something like city column where:

partition 1 -> city = "New York" partition 2 -> city = "London"

and so on.

github-actions · 2026-05-15T17:13:02Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-ffi v53.1.0 (current)
       Built [  57.638s] (current)
     Parsing datafusion-ffi v53.1.0 (current)
      Parsed [   0.059s] (current)
    Building datafusion-ffi v53.1.0 (baseline)
       Built [  57.775s] (baseline)
     Parsing datafusion-ffi v53.1.0 (baseline)
      Parsed [   0.060s] (baseline)
    Checking datafusion-ffi v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.232s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [ 117.281s] datafusion-ffi
    Building datafusion-physical-expr v53.1.0 (current)
       Built [  24.334s] (current)
     Parsing datafusion-physical-expr v53.1.0 (current)
      Parsed [   0.043s] (current)
    Building datafusion-physical-expr v53.1.0 (baseline)
       Built [  24.338s] (baseline)
     Parsing datafusion-physical-expr v53.1.0 (baseline)
      Parsed [   0.044s] (baseline)
    Checking datafusion-physical-expr v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.319s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/enum_variant_added.ron

Failed in:
  variant Partitioning:Expr in /home/runner/work/datafusion/datafusion/datafusion/physical-expr/src/partitioning.rs:121

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  50.073s] datafusion-physical-expr
    Building datafusion-physical-plan v53.1.0 (current)
       Built [  32.255s] (current)
     Parsing datafusion-physical-plan v53.1.0 (current)
      Parsed [   0.122s] (current)
    Building datafusion-physical-plan v53.1.0 (baseline)
       Built [  32.175s] (baseline)
     Parsing datafusion-physical-plan v53.1.0 (baseline)
      Parsed [   0.122s] (baseline)
    Checking datafusion-physical-plan v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.550s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [  66.410s] datafusion-physical-plan
    Building datafusion-proto v53.1.0 (current)
       Built [  52.788s] (current)
     Parsing datafusion-proto v53.1.0 (current)
      Parsed [   0.139s] (current)
    Building datafusion-proto v53.1.0 (baseline)
       Built [  52.704s] (baseline)
     Parsing datafusion-proto v53.1.0 (baseline)
      Parsed [   0.134s] (baseline)
    Checking datafusion-proto v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   1.660s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/enum_variant_added.ron

Failed in:
  variant PartitionMethod:Expr in /home/runner/work/datafusion/datafusion/datafusion/proto/src/generated/prost.rs:2069
  variant PartitionMethod:Expr in /home/runner/work/datafusion/datafusion/datafusion/proto/src/generated/prost.rs:2069

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 110.063s] datafusion-proto

stuhood · 2026-05-15T17:13:30Z

+            Partitioning::Expr(_) => {
+                not_impl_err!(
+                    "Expression partitioning is not supported by RepartitionExec"
+                )
+            }


So, it's worth discussing this in more detail I think.

Expr partitioning is much, much more general than Range partitioning.

In Range partitioning, deciding which partition a row maps to involves either a binary search or sorted map lookup. But in Expr partitioning, it will always be a linear scan through the expressions, unless the consumer has reverse-engineered the fact that it is actually Range partitioning under the hood.

So this operator will be much more expensive than it might be otherwise.

What is the reasoning around using expressions here, and not literally ranges?

My intent wasn't for ExprPartitioning to be efficient execution format for physically repartitioning rows. I was thinking of this as partitioning for sources/plans that already have known partitioning and declare it to preserve in the plan to unlock optimizations.

In follow-ups:

add explicit compatibility/satisfaction APIs around this metadata we can ask structured questions without doing row-wise linear scans. This would eliminate uneeded repartitions in cases where different partitioning types satisfy one another.

keep hash repartitioning as the preferred general execution path when DataFusion needs to repartition arbitrary input, unless we later add a more specialized repartitioning strategy.

Let me know thoughts on that 👍

My intent wasn't for ExprPartitioning to be efficient execution format for physically repartitioning rows. I was thinking of this as partitioning for sources/plans that already have known partitioning and declare it to preserve in the plan to unlock optimizations.

That works for the first join, but not for followup joins. For example:

If you have a 3 table join, the first join will be able to use an equality match on range partitioning to say: no re-partitioning needed at all because the two tables are partitioned the same way! Great.

But its very likely that the second join does need to re-partition one of its inputs (assuming different join keys between the two joins): the output of join one needs to be re-partitioned to match the third table. Now, technically you can just repartition both sides (i.e. switch to hash or something). But if you instead re-partition to match the third table, then you might be able to significantly cut down on data movement.

So, yes: I think that it is important to be able to efficiently re-partition by this strategy. If we don't have concrete use-cases for generic expression partitioning, then it would not be my first choice here.

Add expression partitioning enum variant

366c4ac

github-actions Bot added physical-expr Changes to the physical-expr crates proto Related to proto crate ffi Changes to the ffi crate physical-plan Changes to the physical-plan crate labels May 15, 2026

gene-bordegaray mentioned this pull request May 15, 2026

[DISCUSSION] Extending Partitioning to Support More Variants #21992

Open

Dandandan reviewed May 15, 2026

View reviewed changes

github-actions Bot added the auto detected api change Auto detected API change label May 15, 2026

stuhood reviewed May 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add expression partitioning enum variant#22207

Add expression partitioning enum variant#22207
gene-bordegaray wants to merge 1 commit into
apache:mainfrom
gene-bordegaray:gene.bordegaray/2026/05/expr_partitioning_enum_mechanical

gene-bordegaray commented May 15, 2026

Uh oh!

Dandandan May 15, 2026

Uh oh!

Dandandan May 15, 2026

Uh oh!

Dandandan May 15, 2026

Uh oh!

gene-bordegaray May 15, 2026

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

stuhood May 15, 2026 •

edited

Loading

Uh oh!

gene-bordegaray May 15, 2026

Uh oh!

stuhood May 15, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

gene-bordegaray commented May 15, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Dandandan May 15, 2026

Choose a reason for hiding this comment

Uh oh!

Dandandan May 15, 2026

Choose a reason for hiding this comment

Uh oh!

Dandandan May 15, 2026

Choose a reason for hiding this comment

Uh oh!

gene-bordegaray May 15, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

stuhood May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gene-bordegaray May 15, 2026

Choose a reason for hiding this comment

Uh oh!

stuhood May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stuhood May 15, 2026 •

edited

Loading

stuhood May 15, 2026 •

edited

Loading