Skip to content

[Iceberg] Add basic iceberg integration - #595

Merged
gabotechs merged 6 commits into
iceberg-0.10from
gabrielmusat/add-basic-iceberg-integration
Aug 25, 2026
Merged

[Iceberg] Add basic iceberg integration#595
gabotechs merged 6 commits into
iceberg-0.10from
gabrielmusat/add-basic-iceberg-integration

Conversation

@gabotechs

@gabotechs gabotechs commented Aug 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Closes #600

Note

The PR is large because it contains a lot of boilerplate for a basic Iceberg integration, but it was crafted by hand, and it contains just whatever is minimal to get some tests e2e running

Adds a very basic Iceberg integration as a new crate that depends on datafusion, datafusion-distributed and iceberg-rust.

This is mostly a rewrite of https://github.com/apache/iceberg-rust/tree/main/crates/integrations/datafusion, with a few exceptions for helpers like expression transpilation.

It heavily relies on WorkUnitFeeds for streaming FileScanTasks at runtime, following a pattern that looks like this today in single-node:

┌────────────────────────────────────────────┐
│             IcebergDataSource              │
│                                            │
│┌──────────────────────────────────────────┐│
││           IcebergWorkUnitFeed            ││
││┌────────────┐┌────────────┐┌────────────┐││
│││   Feed 0   ││   Feed 1   ││   Feed 2   │││
││└──────┬─────┘└──────┬─────┘└──────┬─────┘││
│└───────┼─────────────┼─────────────┼──────┘│
│  .─────▼─────. .─────▼─────. .─────▼─────. │
│ (FileScanTask (FileScanTask (FileScanTask )│
│  .───────────. `─────┬─────' .───────────. │
│ (FileScanTask )      │      (FileScanTask )│
│  `─────┬─────'       │       .───────────. │
│        │             │      (FileScanTask )│
│        │             │       `─────┬─────' │
│        │             │             │       │
│ ┌──────▼─────┐┌──────▼─────┐┌──────▼─────┐ │
│ │Partition 0 ││Partition 1 ││Partition 2 │ │
│ │ArrowReader ││ArrowReader ││ArrowReader │ │
│ └──────┬─────┘└──────┬─────┘└──────┬─────┘ │
│        │             │             │       │
│  .─────▼─────.       │       .─────▼─────. │
│ ( RecordBatch ).─────▼─────.( RecordBatch )│
│  `─────┬─────'( RecordBatch ).───────────. │
│        │       `─────┬─────'( RecordBatch )│
│        │             │       `───────────' │
└────────┼─────────────┼─────────────┼───────┘
         ▼             ▼             ▼

And that will look like this once distribution is supported:

 ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
                                     Coordinating Context                                   │
 │
  ┌────────────────────────────────────────────────────────────────────────────────────────┐│
 ││                                  IcebergWorkUnitFeed                                   │
  │┌─────────────┐┌─────────────┐┌────────────┐┌────────────┐┌─────────────┐┌─────────────┐││
 │││   Feed 0    ││   Feed 1    ││   Feed 2   ││   Feed 3   ││   Feed 4    ││   Feed 5    ││
  │└──────┬──────┘└─────┬───────┘└────┬───────┘└───────┬────┘└───────┬─────┘└──────┬──────┘││
 └└───────┼─────────────┼─────────────┼────────────────┼─────────────┼─────────────┼───────┴
    .─────▼─────. .─────▼─────. .─────▼─────.    .─────▼─────. .─────▼─────. .─────▼─────.
   (FileScanTask (FileScanTask (FileScanTask )  (FileScanTask (FileScanTask (FileScanTask )
    .───────────. `─────┬─────' .───────────.    `─────┬─────' .───────────. `─────┬─────'
   (FileScanTask )      │      (FileScanTask )         │      (FileScanTask )      │
    `─────┬─────'       │       .───────────.          │       `───────────'       │
          │             │      (FileScanTask )         │             │             │
 Worker 0 │             │       `─────┬─────'          │             │             │ Worker 1
┌ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ┐┌ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ┐
  ┌───────┼─────────────┼─────────────┼───────┐┌───────┼─────────────┼─────────────┼───────┐
│ │       │     IcebergD│taSource     │       ││       │     IcebergD│taSource     │       │ │
  │       │             │             │       ││       │             │             │       │
│ │┌──────▼─────┐┌──────▼─────┐┌──────▼─────┐ ││┌──────▼─────┐┌──────▼─────┐┌──────▼─────┐ │ │
  ││Partition 0 ││Partition 1 ││Partition 2 │ │││Partition 0 ││Partition 1 ││Partition 2 │ │
│ ││ArrowReader ││ArrowReader ││ArrowReader │ │││ArrowReader ││ArrowReader ││ArrowReader │ │ │
  │└──────┬─────┘└──────┬─────┘└──────┬─────┘ ││└──────┬─────┘└──────┬─────┘└──────┬─────┘ │
│ │       │             │             │       ││       │             │             │       │ │
  │ .─────▼─────.       │       .─────▼─────. ││       │             ▼             ▼       │
│ │( RecordBatch ).─────▼─────.( RecordBatch )││ .─────▼─────. .───────────. .───────────. │ │
  │ `─────┬─────'( RecordBatch ).───────────. ││( RecordBatch ( RecordBatch ) RecordBatch )│
│ │       │       `─────┬─────'( RecordBatch )││ `─────┬─────' `───────────' `─────┬─────' │ │
  │       │             │       `───────────' ││       │      ( RecordBatch )      │       │
│ │       │             │             │       ││       │       `─────┬─────'       │       │ │
  └───────┼─────────────┼─────────────┼───────┘└───────┼─────────────┼─────────────┼───────┘
│         ▼             ▼             ▼       ││       ▼             ▼             ▼         │
 ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─

However, this PR just adds a very basic integration that includes:

  • Basic plumbing for TableProvider and TableProviderFactory implementations, very similar to what https://github.com/apache/iceberg-rust/tree/main/crates/integrations/datafusion has today on this front
  • An IcebergWorkUnitFeed implementation that streams FileScanTasks messages at execution time as new files are discovered
  • An IcebergDataSource that reads the FileScanTask streams, executing the tasks and yielding Arrow RecordBatches.
  • Integration tests, for existing functionality and future one.

It's still laking:

  • Any kind of distribution
  • Any kind of [de]serialization for FileScanTasks and IcebergDataSources
  • No filter pushdowns
  • No sort pushdowns
  • No benchmarks
  • No sense of data partitioning

Stack created with GitHub Stacks CLI • Give Feedback 💬

@gabotechs gabotechs changed the title gabrielmusat/add basic iceberg integration Add basic iceberg integration Aug 9, 2026
@gabotechs
gabotechs force-pushed the gabrielmusat/add-basic-iceberg-integration branch from eb071d9 to 72203a0 Compare August 9, 2026 13:40
@gabotechs
gabotechs changed the base branch from gabrielmusat/add-taxi-iceberg-dataset to gabrielmusat/add-empty-iceberg-crate August 9, 2026 13:41
This was referenced Aug 9, 2026
Merged
Open
shinzoxD

This comment was marked as low quality.

Sign in to view

@sandugood

sandugood commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Lgtm, looks nice as an initial integration.
Thanks @gabotechs

@gabotechs gabotechs changed the title Add basic iceberg integration [Iceberg] Add basic iceberg integration Aug 12, 2026
alexanderbianchi
alexanderbianchi reviewed Aug 20, 2026
View reviewed changes
Comment thread iceberg/src/common/error.rs
use iceberg::{Error, ErrorKind};

/// Converts a datafusion error into an iceberg error.
pub fn iceberg_err(error: DataFusionError) -> Error {

@alexanderbianchi alexanderbianchi Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

  pub fn iceberg_err(error: DataFusionError) -> iceberg::Error {
       match error {
           DataFusionError::External(error) => {
               match error.downcast::<iceberg::Error>() {
                   Ok(error) => *error,
                   Err(error) => iceberg::Error::new(
                       ErrorKind::Unexpected,
                       format!("DataFusion execution failed: {error}"),
                   ),
               }
           }
           error => iceberg::Error::new(
               ErrorKind::Unexpected,
               format!("DataFusion execution failed: {error}"),
           ),
       }
   }

Could we have structured error handling here instead of the string? Can be follow up.

@gabotechs gabotechs Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The challenge with this is that this approach would not survive a serialization roundtrip.

When propagating DataFusionError messages across the network, we use https://github.com/datafusion-contrib/datafusion-distributed/blob/9dd018a392185c6853f4d32a43d839219b31730d/src/protocol/grpc/errors, and any error that relies on type downcasting is lost in the way.

alexanderbianchi
alexanderbianchi approved these changes Aug 20, 2026
View reviewed changes

@alexanderbianchi alexanderbianchi left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as an initial start thanks!

@gabotechs
gabotechs force-pushed the gabrielmusat/add-basic-iceberg-integration branch from 72203a0 to bd7e471 Compare August 24, 2026 19:10
@gabotechs
gabotechs force-pushed the gabrielmusat/add-basic-iceberg-integration branch from bd7e471 to e840655 Compare August 24, 2026 19:35
This was referenced Aug 24, 2026
Closed
Closed
@gabotechs
gabotechs force-pushed the gabrielmusat/add-basic-iceberg-integration branch from 61c91e2 to 1408dd7 Compare August 24, 2026 20:02
@gabotechs
gabotechs force-pushed the gabrielmusat/add-basic-iceberg-integration branch from 1408dd7 to 0f30c31 Compare August 25, 2026 05:32
@gabotechs
gabotechs force-pushed the gabrielmusat/add-basic-iceberg-integration branch from 0f30c31 to 5bb08bb Compare August 25, 2026 05:42
@gabotechs
gabotechs force-pushed the gabrielmusat/add-basic-iceberg-integration branch from 5bb08bb to 872eebe Compare August 25, 2026 05:45
Base automatically changed from gabrielmusat/add-empty-iceberg-crate to iceberg-0.10 August 25, 2026 06:09
gabotechs added a commit that referenced this pull request Aug 25, 2026
@gabotechs
[Iceberg] Add Iceberg crate dependencies (#597)
ce5bb32 
Introduces the empty `datafusion-distributed-iceberg` crate, its
dependency set, and the corresponding lockfile updates.

This is the first layer of the Iceberg integration stack. The
implementation follows in #595.
@gabotechs
gabotechs force-pushed the gabrielmusat/add-basic-iceberg-integration branch from a3b361d to 6a5767e Compare August 25, 2026 06:09
@gabotechs
gabotechs force-pushed the gabrielmusat/add-basic-iceberg-integration branch from 6a5767e to a6cb6af Compare August 25, 2026 06:31
@gabotechs

gabotechs commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

🤔 interesting, this passes on my Mac, but fails on the CI:

It sounds like a bug in iceberg-rust

@gabotechs

gabotechs commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

Ah no, it was just that git lfs was disabled, we are good now

@gabotechs
gabotechs merged commit 29ae9d5 into iceberg-0.10 Aug 25, 2026
32 checks passed
@gabotechs
gabotechs deleted the gabrielmusat/add-basic-iceberg-integration branch August 25, 2026 08:41
@gabotechs

gabotechs commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

Merged!

For further contributions to iceberg, let's make them to the iceberg-0.10 branch, as main is already running on arrow@59 that is incompatible with iceberg-rust@0.10.

Once iceberg-rust picks up arrow@59 we can pull everything into main.

cc @alexanderbianchi @sandugood

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers
@shinzoxD shinzoxD shinzoxD requested changes
@alexanderbianchi alexanderbianchi alexanderbianchi approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Iceberg] Establish a minimal e2e-tested read integration

4 participants

@gabotechs @sandugood @alexanderbianchi @shinzoxD