Skip to content

Support nullable Iceberg partition writes - #99

Merged
osipovartem merged 2 commits into
embucket-sync-df55.0.0from
fix/nullable-partition-writes
Oct 9, 2026
Merged

osipovartem merged 2 commits into
embucket-sync-df55.0.0from
fix/nullable-partition-writes

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 9, 2026 •

Copy link
Copy Markdown

Summary

  • Keep NULL partition tuples through Arrow batch grouping, Parquet data-file metadata, and partition paths.
  • Mark contains_null in manifest summaries and treat missing bounds conservatively during manifest selection and file splitting.
  • Expose nullable snapshot partition bounds from one manifest-list read so downstream MERGE pruning can retain NULL target rows. This does not change the existing bounds API.
  • Preserve the no-NULL vectorized extraction path.

Verification

  • In-memory DataFusion/Iceberg integration: NULL-only, non-NULL, and mixed-batch INSERTs; inspect data-file partition tuples and manifest summaries; reload the table from the catalog and query with IS NULL and equality predicates. Assert nullable snapshot bounds for all-NULL and mixed manifests.
  • Focused partition, manifest, and geometry unit tests; existing partitioned and TIMESTAMPTZ INSERT regressions.
  • cargo clippy -p iceberg-rust -p datafusion_iceberg --all-targets -- -D warnings, fmt, and diff-check passed.
  • Local debug benchmark of no-NULL key extraction (base 988ea41a vs PR, separate build targets): Int64 6.1M vs 6.4M rows/s; TIMESTAMPTZ 6.2M vs 6.4M rows/s. This detects no obvious regression but is not a release benchmark.
  • Independent read-only review approved correctness, performance fast path, and tests.
  • Downstream Rustice#564 passes NULL-partition INSERT/UPDATE/DELETE/MERGE tests against this pinned commit.

API note

Public partition-value helpers now represent nullable values as Option<Value>; external Rust callers need to migrate. This PR does not claim NULL-partition DELETE coverage in this fork itself.

@osipovartem
osipovartem merged commit 34b5758 into embucket-sync-df55.0.0 Oct 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant