Skip to content

Add metadata-only truncate snapshots and ID-aligned historical reads - #114

Merged
osipovartem merged 5 commits into
embucket-sync-df55.0.0from
feat/metadata-only-truncate-df55
Oct 15, 2026
Merged

osipovartem merged 5 commits into
embucket-sync-df55.0.0from
feat/metadata-only-truncate-df55

Conversation

@osipovartem

Copy link
Copy Markdown

Summary

  • Add metadata-only Iceberg TRUNCATE snapshots with an empty manifest list, including v3 row-lineage accounting and commit conflict checks.
  • Let historical DataFusion scans use an explicit Iceberg read schema. Align renamed, reordered, and added nested STRUCT/LIST/MAP fields by Iceberg field ID in Arrow batches; a reused name with a different ID yields a typed null.
  • Reject non-increasing snapshot sequence numbers in local catalog commits.

Performance

TRUNCATE neither scans nor rewrites previous Parquet files. Unchanged-schema reads retain the existing adapter fast path. Schema-evolution mappings are planned once per file and applied using Arrow arrays and shared buffers, without per-row ScalarValue conversion.

Verification

  • cargo +1.99.0 test -p iceberg-rust --lib truncate_: 8 passed.
  • cargo +1.99.0 test -p datafusion_iceberg --lib: 113 passed, 2 manual benchmarks ignored.
  • Targeted Clippy passed with only three pre-existing Rust 1.99 lints excluded.
  • Rustice v3 TRUNCATE/CHANGES end-to-end tests passed against the pinned final commit.
  • Independent read-only review: approved.

Parquet-backed LIST/MAP schema-evolution scans remain a follow-up; the adapter has focused Arrow-batch tests.

@osipovartem
osipovartem merged commit bca4ad1 into embucket-sync-df55.0.0 Oct 15, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant