Skip to content

Preserve source partition specs during COW rewrites - #110

Merged
osipovartem merged 3 commits into
embucket-sync-df55.0.0from
fix/cow-partition-spec-evolution
Oct 15, 2026
Merged

osipovartem merged 3 commits into
embucket-sync-df55.0.0from
fix/cow-partition-spec-evolution

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 15, 2026 •

Copy link
Copy Markdown

Summary

  • Preserve source partition specs and schemas when COW rewrites files from an older spec; replacement files use the current spec.
  • Keep old-spec tombstones in the source manifest with correct snapshot IDs, minimum live sequence numbers, partition bounds, and v3 row lineage.
  • Re-encode manifest entries for v2-to-v3 upgrades, exclude NaN values from bounds, and record the exact length of rewritten manifests.
  • Fail closed when one transaction changes the default spec and writes files; empty no-op writes remain allowed until full transaction staging is supported.

Verification

  • iceberg-rust-spec: 195 active tests passed.
  • iceberg-rust library: 192 active tests passed; focused old-spec COW and no-op transaction tests passed.
  • Clippy for iceberg-rust library/tests passed with -D warnings except the pre-existing clippy::useless_borrows_in_formatting lint in iceberg-rust-spec; fmt and diff checks passed.
  • Ignored 1024-file/512-replacement debug benchmark: median of 3 runs 62.9 ms same-spec, 74.4 ms evolved-spec. This is a smoke comparison, not a before/after regression claim.
  • Independent read-only review found two issues (manifest length and same-transaction spec/write safety). Both were fixed and follow-up review explicitly approved.

Live Horizon verification (2026-10-15)

A direct Rustice executor test used a disposable Snowflake-managed Iceberg v3 table. Snowflake wrote four rows under identity partitioning, changed to BUCKET(16, PART_KEY), and wrote two more rows. Rustice verified both specs had live files, read six exact rows, and committed UPDATE, DELETE, and MERGE. Snowflake read back the expected six rows. Snowflake then committed UPDATE and INSERT; a fresh Rustice session read the new snapshot and committed UPDATE and DELETE. Both engines returned final IDs 1, 3, 4, 5, 7, 8, COUNT(*) = 6, and SUM(ID) = 28. The disposable schema was dropped and the warehouse verified SUSPENDED. This validates sequential COW and alternating writers, not concurrent commits or merge-on-read.

Rustice integration test and reproduction steps: Embucket/rustice#704. Fork GitHub checks were cancelled while building to conserve Actions quota; they did not report test failures.

@osipovartem
osipovartem marked this pull request as ready for review October 15, 2026 02:53
@osipovartem
osipovartem merged commit c897543 into embucket-sync-df55.0.0 Oct 15, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant