Skip to content

Preserve source columns across Iceberg partition spec changes - #111

Merged
osipovartem merged 2 commits into
embucket-sync-df55.0.0from
mixed-spec-scan-physical-columns
Oct 15, 2026
Merged

osipovartem merged 2 commits into
embucket-sync-df55.0.0from
mixed-spec-scan-physical-columns

Conversation

@osipovartem

Copy link
Copy Markdown

Summary

  • Read historical Iceberg partition source columns from Parquet when live data files use a different partition spec, instead of substituting current-spec partition tuples.
  • Match equality/position deletes by partition spec ID and tuple; apply unpartitioned equality deletes globally with bounded plan fan-out.
  • Keep CHANGES scans correct for historical deleted-only manifests and restrict equality-delete metrics pruning to equality keys.

Verification

  • Independent read-only review: approved.
  • Targeted lib tests: 110 passed, 2 ignored.
  • Equality-delete integration: 2 passed; position-delete integration: 3 passed.
  • Targeted all-targets Clippy passed (two pre-existing lint categories suppressed).
  • Debug-build grouping benchmark, 10k files / 20 plans: 846 ms without global deletes, 911 ms with four global deletes (+7.7% for grouping only, not end-to-end query latency).

Live Snowflake/Horizon read and COW validation follows in Embucket/rustice#700.

@osipovartem
osipovartem merged commit ea96b5f into embucket-sync-df55.0.0 Oct 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant