Skip to content

Prune Iceberg data files by partition values within a manifest - #100

Merged
osipovartem merged 3 commits into
embucket-sync-df55.0.0from
fix/year-partition-data-file-pruning
Oct 9, 2026
Merged

osipovartem merged 3 commits into
embucket-sync-df55.0.0from
fix/year-partition-data-file-pruning

Conversation

@osipovartem

Copy link
Copy Markdown

Summary

  • Project safe Iceberg partition predicates onto each data file after manifest selection. This avoids opening unrelated files that share a manifest, while retaining old-spec, unknown-partition, and delete files.
  • Fold exact cast literals before partition projection; refuse sub-microsecond timestamp values.

Validation

  • cargo +1.97.0 test -p datafusion_iceberg pruning_statistics::tests --lib: 32 passed.
  • cargo +1.97.0 fmt --all --check and git diff --check: passed.
  • Independent read-only review approved correctness, regressions, performance, API compatibility, and focused test coverage.

The downstream Rustice test measures a five-file year-partition table with a single manifest. File pruning is asserted for microsecond-precision timestamp predicates; default-precision timestamp coercion remains a separate planner follow-up.

@osipovartem
osipovartem merged commit 1d8e669 into embucket-sync-df55.0.0 Oct 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant