Skip to content

Build manifest datafile stream lazily - #116

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
perf/lazy-manifest-datafiles
Oct 15, 2026
Merged

osipovartem merged 1 commit into
embucket-sync-df55.0.0from
perf/lazy-manifest-datafiles

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 15, 2026 •

Copy link
Copy Markdown

Summary

  • Create fetch/decode futures only as the bounded manifest stream is polled, avoiding an extra O(manifest-count) allocation and URI clones during scan setup.
  • Reject masks whose length differs from the manifest list instead of silently truncating.
  • Cover mismatched and all-filtered masks; include an ignored stream-setup microbenchmark.

Validation

  • cargo +1.99.0 test -q -p iceberg-rust --lib: 198 passed, 2063 ignored on final commit.
  • DataFusion-Iceberg: 18 filter, 2 equality-delete, 3 position-delete tests passed.
  • cargo +1.95.0 clippy -p iceberg-rust --lib --tests -- -D warnings passed.
  • cargo +1.95.0 fmt --all -- --check and git diff --check passed.
  • Manual 100k-manifest stream-setup benchmark: 45.3 ms before, 0.5-6 us after in local debug runs. This measures stream construction only, not manifest I/O or end-to-end query latency.
  • Independent read-only review: approved; no blocking findings.

@osipovartem
osipovartem merged commit 37ab525 into embucket-sync-df55.0.0 Oct 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant