Skip to content

[core] Support bucket pruning for manifest files - #9791

Merged
JingsongLi merged 1 commit into
apache:masterfrom
jianguotian:perf/manifest-bucket-pruning-only
Sep 13, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
jianguotian:perf/manifest-bucket-pruning-only

Conversation

@jianguotian

Copy link
Copy Markdown
Contributor

Purpose

Bucket-key point lookups spanning many partitions can spend most of their planning time opening manifest files whose bucket ranges cannot match. This PR extracts only manifest bucket pruning from #9784, following the review feedback.

Changes

  • persist one nullable totalBuckets value in manifest-list metadata
  • record it only when every ADD and DELETE entry has the same positive total bucket count
  • keep the value unknown for legacy manifests, copied unknown metadata, mixed counts, and non-positive counts
  • reuse BucketSelector cached bucket sets to test whether candidate buckets intersect a manifest bucket range
  • conservatively keep manifests when metadata or a usable predicate is missing, or when negative buckets are present

Manifest sorting, bucket-first layout, forced rewrites, and other independent optimizations are intentionally excluded and can be handled in follow-up PRs.

Verification

  • focused manifest and bucket-selector tests: 106 tests, 0 failures, 0 errors
  • paimon-core reactor tests: 5,389 tests, 0 failures, 0 errors, 34 skipped (PostgresqlCatalogTest excluded locally because Docker is unavailable)
  • Checkstyle, Spotless, and Maven Enforcer: BUILD SUCCESS

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit e600578 into apache:master Sep 13, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants