Skip to content

[core] Prune manifests by bucket metadata - #9783

Closed
jianguotian wants to merge 1 commit into
apache:masterfrom
jianguotian:perf/manifest-bucket-pruning
Closed

[core] Prune manifests by bucket metadata#9783
jianguotian wants to merge 1 commit into
apache:masterfrom
jianguotian:perf/manifest-bucket-pruning

Conversation

@jianguotian

@jianguotian jianguotian commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Purpose

Bucket-key point lookups spanning many partitions can spend most of their time opening manifests whose bucket ranges cannot match. Manifest metadata currently lacks total-bucket bounds, and existing compacted manifests cannot be reorganized under an opt-in bucket-first layout.

Changes

  • persist nullable min/max bucket-count metadata in manifest lists
  • prune manifests conservatively with bucket and total-bucket ranges
  • push partition/bucket filters into projected delete-entry reads
  • add an opt-in bucket-first manifest sort order
  • add a one-shot forced manifest rewrite option for maintenance
  • expose the new ranges in the manifests system table

Legacy manifests without the new nullable fields fall back to conservative reads. Both new options default to false. Bucket enumeration is bounded and also falls back conservatively.

Performance evidence

On a production-shaped 2.54 TB test branch using the same point-lookup SQL and result set, the combined layout and pruning path reduced opened manifests from 136 to 1 and planning from 22.6 s to 2.0 s. Resulted data files remained 54 and splits remained 25, isolating the change to planning rather than data scanning.

Verification

  • focused paimon-core suite: 178 tests, 0 failures, 0 errors, 1 skipped
  • paimon-core plus required modules package: BUILD SUCCESS (9 modules)
  • paimon-core Spotless check: BUILD SUCCESS
  • generated core option documentation updated

@jianguotian
jianguotian force-pushed the perf/manifest-bucket-pruning branch from 749174e to 0edd16d Compare September 13, 2026 06:38
@jianguotian
jianguotian force-pushed the perf/manifest-bucket-pruning branch from 0edd16d to d14e4c5 Compare September 13, 2026 06:43
@jianguotian
jianguotian marked this pull request as ready for review September 13, 2026 06:45
@jianguotian

jianguotian commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

CI note: the two currently failed jobs both stop in the unrelated paimon-s3-impl S3FileIOTest because Testcontainers cannot pull minio/minio:RELEASE.2022-02-07T08-17-33Z; the registry returns 404 / pull access denied. There are no compilation, assertion, formatting, or changed-module failures in those jobs. I attempted to rerun the failed jobs, but GitHub requires repository admin permission for that action; the remaining matrix is still running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant