Skip to content

feat(datafusion): populate partition stats in $manifests system table#308

Open
QuakeWang wants to merge 1 commit intoapache:mainfrom
QuakeWang:manifests-partition-stats
Open

feat(datafusion): populate partition stats in $manifests system table#308
QuakeWang wants to merge 1 commit intoapache:mainfrom
QuakeWang:manifests-partition-stats

Conversation

@QuakeWang
Copy link
Copy Markdown
Contributor

@QuakeWang QuakeWang commented May 5, 2026

Purpose

The DataFusion $manifests system table already defines min_partition_stats and max_partition_stats, but currently returns NULL for both columns. This change decodes the partition stats stored in ManifestFileMeta and renders them with Java-compatible row cast semantics.

Brief change log

  • Add a DataFusion-internal formatter for Java-style BinaryRow row-to-string output.
  • Populate $manifests.min_partition_stats and $manifests.max_partition_stats.
  • Avoid formatting partition stats when the columns are not projected.
  • Handle empty stats bytes for unpartitioned tables, all-null partition stats, and unknown stats.
  • Add focused formatter tests and a DataFusion integration test for partition stats.

Tests

API and Format

Documentation

Signed-off-by: QuakeWang <1677980708@qq.com>
@QuakeWang QuakeWang changed the title feat: Display partition stats in manifests system table feat(datafusion): populate partition stats in $manifests system table May 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant