Skip to content

[core] Support filesystem partitions in SystemRead - #1138

Merged
JingsongLi merged 4 commits into
apache:mainfrom
JingsongLi:codex/native-partition-metadata
Oct 16, 2026
Merged

JingsongLi merged 4 commits into
apache:mainfrom
JingsongLi:codex/native-partition-metadata

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

PyPaimon already exposes filesystem $partitions metadata, including deletion-vector cardinalities, but Table.new_system_read("partitions") was unsupported. Add this missing core implementation so Python can reuse the existing thin SystemRead binding.

Changes

  • Read the 12 Java PartitionsTable columns from the resolved manifest snapshot. Files and deletion-vector metadata share one pinned snapshot, including fixed-empty and historical views.
  • Share typed partition aggregation with filesystem catalog listing and Table.partition_stats(). Preserve declared total_buckets rather than counting occupied buckets, and resolve file creation times as instants before aggregation.
  • Sum deletion-vector cardinalities across buckets, preserve unknown cardinalities as NULL, and skip index-manifest reads when neither projection nor predicates need the count. Normalize each partition through Java's legacy/SQL spec round trip before both display and deletion lookup. Respect default/NULL, binary UTF-8, legacy TIME precision, TIMESTAMP precision, and LTZ DST gap/overlap rules.
  • Render typed partition specs with Java casts, default partition names, Java whitespace and UTF-8 rules. Reuse the existing Java floating formatter.
  • Convert DataFusion partition epoch timestamps back to system-local TIMESTAMP columns, matching Java and core SystemRead.
  • Format BinaryRow timestamp values directly from millisecond/sub-millisecond storage, sharing formatting with schema evolution. This avoids Arrow nanosecond overflow for high-precision timestamps outside 1677–2262.

This covers the filesystem metadata behavior already implemented by Python. Catalog-owned partition audit metadata and the DataFusion catalog-backed provider retain their existing paths. No new public builder or Python binding API is introduced.

Validation

  • Core system-table tests under TZ=America/New_York: 39 passed.
  • Core partition metadata end-to-end tests: 2 passed.
  • DataFusion system-table integration tests under TZ=Asia/Shanghai: 29 passed; SQL/core/files creation timestamps agree.
  • Workspace Clippy, Rust 1.98.0, --all-targets --features fulltext,vortex -- -D warnings: passed.
  • cargo fmt --all -- --check and git diff --check: passed.
  • Companion Python Native partition metadata tests against the rebuilt binding: 27 passed in UTC (4 New York golden cases skipped), 31 passed in America/New_York. Another 72 system-table regressions passed in UTC.
  • Original review reproductions now match Java for legacy TIME deletion counts, legacy LTZ display, and non-legacy LTZ overlap deletion keys.

Independent verification completed; actionable findings were fixed with regression tests.

Companion PyPaimon PR: apache/paimon#10605 (draft until this PR is merged).

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 4e6c25a into apache:main Oct 16, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants