Skip to content

[python] Read partition metadata with Native SystemRead - #10605

Merged
JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/native-partition-metadata
Oct 16, 2026
Merged

JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/native-partition-metadata

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Use Rust core for the filesystem $partitions system table, which Python already supports. Reuse Table.new_system_read("partitions") and the existing in-memory system split; partition aggregation and deletion-vector counting execute in Rust.

Changes

  • Enable Native partition metadata on the existing REST path.
  • Align the Python implementation with Java: honor captured/time-travel snapshots, preserve manifest-declared bucket counts, render typed and unescaped partition specs with Java null/whitespace semantics, and sort output.
  • Compare file creation times as instants, resolving DST gaps and overlaps like Java, before rendering local TIMESTAMP metadata.
  • Group using serialized partition values so identical display strings cannot merge different partitions. Reuse one normalized typed row for display and deletion lookup, respecting Java legacy/SQL specs, TIME and TIMESTAMP precision, and LTZ DST rules.
  • Run New York partition golden cases in the existing non-UTC Native CI processes.
  • Cover real Avro index-manifest cardinalities, unknown and absent counts, occupied versus declared buckets, scalar partition casts, default names, and captured splits. New Native tests are marked and skipped when the binding is absent.

Depends on apache/paimon-rust#1138. Keep this PR draft until that change reaches Apache paimon-rust@main; CI continues to build Native from main.

Validation

  • Rebuilt the local Native binding from the companion Rust PR with both review fixes.
  • Native partition metadata tests: 27 passed in UTC (4 New York golden cases skipped), 31 passed in America/New_York.
  • Another 72 system-table regressions passed in UTC with Native plan/read enabled.
  • The 4 New York golden cases also passed with the Native plan/read/write/commit CI settings.
  • Ordinary Python partition system-table tests: 6 passed.
  • Changed Python files parse with Python 3.6 grammar; changed-file Flake8 and git diff --check passed.

Independent verification completed. Tests assert Java golden outputs for default legacy TIME deletion counts and LTZ normal/overlap cases, in addition to Python/Native parity.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi marked this pull request as ready for review October 16, 2026 12:28
@JingsongLi
JingsongLi merged commit a6ae9d8 into apache:master Oct 16, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants