Skip to content

[python] Read REST system tables with Native SystemRead - #10604

Merged
JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/native-system-table-reads
Oct 16, 2026
Merged

JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/native-system-table-reads

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Read the existing PyPaimon system tables through Rust core when Native reads are enabled on a supported REST FileStoreTable.

  • Materialize metadata through Table.new_system_read(name).read(), returning a single PyArrow RecordBatch. The Native SystemRead also exposes read_type, with_projection, with_filter and with_limit.
  • Store the full result in the existing Python SystemSplit. Existing in-memory projection/filter/limit, split reuse and pickle continue to work without recreating REST credentials or file IO on workers. No Native scan/plan/split classes are introduced.
  • Delegate snapshots, schemas, options, manifests, files, buckets, tags, branches, file_key_ranges, table_indexes and consumers. Partitions retain their catalog-backed implementation.
  • Match Java metadata semantics in Python: row-cast partitions, object-form keys/statistics, logical escaped file paths, actual file formats, physical-column/schema-evolved statistics, captured views and local metadata instants.
  • Preserve declared schemas and empty results. Propagate Native read failures.

The changed metadata text follows Java. Logical escaped file paths can be passed through to_file_io_path before Python URI-based FileIO operations.

Dependency / CI

The companion apache/paimon-rust#1137 is merged. CI continues to build Apache paimon-rust main, with no fork or feature-branch pin.

Review follow-ups

  • Format Python LTZ partition values from their epoch milliseconds and submillisecond fraction, without narrowing them into Arrow i64 nanoseconds. Preserve local-zone history/DST and submicrosecond fractions.
  • Keep explicitly empty snapshot views empty when reading table_indexes; never pass the empty-view sentinel to scan(None), which requests latest.
  • Construct metadata file paths from logical relative bucket paths, retaining file: and escaped components with data-file.path-directory.

Verification

  • Native REST/system-table tests and existing system-table tests: 178 passed in UTC and 178 passed in America/New_York, using the companion extension.
  • Added regressions failed before the fixes (9 failing cases) and passed afterward (14 focused cases). Shanghai timezone LTZ/path cases: 13 passed.
  • Direct Native API tests cover filtering unprojected columns before projection/limit, read_type, zero limits and repeated/zero-column projections.
  • Captured Python splits support another read configuration and pickle/offline reads without recreating REST.
  • Prior global-index builder / SQL system-table regressions: 48 passed, 27 subtests passed. Escaped partition-source Native merge regressions: 24 passed.
  • Flake8, license headers, Python 3.6 syntax and diff checks passed.
  • Independent review of the simplified core/Python integration completed without P1/P2 findings.

@JingsongLi JingsongLi changed the title [python] Read REST system tables through Native metadata providers [python] Read REST system tables through Native core APIs Oct 16, 2026
@JingsongLi JingsongLi changed the title [python] Read REST system tables through Native core APIs [python] Read REST system tables with Native SystemRead Oct 16, 2026
@JingsongLi
JingsongLi marked this pull request as ready for review October 16, 2026 07:11

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 5d019f5 into apache:master Oct 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants