Skip to content

feat(core): add SystemRead for Native metadata reads - #1137

Merged
JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/native-system-table-reads
Oct 16, 2026
Merged

JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/native-system-table-reads

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Expose the 11 system tables already supported by PyPaimon through a small Rust core reader:

let mut read = table.new_system_read("files")?;
read.with_projection(&["file_name", "row_count"])?;
read.with_filter(predicate)?;
read.with_limit(100);
let fields = read.read_type();
let batch = read.read().await?;

System metadata is small enough to materialize in one RecordBatch. There are no separate system-table, builder, scan, plan or split types in the core/Native API.

  • Move metadata schemas and extraction from DataFusion providers into Rust core. DataFusion and PyO3 adapt SystemRead.
  • Support snapshots, schemas, options, manifests, files, buckets, tags, branches, file_key_ranges, table_indexes and consumers. Catalog-dependent partitions retain their existing path.
  • Reuse the base table's existing FileIO, REST credentials, branch and captured snapshot. The returned batch is self-contained and preserves its schema when empty.
  • Filter before output projection and limit; predicates can reference unprojected columns. Preserve consumer ID point lookup. DataFusion retains residual filtering and does not push a limit ahead of it.
  • Expose the same SystemRead methods in Python, with read() returning a PyArrow RecordBatch. PyPaimon stores the full result in its existing in-memory SystemSplit, preserving pickle and offline worker reads.
  • Complete Java metadata semantics: persisted latest-schema options, snapshot writer_version, deleteRowCount, composite index fields, local metadata instants and physical-column/schema-evolved statistics. File keys/statistics use Java object strings; partitions use Java CAST strings.

Verification

  • Core metadata unit tests: 27 passed.
  • DataFusion system-table unit tests: 10 passed.
  • DataFusion system-table integration tests: 28 passed.
  • Built the companion Python extension with maturin. REST/system-table suites: 165 passed under UTC and 165 passed under America/New_York.
  • Covers empty/populated tables, fixed/empty/tag views, branches, PK ranges, vertical writes, schema evolution, consumer pruning, indexes, authorization, filters on unprojected columns, limits, repeated/zero-column projections and captured Python split pickle without REST access.
  • Workspace Clippy with Rust 1.98.0 and fulltext/vortex, rustfmt and diff checks passed.
  • Independently reviewed the simplified API against the previous implementation: no remaining P1/P2 findings.

Companion: apache/paimon#10604 (Draft). Python CI continues to build Apache paimon-rust main and should merge after this Rust change.

@JingsongLi JingsongLi changed the title feat(datafusion): complete system-table reads for Python Native feat(core): expose system-table scan and read APIs Oct 16, 2026
@JingsongLi JingsongLi changed the title feat(core): expose system-table scan and read APIs feat(core): add SystemRead for Native metadata reads Oct 16, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit e9bae97 into apache:main Oct 16, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants