Skip to content

[python] Use Native for transformed REST authorization filters - #10548

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-rest-filter-transforms
Oct 14, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-rest-filter-transforms

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

PyPaimon already supports transformed REST query-auth row filters, but those filters prevented Native execution. Add strict Native coverage using the Rust core implementation in apache/paimon-rust#1124 and align the existing Python fallback with Java semantics.

Changes

  • Evaluate AND/OR and ordered response.filter entries only on undecided rows, using Arrow filtering/scattering and a fast path when every row remains active.
  • Validate all children and typed literals before row evaluation, including ignored literals and unreachable branches. Accept Java's omitted/null lists on constant and null-test leaves; reject malformed empty compounds.
  • Enforce CAST character/byte widths, padding and default lengths; preserve the original executor's runtime class when field references are rebound.
  • For rebound NumericToString, print Java's DATE/TIME internal integers and decimal plain text. Explicitly reject unsupported rebound object formatting rather than authorize rows with Arrow-specific text.
  • Match Java's empty CONCAT and rejection of non-NULL binary predicate literals.
  • Add Native regression coverage for all existing string transforms, leaf functions, the validated CAST subset and all 13 calendar extractors across DATE and millisecond/microsecond/nanosecond TIMESTAMP.
  • Cover append, primary-key and data-evolution tables, hidden operands, authorization before masking/user predicates/LIMIT, and captured-policy Python fallback after wire serialization or reader setup failure.

No new binding API is needed. Transform/filter execution stays in Rust core; Python continues to transport the existing REST grant. This does not claim all Java CAST executors.

Dependency

Rust dependency apache/paimon-rust#1124 has been merged. Native CI continues building Rust main; no fork or PR SHA is pinned.

Verification

  • Reproduced the previous eager compound evaluation failure on a row that Java short-circuits.
  • 351 authorization/parser/validation/masking regression tests passed (2113 subtests).
  • 247 strict Native/Python authorization tests passed using the locally built Rust binding (481 Native reads, 517 Native plans, 246 Native writes and commits).
  • Final transformed-filter suite: 141 passed, exercising 276 Native reads, 285 Native plans, 141 Native writes and commits.
  • Actual Java oracle comparison: 103 existing transformed-filter/literal/width cases had zero differences; 83 supported rebound DATE/TIME/decimal and related outputs also matched Java.
  • Flake8, changed-file whitespace checks and Python 3.6 syntax checks passed.
  • Independent reviews checked authorization enforcement, fallback policy capture, Java executor semantics and short-circuit regression coverage.

CI fixes

  • Add the generated calendar Java JSONL fixture to the exact-file RAT exclusions. This resolves the license gate that blocked all six regular Python jobs.
  • Pass the required owner to the JindoInputFile video test fixture after the merged Jindo lifetime change.
  • Reproduced both failures locally. The parent RAT check now reports zero unapproved/unknown licenses; all 19 video tests and 11 subtests pass.
  • The previous 141 transformed-filter Native failures were caused by CI building Rust main before [bug][core] Fix loser tree test #1124 merged. That dependency is now available on main.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the Python fallback changes, Java-compatible evaluation order, literal/type validation, CAST width/remapping behavior, and regression coverage. The targeted Python tests (163/163) pass. The Native CI failures are expected until paimon-rust#1124 is merged, and the Maven RAT failure is from the existing unlicensed calendar oracle resource on the base revision; no blocking issues found. Approving.

@JingsongLi
JingsongLi marked this pull request as ready for review October 14, 2026 00:49
@JingsongLi
JingsongLi merged commit 8de6fdb into apache:master Oct 14, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants