Skip to content

[read] Support Java-compatible REST query authorization masks - #1122

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-query-auth-masks
Oct 13, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-query-auth-masks

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

What changes

Fill an existing PyPaimon Native gap: REST query authorization column masks using SUBSTRING, TRIM, LENGTH, BIT_LENGTH, PAD, TRANSLATE and OVERLAY. Parsing, dependency collection, validation and Arrow execution live in Rust core; no additional Python binding API is needed.

Match Java character/position semantics, including Unicode characters, UTF-8 bit length, mixed field/literal arguments, stored FieldRef type validation, checked integer conversion, SUBSTRING's lazy position parsing, NULL evaluation order, negative positions and wrapping OVERLAY arithmetic.

Carry the original REST response in Java-compatible QueryAuthSplit (SplitSerializer type 5). Metadata serialization preserves rules; deserialization retains the authorization-required marker without reconstructing a catalog-issued grant. Bare DataSplit serialization still refuses to strip authorization.

Keep a REST authorization session when an externally resolved schema has the same definition and branch. Ignore client-synthesized creation timestamps, preserve immutable options, support mutable option removals, and discard cached snapshot resolution. Changed schema definitions, branches and already resolved historical/pinned views still lose the session. A regression test prevents a resolved-schema copy from clearing historical flags while reusing an old grant.

Verification

  • Actual Java 11 Paimon transforms: 9,000 execution cases and 1,165 SUBSTRING parsing boundary cases compared with Rust, without mismatches.
  • Java-generated QueryAuthSplit golden; UTF-16 key ordering, UTF-8/NUL strings, large policies, malformed lengths and bounded nesting.
  • Rust: 5 transform, 39 authorization, 107 predicate, 87 split/source and 36 REST query-authorization tests passed.
  • CI command: cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings; cargo fmt --all -- --check.
  • Locally rebuilt Python binding: authorization scenarios exercised Append, PK and Data Evolution tables, Native planning/reading, hidden dependencies, masked filters, LIMIT and safe Python fallback.

Scope

This adds column-mask transforms. Transformed REST row-filter predicates remain outside this change. Companion PyPaimon PR: apache/paimon#10539. It uses the captured policy when planning or reading falls back and remains draft until this change reaches Rust main.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete change at a743e5e. The Rust unit tests and Python/native authorization validation passed; no correctness issues found.

@JingsongLi
JingsongLi merged commit 31201c0 into apache:main Oct 13, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants