Skip to content

[read] Complete Java scalar REST authorization casts - #1125

Merged
JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/native-rest-numeric-casts
Oct 14, 2026
Merged

JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/native-rest-numeric-casts

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Complete the REST authorization CAST subset that PyPaimon already supports, using Java execution and row getter semantics. The measured 20 × 20 matrix has 212 legal Python/Java executor pairs; Native now covers each of them. Reject the 16 pairs that Python previously admitted but Java has no executor for.

Changes

  • Bind the original Java CAST executor in Rust core alongside the current field expression. Preserve it after field rebinding and include it in grant equality; the public Transform and Python binding APIs stay unchanged.
  • Complete floating/character, binary/character, date, time, timestamp and LTZ conversions in addition to the numeric/decimal/Boolean work in the earlier commit.
  • Match JDK 8/11/17 floating text, decimal and hexadecimal floating parsing, strict integer/Boolean parsing, declared widths, SMART date/time parsing, timestamp precision, default zone and Java long/int wrapping. NumericToString rebinding to Timestamp uses the object's ISO text.
  • Validate executor definitions before execution, but defer source/output object class checks until a non-NULL active row. Ordered filters and AND/OR short circuiting can therefore skip invalid row values like Java.
  • Allow masks on NOT NULL columns and preserve nullable output fields and metadata separately from the canonical physical reader fields. Make DataFusion authorization output fields nullable before SQL optimization.
  • Adapt Decimal/Timestamp and Integer-class mask outputs only when Arrow can preserve their exact coefficients/epoch ticks. Never round or silently turn an unrepresentable value into NULL.
  • Preserve shared schema-evolution/default helpers; no new dependency or binding API.

Verification

  • Permanent actual-Java fixtures: 165 groups in UTC, Shanghai and New York (5,799 input executions), plus the independent 400-pair capability matrix and earlier 68 numeric/predicate groups.
  • Independent Java probes: 72,053 floating cases against JDK 8/11/17; 36,068 date/time cases; 27,984 rebound executions; 30,147 object timestamp/time boundary cases. No remaining observed differences for representable results.
  • Rust authorization: 80 tests; completion/matrix: 2 tests; evolution: 26; nested evolution: 29; REST authorization: 39 (full REST catalog: 124); DataFusion authorization SQL: 9. Formatting and full workspace Clippy 1.98.0 with fulltext,vortex pass.
  • Local Python binding rebuilt. Paired real REST/Parquet Native completion suite: 510 passed; final mask/storage/captured-schema/NULL-guard cases: 27 passed. Existing Native regressions and independent process-zone runs are also verified in the companion PR.

Representation boundaries

Arrow UTF8 cannot preserve Java BinaryString's invalid byte sequences. Arrow timestamps also have a narrower fixed-unit representation than Java's millis+nanos object. Those results fail explicitly; byte-array Object.toString identity hashes remain unsupported. No NaN or signed-zero vectors were added.

Companion: apache/paimon#10551. Its Native CI continues to build Rust main, so it remains a draft until this Rust change merges.

@JingsongLi JingsongLi changed the title [read] Support Java numeric REST authorization casts [read] Complete Java scalar REST authorization casts Oct 14, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the latest head c747879. I found no blocking issues. The Java CAST executor matrix, rebinding behavior, nullable mask output handling, and DataFusion authorization projection changes are consistent with the Java semantics. Local verification passed: cargo test --locked -p paimon cast --lib (39), cargo test --locked -p paimon query_auth --lib (80), cargo test --locked -p paimon-datafusion --test query_auth (9), cargo test --locked -p paimon --lib (4037 passed, 6 ignored), cargo test --locked -p paimon --test rest_catalog_test (124), and cargo fmt --all -- --check. The latest GitHub Actions run 38525943981 had all completed jobs passing; the remaining jobs were still queued at review time.

@JingsongLi
JingsongLi merged commit 2cd024d into apache:main Oct 14, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants