Skip to content

[read] Support Java calendar query authorization masks - #1123

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-rest-calendar-masks
Oct 13, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-rest-calendar-masks

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Fill an existing PyPaimon Native gap: REST query authorization column masks using YEAR, MONTH, DAY, HOUR, MINUTE, SECOND, QUARTER, DAY_OF_YEAR, WEEKDAY, ISO_DAY_OF_WEEK, DAY_OF_WEEK, WEEK and YEAR_OF_WEEK.

PyPaimon already evaluates these masks in Python. Rust currently rejects the calendar transforms, preventing the same existing reads from using Native.

Brief change log

  • Add a calendar Transform and DatePart in Rust core; parse the Java REST names and collect their field dependencies.
  • Validate stored and rebound DATE / TIMESTAMP families as Java DateExtractTransform does. Reject LTZ, non-calendar types, malformed spellings, invalid precision and atomic object types.
  • Evaluate the 13 parts on Arrow DATE32 and timestamp(s, ms, us, ns) arrays as INT32 with NULL propagation. Use proleptic Gregorian arithmetic and ISO week boundaries; preserve Java Timestamp.toLocalDateTime's narrowing to a signed int epoch day, including large and negative timestamps.
  • Integrate with the existing query authorization mask execution, output type checks, actual source nullability and cross-mask dependency rejection. No additional Python binding API is needed.

Tests

  • 146 actual Java-generated calendar vectors, each checked for all 13 parts plus NULL; DATE32 / timestamp extrema, BCE, leap days, ISO years, negative fractions and ordinary Arrow fast-path boundaries.
  • Predicate tests: 109 passed; authorization unit tests: 41 passed; date extraction test filter: 3 passed; REST query authorization integration tests: 37 passed.
  • REST integration exercises append, primary-key and data-evolution tables, hidden DATE / TIMESTAMP operands, masked user predicates and LIMIT.
  • Rebuilt the Python binding locally; companion Python tests passed for strict Native reads and safe Python fallback.
  • cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings and cargo fmt --all -- --check passed.

API and Format

Extends the existing Transform API with DateExtract and DatePart. No storage format or Python binding API changes. This change supports column masks; transformed REST row-filter predicates remain outside its scope.

Documentation

The new enum documents calendar numbering and ISO week semantics. The companion Python PR remains draft until this implementation reaches Rust main; its Native CI keeps building Rust main.

Companion PyPaimon PR: apache/paimon#10545.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete change at 1622761. Rust calendar extraction, QueryAuth regression, REST integration, and Python native calendar validation passed; no correctness issues found.

@JingsongLi
JingsongLi merged commit 272cce9 into apache:main Oct 13, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants