Skip to content

[python] Support LENGTH, BIT_LENGTH, OVERLAY, PAD and TRANSLATE in query-auth rules - #10179

Open
jackylee-ch wants to merge 1 commit into
apache:masterfrom
jackylee-ch:python-query-auth-string-transforms
Open

jackylee-ch wants to merge 1 commit into
apache:masterfrom
jackylee-ch:python-query-auth-string-transforms

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Purpose

A REST catalog can hand PyPaimon a query-auth column-masking / row-filter
rule whose transform is any of the Java predicate transforms. The Python
predicate_json_parser handled CAST / UPPER / LOWER / CONCAT /
CONCAT_WS / SUBSTRING / TRIM / NULL but raised Unknown transform type for the string transforms LENGTH, BIT_LENGTH, OVERLAY, PAD and
TRANSLATE — so a table whose auth rule used one of them failed to read at
all.

Change

  • Add the five transforms to _apply_predicate_transform, mirroring Java:
    LENGTH = char count, BIT_LENGTH = UTF-8 byte count × 8 (both INT);
    TRANSLATE = code-point map (first mapping wins, a 0/absent replacement
    deletes); OVERLAY reuses the existing SQL-substring helper (replaced
    length defaults to the replacement length when omitted or negative);
    PAD (LPAD/RPAD) truncates when the target is shorter than the source.
  • Null arguments propagate to a null result, as in SQL / the Java transforms.

Tests

  • Per-transform output against the Java semantics (map/delete, overlay with
    and without length, left/right pad and truncation, null propagation) plus
    one wired through a LEAF filter. Verified non-vacuous (removing the
    branches makes the six feature cases raise Unknown transform type).

Written with Claude Code; verification is mine.

@JingsongLi

Copy link
Copy Markdown
Contributor

Reviewed the current head (e4174f7) for the REST query-auth use case. This has end-to-end value: a Java-authored rule using these string transforms can otherwise prevent PyPaimon reads. I compared the five implementations with their Java transforms, including Unicode/code-point behavior, UTF-8 bit length, null handling, overlay length defaults, and PAD truncation. I found no blocking correctness issue.

Validation: PYTHONPATH=paimon-python python -m unittest pypaimon.tests.predicate_json_parser_test (87 passed) and the auth masking/table/validation/widening suites (240 passed); git diff --check passed. CI run 36106876365 is green. These tests exercise the parser and auth consumers; a live REST catalog integration remains the production-path check. Please also remove the two PLACEHOLDER_* comments before merge.

…ery-auth rules

Purpose:
A REST catalog can hand PyPaimon a query-auth column-masking / row-filter
rule whose transform is any of the Java predicate transforms. The Python
predicate_json_parser handled CAST/UPPER/LOWER/CONCAT/CONCAT_WS/SUBSTRING/
TRIM/NULL but raised "Unknown transform type" for the string transforms
LENGTH, BIT_LENGTH, OVERLAY, PAD and TRANSLATE, so a table whose auth rule
uses one of them failed to read at all.

Change:
- Add the five transforms to _apply_predicate_transform, mirroring Java:
  LENGTH = char count, BIT_LENGTH = UTF-8 byte count * 8 (both INT);
  TRANSLATE = code-point map, first mapping wins, a 0/absent replacement
  deletes; OVERLAY reuses the existing SQL-substring helper
  (replaced length defaults to the replacement length when omitted or
  negative); PAD (LPAD/RPAD) truncates when the target is shorter.
- Null arguments propagate to a null result, as in SQL / the Java transforms.

Tests:
- Per-transform output against the Java semantics (map/delete, overlay with
  and without length, left/right pad and truncation, null propagation) and
  one wired through a LEAF filter. Verified non-vacuous (removing the branches
  makes the six feature cases raise "Unknown transform type").

Written with Claude Code; verification is mine.
@jackylee-ch
jackylee-ch force-pushed the python-query-auth-string-transforms branch from e4174f7 to 820a310 Compare September 25, 2026 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants