Skip to content

[python] Use native primary-key full-text search - #10586

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-pk-full-text-results
Oct 15, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-pk-full-text-results

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Enable Native execution for the primary-key full-text searches already supported by PyPaimon, using Java's physical-position result semantics.

Depends on apache/paimon-rust#1132 reaching Rust main. Native CI continues to build main; keep this PR draft until the dependency merges.

Changes

  • Send standard REST PK full-text execute_local() calls to Rust core, retaining the existing FAST-only contract and propagating execution errors without a Python retry.
  • Reuse one metadata converter for Native PK vector and full-text results. Core owns snapshot selection, query execution, Top-K, score alignment and selected file ranges; Python converts the captured result.
  • Use Java-compatible BinaryRow serialization for Python search-position partitions. The previous repr(tuple(...)) form disagreed with both Java and Rust and changed the cross-partition tie breaker. No compatibility path is retained.

Verification

  • 422 Native cases passed across full-text, PK full-text, vector and hybrid regression suites (421 in the combined run plus the additional delayed-read test).
  • 59 original Python tests passed, plus 7 subtests.
  • The new 38-case REST suite copies the existing Java PK full-text golden data and compares Python/Native historical snapshots, physical positions, BM25 scores, selected splits and materialized rows.
  • Tests cover projection/zero projection, reusable captured plans and readers, context mismatches, FAST-only errors and no Python retry after Native failures. Temporarily moving user Parquet files still permits index search; only reading user columns fails.
  • Nonempty/empty GenericRow and BinaryRow partitions now agree, and PK vector tests also compare partition bytes.
  • Rebuilt the final extension; its 8 vector binding tests and 39 final PK/DE full-text smoke cases passed.
  • Flake8, license headers and Python 3.6 syntax checks passed.

The production adapter changes remain small; the query implementation is in the companion Rust core PR.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the native primary-key full-text path against the Java implementation and the merged Rust dependency (#1132). The Python changes correctly preserve FAST-only behavior, snapshot-scoped physical positions, deletion vectors, serialized splits, score ordering, and the shared PK result conversion. No blocking code issues found. The failing Native CI was built before Rust #1132 reached main, and the Python 3.6 failure is an unrelated Debian archive dependency 404.

@JingsongLi
JingsongLi marked this pull request as ready for review October 15, 2026 10:43
@JingsongLi
JingsongLi merged commit 1871cc0 into apache:master Oct 15, 2026
13 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants