Skip to content

Support primary-key full-text results through scan and read - #1132

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-pk-full-text-results
Oct 15, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-pk-full-text-results

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

PyPaimon already searches primary-key full-text indexes and reads the resulting physical file positions. Native full-text scan/read and its Python binding currently expose only global row IDs, so this existing operation stays on the Python path.

Changes

  • Support PK full-text queries through FullTextScan -> FullTextRead -> SearchResult, following Java's snapshot-scoped scan/read and PrimaryKeyScoredResult design.
  • Reuse the existing vector/full-text shared result: global row IDs for DE, file positions for PK, original scores, serialized selected splits and projected materialization after search.
  • Keep the PK FAST-only restriction, reject non-partition row filters and global row-ID allowlists, and validate plan context even for empty plans.
  • Reuse the shared materializer from the existing direct PK full-text read, removing its duplicate materialization logic.
  • Move shared result code into search_result modules in both core and Python bindings. Full-text bindings now return the existing SearchResult class; remove the separate global-only FullTextSearchResult wrapper and update the stub.

execute_scored() now returns the common table::SearchResult. The ranges-only execute() continues to reject PK positions, which cannot be interpreted as global row IDs. No dependency is added.

Verification

  • Core tests passed: full-text 116, shared results 21, vector builders 28, hybrid search 20 (some groups overlap).
  • Python binding tests: 8 passed; extension rebuilt locally.
  • PyPaimon Native regression: 422 cases passed across full-text, PK full-text, vector and hybrid search; original Python tests: 59 passed.
  • Java golden archives verify historical snapshots, original BM25 scores and physical positions. Tests also cover delayed/projected reads, zero projections, plan reuse, context mismatch, FAST-only errors and no retry after Native failures.
  • Removing the data Parquet files temporarily still permits searching their indexes; materialization alone fails on the missing files.
  • cargo fmt --all --check
  • cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
  • Default-feature cargo check -p paimon --locked

Companion PyPaimon PR: apache/paimon#10586. It enables this route once the Rust change reaches main.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the scan/read split and compared the primary-key full-text path with the Java implementation, including snapshot pinning, source-file reconciliation, deletion vectors, physical-position materialization, score ordering, and the unified SearchResult API. Focused Rust full-text and primary-key tests pass, and all PR CI checks are green. No blocking issues found.

@JingsongLi
JingsongLi merged commit eb1493d into apache:main Oct 15, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants