Skip to content

Support supplied global-index results in batch scans - #1133

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-index-result-scan
Oct 15, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-index-result-scan

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Support the existing PyPaimon with_global_index_result read path in Rust core. Primary-key vector/full-text results carry physical IndexedSplits, not global row IDs. Ordinary scans currently cannot consume those captured selections, forcing the Python scanner path.

Match Java PrimaryKeyBatchScan and DataEvolutionBatchScan: preserve PK result snapshots/splits before ordinary planning, and apply DE row-ID selections and scores to the final planned ranges. Query-authorized scans ignore external raw-index selections.

PyPaimon integration: apache/paimon#10591.

Brief change log

  • Unify table vector/batch-vector/full-text/hybrid search results and scan selections as one core GlobalIndexResult, following Java. Retain optional source context internally for ranked reads.
  • Add TableScan::with_global_index_result and IncrementalScan::with_global_index_result; searches pass their result directly without a conversion API.
  • Keep DE incremental snapshot windows and final score alignment in core; reject incompatible coordinate spaces and selection modes.
  • Retain scored Java IndexedSplit frames, validate score lengths and truncated payloads, and preserve absent totalBuckets rather than materializing it during serialization.
  • Expose a single Python GlobalIndexResult and GlobalIndexResultReadBuilder, with thin static transports for external ranges/scores/splits. Remove the separate table SearchResult, Python HybridSearchResult, and explicit conversion. Update type stubs and C/DataFusion callers.

Tests

  • 14 global-index-result tests covering direct handoff, ranked reads, source snapshots, external read-context rejection, score alignment, incremental windows, authorization, and invalid inputs.
  • 490 vector-related, 116 full-text-related, and 20 hybrid core tests passed.
  • 188 Python binding vector/read/resolved-table tests passed.
  • Cross-repository PyPaimon: 132 planner/result tests plus 46 subtests, and 188 PK full-text/vector/hybrid search tests passed against the locally built binding.
  • All four search families return GlobalIndexResult and support ranked reads and direct scan handoff, with exact final split-score transport checks.
  • cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
  • cargo fmt --all --check

API and Format

Adds the core selection APIs and unifies the public table-search result type as GlobalIndexResult (no compatibility aliases). Existing plain-split callers keep their behavior. No table storage-format changes; the split wire reader now accepts the existing Java float-score representation.

Documentation

Updated API documentation and Python type stubs.

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 1992264 into apache:main Oct 15, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants