[python] Complete native index lifecycle and explicit search plans

[JingsongLi]

Purpose

Use Rust for the index capabilities PyPaimon already supports: global-index build/drop and explicit vector, batch-vector and full-text Scan → Plan → Read. Align correctness with Java rather than preserving Python-specific behavior.

The companion Rust PR apache/paimon-rust#1136 is merged (33a14cc). CI continues building apache/paimon-rust main; no fork or branch pin is introduced.

Changes

• Delegate BTree/Bitmap and all five Vindex global-index builds on deletion-vector tables and row-tracked append tables. Remove unnecessary data-evolution and global-index.enabled gates. Full-text retains its existing deletion-vector restriction.
• Keep physical row-ID coverage in index-build splits, matching Java. Query snapshots still apply their own deletion vectors. Structural validation uses the actual table configuration.
• Delegate index-drop preparation to Rust while retaining the public Python commit/dry-run API. Failed preparation is not retried; ambiguous commit failures preserve files.
• Use Rust core planning and local reading for the existing explicit vector, batch-vector and full-text scan/read APIs, including primary-key indexes. Keep ordinary Python split/snapshot metadata for pickle and Ray transport, pin the captured snapshot, and preserve caller changes to source modes, reader options or split lists.
• Enable native full-text and hybrid search on eligible non-data-evolution row-tracked tables.
• Use the Rust core Vindex reader underneath Python index inputs, including transported plans. Handle actual hit counts smaller than K, live NULL vectors and sparse filters, with Java-compatible score conversion and query parameter validation.
• Fix Python’s cross-family BTree/Bitmap reader composition to union results, matching Java and preserving rows when the families cover different ranges.

Python remains responsible for API adaptation, option/predicate conversion and transport; index algorithms and native source planning live in Rust core. Existing unsupported table/authentication combinations keep their established qualification behavior.

Validation

Built the companion Rust branch locally with maturin, then ran:

• Broader REST Native index-build/search/drop suite: 816 passed.
• Final explicit search-plan and append-search suite: 107 passed (112 native plans and 100 native reads exercised). Includes single/batch vector and full-text modes, primary-key updates after planning, pickle/Ray transport, mutable splits, reader configuration changes, raw Vindex/Lumina defaults and mixed-family rejection.
• Final low-level Vindex/input/metric/live-NULL suite: 59 passed, 55 subtests passed.
• Ordinary build, family-union and evaluator suites: 68 passed, 56 subtests passed.
• Direct Rust vector/Blob Python binding regressions: 48 passed.
• Without an importable Native package, all 147 cases in the four new Native-only modules collected and skipped correctly.
• Flake8, Python 3.6 syntax checks for all 28 changed/new Python files, all 992 license headers, and diff checks: passed.
• Independent Rust/Java and Python transport review: no remaining actionable P1/P2 findings.

CI follow-up

• Rebased on current master, including [python] Align empty and metadata-only results with runtime read schemas #10578's runtime-schema fixes.
• Removed the obsolete AND-padding machinery from Python's field-reader union. Like Java, an index result contains actual hits; uncovered ranges remain raw-fallback work. Updated the extra-field grouping regressions to check both true union hits and the absence of invented hits.
• Re-ran all 128 cases from the previous Native CI failure list: all passed against a binding whose Rust source tree matches merged main. Ordinary vector-filter/exactness/scalar-mode regressions: 164 passed, 31 subtests passed.
• With all five Native CI switches enabled, the new search/drop/reader suites plus all seven update entry points passed: 156 tests; native update counters were positive for row_id, grouped, predicate, upsert, incremental, merge and shard.
• Flake8, Python 3.6 syntax, license headers and diff checks passed.
• Python 3.13's previous Java build failure was a Maven Central HTTP 502 while downloading doxia-sitetools; the new head reruns that job.

Native CI qualification regression

The subsequent full run passed 12,317 tests and failed only the PK builder routing unit test: once main provided search-plan metadata, its schema-only table stub entered native qualification without an options attribute. Native qualification now treats the missing native-read configuration as ineligible. The regression forces bridge availability so ordinary CI exercises the path even without a Native package.

• Native vector search and PK definitions: 112 passed, 7 subtests passed (51 native plans and 50 native reads).
• PK definitions and query-authorization regressions: 19 passed, 21 subtests passed.
• Routing regression with the Native package unavailable: passed.
• Flake8, Python 3.6 syntax and diff checks: passed.

[leaves12138]

LGTM