Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench

[nvzm123]

Summary

This PR adds a built-in pylucene backend to cuVS Bench for building and searching local Lucene vector indexes through PyLucene and cuVS-Lucene.

It supports:

• Lucene101AcceleratedHNSWCodec for cuVS-assisted HNSW construction with Lucene HNSW search;
• CuVS2510GPUSearchCodec for GPU CAGRA construction and search;
• packaged configurations, deterministic parameter sweeps, automatic tuning, dry runs, index reuse, forced rebuilds, query batching, and standard cuVS Bench CSV results; and
• h5py as an explicit dependency for the existing dataset-preparation path.

This PR also owns the PyLucene end-to-end suite. Pytest owns its scenarios, assertions, parameterization, and reporting under cuvs_bench/tests/pylucene; reusable Python helpers remain non-test modules, and test-only Java adapters live under python/cuvs_bench/tests/java. Those adapters are compiled into a temporary directory before the process-wide JVM starts and are excluded from published packages.

HNSW parameters and topology

The HNSW backend exposes these benchmark parameters:

• m maps to AcceleratedHNSWParams.maxConn;
• ef_construction maps to AcceleratedHNSWParams.beamWidth;
• both inputs use cuVS-Lucene's SAME_GRAPH_FOOTPRINT heuristic to derive the underlying CAGRA build parameters;
• num_candidates controls Lucene's KnnFloatVectorQuery candidate budget and may be swept independently for each built index; and
• direct_single_segment requests one direct Lucene segment without a force merge and verifies the committed topology.

Stock PyLucene instantiates codecs through no-argument constructors. The backend therefore compiles a small Java adapter before JVM startup that delegates to the production accelerated codec with the requested heuristic parameters. It does not reimplement the production codec.

Automatic tuning covers m, ef_construction, and num_candidates, with the candidate lower bound resolved from top_k. Build parameters and segment count are recorded in schema-v4 provenance so incompatible indexes cannot be silently reused.

Runtime and integrity behavior

The backend validates the PyLucene 10.2 ABI, JAR layout, native-library paths, Lucene SPI codecs, dataset shape, and index location before execution. PyLucene and the JVM are initialized lazily.

HNSW preserves cuVS-Lucene's production behavior: it uses the accelerated writer when available and intentionally falls back to Lucene's CPU writer otherwise. CAGRA requires GPU support and is validated fail-closed. CAGRA segment metadata, vector dimensions and counts, file coverage, headers, footers, and checksums are verified before results are accepted.

Atomic, commit-bound provenance protects index reuse. Failed new builds remove only their partial output, and preflight failures preserve an existing index. In-process backend results use the shared direct CSV exporter and preserve build/search identity, latency percentiles, and failure handling.

Requirements and limits

• FLOAT32 Euclidean/L2 datasets, at most 4096 dimensions, and at least two indexed vectors
• latency mode with one search thread; query batching through --batch-size
• num_candidates >= top_k for HNSW; this is Lucene's candidate budget, not a direct cuVS ef_search setting
• CAGRA searches with k <= 1024
• direct_single_segment remains subject to Lucene's per-indexing-thread hard RAM limit and fails if Lucene commits more than one segment
• process-wide PyLucene JVM settings cannot change after initialization

PyLucene 10.2 remains a source-built external dependency. This PR depends on the production codec and compatibility changes in NVIDIA/cuvs#2475 until that PR is merged. The Bench guide pins the exact tested producer revision rather than a moving PR head.

Test coverage

Non-live pytest coverage includes backend registration, configuration expansion and tuning, parameter validation, adapter compilation and classpath failures, JVM and SPI validation, index build/reuse/cleanup, single- and multi-segment topology, search candidate propagation, result export, provenance mismatches, CAGRA corruption detection, package-resource ownership, and CLI failure handling.

The opt-in live matrix explicitly distinguishes:

• CPU HNSW build and HNSW search;
• GPU CAGRA build followed by one-layer or three-layer HNSW search; and
• GPU CAGRA build and CAGRA search.

GPU-required cases assert the accelerated writer, reader, and query implementations and fail on unavailable cuVS or CPU fallback. CPU cases explicitly verify and report stock Lucene HNSW execution.

Coverage includes a single live document, one and ten segments, 10-to-1 and 100-to-10 force merges, CAGRA searchWidth values 1, 16, and 32, deletion, selective filtering, persisted HNSW graph degree/layers, an exact Lucene filtered-search boundary, deterministic brute-force recall, duplicate-hit exclusion, inactive/filter-rejected document exclusion, and rank-one self matches. The Java-side query bridge verifies the retained production searchWidth value rather than echoing Python configuration.

CAGRA configurations use graphDegree=32, intermediateGraphDegree=64, and enough documents to avoid cuVS clamping warnings, including 24,832 vectors for the three-layer case.

Artifact gates require the production Java adapter and PyLucene YAML resources in built distributions while rejecting test Java, PyLuceneTestSupport, and .class payloads. A Bench-owned formatting POM, the pre-commit matcher, and the Spotless wrapper cover both production and test Java roots without adding a Maven artifact to the package tree.

Validation

Final revisions:

• cuVS Bench PR Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench #2385: 7816d6f5ce01111e8b4e9cd381f4b8f021cd5c00
• cuVS producer PR Add PyLucene integration and CPU/GPU end-to-end tests #2475: d6fcab0946837d7d3997cec4ed18189d3faa12e6

The GitHub pull/2475/head ref was fetched into a separate checkout and detached at the exact producer revision. The producer JAR was built from that checkout. GPU tests used a compatible native cuVS runtime reporting 26.10 and a 26.10 cuvs-java JAR; a clean rebuild of all 5,420 current-main native targets was stopped as impractical for this Java/Python-only validation.

Producer validation

• detached-checkout ./build.sh lucene: passed
• detached-checkout mvn clean verify: 332 tests, 0 failures, 0 errors, 30 skipped
• ThinJarContentsIT: 1 test, 0 failures, 0 errors, 0 skipped
• focused TestBackCompat,TestLucene102FormatConstruction: 20 passed

cuVS Bench validation

• non-live PyLucene suite: 293 passed, 43 deselected
• focused CAGRA searchWidth 16/32 cases: 2 passed, 18 deselected; recall 1.0
• full GPU PyLucene suite against the exact producer JAR: 43 passed, 293 deselected in 47.26 seconds
• full python/cuvs_bench pytest suite: 487 passed, 71 skipped in 72.68 seconds
• CUDA wheel build and manifest inspection: passed; 103 wheel entries, required resources present, no test Java or class payloads
• conda-style wheel build and isolated installed-distribution manifest inspection: passed; 168 distribution entries, required resources present, no forbidden payloads
• Ruff check/format, Bench Java Spotless apply/check, shell syntax check, and git diff --check: passed

The full producer suite retains existing randomized small-dataset graph-clamping, intentional fallback/invalid-configuration, JVM vector/native-access, and Javadoc-plugin warnings. The live PyLucene matrix emitted no cuVS graph-clamping or CPU-fallback warnings; it emitted the expected JVM notice for the incubating vector module. The isolated wheel install emitted pip's expected root-user warning because it used a temporary --target directory.

Related work

• NVIDIA/cuvs#2475
• NVIDIA/cuvs#1907
• NVIDIA/cuvs-lucene#174 (historical source of the producer-side PyLucene compatibility work)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.