Make performance evidence reproducible

[tvanreenen]

Why this change

The README included a useful-looking performance table, but only identified the hardware. It did not preserve the source revision, Python or NumPy version, BLAS implementation, thread environment, seeded workload, warmups, repetitions, raw samples, or summary method. That made the figures difficult to reproduce, compare, or refresh as the implementation changed.

This PR replaces that unsupported table with repeatable evidence. It adds benchmark commands for the two performance-sensitive workflows already described by the library—ingestion and exact search—without adding a runtime dependency or turning variable CI machines into latency gates.

What changes

The repository now provides ingest and search subcommands through uv run python benchmarks/benchmark.py. Each command writes one JSON record containing:

• the complete workload, including rows, dimensions, batch or query count, normalization mode, seed, and timed-region description;
• every measured nanosecond sample, discarded warmup count, median duration, and a derived per-query or rows-per-second value;
• the package version, Git revision and dirty state, Python and NumPy versions, platform, processor, timer implementation and resolution, numerical-library thread environment, and NumPy build/BLAS configuration;
• SHA-256 digests of the prepared arrays so two records can verify whether they used the same generated inputs.

Inputs use an explicit NumPy Generator(PCG64). The record still captures the NumPy version and array digests because NumPy does not promise identical distribution streams across every version. Data generation, digest calculation, environment inspection, explicit garbage collection, and final-result disposal occur outside timed regions. Cyclic garbage collection is temporarily disabled during measured operations and restored afterward.

The ingestion benchmark constructs and fills a fresh store on every trial. The search benchmark prepares the store once, then runs the same seeded query sequence through the selected public unfiltered search method. Search supports cosine, dot-product, and Euclidean metrics as well as normalized and raw stores.

Updated performance evidence

The README now reports clean measurements from commit 801de9a on a 24 GB Apple M4 Mac mini with macOS 26.6.1, CPython 3.13.5, NumPy 2.3.3, and Accelerate BLAS. The search table uses the same 1,000-, 10,000-, and 100,000-row counts at 384, 1,536, and 3,072 dimensions, so readers can compare row scaling and embedding-width scaling directly. Each cell shows both stored-vector memory and the median duration of seven measured 20-query trials divided by 20, after two warmup trials.

For a prepared 10,000-by-384 input, single-row ingestion measured 77.3 ms at the median, while batches of 1,000 rows measured 6.28 ms. Those figures demonstrate why callers should pass an existing batch as a batch, even though the store's geometric capacity growth keeps repeated additions from recopying every prior row.

These measurements are reference points, not portable latency guarantees. The README and benchmark guide state which memory is counted, which work is timed, and which hardware and software differences can change the result.

The README also makes the intended scale explicit: the benchmark stops at 100,000 rows because this package targets small-to-medium, in-process exact search. That row count is an upper reference rather than a promised limit or a target for continued scaling. Workloads that routinely reach millions of vectors generally need an indexed or service-backed system.

Complexity guarantees

The existing suite already verifies that repeated additions grow capacity geometrically and that unfiltered search uses the stored vector matrix without a full copy. This PR adds the remaining structural regression check named in the roadmap: a small top_k search must use partial selection and sort only the selected rows rather than fully sorting every candidate.

Wall-clock thresholds remain out of shared CI. CI continues checking algorithmic structure and observable correctness, which avoids failures caused only by transient runner load.

User impact

There is no change to the VectorStore runtime API, archive format, package dependencies, or runtime package contents. Users evaluating the library get current, qualified performance reference points and commands they can run on their own workload and machine. Contributors get an inspectable JSON artifact for release evidence and performance investigations.

The benchmark command and guide are included in the source distribution. The wheel remains limited to the runtime package. The package version remains 0.5.0 on this branch; the dedicated release PR will perform the 0.6.0 version change.

Documentation and ancillary changes

• Replace the old timing grid and one-line hardware footnote with measured search and ingestion results, methodology, exact commands, and non-guarantees.
• Add a benchmark guide covering timed regions, options, output fields, threading, input reproducibility, and result interpretation.
• Add command-level tests for both JSON records and a focused test proving cyclic garbage collection stays outside measured trials.

Verification

• uv run ruff check .
• uv run ruff format --check .
• uv run mypy src benchmarks
• uv run pytest -W error -q — 298 tests passed on the locked Python 3.13 environment
• uv run --no-project --isolated --python 3.11 --with numpy==1.23.2 --with pytest==8.4.2 --with-editable . pytest -W error -q — 298 tests passed after explicitly verifying Python 3.11 and NumPy 1.23.2 were imported
• Source and wheel distributions built successfully; source contents include the benchmark command, guide, and tests, while the wheel contains only the runtime package.
• Clean benchmark records were collected from commit 801de9a with dirty: false; shared workloads produced matching input digests.

Commit organization

1. Add the ingestion and search benchmark commands with command-level tests.
2. Protect partial top-k selection with a structural regression test.
3. Record verifiable input provenance and control garbage collection around timing.
4. Replace stale performance claims with reproducible evidence and a benchmark guide.
5. Keep the no-copy input digest implementation clean under static type checking.
6. Correct the benchmark hardware provenance to the verified Apple M4 machine.
7. Align every embedding width on the same row-count benchmark matrix.
8. Clarify the intended small-to-medium benchmark boundary.