research: witness-chained retrieval receipts on a real multi-layer HNSW index

[ruvnet]

Hypothesis

ruvector-retrieval-receipt (ADR-304, 2026-08-13 nightly) proved witness-chained
provenance receipts detect 100% of tested tampering on ANN results, but
deliberately measured overhead against a brute-force index to isolate the
provenance layer's cost from ANN recall. Both that report and ADR-304's own
Rejection Criteria explicitly flagged the open question: does the overhead
ratio hold up against a real approximate index?

Given a real multi-layer HNSW index (ruvector-hnsw-repair::HnswGraph) built
by ingesting N vectors through ruvector_proof_gate::HashChainGate,

when top-k approximate search results are wrapped with a retrieval receipt
(PerResultReceipt or MerkleReceipt, unmodified from ruvector-retrieval-receipt),

then (a) both variants achieve 100% verify_full success, (b) MerkleReceipt's
worst-case proof stays smaller than PerResultReceipt's, and (c) receipts
never perturb search results,

subject to Merkle receipt-build p50 latency staying under 50% of raw HNSW
search p50 latency (pre-registered, not adjusted after seeing results).

Architecture

New crate ruvector-hnsw-receipt composes three existing crates —
ruvector-proof-gate (write provenance), ruvector-retrieval-receipt
(receipt cryptography, reused unmodified via dependency, zero new crypto
code), and ruvector-hnsw-repair (a real from-scratch multi-layer HNSW
graph) — without modifying any of them.

flowchart LR
    V[vectors] --> GATE[HashChainGate::admit] --> WR[WriteReceipt]
    V --> HNSW[HnswGraph::insert]
    Q[query] --> SEARCH["HnswGraph::search\n(real multi-layer ANN)"]
    SEARCH --> ITEMS["ResultItem\nscore + write_receipt"]
    WR -.-> ITEMS
    ITEMS --> BUILD["RetrievalReceipt::build\n(PerResult | Merkle, unmodified)"]
    BUILD --> VERIFY[verify_full — offline]

Loading

Files changed

• crates/ruvector-hnsw-receipt/ — new crate: HnswReceiptIndex
  (search_raw, search_items, brute_force_topk), 6 unit tests, release
  benchmark binary.
• Cargo.toml / Cargo.lock — workspace member registration.
• docs/research/nightly/2026-08-26-hnsw-witness-receipts/README.md — full
  nightly research report with raw benchmark transcripts.
• docs/adr/ADR-340-hnsw-witness-receipts.md — follow-up ADR to ADR-304.
• docs/adr/INDEX.md — regenerated via node scripts/adr-index.mjs (also
  refreshed stale "Last commit" dates across the table as an incidental,
  correct byproduct of running the canonical generator; not new content).
• docs/research/nightly/2026-08-26-hnsw-witness-receipts/gist.md — public
  technical write-up.

Benchmark command

cargo build --release -p ruvector-hnsw-receipt
./target/release/benchmark <n> <dims> <k> <ef> <queries>

Hardware: 4 logical CPUs, Intel(R) Xeon(R) @ 2.80GHz, Linux 6.18.44-fc-v21.
Toolchain: rustc 1.94.1, cargo 1.94.1.

Real benchmark results (unedited)

N	dims	Merkle build p50 overhead vs. raw search p50	verify_full success	Recall@10 (context only)
5,000	64	4.38%	300/300	0.58
5,000	64 (repeat)	4.66%	300/300	0.58
20,000	128	1.35%	300/300	0.31

Merkle worst-case proof: 160 bytes vs. PerResult's 320 bytes at k=10 (ratio
0.5, matching the brute-force experiment). Overhead shrinks with scale
because receipt-build cost is O(k), flat, while HNSW search cost grows with
graph traversal — the opposite of what a brute-force-only measurement could
show.

Acceptance result

ACCEPT. All three pre-registered conditions pass at both scales:
100% verify_full success, Merkle proof bytes < PerResult proof bytes,
Merkle overhead p50 < 0.50x raw search p50 (measured 0.0135x–0.0466x — also
clears ADR-304's original, tighter 0.15x rejection threshold for this exact
follow-up measurement).

Darwin / Flywheel

Neither npx ruvector harness darwin nor flywheel CLI was resolvable in
this environment (npm error could not determine executable to run) —
recorded honestly in the nightly report's Capability Discovery table rather
than assumed. No bounded evolution was run; two dataset scales were varied
manually instead.

Security review

No new cryptographic code — the receipt cryptography is reused unmodified
from ruvector-retrieval-receipt (already reviewed under ADR-304). No
unsafe code, no new external dependencies beyond the crates being
composed. Threat model (unsigned commitments, detect post-issuance mutation
only, no write-chain membership proof) is unchanged and stated in full in
the ADR.

Main limitations

• Only N=5,000 and N=20,000 tested; ADR-304 itself named N≥100k as the scale
  needing re-confirmation — not reached here (top item in Next Research).
• Recall@10 (0.31–0.58) is honestly low due to a cosine-vs-internal-L2
  ranking metric mismatch and untuned HnswConfig defaults — explained in
  the report, not hidden, and irrelevant to the overhead/verification
  claims actually under test.
• Composed against ruvector-hnsw-repair's research/repair-focused graph,
  not the workspace's primary production index (ruvector-core).
• No root/receipt signing (shared open item with ADR-304).

Production recommendation

Promote to an available, feature-flagged (opt-in) composition alongside
ruvector-agent-memory, following ADR-304's own proposed integration path.
Root/receipt signing remains the correct gate before any compliance-grade
claim.

Research document / ADR / gist

• Research: docs/research/nightly/2026-08-26-hnsw-witness-receipts/README.md
• ADR: docs/adr/ADR-340-hnsw-witness-receipts.md
• Gist: docs/research/nightly/2026-08-26-hnsw-witness-receipts/gist.md

---

Generated by Claude Code

[ruvnet]

Portfolio review verdict: INCONCLUSIVE.

The frozen receipt-overhead hypothesis is well supported by the recorded benchmarks, and the research-nightly, security, clippy, native-module, regression, and supply-chain jobs passed. The repository-wide promotion gate is not complete because Tests (core-and-rest) was cancelled, so the full Workspace CI run is not green.

The experiment may be accepted after that cancelled shard passes on the exact head commit. Keep the production recommendation separate: recall@10 of 0.31 to 0.58 on a research-focused graph with a known metric mismatch is not a production integration result. Promotion should remain opt-in research until the same receipt path is measured on the primary production index with the metric aligned and a predeclared recall floor, preferably at N at least 100,000.