research(nightly): content-defined chunking for incremental, witness-chained index checkpoints

[ruvnet]

Summary

Nightly research + implementation: content-defined chunking (CDC) for incremental, witness-chained index checkpoints, addressing the fact that ruvector-snapshot-style full re-snapshotting scales checkpoint cost with collection size instead of edit size.

• New crate crates/ruvector-cdc-checkpoint: a deterministic FastCDC-style gear-hash chunker, a SHA-256-keyed content-addressed chunk store, and a witness hash chain (mirroring ruvector-retrieval-receipt's domain-separated chaining pattern) over checkpoint manifests.
• Connects 5 ecosystem capabilities: vector/graph index durability, witness/provenance, agent memory (churn model), RVF (portable-artifact analysis), ruFlo (scheduled checkpoint workflow).
• docs/adr/ADR-340-cdc-witness-checkpoint.md — full ADR with hypothesis, decision, evidence, threat model, rejection criteria.
• docs/research/nightly/2026-08-27-cdc-witness-checkpoint/{README.md,gist.md} — full nightly research report and standalone public gist.

Hypothesis

Given an HNSW-style index checkpointed under realistic agent-memory churn
(~0.2%/0.1%/0.3% insert/delete/update per round, 20,000 rows, 30 rounds),

when checkpoints use content-defined chunking + a content-addressed store
instead of full re-snapshot or fixed-block chunking,

then steady-state incremental bytes/round should be substantially lower
under CDC (<=50% of fixed-block, <=20% of full-snapshot),

subject to bit-identical reconstruction (witness-verified) every round,
every variant, and chunking throughput >= 20 MB/s.

Result: ACCEPT — CDC hit 11.84% of fixed-block's bytes/round and 5.19% of full-snapshot's, throughput ~23x the floor, 100% reconstruction correctness (90/90 checks).

Benchmark command

cargo run --release -p ruvector-cdc-checkpoint --bin benchmark

Real benchmark results

--- Steady-state results (rounds 1..30, round 0 cold-start excluded) ---
full_snapshot  avg_new_bytes/round(steady)=  11734378  final_resident= 351836960  final_chunk_count=       1  throughput=  1084.7 MB/s
fixed_block    avg_new_bytes/round(steady)=   5146173  final_resident= 160779040  final_chunk_count=    2910  throughput=  1140.4 MB/s
cdc            avg_new_bytes/round(steady)=    609120  final_resident=  29204492  final_chunk_count=    4575  throughput=   460.7 MB/s

cdc/fixed_block new_bytes ratio : 0.1184  (threshold <= 0.5)
cdc/full_snapshot new_bytes ratio: 0.0519  (threshold <= 0.2)
cdc chunking throughput          : 460.7 MB/s (threshold >= 20)
reconstruction correctness       : 100% (asserted every round, every variant, in-loop above)

--- Darwin-lite bounded sweep over CDC avg_size (1 generation x 4 candidates) ---
  avg_size=  1024  avg_new_bytes/round(steady)=    341856  chunk_count=   9115  fitness=0.5836
  avg_size=  2048  avg_new_bytes/round(steady)=    609120  chunk_count=   4575  fitness=0.5109
  avg_size=  4096  avg_new_bytes/round(steady)=   1096634  chunk_count=   2257  fitness=0.4345
  avg_size=  8192  avg_new_bytes/round(steady)=   1912012  chunk_count=   1129  fitness=0.3750

ACCEPTANCE: ACCEPT

cargo test -p ruvector-cdc-checkpoint: 15 passed, 0 failed. cargo clippy -p ruvector-cdc-checkpoint --all-targets: clean.

Acceptance result

ACCEPT — all three hypothesis clauses held with wide margin. See the ADR's Rejection Criteria for what would falsify this at larger scale / against the real ruvector-snapshot binary format (not yet integrated — see Limitations).

Darwin result

No ruvector harness darwin CLI is installed in this environment (verified: npx ruvector harness doctor --json → npm error could not determine executable to run). Ran a bounded, disclosed in-crate substitute: 1 generation × 4 candidates sweeping CDC's target chunk size. Winner: avg_size=1024 (fitness 0.5836) vs. the avg_size=2048 used in the headline comparison (fitness 0.5109) — a real bytes-vs-chunk-count-overhead tradeoff, not free improvement. No candidate violated a hard constraint, so none were rejected.

Flywheel result

No ruvector harness flywheel CLI is installed either (same capability check). Evidence is instead retained as this PR's ADR + nightly README + raw benchmark output, per repo convention for prior nightly research (see e.g. ADR-304 / docs/research/nightly/2026-08-13-retrieval-receipts/).

Security review

• No unsafe code. Only dependency is sha2 (already used by ruvector-proof-gate / ruvector-retrieval-receipt).
• Domain-separated hashing prevents leaf/chain/content hash confusion.
• Threat model stated precisely in the ADR: detects post-issuance corruption/forgery of a manifest or its chunks; does not prove the checkpoint producer itself was honest (no proof-gated write-path integration here).
• Witness manifests are commitments, not signatures — same open item as ruvector-proof-gate/ruvector-retrieval-receipt.

Main limitations

• Uses a synthetic index format (same shape as ruvector-snapshot, not its actual binary layout) — integrating against the real format is the named next step.
• Single collection size / churn profile measured; a full-blob-rewrite churn pattern would erase CDC's advantage (disclosed, not hidden).
• No RVF/RVM wire-format integration, cross-platform, or WASM measurement performed — analysis only, per the mandatory RVF/RVM-implications steps.

Production recommendation

Promising enough to pursue integration against ruvector-snapshot's real format and re-measure at scale before any production adoption decision — not yet production-ready as implemented (synthetic format, unsigned commitments).

Files changed

• crates/ruvector-cdc-checkpoint/ — new crate (chunker, store, witness chain, synthetic workload, benchmark, 15 tests)
• Cargo.toml / Cargo.lock — new workspace member
• docs/adr/ADR-340-cdc-witness-checkpoint.md — new ADR
• docs/adr/INDEX.md — regenerated via node scripts/adr-index.mjs
• docs/research/nightly/2026-08-27-cdc-witness-checkpoint/{README.md,gist.md} — nightly research report + public gist

Test plan

• cargo build --release -p ruvector-cdc-checkpoint
• cargo test -p ruvector-cdc-checkpoint (15/15 passing)
• cargo clippy -p ruvector-cdc-checkpoint --all-targets (clean)
• cargo fmt -p ruvector-cdc-checkpoint
• cargo run --release -p ruvector-cdc-checkpoint --bin benchmark (run twice; byte/chunk metrics bit-identical across runs, confirming determinism)

---

Generated by Claude Code

[ruvnet]

Tests (ml-research-rest) failed on the first CI run (job 98444751733), but this isn't this PR's failure: the job died during environment setup, before any test ran, because sudo apt-get install -y libfontconfig1-dev hit a 403 from an unrelated third-party APT mirror (packages.microsoft.com/repos/azure-cli), not from any package this PR touches:

Err:6 https://packages.microsoft.com/repos/azure-cli jammy InRelease
  403  Forbidden [IP: 13.107.246.41 443]
...
E: The repository 'https://packages.microsoft.com/repos/azure-cli jammy InRelease' is no longer signed.
##[error]Process completed with exit code 100.

No fix exists to port (this is a CI-infra dependency-mirror issue, not code in this diff). Re-running once the rest of the workflow run finishes (the run is still in progress, so rerun-failed-jobs currently 403s with "workflow is already running") — will report back if it fails again for a different reason.

---

Generated by Claude Code