research(agent-memory): structural-time-gated compaction for bursty-idle workloads

[ruvnet]

Summary

ruvector-agent-memory::CoherencePolicy scores memory-compaction retention with a recency term based on last_accessed_at — a wall/step clock. This run tests whether swapping that recency signal for emergent-time's Structural Proper Time (internal time driven by embedding movement, not wall-clock ticks) improves retention for agents whose activity is bursty: dense writing sessions separated by long idle gaps (waiting on a human, a slow tool call, an overnight pause).

Result: ACCEPT. On a synthetic bursty-idle workload, wall-clock CoherenceWeighted retains 27.2% of held-out relevant memories after compaction; the new StructuralTime policy retains 59.0% (+31.8pp) — while on a matched steady-activity control workload the two are statistically identical (59.0% vs 59.0%, +0.0pp). Direction reproduces across 8 independent seeds.

Connects 4 existing RuVector pieces (agent memory, emergent-time, vector coherence scoring, witness/provenance hashing) with zero new algorithms and zero new transitive dependencies — emergent-time is itself dependency-free.

Hypothesis (pre-registered)

Given a memory stream with a long idle gap between a dense "phase 1" burst
and a smaller "phase 2" burst right after the agent returns,

when compaction recency is computed from Structural Proper Time instead of
last_accessed_at,

then Recall@10 for held-out queries about the end of phase 1 should exceed
the wall-clock CoherencePolicy baseline by ≥3.0pp,

subject to: no more than 1.0pp regression on a steady (no-gap) control
workload of the same size.

What changed

• crates/ruvector-agent-memory/src/temporal_compaction.rs (new) — StructuralTimePolicy (candidate A) and GatedStructuralTimePolicy (candidate B), both additive CompactionPolicy implementations. CoherencePolicy's default behavior is untouched.
• crates/ruvector-agent-memory/src/memory.rs — adds MemoryStore::advance_clock() to model an idle period.
• crates/ruvector-agent-memory/examples/temporal_compaction_bench.rs (new) — the benchmark, with embedded acceptance tests + an 8-seed robustness test.
• Cargo.toml — one new path dependency: emergent-time (already in-workspace, dependency-free).
• docs/adr/ADR-340-structural-time-memory-compaction.md, docs/research/nightly/2026-08-29-structural-time-memory-compaction/{README,gist}.md + raw benchmark evidence.

Benchmark command

cargo run --release -p ruvector-agent-memory --example temporal_compaction_bench

Real benchmark results

[bursty-idle]                     Recall@10
LRU                                   66.7%
CoherenceWeighted (baseline)          27.2%
StructuralTime (candidate A)          59.0%   (+31.8pp vs baseline)
GatedStructuralTime (candidate B)     59.0%   (+31.8pp vs baseline)

[steady control]                  Recall@10
CoherenceWeighted (baseline)          59.0%
StructuralTime (candidate A)          59.0%   (+0.0pp — no regression)

Acceptance: [A] +31.8pp ≥ 3.0pp → PASS. [B] same → PASS. [C] steady regression +0.0pp, tolerance −1.0pp → PASS. Verdict: ACCEPT.

Witness (FNV-1a over seed+params+rounded recall, reusing emergent_time::witness::fnv1a64 — no new hash): e0d3b9cf5b37176e, reproduced byte-for-byte across every run used for this report.

Robustness: structural_time_wins_across_multiple_seeds re-checks the direction of the effect across 8 independent seeds (1,2,3,4,5,340,7777,99991) — all pass.

Honest disclosures (see README "Reading the numbers honestly")

• GatedStructuralTimePolicy adds nothing measurable on this dataset — identical to StructuralTimePolicy in both workloads. The jitter gate never binds at the tested noise level. Reported as a neutral finding, not hidden.
• Plain LruPolicy also beats CoherenceWeighted on the bursty-idle workload (66.7% vs 27.2%). Not the paper's claim (not a like-for-like comparison), but disclosed because it shows the coherence term — when its context window is realistically unrelated to what's later queried — actively degrades an otherwise-cleaner recency cutoff under the idle gap, which StructuralTime (same coherence formula) does not.
• Methodology note: the first dataset design (50% compaction ratio, context window matching the recall topic) left every policy at a 100% recall ceiling — not discriminative, and risked leaking the evaluation answer via coherence. Fixed (tighter 35% ratio, context restricted to an unrelated topic) before any acceptance number was computed; the pre-registered 3.0pp/1.0pp thresholds themselves were never touched. A separate bug (rebuilding stores via insert(), which silently discarded idle-gap timestamps) was also caught and fixed before any acceptance number was recorded.

Limitations (explicitly not claimed)

Synthetic dataset only (no real agent trace), one gap magnitude, one two-phase topic-disjoint dataset shape, only the embedding channel of StructuralMetric is honestly populated (no entropy/graph/prediction-error signal available in this crate), and candidate B is not validated as beneficial. See the ADR's "Rejection Criteria" and README "Failure modes" for the full list.

Security review

Pure scoring-function change inside an existing, already-gated compact() call. No new external input, no new authority, no new serialization format, no reward-hack surface (ground truth and dataset generator are fixed before any policy result is computed; acceptance thresholds are constants the candidate policies cannot see or influence).

Production recommendation

Not proposed for default-on promotion from this single run. Ships as an additional, independently-selectable CompactionPolicy behind no feature flag — adoptable incrementally with zero migration cost. Recommended before any real-deployment use: validate against a corpus of real agent-memory traces (this crate's own ledger/observation modules could source one).

MetaHarness / Flywheel / Darwin status

Checked, not assumed: npx metaharness --help resolves (a project-scaffolding CLI, not an in-repo research daemon; not applicable to a single bounded hypothesis like this one). npx ruvector harness doctor/flywheel/darwin --json — no such CLI exists in this repository; no such tool call is fabricated in the report. Evidence retention, evolutionary search, and promotion gating were performed directly and are documented in the ADR/README instead.

---

Generated by Claude Code