[Hillary Mutisya (hillary-mutisya)]

Summary

Adds reproducible BEAM evaluation against the durable memory service and addresses ingestion, background-maintenance and retrieval issues encountered when processing long conversations.

This is not a benchmark-only change: the token-safety, scheduling and indexed-answer improvements also apply to production memory clients.

Changes

1. TypeScript BEAM evaluation

• Add a harness and in-process memory-service bridge.
• Support smoke, qualification and full runs across BEAM’s 100k, 500k, 1m and 10m tiers, using BM25 RAG, content RAG, native content answers and event RAG.
• Pin dataset and scoring-protocol provenance, fingerprint product/model configuration, and checkpoint ingestion, predictions and judging for resumable runs.
• Report provider-blocked and malformed responses separately from scored answers. Infrastructure failures remain fatal; accuracy and coverage are reported independently.

2. Token-safe long-document ingestion

• Route plain text, Markdown, HTML and VTT through a shared post-parser splitting/packing stage. Plain text remains literal rather than being reinterpreted as Markdown.
• Separate 24k-token extraction parts from lossless 7.5k-token embedding windows, with final 8k-token embedding-input guards at indexing and hosted API boundaries.
• Preserve Unicode, whitespace, source/revision metadata and transcript timestamps. Packing does not cross incompatible source, revision or transcript-turn boundaries.
• Bound aggregate hosted embedding requests to sequential 8k-token subbatches, retaining input order and validating each subbatch’s result count.
• Increase the default embedding-attempt timeout from 30 to 180 seconds to accommodate provider retry waits, while preserving explicit caller deadlines. Correct character-limit forwarding so it does not accidentally alter concurrency.

3. Coalesced maintenance and bounded ingestion publication

• Add automatic/manual maintenance policies, nested foreground holds, status reporting and deadline/cancellation-aware draining.
• Coalesce autonomous work by corpus/family, defer maintenance during active imports, and reconstruct pending work from durable state after restart.
• Add opt-in candidate, pass and request budgets, with persisted admission decisions and bounded continuation.
• Keep partial-build target errors visible without blocking independent targets or subsequent passes. Preserve the existing two-attempt limit for unchanged failed inputs and the blocking path for infrastructure failures.
• Support document/byte-bounded publication groups with bounded admission waits. Recovery reconciles already-published revisions rather than needlessly reimporting committed data.

4. Retrieval efficiency and indexed-answer coverage

• Reuse identical query embeddings within a request and compatible fallback passes. Reuse is bounded and isolated by model, embedding identity and exact text; it is not a persistent query or answer cache.
• Balance indexed evidence across relevant sources and query terms, and retrieve eligible neighboring conversation blocks for sequence-relative questions.
• Allow one refinement round after an evidence-related noAnswer: at most two focused searches, retaining the original source scope, temporal context and request budget.
• Merge refinement evidence without overlapping ranges, retain bounded final context, and revalidate exact citations before delivery. Diagnostics preserve the initial and refined reader contexts.
• Extend opt-in timing traces with batch correlation, maintenance queue/work measurements and embedding-cache counters.

Behavior and compatibility notes

• Product defaults remain automatic maintenance and two-document publication groups. Benchmarks default to manual maintenance and ten-document groups, with an 8 MiB group budget and 250 ms admission window.
• Deferred maintenance is still pending work—not eliminated cost. Privacy removal and source-eligibility checks remain immediate.
• Existing stored messages are not silently rechunked because that would change indexed locators. Evaluating the new chunking policy requires reimport/rebuild and fresh benchmark stores.
• Answer refinement can add two searches and another reader call. This is an intentional coverage/latency tradeoff, not an unbounded retry loop.

Validation

• Full workspace build, formatting and all four quality gates passed on the merged branch.
• Memory-service regression run after integrating the retrieval changes: 223 tests across 16 suites passed. A further 49 targeted memory tests passed after merging the latest main.
• Added offline coverage for token boundaries, Unicode and metadata preservation, embedding retries, grouped-ingestion recovery, maintenance continuation, query-cache isolation, scoped refinement and benchmark resume/error reporting.
• A full 100k content-RAG replay processed all 400 questions: 398 scored, two provider-blocked, 189 correct, giving 47.49% scored-only accuracy at 99.5% coverage. The earlier runaway latency pattern did not recur.

Benchmark limitation: that live replay used commit 0b8232a6d, before the latest indexed-answer refinements. It is not a score for the final merged HEAD. Maintenance was deferred rather than drained, and the complete cross-tier/provider matrix has not been run.