fix(memory): skip unhealthy query embeddings

[yyhhyyyyyy]

Summary

Fixes #2118 by adding bounded circuit-breaker behavior to Memory query embeddings.

When query embeddings repeatedly time out or fail, subsequent Memory recalls now skip the unhealthy vector path and immediately use the existing FTS fallback instead of paying the embedding deadline on every turn.

Changes

• Add a per-Agent circuit breaker for the current embedding provider/model.
• Open the circuit after two qualifying failures within 30 seconds.
• Skip query embedding and use FTS immediately while the circuit is open.
• Allow one half-open recovery probe after a 30-second cooldown.
• Close the circuit after a successful probe or reopen it after another failure.
• Exclude cancellation, AbortError, and local capacity rejection from provider-health failures.
• Reset circuit state when the embedding configuration changes, an Agent is cleaned up, or Memory is disposed.
• Preserve query embedding de-duplication and concurrent consumer cancellation semantics.
• Keep vector readiness intact when only query embedding fails.
• Add content-free circuit state and counters to Memory health diagnostics.
• Add the embeddingCircuitOpen retrieval degradation cause.
• Update Memory architecture and observability documentation.
• Add regression coverage for timeout thresholds, cooldowns, recovery, cancellation, concurrent settlement, Agent/model isolation, cleanup, and diagnostics.

Validation

• pnpm run test:memory — 888 tests passed
• pnpm run test:main — 6,622 tests passed
• Memory diagnostics renderer tests — 22 tests passed
• pnpm run typecheck
• pnpm run lint
• pnpm run i18n
• pnpm run format:check

Summary by CodeRabbit

• New Features

  • Added automatic protection against repeated query-embedding failures.
  • Temporarily falls back to text-based retrieval when embedding services are unavailable, then probes recovery automatically.
  • Added runtime health diagnostics for circuit state, failures, openings, and skipped queries.
  • Reports retrieval degradation when the embedding circuit is open.

• Documentation

  • Documented circuit-breaker behavior, recovery rules, diagnostics, and failure classification.

• Tests

  • Added coverage for failure thresholds, recovery, cancellation handling, configuration changes, cleanup, and diagnostics.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ec7dff95-6a27-4d8e-aecc-ab2fcf7d6a03

📥 Commits

Reviewing files that changed from the base of the PR and between afc90b5 and 33b4bac.

📒 Files selected for processing (14)

• docs/architecture/memory-quality-gates-and-observability/metrics.md
• docs/architecture/memory-quality-gates-and-observability/spec.md
• docs/architecture/memory-system.md
• docs/issues/memory-query-embedding-circuit-breaker/spec.md
• src/main/memory/index.ts
• src/main/memory/infra/diagnostics/memoryDiagnosticsCollector.ts
• src/main/memory/runtimeConstants.ts
• src/main/memory/services/retrievalService.ts
• src/shared/contracts/routes/memory.routes.ts
• src/shared/types/agent-memory.ts
• test/main/memory/managementService.test.ts
• test/main/memory/memoryDiagnosticsCollector.test.ts
• test/main/memory/serviceTestSupport.ts
• test/main/routes/memoryDto.test.ts

---

📝 Walkthrough

Walkthrough

Warm recall now uses a bounded per-Agent/provider/model query-embedding circuit breaker. Open circuits skip vector retrieval and use FTS. Half-open probes restore vector retrieval. Diagnostics, health contracts, lifecycle resets, tests, and documentation cover the behavior.

Changes

Memory query-embedding circuit breaker

Layer / File(s)	Summary
Circuit contracts and diagnostics src/main/memory/runtimeConstants.ts, src/shared/types/agent-memory.ts, src/shared/contracts/routes/memory.routes.ts, src/main/memory/infra/diagnostics/memoryDiagnosticsCollector.ts, docs/architecture/memory-quality-gates-and-observability/*	Defines breaker thresholds, degradation causes, circuit diagnostics, default values, state validation, and aggregate counters.
Retrieval circuit flow src/main/memory/services/retrievalService.ts	Tracks per-Agent query-embedding state, classifies provider failures, supports cooldown probes, skips open circuits, propagates cancellation, and preserves FTS fallback.
Lifecycle and health wiring src/main/memory/index.ts, src/main/memory/services/retrievalService.ts	Resets stale state after embedding identity changes and cleanup. Health responses expose the current circuit state.
Validation and documented behavior test/main/memory/managementService.test.ts, test/main/memory/memoryDiagnosticsCollector.test.ts, test/main/memory/serviceTestSupport.ts, test/main/routes/memoryDto.test.ts, docs/architecture/memory-system.md, docs/issues/memory-query-embedding-circuit-breaker/spec.md	Tests cover opening, skipping, probing, recovery, isolation, cancellation, races, cleanup, diagnostics, and DTO serialization. Documentation records the design and validation results.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RetrievalService
  participant ProviderGateway
  participant MemoryDiagnosticsCollector
  participant FTSRecall

  RetrievalService->>ProviderGateway: Start query embedding with abort signal
  ProviderGateway-->>RetrievalService: Return result or provider failure
  RetrievalService->>MemoryDiagnosticsCollector: Record circuit event
  RetrievalService->>FTSRecall: Use FTS candidates when circuit is open
  RetrievalService->>ProviderGateway: Send one cooldown probe
  ProviderGateway-->>RetrievalService: Return successful probe
  RetrievalService->>MemoryDiagnosticsCollector: Record circuit closure

Loading

Possibly related PRs

• ThinkInAIXYZ/deepchat#1871: Earlier recall query-embedding reliability work in the same retrieval path.
• ThinkInAIXYZ/deepchat#1947: Introduced related diagnostics, health contracts, and retrieval infrastructure.
• ThinkInAIXYZ/deepchat#1966: Related provider cancellation handling through the embedding stack.

Suggested reviewers: zerob13

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: skipping query embeddings when the embedding path is unhealthy.
Linked Issues check	✅ Passed	The changes address issue #2118 through bounded opening, FTS fallback, recovery probes, resets, cancellation handling, diagnostics, and deterministic tests.
Out of Scope Changes check	✅ Passed	The implementation, contracts, documentation, diagnostics, and tests directly support the linked circuit-breaker objective.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/memory-embedding-circuit-breaker

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}