fix: prune deleted BM25 vocabulary entries

[hxaxd]

Related Issues

• fixes bug: InMemoryDocumentStore retains zero-frequency vocabulary entries after document deletion, affecting BM25 scoring #11993

Proposed Changes:

InMemoryDocumentStore.delete_documents() now removes vocabulary entries whose document frequency reaches zero after subtraction. This prevents terms that exist only in deleted documents from changing the epsilon-adjusted average IDF used by BM25Okapi.

PR #7549 introduced incremental BM25 indexing to avoid rebuilding the corpus for each query. Its deletion path used Counter.subtract(), which retains zero-count keys, while the scoring path treated every Counter key as part of the active vocabulary. The change repairs that state invariant at deletion time without giving up incremental indexing.

The PR also adds a regression test comparing a fresh store with a store that previously contained a deleted document, plus a release note.

How did you test it?

• Added TestMemoryDocumentStore::test_bm25_okapi_scores_do_not_depend_on_deleted_documents.

Notes for the reviewer

The cleanup is limited to the distinct terms in each deleted document.

Checklist

• I have read the contributors guidelines.
• I have updated the related issue with new insights and changes.
• I added unit tests; no public docstring change is needed.
• The planned title follows the conventional-commit format: fix: prune deleted BM25 vocabulary entries.
• The user-visible behavior is documented in a release note.
• I have added the required release note.
• I have run the full pre-commit checks.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[vercel]

@hxaxd is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/document_stores/in_memory
document_store.py
Project Total

This report was generated by python-coverage-comment-action

[anakin87]

Thank you. Looks good!