fix: DocumentRecallEvaluator single_hit reports false 1.0 recall on missing comparison values

[Otis0408]

Anyone using DocumentRecallEvaluator in single_hit mode with a document_comparison_field that is not present on the documents (for example meta.file_id when that key was never set) gets a silently wrong evaluation. _get_comparison_value returns None for every such document, so both the ground-truth set and the retrieved set collapse to {None}. The set intersection is non-empty (None matches None), and the evaluator reports a perfect recall of 1.0 even when the ground-truth and retrieved documents are completely unrelated. The same happens when all documents have empty-string comparison values.

multi_hit mode already guards against this: it returns 0.0 with a warning when the ground-truth or retrieved comparison values are empty or a subset of {"", None}, and the MAP/MRR/NDCG evaluators all filter out None values too. Only the single_hit path was missing the guard, so it treated a missing/empty value as a legitimate match — most likely overlooked when the guard was added to multi_hit.

The fix adds the same guard to _recall_single_hit: before matching, if the unique ground-truth values (or the unique retrieved values) are empty or a subset of {"", None}, the method logs a warning and returns 0.0. Valid comparisons are unaffected.

Before/after with the public API:

from haystack import Document
from haystack.components.evaluators import DocumentRecallEvaluator

evaluator = DocumentRecallEvaluator(mode="single_hit", document_comparison_field="meta.file_id")
result = evaluator.run(
    ground_truth_documents=[[Document(content="real ground truth")]],
    retrieved_documents=[[Document(content="totally unrelated doc")]],
)
print(result["score"])
# before this fix: 1.0  (the documents share no file_id and are unrelated)
# after this fix:  0.0

A regression test (test_missing_comparison_field_on_both_sides) plus the empty / empty-string cases already covered for multi_hit are added to the single_hit test class. The new test fails on the current code and passes with the fix.

A release note is included at releasenotes/notes/document-recall-single-hit-missing-values-c3d4e5f607182930.yaml.

[vercel]

@Otis0408 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/evaluators
document_recall.py
Project Total

This report was generated by python-coverage-comment-action