Skip to content

fix: raise a clear error for empty inputs in retrieval and answer evaluators#11958

Open
Otis0408 wants to merge 2 commits into
deepset-ai:mainfrom
Otis0408:fix/evaluators-zero-division-empty
Open

fix: raise a clear error for empty inputs in retrieval and answer evaluators#11958
Otis0408 wants to merge 2 commits into
deepset-ai:mainfrom
Otis0408:fix/evaluators-zero-division-empty

Conversation

@Otis0408

Copy link
Copy Markdown
Contributor

Summary

AnswerExactMatchEvaluator, DocumentMAPEvaluator, DocumentMRREvaluator and DocumentRecallEvaluator average their per-query scores by dividing by the number of input queries, without guarding against an empty batch. Two empty lists satisfy the equal-length check these evaluators perform, so the code proceeds to sum(...) / len(...) and crashes with an opaque ZeroDivisionError: division by zero.

DocumentNDCGEvaluator already handles this case: its validate_inputs rejects empty input with a descriptive ValueError ("ground_truth_documents and retrieved_documents must be provided."). This PR makes the other four evaluators consistent with that contract, so an empty batch surfaces a clear, actionable error instead of a bare arithmetic failure.

Before

from haystack.components.evaluators import DocumentRecallEvaluator

DocumentRecallEvaluator().run(ground_truth_documents=[], retrieved_documents=[])
# ZeroDivisionError: division by zero

After

from haystack.components.evaluators import DocumentRecallEvaluator

DocumentRecallEvaluator().run(ground_truth_documents=[], retrieved_documents=[])
# ValueError: ground_truth_documents and retrieved_documents must be provided.

The same change applies to AnswerExactMatchEvaluator (message: ground_truth_answers and predicted_answers must be provided.), DocumentMAPEvaluator and DocumentMRREvaluator.

Tests

Added test/components/evaluators/test_evaluators_empty_input.py, a parametrized regression test asserting all four evaluators raise ValueError on empty input. It fails on the pre-fix source (ZeroDivisionError) and passes with the fix. All existing evaluator test suites continue to pass.

AnswerExactMatchEvaluator, DocumentMAPEvaluator, DocumentMRREvaluator and
DocumentRecallEvaluator averaged their per-query scores by dividing by the
input length without guarding against an empty batch. Passing two empty
lists (which satisfies the equal-length check) therefore raised an opaque
ZeroDivisionError. They now raise a descriptive ValueError, matching the
existing contract of DocumentNDCGEvaluator.
@Otis0408
Otis0408 requested a review from a team as a code owner July 10, 2026 14:19
@Otis0408
Otis0408 requested review from bogdankostic and removed request for a team July 10, 2026 14:19
@vercel

vercel Bot commented Jul 10, 2026

Copy link
Copy Markdown

@Otis0408 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added topic:tests type:documentation Improvements on the docs labels Jul 10, 2026
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/components/evaluators
  answer_exact_match.py
  document_map.py
  document_mrr.py
  document_recall.py
Project Total  

This report was generated by python-coverage-comment-action

@bogdankostic bogdankostic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks already good in principle, let's just place the tests in the existing evaluator test files.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not add this test file containing only a single test function. Instead, let's add this test to the existing evaluator test files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:tests type:documentation Improvements on the docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants