Align evaluator metric mapping for standardized single-metric outputs

[Copilot]

DocumentRetrievalEvaluator and RougeScoreEvaluator were updated to emit standardized single-primary-metric outputs, but _EvaluatorMetricMapping.EVALUATOR_NAME_METRICS_MAPPINGS still pointed to legacy multi-metric lists. This caused AOAI conversion paths to treat non-emitted metrics as primary/expected.

• Constants alignment (_constants.py)

  • Updated:
    • document_retrieval: ["document_retrieval"]
    • rouge_score: ["rouge"]
  • Left all other evaluator mappings unchanged.
  • Left EVAL_CLASS_NAME_MAP unchanged (DocumentRetrievalEvaluator -> document_retrieval, RougeScoreEvaluator -> rouge_score).

• Changelog update (CHANGELOG.md)

  • Added an unreleased breaking-change note that these evaluators now report single primary metrics, with prior sub-metrics represented under *_properties.

# before
"document_retrieval": ["xdcg@3", "ndcg@3", "fidelity", ...],
"rouge_score": ["rouge_f1_score", "rouge_precision", "rouge_recall"],

# after
"document_retrieval": ["document_retrieval"],
"rouge_score": ["rouge"],

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• pypi.org
  • Triggering command: /home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/.venv/bin/python /home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/.venv/bin/python /home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/.venv/lib/python3.10/site-packages/pip/__pip-REDACTED__.py install --ignore-installed --no-user --prefix /tmp/pip-build-env-chn4lfg6/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i REDACTED -- setuptools>=40.8.0 (dns block)
• scanning-api.github.com
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

Context

PR #46436 (Standardize Output Schema for Evaluators, branch mohessie/standardize_output_schema) updated several evaluator classes — including DocumentRetrievalEvaluator and RougeScoreEvaluator — to emit the unified output schema (a single primary metric plus _score/_passed/_result/_reason/_status/_threshold/_properties).

However, the _EvaluatorMetricMapping.EVALUATOR_NAME_METRICS_MAPPINGS dictionary in sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_constants.py still references the old multi-metric lists for these two evaluators. This is the static lookup that powers the AOAI result conversion in _evaluate.py (_extract_metrics_from_evaluator_name, _extract_from_evaluator_base, _is_primary_metric, _calculate_aoai_evaluation_summary, _get_metric_from_criteria), so it must be kept in lockstep with what the evaluators actually emit. Otherwise:

• _is_primary_metric keeps treating the old (no-longer-emitted) first list entry ("xdcg@3" / "rouge_f1_score") as the primary metric, which makes _calculate_aoai_evaluation_summary.result_counts.passed/failed drop to 0 for these evaluators.
• _add_error_summaries (via _extract_from_evaluator_base) advertises 9 (doc_retrieval) / 3 (rouge_score) metrics that are no longer emitted at the top level, producing spurious error-status entries in _evaluation_results_list.
• _get_metric_from_criteria will try prefix-matching the new document_retrieval_* / rouge_* column suffixes against the old metric names and only match by accident (falling through to step 4).

New emitted top-level keys (already in this branch)

DocumentRetrievalEvaluator (see _document_retrieval.py)

{
    "document_retrieval": ndcg_score,
    "document_retrieval_score": ndcg_score,
    "document_retrieval_passed": ndcg_passed,
    "document_retrieval_result": EVALUATION_PASS_FAIL_MAPPING[ndcg_passed],
    "document_retrieval_reason": None,
    "document_retrieval_status": "completed",
    "document_retrieval_threshold": self._threshold,
    "document_retrieval_properties": metrics,  # the old 9 sub-metrics live here now
}

RougeScoreEvaluator (see _rouge.py)

{
    "rouge": rouge_f1_score,
    "rouge_score": rouge_f1_score,
    "rouge_passed": is_passed,
    "rouge_result": EVALUATION_PASS_FAIL_MAPPING[is_passed],
    "rouge_reason": None,
    "rouge_status": "completed",
    "rouge_threshold": self._threshold["f1_score"],
    "rouge_properties": {
        "rouge_precision": ...,
        "rouge_recall": ...,
        "rouge_f1_score": ...,
        # plus per-sub-metric results/passed/thresholds
    },
}

Required change

Update _EvaluatorMetricMapping.EVALUATOR_NAME_METRICS_MAPPINGS in sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_constants.py so that the entries match the new single-primary-metric schema:

# Before
"document_retrieval": [
    "xdcg@3",
    "ndcg@3",
    "fidelity",
    "top1_relevance",
    "top3_max_relevance",
    "holes",
    "holes_ratio",
    "total_retrieved_documents",
    "total_ground_truth_documents",
],
...
"rouge_score": ["rouge_f1_score", "rouge_precision", "rouge_recall"],

# After
"document_retrieval": ["document_retrieval"],
...
"rouge_score": ["rouge"],

Notes:

• The EVAL_CLASS_NAME_MAP entries for "DocumentRetrievalEvaluator": "document_retrieval" and "RougeScoreEvaluator": "rouge_score" are already correct and must be left as-is. The lookup chain is class name → evaluator key (here) → metric list, so the metric list is the only piece that needs to change.
• Keep the comment at the top of _EvaluatorMetricMapping that says the mapping is "based on assets.json" — once the corresponding Azure/azureml-assets document_retrieval spec.yaml and rouge_score spec.yaml output schemas are updated to a single metric, both sides will be aligned.

Acceptance criteria

1. EVALUATOR_NAME_METRICS_MAPPINGS["document_retrieval"] equals ["document_retrieval"].
2. EVALUATOR_NAME_METRICS_MAPPINGS["rouge_score"] equals ["rouge"].
3. No other entries in EVALUATOR_NAME_METRICS_MAPPINGS or EVAL_CLASS_NAME_MAP are changed.
4. Existing unit tests for _get_metric_from_criteria, _is_primary_metric, and _calculate_aoai_evaluation_summary in `sdk/evaluation/azure-ai-evaluation/tests/unittests...

This pull request was created from Copilot chat.