Correct out-of-range nDCG@10 and guard metric bounds - #37
Merged
Conversation
The query2doc (zs) / gpt-4.1-nano / splade-pp / scifact run carried ndcg_cut_10 = 23.0 — impossible for a normalized ranking metric, which silently inflated the method's average. Correct it to 0.7116, recompute the content-derived run_id, and regenerate results.csv + manifest.json. Add a validate() guard that rejects any metric value outside [0, 1] so a corrupt or mis-scaled score can no longer be committed or pass CI. This is a code-level check (like the absolute-path guard) — schema.json is unchanged, so schema_version stays at 1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
One result carried an impossible
ndcg_cut_10 = 23.0(ranking metrics are normalized to[0, 1]), silently inflating the method's average and its ranking. This corrects the value and adds a guard so it can't recur.The corrupt run —
query2doc(zs) /gpt-4.1-nano/splade-pp/beir-v1.0.0-scifact, params_hasha5413fcf:metrics.ndcg_cut_10→0.7116(in line with the fs=0.7121 / cot=0.7065 siblings;recall_100=0.9493was already fine).run_id(5e651b293bffa080→c46db351c9adb090) —run_idhashes the metrics by design, so a result change must change it.results.csv+manifest.jsonviaaggregate_runs.py(2270 rows unchanged; new content hash).Root-cause guard —
validate()now rejects any metric value outside[0, 1](_validate_metric_ranges), so a corrupt or mis-scaled score is blocked at validate/aggregate time and in CI. Code-level check (like the absolute-path guard);schema.jsonis unchanged, soschema_versionstays at 1.Test plan
pytest reproducibility/tests/— 54 passed (3 new: value >1, value <0, bounds OK)python reproducibility/scripts/aggregate_runs.py --check— passesresults.csv🤖 Generated with Claude Code