Report when the gold check had nothing to check against - #2293
Closed
ATMAECHO wants to merge 1 commit into
Closed
Conversation
`Task.validate` returns `True` unconditionally, so a taskset that does not override it passes the gold check for every item without anything having been checked. Of the five in-tree tasksets defining a `Task` subclass, only `lean` overrides `validate`; `harbor`, `openenv`, `textarena` and `nemo_gym` inherit the base implementation. For those, `validate --only-gold` reports `valid_rate: 1.0`, which reads as a passing check but measures nothing. Mark such rows with `checked: false`, count them as `gold_unchecked` in the summary, and warn once per taskset naming the classes involved. Validation behaviour is unchanged; this only distinguishes "checked and valid" from "never checked".
xeophon
added a commit
that referenced
this pull request
Aug 29, 2026
## Overview Represent model-free gold validation as a tri-state result so tasksets without a gold check are reported as unchecked instead of valid. This is an alternative implementation of the issue identified by @ATMAECHO in #2293. It makes the absence of a check part of the Task.validate contract instead of inferring it from method overrides. ## Details - Make Task.validate return None by default while preserving explicit True and False results. - Persist unchecked as a terminal result with valid: null, including resume and summary handling. - Preserve invalid, timeout, and error precedence when gold and setup checks are combined. - Show unchecked outcomes in the validation dashboard and leave valid_rate unset when nothing was checked. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changing the default `Task.validate` from `True` to `None` shifts reporting for any task that relied on the inherited default without overriding the method. > > **Overview** > Gold validation is now **tri-state**: `Task.validate` may return `None` when there is no model-free check, instead of implicitly counting those tasks as valid. > > The default `Task.validate` returns **`None`** (was `True`). The validate CLI persists **`reason: "unchecked"`** with **`valid: null`**, treats unchecked as **terminal** for resume/summary, and shows it in the **dashboard** (dim style). **`valid_rate`** is computed only over tasks that were actually checked (valid + invalid); it stays unset when nothing was checked. Combined gold+setup runs keep **error/timeout/invalid** precedence and surface **unchecked** when neither sub-check failed. > > **Lean** tasks without a gold proof now return **`None`** instead of passing validation by default. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit d8b1e4a. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Add `unchecked` tri-state outcome to validation pipeline and dashboard > Introduces `valid: bool | None` across the validation system so tasks without a model-free check are reported as `unchecked` instead of implicitly passing. > > - Changes `Task.validate` and `LeanTask.validate` to return `None` when no gold proof exists, rather than defaulting to `True` > - Updates `FINAL_VALUES`, `_classify`, `_row`, `_all_reason`, and `_all_error` in [validate.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2466/files#diff-f9da541290540aab599f22057f02b6cfcd418d89dfa7bf9cbe3c325d970d5a67) to handle the tri-state: `unchecked` is terminal, excluded from error aggregation, and `valid_rate` is computed over checked items only > - Adds `unchecked` styling and mark to the dashboard in [validate.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2466/files#diff-2ded55a5b55a7c27d925b91343c516de853d005e99d07590ceb341bdc9e9149e) > - Behavioral Change: `LeanTask.validate` no longer returns `True` for tasks without a gold proof — callers that treated any truthy return as "valid" will now see `None`; persisted result rows store `valid=None` for unchecked outcomes > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized d8b1e4a.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here -->
Member
|
superseded by #2266 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Task.validatereturnsTrueunconditionally, so a taskset that does not override itpasses the gold check for every item without anything having been checked.
Of the five in-tree tasksets defining a
Tasksubclass, onlyleanoverridesvalidate.harbor,openenv,textarenaandnemo_gyminherit the base implementation. For those,validate --only-goldreportsvalid_rate: 1.0— which reads as a passing check, butmeasures nothing. The failure is silent and flattering: absence of a check is reported as
agreement.
The change
34 lines in
verifiers/v1/cli/validate.py:gold_implemented(task)— whether the taskset actually wrote a gold check(
type(task).validate is not Task.validate, so inheriting from an overriding classstill counts).
_run_goldmarks each rowchecked: true|false.summarizereportsgold_uncheckedingoldandallmodes (_gold_rowreads thenested gold row in
allmode).run_validatewarns once, naming the classes that don't overridevalidate.Validation behaviour is unchanged. This only separates "checked and valid" from
"never checked".
A stronger alternative, if you'd prefer it
Add an
uncheckedvalue toreasonso it fails loudly rather than being reported in thesummary. I kept this additive because that would touch
FINAL_REASONS,_is_finalandresume semantics — happy to switch if you'd rather have the louder version.
Verification
Per AGENTS.md ("to check your own work, write a temporary script instead of committing new
tests") I used a temporary script, not a committed test. It checks:
Task.validate, overrides it, and inherits aclass that overrides it;
leanchecked;summarizeingold,allandsetupmodes, including a row with nocheckedfield;passing vacuously.
All pass.
uv run ruff checkanduv run ruff format --checkare clean.One caveat I'd rather state than hide: I could not run
uv run pytest tests/on thismachine.
verifiers/v1/runtimes/limiters.pyimportsfcntl, which is Unix-only, so thepackage doesn't import on Windows. I stashed the change, collected, un-stashed, and
collected again — the failure set is byte-identical (38 both ways, same 6 collection
errors), so nothing here is caused by this change. Still worth a CI run on Linux.
Note
Report unchecked gold validations when a task does not override
Task.validategold_uncheckedcount to the summary ingoldandallmodes, tracking results that passed without a real gold check being implemented._run_goldnow includes acheckedboolean derived from whether the task's class overridesTask.validate.run_validatelogs a warning listing any task classes whose gold checks will always pass due to missingvalidateoverrides.Macroscope summarized 4a74e93.