Report unchecked gold validation

[xeophon]

Overview

Represent model-free gold validation as a tri-state result so tasksets without a gold check are reported as unchecked instead of valid.

This is an alternative implementation of the issue identified by @ATMAECHO in #2293. It makes the absence of a check part of the Task.validate contract instead of inferring it from method overrides.

Details

• Make Task.validate return None by default while preserving explicit True and False results.
• Persist unchecked as a terminal result with valid: null, including resume and summary handling.
• Preserve invalid, timeout, and error precedence when gold and setup checks are combined.
• Show unchecked outcomes in the validation dashboard and leave valid_rate unset when nothing was checked.

---

Note

Medium Risk
Changing the default Task.validate from True to None shifts reporting for any task that relied on the inherited default without overriding the method.

Overview
Gold validation is now tri-state: Task.validate may return None when there is no model-free check, instead of implicitly counting those tasks as valid.

The default Task.validate returns None (was True). The validate CLI persists reason: "unchecked" with valid: null, treats unchecked as terminal for resume/summary, and shows it in the dashboard (dim style). valid_rate is computed only over tasks that were actually checked (valid + invalid); it stays unset when nothing was checked. Combined gold+setup runs keep error/timeout/invalid precedence and surface unchecked when neither sub-check failed.

Lean tasks without a gold proof now return None instead of passing validation by default.

^{Reviewed by Cursor Bugbot for commit d8b1e4a. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add unchecked tri-state outcome to validation pipeline and dashboard

Introduces valid: bool | None across the validation system so tasks without a model-free check are reported as unchecked instead of implicitly passing.

• Changes Task.validate and LeanTask.validate to return None when no gold proof exists, rather than defaulting to True
• Updates FINAL_VALUES, _classify, _row, _all_reason, and _all_error in validate.py to handle the tri-state: unchecked is terminal, excluded from error aggregation, and valid_rate is computed over checked items only
• Adds unchecked styling and mark to the dashboard in validate.py
• Behavioral Change: LeanTask.validate no longer returns True for tasks without a gold proof — callers that treated any truthy return as "valid" will now see None; persisted result rows store valid=None for unchecked outcomes

^{Macroscope summarized d8b1e4a.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-08-29T07:57:35.514007Z	0cbb90e	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0cbb90ee26

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[macroscopeapp]

Approvability

Verdict: Approved at d8b1e4a

Macroscope's review found this PR approvable — This is a localized validation-reporting fix: tasks without model-free checks now produce an explicit unchecked terminal result, while checked validation and runtime setup remain unchanged. The CLI summaries, resume handling, and dashboard are updated consistently without major schema, security, billing, or infrastructure impact.

^{You can add or adjust custom eligibility rules. Learn more.}