Skip to content

RQ-61-EVIDENCE (#1085): R7/R8/R9 — evidence must belong to the release, the release field must match the file, code-existence is not a predicate - #1099

Merged
avrabe merged 1 commit into
mainfrom
feat/evidence-rules-1085
Aug 27, 2026
Merged

RQ-61-EVIDENCE (#1085): R7/R8/R9 — evidence must belong to the release, the release field must match the file, code-existence is not a predicate#1099
avrabe merged 1 commit into
mainfrom
feat/evidence-rules-1085

Conversation

@avrabe

@avrabe avrabe commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Closes nothing yet — implements RQ-61-EVIDENCE (#1085): three rules for scripts/status_evidence_check.py, all found by USING the v0.60 gate rather than reasoning about it.

R7 — evidence must belong to the release

For every contains:/file: done-when that HOLDS, the oldest first-parent commit introducing the signature (git log -S<literal> for contains:, --diff-filter=A for file:) must not be an ancestor of the previous minor's highest tag (vX.(Y-1).*, so "shipped in the previous patch" is caught too; a main-line commit that is an ancestor of a patch tag predates the patch branch point, so a later-tagged patch cannot mis-attribute post-minor work).

  • Replay: the RQ-60-CANARY shape (gate merged 08:50 → v0.59.0 tagged 15:19 → v0.60 plan 18:56) reconstructed in a real git fixture with real ancestry.
  • Escape hatch: shipped-in: vX.Z (per-case, version-shaped — shipped-in: earlier is red — reason written beside it). RQ-60-CANARY on main is the live green instance; the gate accepts it.
  • Shallow clones, stated: no-git / shallow / missing-tag / uncommitted-signature SKIPS LOUDLY — an R7-SKIP warning per artifact plus a skip count in the machine-read summary line. The CI grep (updated in this PR) pins (0 skipped) and ≥ 1 archaeology check, so the loud skip cannot quietly pass in the required job; a shallow CI checkout already reds the R4 delivery floor (28) before R7 is even reached. fetch-depth: 0 in the claim-check job provides both history and tags.
  • Live: 5 archaeology checks performed, 0 skipped, 0 failures (this PR's own signature was R7-SKIP: uncommitted until committed — the skip contract demonstrated itself).

R8 — the release: field must equal the file's version

The checker version-gates by PATH; rivet's readiness query reads the FIELD; nothing asserted they agree.

  • Red fixture (the harmful direction nobody has hit): release: v0.60 parked in release-v0.59.yaml — before this PR it passed every rule (no done-when demanded, path < v0.60) while rivet counts it in v0.60's scope. Now red with a "version-gate bypass" message, and R8 is the only rule that fires on it.
  • Disposition of the 6 measured mismatches: KEPT, under a now-STATED allowance. A field naming a patch of the file's previous minor (vX.(Y-1).Z, Z ≥ 1) is the parked-patch practice — real v0.56.1/v0.56.2 artifacts written up in release-v0.57.yaml during the v0.57 cycle. Correcting the fields would falsify what rivet's release query says actually shipped them, and moving 6 artifacts between shipped-history files churns the graph for zero information; stating the rule is the honest disposition. No historical release file was edited. Missing, unparseable, ahead-of-path, and behind-without-patch fields are all red; the six are replayed as a green fixture.

R9 — a contains: into crate source is weaker than the gate that exercises it

Under a CLAIMING status, a signature whose path is crates/** excluding /tests/ components requires a written verified-by saying why code-existence genuinely IS the outcome — or, better, re-point the signature at the gate (what both #1090 corrections did).

  • Predicate argued from the tree, not taste: every honest live signature already points at gate-shaped surface — scripts/repro/ (executed oracle), scripts/test_*.py (required job), coq/** (kernel-checked by verify_proofs), .github/workflows/ (the wiring itself). Crate integration-test dirs (crates/*/tests/) are executed by the required Test job (e.g. wcet_bound_gate.rs) and are gate-shaped, so they are deliberately not matched. The only failing class is crate source — the RQ-60-A64IMPORT specimen ("the acceptance number is the deliverable" pinned on a symbol existing).
  • The rule caught its own history: the pre-fix(#1085): two v0.60 artifacts pinned evidence that could not fail on the failure they define #1090 CANARY done-when used as a green control in test_4 is exactly the shape R9 reds — the fixture was updated to the fix(#1085): two v0.60 artifacts pinned evidence that could not fail on the failure they define #1090 corrected predicate, with a comment saying why.
  • Residual, stated: R9 verifies the signature points at gate-shaped surface; whether a scripts/ signature names a gate some CI job actually runs is oracle_wiring_check.py's surface.

Red-first / mutation evidence

Pre-change checker on all three defective fixtures: 0 failures each (transcript in lane log). Post-change: each fails on exactly the rule under test and no other (asserted in the tests themselves: [f for f in fails if not f.startswith("R7 ")] == [] etc.).

Mutations (needle count asserted before each; byte-identical restore verified):

mutation needles result
R7 ancestor check disabled 1 KILLED (3 tests)
R8 rule loop disabled 1 KILLED (4 tests)
R8 patch-park allowance widened to never 1 KILLED (1 test — the six-mismatch replay)
R9 crate-source trigger disabled 1 KILLED (1 test)
R7 loud-skip accounting silenced 3 KILLED (1 test)

Files changed, and why

  • scripts/status_evidence_check.py — R7/R8/R9 + docstring (rules, dispositions, residuals); summary line now carries N release-scope archaeology checks (N skipped).
  • scripts/test_status_evidence_check.py — 12 new tests (42 total, 7-instance replay intact); fixtures now stamp release: from the file's own version, matching every artifact on shipped history.
  • .github/workflows/ci.yml — the status-evidence summary grep extended for the new line shape and pinning (0 skipped) + ≥ 1 archaeology check (same PR as the line-shape change, so the grep cannot go stale).
  • artifacts/release-v0.61/RQ-61-EVIDENCE.yaml — status → implemented with the dispositions written into the description, done-when pointing at the R7 replay class in the unit suite (R9 applied to itself), and a written verified-by. No other release artifact file touched (the R8 disposition required none).

Gates

status_evidence_check 66 artifacts / 0 failures / 5 archaeology checks (0 skipped) · test_status_evidence_check 42 OK · claim_check 52/52 · test_claim_check 50 OK · cargo fmt --all clean · cargo clippy --workspace --all-targets -- -D warnings exit 0.

Honest residual

  • R7 checks only the "arrived too early" direction; evidence landing after a release's own tag (scoped vX.Y, delivered vX.Y+1) remains the release-notes review's problem — written into the script's not-covered list.
  • shipped-in's and verified-by's truth are trusted as written; the gate forces them to be written where the release-query reader sees them, it cannot judge them.
  • R7 archaeology trusts git log -S occurrence-count semantics; a literal added, fully removed, and re-added later attributes to its first introduction. The escape hatch covers the pathological case.

Refs #1085, #1090, #1064, #242.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L

…e, the release field must match the file, code-existence is not a predicate

Three rules, all found by USING the v0.60 status-evidence gate rather than
reasoning about it. Red-first each: all three defective fixtures pass the
pre-change checker with 0 failures, then fail on exactly the rule under
test and no other.

R7 — EVIDENCE MUST BELONG TO THE RELEASE. For every contains:/file:
done-when that HOLDS, the oldest first-parent commit introducing the
signature (git log -S for contains:, --diff-filter=A for file:) must not
be an ancestor of the previous minor's HIGHEST tag (so evidence shipped in
a previous PATCH is caught too). The RQ-60-CANARY shape — gate merged
08:50, v0.59.0 tagged 15:19, v0.60 plan written 18:56 — is replayed in a
real git fixture; the shipped-in: escape hatch (per-case, version-shaped,
reason written beside it, #1090) is the green control exactly as it stands
on main. Reliability stated, not silent: no-git/shallow/missing-tag/
uncommitted-signature SKIPS LOUDLY (R7-SKIP warning + a skip count in the
machine-read summary), and the CI grep pins '(0 skipped)' with >= 1
archaeology check — a shallow CI checkout already reds the R4 delivery
floor before R7 is reached. Live: 4 checks performed, 0 skipped.

R8 — THE release: FIELD MUST EQUAL THE FILE'S VERSION. The checker
version-gates by PATH; rivet's readiness query reads the FIELD; nothing
asserted agreement. The harmful never-hit direction (release: v0.60
parked in a pre-v0.60 file, exempt from every >= v0.60 rule while rivet
counts it in v0.60's scope) is the red fixture. THE 6 MEASURED MISMATCHES
ARE KEPT under the one now-STATED allowance: a field naming a PATCH of
the file's previous minor (vX.(Y-1).Z, Z >= 1) is the parked-patch
practice (v0.56.1/v0.56.2 in release-v0.57.yaml) and green; missing,
unparseable, ahead-of-path, or behind-without-patch fields are red. No
historical release file needed edits.

R9 — A contains: INTO CRATE SOURCE IS WEAKER THAN THE GATE. Under a
CLAIMING status, a signature whose path is crates/** excluding /tests/
components (crate integration-test dirs are executed by the required Test
job and are gate-shaped, like scripts/, coq/ and workflows — measured,
every honest live signature already points at one of those) requires a
written verified-by saying why code-existence IS the outcome; the better
fix is re-pointing at the gate, which is what both #1090 corrections did.
The RQ-60-A64IMPORT specimen is the red fixture; both correction forms
are green controls. test_4's green control updated to the #1090 corrected
done-when — the old fixture carried the exact shape R9 now reds, and R9
caught it.

Gates: 42 unit tests OK (7-instance replay intact); live tree 66
artifacts / 0 failures / 4 archaeology checks (0 skipped); 5 mutations
each kill >= 1 test with needle counts asserted before mutating
(R7 ancestor check, R8 loop, R8 allowance, R9 trigger, R7 skip
accounting) and byte-identical restore; claim_check 52/52;
test_claim_check 50 OK; ci.yml summary grep extended in the same PR.
Artifact flipped implemented with done-when pointing at the R7 replay
class in the unit suite (R9 applied to itself) and a written verified-by.

Refs #1085, #1090, #1064, #242.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
@avrabe
avrabe force-pushed the feat/evidence-rules-1085 branch from 06db00a to 4442e77 Compare August 27, 2026 21:05
@avrabe

avrabe commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto 59dbe55b and re-running CI — the previous 9/9 was earned against a stale base.

#1098 merged .................. 2026-08-27T20:55:05Z
#1099 Claim Check completed ... 2026-08-27T20:50:41Z   <- BEFORE the merge
#1099 baseRefOid .............. 88abab32   (origin/main was already 59dbe55b)

Claim Check is the one required job that runs status_evidence_check.py over the artifact set — and it passed against a tree that did not contain #1098's five new v0.61 artifacts, while this PR adds three new rules (R7/R8/R9) that run against exactly those five. Rivet Validation, Format and Version Pin Sweep were also pre-merge.

Nothing here is suspected broken. I verified the real post-merge tree locally before pushing:

status-evidence: 71 artifacts across 16 release files, 38 delivery commits matched,
                 19 done-when predicates evaluated,
                 5 release-scope archaeology checks (0 skipped), 0 failures
claim_check 52/52 · test_status_evidence_check 42 OK

But a green check is a statement about the tree that was tested, and that tree no longer exists. The commit content is unchanged — only the base moved — so this is a re-run, not a revision.

Worth recording because it is the concrete case for the name-conditioned rule: the earlier framing that the queued jobs were "untouched by this change" was true of the diff and false of the tree.

@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@avrabe
avrabe merged commit b2abf95 into main Aug 27, 2026
58 checks passed
@avrabe
avrabe deleted the feat/evidence-rules-1085 branch August 27, 2026 21:58
avrabe added a commit that referenced this pull request Aug 28, 2026
… idle rust-cpu pool (#1101)

The smallest possible first step, and deliberately a NON-REQUIRED job.

MEASURED, not argued. Four PRs opened within ~30 minutes each reached
EXACTLY 3/9 required contexts within minutes and then stalled for hours.
The three that completed were the same three every time — Format, Version
Pin Sweep, Rivet Validation — and those are precisely the three that run
SELF-HOSTED. The six that stalled are the six on `ubuntu-latest`. Same
commits, same moment, same queue; the split is not explained by job cost
(Format is not cheaper than Clippy) but entirely by which pool the job
asks for.

    runs-on census:  56 ubuntu-latest · 4 rust-cpu · 2 light · 1 macos
    org runners:     12 online, 4 busy (OTHER repos), 8 IDLE (7 rust-cpu, 1 light)

WHY THIS JOB. `synth-provenance-v1 reconciliation gate` needs only
`dtolnay/rust-toolchain` — no apt, no Nix, no `cargo install`, no network
package fetch. That is the same shape as `Rivet Validation`, which already
runs on `rust-cpu` successfully, so the pool is proven for cargo-shaped
work and this move tests scheduling rather than toolchain.

WHY NOT A REQUIRED JOB FIRST, even though the required six are what
actually blocks. A REQUIRED context naming a label no online runner
satisfies does not fail — it NEVER RUNS, and a non-running required check
deadlocks every merge in the repo. A non-required job that fails is a red
someone can read and revert; a required job that never starts is a repo
that cannot merge its own revert. One job per PR, each with its own full
green cycle, in the risk order posted to #1062:

    Claim Check ... checkout + python ONLY .... next, after this cycle
    Clippy ........ rust-toolchain + cache .... then
    Kani .......... cargo install ............. then
    Test / Z3 ..... sudo apt-get .............. only after apt is confirmed
    Bazel ......... nix + setup-bazel ......... last or never — an
                                                environmental failure there
                                                looks exactly like a real
                                                proof failure

THE COST BEING PAID, measured this wave: queue latency converts directly
into wasted CI cycles. Two PRs each accumulated green against a base that
moved underneath them (#1096 at 8/9, #1099 at 9/9) because GitHub re-runs
checks when the BRANCH moves, never when the BASE does. Both had to be
rebased and re-run from scratch; #1096 twice. The longer a PR waits, the
likelier its green describes a tree that no longer exists.

Change is one `runs-on` line: ci.yml ubuntu-latest 50 -> 49, all workflows
56 -> 55. Nothing else moved.

Refs #1062, refs #242.


Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant