Release machinery: commit traceability (FEAT-091, #161) + required-check drift gate (FEAT-093, #130) - #167
Conversation
rivet ships commit-to-artifact traceability and it shipped here unconfigured, so `Broken refs` read 0 for want of any reference to check. That counter is what would have caught PR #159 merging with a commit naming FEAT-086 before the artifact existed (#160) -- found by grepping, not by a gate. TWO MEASUREMENTS CORRECT #161's PREMISE, and both changed the design. (1) #161 says "zero of 125 commits carry an artifact trailer". Configure a `Verifies` mapping and rivet instead reports linked:4, broken_refs:4 AND malformed_refs:4 -- the SAME four commits inflating three counters. None carry a trailer. All four are the prose line Verified: bazel build //:scry green + wasm-tools validate ok. which rivet's FUZZY trailer-key matcher reads as a malformed `Verifies:`. Five commits in this history carry such a line. A/B measured: with `Verifies` mapped the counters read 4/4/4; unmapped they read 0/0/0 and orphans rises 81 -> 85. So `linked: 4` was never real, and the honest baseline is linked:0 orphans:85 broken_refs:0. The mapping is kept to `Refs: traces-to` ALONE -- a finding, not laziness: a gate that fails on a normal English sentence gets switched off, and "Verified:" is an entirely natural thing to write. Reported upstream. (2) `rivet commits --strict` cannot BE the gate. It also promotes the repo-wide unimplemented-artifacts warning -- `Artifact coverage: 1/257 (0.4%)` -- which no pull request can satisfy. Measured: a range with one correctly-linked commit, zero orphans and zero broken refs STILL exits 1 under --strict. A gate no PR can pass is not a gate. So tools/check-commit-trailers.py reads the JSON summary and fails on only the three counters a PR owns -- orphans, broken_refs, malformed_refs -- ignoring unimplemented and artifact_coverage. Forward-only by construction: scoped to origin/main..HEAD, so the 85 pre-existing orphans never enter it and no history rewrite is needed. Its own CI job, so a traceability failure is legible in the checks list rather than buried behind artifact validation. VERIFIED END-TO-END on the real repo, three directions: crates/ commit, no trailer -> orphans=1, exit 1 same commit, `Refs: REQ-005` -> linked=1, exit 0 `Refs: FEAT-99999` (the #160 bug) -> broken_refs=1, exit 1 Plus: parse the JSON even when rivet exits non-zero (bailing on the exit code alone degraded an informative failure to "rivet exited 1"), and fail CLOSED when rivet is absent. Self-test of 6 cases runs BEFORE the real check. tests=0 fmt=0 rivet=0 claim-check=0 gate-coverage=0 trailers-self-test=0. Refs: FEAT-091 Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
📐 rivet artifact deltaPR: #167 Base SHA: Validationhead — `rivet validate` resultbase — `rivet validate` result (for comparison)Artifact stats
full stats — headDiff (base → head)AADL model — headPosted by the |
… exist ci.yml said "Required-status-check names that temper's branch protection expects: Format, Clippy, Test". The ruleset required NOTHING -- so the file documented a gate the repo did not have, and every "merged green" rested on whoever merged remembering to check by hand. Now accurate: all ten CI jobs are required as of 2026-08-27, verified by every open PR flipping to mergeStateStatus=BLOCKED. Also records the constraint that decides which checks may be required at all: `Rivet artifact delta` lives in the PATH-FILTERED rivet-delta.yml, so it never reports on a code-only PR. Requiring it would block such PRs forever. Same trap applies to any job added to this file -- require it only after its name exists on every PR, i.e. after it has merged to main. Refs: FEAT-091 Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
scry#130 was enforced on 2026-08-27 -- all ten ci.yml jobs are now required. Verified in BOTH directions, which is what separates a gate from a setting: every open PR flipped to mergeStateStatus=BLOCKED on apply, and #164 merged CLEAN once its ten checks passed. A setting that was wrong for months can be wrong again, and it would be invisible -- CI still runs, PRs still go green, nothing announces that a job stopped being enforced. Three ways it rots: 1. a job is ADDED and never required -- scry#130 one job at a time, no symptom 2. a job is RENAMED -- the stale context never reports, every PR deadlocks 3. the ruleset is RESET -- it is named `temper-`, so something may regenerate it, and required_status_checks would simply vanish THE DEADLOCK RULE is why this is not set-equality. A check may be required only if it reports on EVERY PR. rivet-delta.yml is PATH-FILTERED, so `Rivet artifact delta` never runs on a code-only PR and requiring it would block such PRs forever. The gate fails if a path-filtered or `if:`-conditional job is ever required, and carries that exclusion WITH its reason so it is auditable rather than folklore. NEW JOBS ARE NOT FAILURES. A job added in the current PR does not exist on main and cannot be required yet. The gate reads ci.yml from BOTH origin/main and the PR and reports such jobs as PENDING -- without which it would fail on the very PR that introduces it. This PR introduces two and still exits 0. MUTATION-CHECKED AGAINST THE LIVE RULESET, not a fixture: `Format` was removed from the real required set (asserted 10 -> 9 before applying), the gate failed naming exactly that job, and restoring returned 10 contexts and exit 0. Also corrects ci.yml, which claimed "temper's branch protection expects Format, Clippy, Test" while the ruleset required NOTHING -- the repo was documenting a gate it did not have. rivet=0 claim-check=0 fmt=0 drift-self-test=0 trailer-self-test=0. Refs: FEAT-093 Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
Scope grew — this PR now carries a second feature, and the title reflects it. FEAT-093: the required-check set can't silently rot back to #130#130 was enforced while this PR was open: the ruleset But a setting that was wrong for months can be wrong again, and it would be invisible: CI still runs, PRs still go green, nothing announces that a job stopped being enforced.
The deadlock rule is why this isn't set-equality. A check may be required only if it reports on every PR. New jobs are not failures. A job added in the current PR doesn't exist on main and can't be required yet. The gate reads Mutation-checked against the live ruleset, not a fixture
Restoring returned it to 10 contexts and exit 0. Also in this PR
|
I broke CI on this branch. The required-checks drift gate ran the LIVE-ruleset
check in CI under `permissions: administration: read`. That is not a grantable
Actions scope, and a workflow requesting it is REJECTED WHOLESALE -- the run
completed as `failure` with ZERO jobs and no log, which reads like a queue
outage rather than a syntax error. It was caught only by noticing the PR showed
1 registered check instead of 12.
A workflow's GITHUB_TOKEN cannot read rulesets at all, so the original design
could never have worked in CI. Two modes now:
--against-file compares jobs to .github/required-checks.txt. No API, so CI
can run it. Catches the rot WE cause in our own PRs -- a job
added or renamed without updating the file.
(default) compares jobs to the LIVE ruleset AND cross-checks the file
against it, because CI gates on the file and a drifted file
means CI is gating on a fiction. Needs admin credentials.
MUTATION-CHECKED, and the two modes fail DIFFERENTLY, which is the point:
deleting `Clippy` from the file gives
file mode: "job 'Clippy' exists on main and runs on every PR, but is NOT required"
live mode: "'Clippy' is required LIVE but missing from .github/required-checks.txt"
Both exit 1; both restore to 0.
HONEST LIMIT, now in the artifact residual: a ruleset RESET is still not caught
by CI and cannot be, since no workflow can see the live setting. It is caught
the next time the live mode is run by hand. Closing that needs a PAT secret,
which is a repo-owner decision.
rivet=0 claim-check=0 gate-coverage=0 drift-self-test=0 drift-file=0
trailer-self-test=0 fmt=0.
Refs: FEAT-093
Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
3ab2976 to
aae8bdf
Compare
ci.yml recorded the rule: require a job only once its name exists on every PR, i.e. after it merges to main. Correct, and incomplete -- a PR opened BEFORE the job existed still does not have it, so it can never report and is blocked forever. Observed rather than theorised. When #167 added two jobs and both were required, PRs #168 and #169 each showed 11 checks with 0 of the 2 new ones. Both would have deadlocked on a check that could never run. Both were rebased and now register 13. The complete procedure is now in ci.yml: 1. merge the PR that adds the job 2. add the context to the ruleset AND to required-checks.txt, ruleset FIRST (CI gates on the file, so a file ahead of the ruleset means CI is gating on a fiction -- the live-mode cross-check says exactly that) 3. REBASE EVERY OPEN PR drift-gate=0 gate-coverage=0 claim-check=0 rivet=0. Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Sync required-checks.txt to the ruleset: 10 -> 12 (scry#130) #167 landed FEAT-091 and FEAT-093, so `Commit traceability (rivet)` and `Required checks track CI jobs (scry#130)` now exist on main and run on every PR. The drift gate immediately moved them from PENDING to FAIL -- which is the transition it was built to make: a requirable job that is not required is scry#130 recurring one job at a time. Both were added to ruleset 16891064 (10 -> 12 contexts), and this syncs the checked-in file so the two agree. Order matters: the RULESET was updated first and the file second, because CI gates on the file -- a file listing checks the ruleset does not require would mean CI gating on a fiction, which the live-mode cross-check reports in exactly those words. Briefly red on main is the honest state during that window. Verified: file mode PASS, live mode PASS with `file agrees: True`. rivet=0 claim-check=0 drift-self-test=0. Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * "Require it after merge" was necessary but not sufficient ci.yml recorded the rule: require a job only once its name exists on every PR, i.e. after it merges to main. Correct, and incomplete -- a PR opened BEFORE the job existed still does not have it, so it can never report and is blocked forever. Observed rather than theorised. When #167 added two jobs and both were required, PRs #168 and #169 each showed 11 checks with 0 of the 2 new ones. Both would have deadlocked on a check that could never run. Both were rebased and now register 13. The complete procedure is now in ci.yml: 1. merge the PR that adds the job 2. add the context to the ruleset AND to required-checks.txt, ruleset FIRST (CI gates on the file, so a file ahead of the ruleset means CI is gating on a fiction -- the live-mode cross-check says exactly that) 3. REBASE EVERY OPEN PR drift-gate=0 gate-coverage=0 claim-check=0 rivet=0. Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Both shipped and CI-verified; leaving them proposed understates the release the same way promoting them early would overstate it. FEAT-091 (#167, 13/13 green) -- commit traceability. Its gate's --self-test PASSES and the real check PASSES on main against HEAD~1..HEAD, and the job runs in CI as a REQUIRED check. FEAT-092 (#172, 13/13 green) -- one operator, one name. Its oracle re-run on main just now; the naming-only claim was measured, not argued (every advisory-code count identical on a real module, leaks 2,800 -> 71). NOT promoted, and the reason matters: FEAT-093 -- merged in #167 and green, but #175 is OPEN and adds ACs to it. Promoting now would produce an `accepted` artifact that immediately gains unmet criteria. It goes accepted after #175 lands, not before. FEAT-089 -- filed, not built; its AC#1 demands a test still red by design. FEAT-057 / FEAT-065 / REQ-021 -- unbuilt. FEAT-064 -- AC1 still falsified, so REQ-020 stays blocked. rivet=0 claim-check=0 drift-gate=0 fmt=0. Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…175) Operating the gate exposed a hole in the procedure it enforces. Landing a job while its required-checks.txt entry arrives in a SEPARATE PR leaves main AND every open PR red until that follow-up merges. #167 did exactly that: the moment it merged, main failed its own drift gate and #168/#169 went red on a file they could not fix. The fix is structural, not another comment. The gate now FAILS a PR that adds a requirable job which is not in required-checks.txt, so the job and its entry ship together and the window does not exist. VERIFIED ON THE REAL REPO, both directions: job added, no file entry -> exit 1, naming it and quoting the instruction same job WITH its entry -> exit 0 (pending the ruleset update only) restored -> exit 0 Plus two new self-test cases covering exactly those, 7 in total. The procedure in ci.yml is now three steps with no red window: 1. in the SAME PR: add the job AND its required-checks.txt entry 2. after merge: add the context to the ruleset (instant, via the API) 3. rebase every open PR Steps 2 and 3 were each learned by getting them wrong. A required context whose job does not exist deadlocks everything; a job whose context does not exist deadlocks nothing. The asymmetry is why the ordering matters. drift-self-test=0 drift-file=0 gate-coverage=0 trailer-self-test=0 claim-check=0 rivet=0 fmt=0. Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Both shipped and CI-verified; leaving them proposed understates the release the same way promoting them early would overstate it. FEAT-091 (#167, 13/13 green) -- commit traceability. Its gate's --self-test PASSES and the real check PASSES on main against HEAD~1..HEAD, and the job runs in CI as a REQUIRED check. FEAT-092 (#172, 13/13 green) -- one operator, one name. Its oracle re-run on main just now; the naming-only claim was measured, not argued (every advisory-code count identical on a real module, leaks 2,800 -> 71). NOT promoted, and the reason matters: FEAT-093 -- merged in #167 and green, but #175 is OPEN and adds ACs to it. Promoting now would produce an `accepted` artifact that immediately gains unmet criteria. It goes accepted after #175 lands, not before. FEAT-089 -- filed, not built; its AC#1 demands a test still red by design. FEAT-057 / FEAT-065 / REQ-021 -- unbuilt. FEAT-064 -- AC1 still falsified, so REQ-020 stays blocked. rivet=0 claim-check=0 drift-gate=0 fmt=0. Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…→4 (#176) * Promote FEAT-091 / FEAT-092 to accepted — v3.4.0 not-ready 7 -> 5 Both shipped and CI-verified; leaving them proposed understates the release the same way promoting them early would overstate it. FEAT-091 (#167, 13/13 green) -- commit traceability. Its gate's --self-test PASSES and the real check PASSES on main against HEAD~1..HEAD, and the job runs in CI as a REQUIRED check. FEAT-092 (#172, 13/13 green) -- one operator, one name. Its oracle re-run on main just now; the naming-only claim was measured, not argued (every advisory-code count identical on a real module, leaks 2,800 -> 71). NOT promoted, and the reason matters: FEAT-093 -- merged in #167 and green, but #175 is OPEN and adds ACs to it. Promoting now would produce an `accepted` artifact that immediately gains unmet criteria. It goes accepted after #175 lands, not before. FEAT-089 -- filed, not built; its AC#1 demands a test still red by design. FEAT-057 / FEAT-065 / REQ-021 -- unbuilt. FEAT-064 -- AC1 still falsified, so REQ-020 stays blocked. rivet=0 claim-check=0 drift-gate=0 fmt=0. Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * Also promote FEAT-071 — v3.3.0 not-ready 6 -> 5 #174 landed FEAT-071 with 13/13 green, so it belongs in this pass rather than a follow-up. Its oracle re-run on main just now: the scope manifest reaches the feed as data, and the anti-drift property (every not-proven item present in BOTH the page and the feed) holds. That property was mutation-checked when it landed -- emitting empty strings in the feed while the page kept them kills the test. Folded in here rather than opened separately because #176 had not started CI, so it costs nothing; had it been mid-run the trade would have gone the other way. rivet=0 claim-check=0 drift-gate=0 fmt=0. Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * Also promote FEAT-093 — its refinement landed, so its criteria are complete FEAT-093 was deliberately held back from the earlier promotion because #175 was OPEN and adding acceptance criteria to it; promoting then would have produced an `accepted` artifact that immediately acquired unmet criteria. #175 has now merged, so the criteria are complete AND met. Evidence re-run on main: the gate's 7-case --self-test passes, file mode passes, and live mode passes with `file agrees: True` -- the ruleset, the checked-in file and the CI jobs are all in agreement. Folded in here rather than opened separately because #176 still had not started CI. Same trade as FEAT-071 a moment ago, and the same reason. rivet=0 claim-check=0 drift-gate=0 fmt=0. Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc Co-authored-by: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Closes the measurement in #161. Two measurements correct that issue's premise, and both changed the design — worth reading before the diff.
(1)
linked: 4was never real — rivet's trailer matcher is fuzzy#161 reports "zero of 125 commits carry an artifact trailer." Configure a
Verifiesmapping and rivet instead reportslinked: 4,broken_refs: 4andmalformed_refs: 4— the same four commits inflating three counters at once.None of them carry a trailer. All four are this ordinary prose line:
rivet's fuzzy trailer-key matcher reads that as a malformed
Verifies:. Five commits in this history carry such a line.A/B measured:
VerifiesRefsonlySo the honest baseline is
linked: 0, orphans: 85, broken_refs: 0.The mapping is therefore kept to
Refs: traces-toalone. That's a finding, not laziness: a gate that fails on a normal English sentence gets switched off, andVerified:is an entirely natural thing to write in a commit body. Reported upstream; widen when the matcher stops guessing.(2)
rivet commits --strictcannot be the gate--strictalso promotes the repo-wide unimplemented artifacts warning —Artifact coverage: 1/257 (0.4%)— which no pull request can satisfy.Measured: a range with one correctly-linked commit, zero orphans and zero broken refs still exits 1 under
--strict. A gate no PR can pass is not a gate; it just gets disabled.So
tools/check-commit-trailers.pyreads the JSON summary and fails on only the three counters a PR actually owns —orphans,broken_refs,malformed_refs— and deliberately ignoresunimplementedandartifact_coverage.Verified end-to-end on the real repo
crates/commit, no trailerorphans=1Refs: REQ-005linked=1Refs: FEAT-99999(the #160 defect)broken_refs=1That middle row is the direction
--strictcould not produce, and is the whole reason for a custom gate.Two fixes found by measuring rather than assuming:
"rivet exited 1"Self-test of 6 cases runs before the real check, and asserts that
linked/exemptnever fail the gate and that a missing key reads 0.Forward-only by construction
Scoped to
origin/main..HEAD, so the 85 pre-existing orphans never enter the gate and no history rewrite is needed.traced-paths: [crates/]scopes it to the code side — which is what #161 measured as disconnected. It runs as its own CI job so a traceability failure is legible in the checks list rather than buried behind artifact validation.Residual, stated
rivet commitswithout a range still reports them — that number is the honest record of when the connection started.Satisfies/Verifiesare unavailable and everything routes throughtraces-to. A real loss of expressiveness, accepted to keep the gate trustworthy. (Fixes:also appears once as prose, so it is likewise unmapped.)tests=0 fmt=0 rivet=0 claim-check=0 gate-coverage=0 trailers-self-test=0gh pr checksby hand.🤖 Generated with Claude Code
https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc