feat(dogfood): apr-dogfood v3.0 — the gate suite gets a denominator (PMAT-742) by noahgift · Pull Request #2587 · paiml/aprender

noahgift · 2026-08-22T14:45:28Z

The finding that drove this

The v2.0 dogfood suite has 19 gates and they work — where they look. A surface audit of 830 features across 28 binaries measured where they look:

	Features	Covered by a gate	Coverage
Total	830	142	17.1%
`apr`	367	142	38.7%
The other 27 binaries	463	0	0.0%

Coverage by quality band is the real signal:

Band	n	Covered
1–2 (broken)	80	58	72.5%	where gates look, they find defects
3–4	49	27	55.1%
5–6	673	29	4.3%
7–8	8	8	100%
9–10	20	20	100%

644 of the 672 features scored 6 are uncovered — scored 6 because no ledger finding exists and no gate covers them. That is not "fine", it is unlooked-at. The 1–2 band is the control: gates find defects at 72.5% when pointed at something.

Every v2.0 gate answers "did this check pass?"; none answers "what fraction of the shipped surface did we check at all?" So the organizing gate of v3.0 is coverage itself, and a coverage regression alone is NO-GO.

Gate bodies are spliced VERBATIM, not re-derived

A re-derived gate body silently loses the specific mutation each one was hardened against — that is why they exist in that exact wording. All 12 slices are asserted as contiguous-block equalities against origin/main:

Gate 13 → G0.3 · Gates 2–3 → Tier 1 · Gates 4–7 → Tier 0 · Gates 8–12, 14–16, 18 → Tier 2 · Gate 17 → T3.1 · plus the exit-code-capture note and Cleanup. Zero v2.0 body lines lost.

Stale references found and FIXED (not passed through)

Claim	Attached	At HEAD	Verdict
`commands_enum.rs:110`	`:110`	`:154` (`:110` is `profile: bool`)	STALE → fixed
`qwen-story-daily.yml:63`, "via `-p apr-cli`"	`:63`	`:62`, `--path crates/apr-cli --features cuda`	STALE → fixed
#2384 "a P0 that is still open"	open	CLOSED COMPLETED 2026-08-11 (#2424)	STALE → fixed
#2376 "P0, open"	open	CLOSED COMPLETED 2026-08-13	STALE → fixed
"24 open P0s"	24	24 is the P0 total; 16 are open	STALE → fixed
"644 features scored 6"	644	672 scored 6; 644 of those uncovered	IMPRECISE → fixed
`APR-BENCH-RFC-001` (G5.2)	cited	zero hits tree-wide	DANGLING → marked to-be-authored; G5.2 SKIPs naming it
`[package.metadata.transports]`	assumed	absent from every Cargo.toml	G4.1 is RED at HEAD — recorded inline

Both ledger rows for #2384/#2376 still show no Fixed by while GitHub has them closed — the ledger and GitHub disagree, noted in the file.

Verified correct and left alone: serve_commands.rs:73 (0 BACKEND_VALUES), extended_commands.rs:85, serve/types.rs:57, dispatch_analysis.rs:1383-1419 (exactly 10 arms), crates/apr-cli/src/main.rs:10-12, dispatch.rs:180-182, apr-cli/Cargo.toml:89. All CSV and ledger tallies recomputed and confirmed.

Thresholds are MEASURED, not chosen

G2.3's table shipped deliberately blank. Filled from measurement at 4bbfeb07f: 142/830 covered, 44 broken-and-ungated, 427 UNKNOWN hardware, 204 low-confidence-and-uncovered — committed as the ratchet baseline. A threshold picked before measurement either never fires or fires constantly.

Contract in the same PR

contracts/apr-dogfood-coverage-v1.yaml — pv validate clean. Its four executable falsifiers were run: GREEN on the clean tree, RED under their registered mutation, GREEN on a no-op copy of the same data. A gate that fires on both measures nothing. F-DOGCOV-002 needs a --emit-features mode on dogfood_surfaces.sh and honestly reports SKIP naming the missing flag rather than PASS.

Verification

12/12 splices verbatim; 0 v2.0 body lines lost
pv validate clean; cargo test -p aprender-contracts --lib 1446 passed
cargo test -p aprender-core --test readme_contract 15 passed (contract count 1778 → 1779)
bashrs ratchet PASS with dogfood.sh included: 91/91 scripts scanned, 127 error lines vs baseline 876
Corruption check: 0 U+FFFD introduced, 0 NUL, valid UTF-8, line counts exact, three artifacts byte-identical to handoff. The single U+FFFD at :1096 is pre-existing in origin/main:854 — an intentional mojibake detector inside Gate 18's coherent()
invariance.py tripped the pre-commit complexity gate (one 85-line main(), CC 30). Refactored by pure extraction to 11 functions, max CC 6; all four guards re-run against a fake two-verb binary and still fire. The gate was right; it was not bypassed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

…PMAT-742) The v2.0 suite has 19 gates and they work — where they look. A surface audit of 830 features across 28 binaries measured where they look: 142 covered, 17.1%. All 142 are in `apr`; 27 of 28 binaries have zero coverage, including aprender-orchestrate (184 features), aprender-test-cli (39) and pv (38). The band split is the finding. Where gates look they find defects at 72.5% (58/80 in the quality 1-2 band). Where they don't, 644 of the 672 features scored 6 are uncovered — scored 6 because no ledger finding exists AND no gate covers them. That is not "fine", it is unlooked-at, and it is the prediction of where the next 201-finding ledger comes from. Every v2.0 gate answers "did this check pass?" and none answers "what fraction of the shipped surface did we check at all?" A suite with no denominator reports a clean sweep over whatever subset it happens to cover — the vacuity failure dogfood_surfaces.sh already guards per-enumeration, applied one level up. So the organizing gate of v3.0 is coverage itself, and a coverage regression alone is NO-GO. Gate bodies are SPLICED VERBATIM from v2.0, not re-derived — a re-derived body loses the specific mutation each was hardened against. All 12 slices are asserted as contiguous-block equalities against origin/main: Gate 13 -> G0.3, Gates 2-3 -> Tier 1, Gates 4-7 -> Tier 0, Gates 8-12/14-16/18 -> Tier 2, Gate 17 -> T3.1, plus the exit-code note and Cleanup. Stale references found and FIXED at HEAD rather than passed through: - commands_enum.rs:110 -> :154 (110 is `profile: bool`; the value_parser is :154) - qwen-story-daily.yml:63 -> :62, and the invocation is `--path crates/apr-cli --features cuda`, not `-p apr-cli` - #2384 and #2376 were cited as open P0s; both CLOSED COMPLETED (08-11, 08-13). Their ledger rows still show no `Fixed by` — ledger and GitHub disagree. - "24 open P0s" -> 24 is the P0 total, 16 are open - "644 features scored 6" -> 672 scored 6, 644 of those uncovered - APR-BENCH-RFC-001 (G5.2) does not exist anywhere in the tree; marked to-be-authored, gate reports SKIP naming it rather than PASS - [package.metadata.transports] is absent from every Cargo.toml, so G4.1 is RED at HEAD — recorded inline as a finding, not left as a hypothetical G2.3 thresholds are MEASURED, not chosen: 142/830 covered, 44 broken-and-ungated, 427 UNKNOWN hardware, 204 low-confidence-and-uncovered, committed as the ratchet baseline at 4bbfeb0. A threshold picked before measurement either never fires or fires constantly. contracts/apr-dogfood-coverage-v1.yaml ships in the same PR. Its four executable falsifiers were run: GREEN on the clean tree, RED under their registered mutation, GREEN on a no-op copy of the same data. A gate that fires on both measures nothing. F-DOGCOV-002 needs a --emit-features mode on dogfood_surfaces.sh and honestly reports SKIP naming the missing flag. invariance.py main() was one 85-line function at cyclomatic 30 and tripped the pre-commit complexity gate. Refactored by pure extraction into 11 functions, max cyclomatic 6; the four guards (skip-under-two-transports, verb-not-in-list, empty-list vacuity, missing-declaration) were re-run against a fake two-verb binary and all still fire. The gate was right; it was not bypassed. The #2332 frontmatter `name:` is preserved — without it this file takes its name from the directory, collides with a user-scope ~/.claude/skills/dogfood/, never appears in the session listing, and edits look effective while changing nothing that runs. That already happened once, to #2357. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

…measured baselines (PMAT-742) Phase 2 of the apr-dogfood v3.0 coverage gate. - scripts/dogfood_surfaces.sh gains --emit-features: a bounded (depth 4) deterministic --help walk, the mechanism F-DOGCOV-002 needed to exist before it could execute rather than SKIP. - scripts/dogfood_reconcile.py: both-directions reconciliation. Both directions are genuinely 0 -- the 52 raw "extra" are internal tree nodes whose leaves the ledger carries, and the 28 raw "missing" are 26 feature-gated (proven by rebuilding with the features, not by reading the #[cfg]) plus 2 artifacts of the emitter's own help filter. - scripts/dogfood_baseline.py: re-derives every committed baseline from the ledger, so no number in the contract cites a command that does not exist. - scripts/dogfood_the44.py + docs/audits/dogfood-the-44.yaml: the 44 quality<=4 ungated features become 44 triage slots, not prose. - contracts/: baselines under metadata.baselines (142/830 = 0.1711, never rounded), F-DOGCOV-006/007/008 added, F-DOGCOV-002 now executes. - surface_audit.csv: 3 rows carried the stale commands_enum.rs:110; the SKILL.md fix had not been applied to the ledger. Patched to :154. The pre-commit complexity gate rejected dogfood_baseline.py (main CC 16 / cognitive 64, compute 16/43) and dogfood_reconcile.py (classify cognitive 36). Refactored by pure extraction rather than bypassed -- max CC is now 7 and 8. Proven behaviour-preserving by a differential test of the old and new classify() over all 830 rows: in_scope sets identical (583), out-of-scope histogram identical (146/70/15/11/5). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

…PR cannot reach it (PMAT-742) Phase 3. The apr-dogfood coverage ratchet becomes a required check. scripts/check_dogfood_coverage.sh fails the build when the audited surface gets worse: G2.1 freshness this branch moves cited evidence and leaves the ledger G2.2 reconciliation a ledger row vanishes, taking its defect with it G2.3 floors coverage falls, or a dark-row count rises G2.4 waivers a quality<=4 feature has neither a gate nor a waiver THE FLOOR IS NOT IN THIS REPOSITORY. There is no `142` and no `830` in the gate. Every floor is derived at run time from git show origin/main:docs/audits/surface_audit.csv The multi-platform dogfood gate kept its floor and its universe as literals in one file, so a single commit editing both defeated it while it reported green. A floor a PR can rewrite in the commit that breaks it is not a floor. M4 of the guard's own mutation table commits exactly that attack and is RED. Three registered mutations, each proven to have ENGAGED by an md5 delta before its verdict was read, each against the real 830-row ledger, each followed by a restore asserted GREEN: G2.1 append to commands_enum.rs, ledger untouched -> RED naming that file G2.2 delete `apr canary check` -> RED on G2.2, G2.3, G2.4 G2.3 flip `apr bench` yes -> no -> RED (141 < 142, apr 141 < 142) Two defects the discipline caught, both of which would have shipped a gate that proved nothing: * The first self-test reported all three mutations RED -- for the wrong reason. `git init` is not hermetic here: init.templatedir installs the pmat hooks into every scratch repo, so each fixture commit failed, the fixture had no HEAD, and every run died on "ledger absent from main". Fixture commits are now checked and abort the self-test on failure. * G2.1 first compared against a `measured_commit:` field the same PR could edit; forward-dating it to HEAD made the check vacuous. The obvious patch (a "you may not claim a date newer than the ledger" rule) then rejected a legitimate re-measure whose output was identical. Replaced with a quantity derived entirely from git, which has neither failure mode. Also fixes three violations of check_apr_bin_pinned.sh that Phase 1 introduced and which would have failed `ci / gate`: scripts/dogfood.sh resolved `pv` through PATH in the script that CERTIFIES A RELEASE -- the exact failure scripts/pv_bin.sh exists to record, where a PATH pv 0.49.0 and the in-tree 0.63.0 disagreed on the binding gate. Where pv_bin.sh is absent (this protocol runs fleet-wide), pv is left unresolved and the contract gates REPORT it rather than falling back to an unknown binary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

…stale ledger (PMAT-742) `guard-runner-labels` went red on the pull request that INTRODUCES this gate, at its first step, skipping the other 30 steps behind it: G2.1 freshness FAIL measured_commit cbb1ccd is not a commit in this repo G2.2 reconciliation PASS no row lost (830 comparand rows all present) G2.3 floors PASS 142/830 covered (17.1%), 28 per-binary floors held G2.4 waivers PASS 44 broken-and-ungated, 44 triaged, 0 new Coverage did not regress. Every floor passed, with the same numbers CI and a developer box both measure. What failed was a provenance probe that cannot mean what it says in the environment it runs in. `git cat-file -e <sha>` returns the same 128 for "you invented this SHA" and for "you were cloned with --depth=1 and this commit is two commits back". This job is checked out at fetch-depth 1 -- resolve_base_ref() says so in as many words, ten lines up, and the reasoning was never carried across to this check. cbb1ccd is the branch's own first commit; the depth-1 checkout has only dd2f69c. The gate was reporting on the checkout, not on the ledger. Reproduced exactly, before the fix, in a depth-1 clone of the PR head: G2.1 freshness FAIL measured_commit cbb1ccd is not a commit in this repo DOGFOOD COVERAGE GATE: FAIL (rc=1, shallow) DOGFOOD COVERAGE GATE: PASS (rc=0, same commit, full clone) Two fixes, because there were two defects: 1. Split the probe into the half that is decidable everywhere and the half that is not. SHAPE -- present, and 7-40 lowercase hex -- is enforced always, so PLACEHOLDER, a typo, a branch name and a deleted line are red in any repository. EXISTENCE is enforced only where the repository can answer it, and the PASS line now PRINTS which of the two it applied. A check that silently narrows its scope is theater; one that names its scope is a measurement. The honest limit is written into the file. 2. `measured_commit` named a commit on the feature branch. A squash merge orphans it, so once this landed the provenance line would have pointed at nothing a fresh clone of main could resolve -- the same red, permanently, for everyone. It now names 4bbfeb0, the merge base on main, which is the commit the surface was actually measured against and the one the contract's own `references:` block already cited. The reason this shipped is that the self-test had no mutation for the provenance branch: three registered mutations, none of them touching measured_commit. M5 adds four cases, and the last two are the discriminating pair -- the SAME well-formed-but-absent SHA must be RED in a full clone and GREEN in a depth-1 clone, or the check is measuring the checkout again. The shallow fixture asserts `is-shallow-repository=true` before trusting its verdict, and a fixture that fails to build fails the self-test rather than quietly not running. bash scripts/check_dogfood_coverage.sh --self-test rc=0 bash scripts/check_dogfood_coverage.sh rc=0 depth-1 clone of this commit, same gate rc=0 bash scripts/check_shell_lint_ratchet.sh rc=0 pv validate contracts/apr-dogfood-coverage-v1.yaml rc=0 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

…ter (PMAT-742) The previous commit gated the existence half of the provenance check on `git rev-parse --is-shallow-repository`, which throws away a verdict the repository was perfectly able to give. Found by running the fixed gate on this dev box and reading its own new output: measured_commit 4bbfeb0: shape ok; existence UNCHECKED (shallow clone) `/home/noah/src/aprender/.git/shallow` carries four graft points about 740 commits back -- someone's depth-limited fetch, months of history ago. The repo resolves 4bbfeb0 without difficulty, and `--is-shallow-repository` still says `true`, because that flag reports whether a graft EXISTS, never whether the object you care about is beyond it. Asking it first downgraded every run on every grafted clone -- which is most working checkouts -- to UNCHECKED. PRESENCE is conclusive in any repository. Only ABSENCE needs the caveat. So the order is now: `cat-file -e` first, and only its failure consults the depth. same commit, this box, shallow-first: shape ok; existence UNCHECKED same commit, this box, presence-first: shape ok, object present The self-test's discriminating pair is unchanged and still passes both ways: the same well-formed-but-absent SHA is RED in a full clone and GREEN in a depth-1 clone, so the reordering did not soften what it was added to catch. bash scripts/check_dogfood_coverage.sh --self-test rc=0 bash scripts/check_dogfood_coverage.sh rc=0 (object present) depth-1 clone of this commit + depth-1 origin/main, gate rc=0 (absence inconclusive) depth-1 clone of this commit + depth-1 origin/main, selftest rc=0 bash scripts/check_shell_lint_ratchet.sh rc=0 (8 errors, unchanged) pv validate contracts/apr-dogfood-coverage-v1.yaml rc=0 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

noahgift · 2026-08-22T18:09:40Z

Superseded by #2613, the 0.64.0 integration batch.

This PR's commits are merged into batch/release-0-64-0 verbatim (--no-ff, never rebased),
and #2613's body carries the full provenance table — PR number, branch, merged head SHA, and
the issues each closes — so the detail survives the squash.

Why batched rather than landed individually: one workspace-test run is ~58 minutes on a
shared box. Thirteen PRs cost thirteen runs whether they go serially or in parallel; one
integration branch costs one. The same approach landed 24 branches previously.

Batching also found four defects that were invisible to every individual PR — most
notably the README contract count: #2548, #2549 and #2587 each add exactly one contract, each
is individually correct at 1779, and three +1s collide on one literal (correct value 1781).
That is the exact class that killed the previous batch.

Closing now, deliberately: an open PR that merges first moves #2613's base and forces
another full run. This is reversible and the branch is untouched — reopen if #2613 is
abandoned.

noahgift enabled auto-merge August 22, 2026 14:45

noahgift and others added 2 commits August 22, 2026 17:20

noahgift mentioned this pull request Aug 22, 2026

feat(dogfood): the per-cluster floor, the sibling-sweep rule, and the cluster_id key ban (PMAT-742) #2600

Closed

noahgift and others added 2 commits August 22, 2026 19:18

noahgift closed this Aug 22, 2026

auto-merge was automatically disabled August 22, 2026 18:09
Pull request was closed

Navigation Menu

Uh oh!

feat(dogfood): apr-dogfood v3.0 — the gate suite gets a denominator (PMAT-742) - #2587

feat(dogfood): apr-dogfood v3.0 — the gate suite gets a denominator (PMAT-742)#2587
noahgift wants to merge 5 commits into
mainfrom
feat/apr-dogfood-v3-coverage-gate

noahgift commented Aug 22, 2026

Uh oh!

noahgift commented Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

noahgift commented Aug 22, 2026

The finding that drove this

Gate bodies are spliced VERBATIM, not re-derived

Stale references found and FIXED (not passed through)

Thresholds are MEASURED, not chosen

Contract in the same PR

Verification

Uh oh!

noahgift commented Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant