feat(dogfood): the per-cluster floor, the sibling-sweep rule, and the cluster_id key ban (PMAT-742) - #2600
Closed
noahgift wants to merge 3 commits into
Closed
Conversation
…T-742)
93.0% of gate effort sits over 26.4% of the surface. Nobody chose that
allocation; it accreted. Three clusters -- apr-lint-diag (66 features, 55
gates), http-apr-serve (44/39) and apr-core-commands (109/38) -- hold 132 of
142 gates; the other eleven clusters share 10 gates over 611 features, and
NINE of the fourteen sit at zero. Nine clusters at zero is nine clusters with
no evidence at all. That is the gap, not the 688 uncovered rows.
Clustering is what makes that visible, so the ledger now carries cluster_id
and cluster_label and the coverage gate grows a THIRD floor beside overall and
per-binary.
ONE LEDGER, NOT TWO. The clustered CSV REPLACED docs/audits/surface_audit.csv
rather than landing beside it. Two ledgers over one surface is the drift hazard
this repo keeps re-finding; the copy goes stale in silence and every consumer
then has to be told which is authoritative. It was a superset in shape -- same
830 rows, same order, two extra columns -- so replacing cost nothing. Three
cells DID disagree: `apr run --backend {cpu,cuda,wgpu}` cited
commands_enum.rs:110 in the clustered snapshot and :154 in the landed ledger.
:154 is the `backend:` arg and :110 a chat-template arg, so the landed value
won. That disagreement, found on the one day both files existed, is the
argument against keeping both.
WHY THE BINARY IS THE WRONG UNIT. aprender-orchestrate's 184 features are
three unrelated subsystems -- 95 Banco HTTP routes, a 56-feature agent stack,
17 Pacha secrets commands. A per-binary floor of ">= 1 gate" lets one gate on
Pacha make all 184 look touched. The cluster is the unit whose members share a
module, a dispatch path and a failure mode, which is the property that makes a
gate on one member evidence about the rest.
The floor keeps the comparand it already had: every number is derived at run
time from `git show origin/main:docs/audits/surface_audit.csv`. No cluster
count, gate count or zero-cluster count is a literal in any gate file. While
main still carries the 8-column ledger the ratchet prints a SCHEMA UPGRADE
banner instead of passing silently; that branch is self-closing and a
half-migrated comparand is a hard failure, not an upgrade.
THE THREE TRAPS, ENFORCED RATHER THAN DOCUMENTED
T1 k-means ids PERMUTE on re-run, so an obligation keyed on one silently
re-points at a different cluster -- the stale-hardcoded-list class in new
clothes. scripts/check_no_cluster_id_keys.sh refuses any contract or gate
keying on cluster_id, with a 27-row must-match/must-not-match case table
that CI runs beside the scan. It found its FIRST real violation in its
own mutation harness: `sed 's|key: <label>|key: <id>|'` is itself a
keying line. Its second was a GitHub Actions step NAME, which sharpened
K2 to require the token to be the whole value rather than merely to
follow the field.
T2 cluster coverage != feature coverage. One gate in a 95-member cluster is
1%, not "covered". enforce_pairing() reads the report back before
printing and fails the gate if any line states a cluster fraction with no
feature fraction beside it -- and fails on an empty report too, because a
pairing rule applied to nothing proves nothing.
T3 clustering is a PRIOR, never evidence. quality_1_10 is never derived from
membership; severity still comes from the 0.63.0 ledger.
SIBLING SWEEP (Phase 3, process). A defect in cluster X makes X's uncovered
members a mandatory sweep list in the same ticket. The prior is measured: the
0.63.0 ledger collapsed 201 findings into 37 root causes, ~5.4 per cause.
Clustering supplies that prior mechanically instead of retrospectively.
REGISTERED MUTATIONS -- six now, each RED with a paired GREEN restore and a
no-op discrimination check, engagement proved by md5 delta BEFORE the verdict
is read:
M5 G2.5 move a cluster's ONLY gate to another cluster. Overall and every
per-binary count are UNCHANGED, so only the per-cluster floor can
explain the RED -- and the finding TEXT is asserted, not just the
exit code, or the mutation would be unattributable.
M6 T2 delete the feature fraction from the emitter -> RED "G2.5 T2 FAIL"
M7 T1 key a fixture contract on the permuting id -> RED
Contract apr-dogfood-coverage-v1 -> 1.1.0: per_cluster baselines, three new
equations, F-DOGCOV-013/014/015, four new proof obligations. `pv validate`
0 errors 0 warnings; `pv audit` no findings; `dogfood_baseline.py --check`
passes.
scripts/dogfood_cluster.py ships the k-selection evidence only (the sweep and
docs/audits/surface_audit_elbow.png). It does NOT write the two columns and
re-running it will not regenerate them -- that follows from T1: the labels are
human-owned. Its bare `surface_audit.csv` path was also fixed; a script that
can only run from inside docs/audits/ is the shipped-but-unreachable class.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR
Found by running M5 against the REAL 830-row ledger instead of only the
fixture. The per-cluster ratchet reuses `_ratchet`, which hardcoded the G2.3
label, so ungating `apr bench` and gating `apr sim emc-check` in one edit
produced:
G2.3 floors PASS 142/830 covered, 28 per-binary floors held
G2.5 per-cluster FAIL
G2.3 floors FAIL: gates in cluster `apr-lint-diag` is 54, must be >= 55
-- the summary line and the finding naming different gates. A misattributed
finding sends the reader to a gate that just passed.
`gate` is now a parameter with the old value as default, so the G2.3 callers
are untouched. The M5 marker in the self-test was tightened to assert the
prefix too, which is what would have caught this in the fixture.
This is the fourth rule of verification discipline paying: extending a helper's
SCOPE required re-mutating in the new scope, and the fixture's proof did not
transfer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR
… guards were narrower than their claim
Three defects in the per-cluster floor, each closed and each proved by an A/B
against the code it replaces rather than by argument.
1. MEMBERSHIP WAS UNRATCHETED, SO THE RELEASE ARM WAS SATISFIABLE BY RELABELLING
The floor ratcheted gate COUNT per label, label PRESENCE and the zero-gate
cluster count. None of those constrains who is IN a cluster. Move an
already-gated feature into a zero-gate cluster, write ONE new gate in the
cluster it left, and every count holds: the source keeps its gate count, the
target's rises, the zero-gate count falls, no label vanishes, overall and
per-binary both rise. Run against the pre-fix module that ledger prints
G2.5 per-cluster PASS clusters gated 3/3 (100.0%)
and exits 0. Nothing was proved about the target's 200 features; the zero
stopped EXISTING instead of being closed. Nine zeros are nine tickets only while
a zero cannot be made to stop existing. It is the same move as deleting a losing
benchmark row, which this repo has done exactly once (d7e0804, 395 deletions,
the only beat deletion in its history, removing the only two losing rows).
Clusters are DERIVED, so a re-cluster legitimately moves members and a
prohibition would forbid the one operation that keeps the ledger true. The
answer is therefore a DECLARATION, in two parts that are different rules:
membership a row present on both the comparand and the working ledger may not
change cluster_label without an entry in
docs/audits/cluster_reassignments.yaml giving from, to and a
reason. Checked in BOTH directions -- an entry whose `to`
disagrees with the ledger is a pre-authorisation, which is how a
declaration becomes a blanket permit.
earned a declaration buys legibility, never evidence. The zero-gate set
and the release arm count EARNED gates: a gate on a feature that
was already in the cluster, or on genuinely new surface. A gate
that walked in from elsewhere is evidence about where it came
from. Writing a gate is the only way off zero.
The same ledger now exits 1 naming "G2.5 membership FAIL" and reports the target
as [1 inherited, 0 earned]. Declared and armed at DOGFOOD_RELEASE=1 it is still
RED, naming "carry no EARNED gate"; the paired GREEN is the same fixture, same
arm, with a gate WRITTEN in the zero cluster instead of moved into it.
2. T1 WAS A BLACKLIST OF FOUR SYNTAXES, NOT A BAN ON THE TOKEN
classify_line() recognised a YAML mapping key, an identity field's value, a dict
subscript and a CLI/query key, and passed everything else. "Nothing may key on
this" is a universal claim and no finite list of syntaxes carries one. An A/B of
the extracted function over twelve ordinary keying constructs -- groupby,
sort_values, .get, attribute access, a dict-comprehension key, a SQL WHERE, ==,
setattr, tuple unpacking, a kwarg, a yaml round-trip and a step name -- returns
OK for all twelve on the old guard.
Inverted. A standalone token is a KEY unless the line says otherwise, and there
are three ways to say otherwise, each mechanical rather than a matter of trust:
the token is backticked (a backtick is a literal in yaml, a syntax error in
python 3 and command substitution in bash, so a backticked token cannot function
as a key in any language this guard scans), it follows a comment marker (read by
nothing), or the line names three or more ledger columns and is the schema
declaration. Plus the pragma that already existed, for the five genuine reads.
The same twelve constructs now return KEY.
Kept: the shared classify_line() between the case table and the scan, the
vacuity guard, and the documented scope choice excluding docs/specifications/.
The case table grew the twelve rows the blacklist walked past. The inversion
turned 18 existing prose lines red -- each one is now backticked, or carries a
pragma where it is a genuine read.
3. T2 WAS ENFORCED OVER ONE PHRASING AND ONE CHANNEL
enforce_pairing() was applied to one generated string: this gate's own report.
The failure it exists to prevent is a RECEIPT that states cluster coverage
without the feature fraction -- the gate then looks STRICTER than before while
measuring LESS, which is vacuity one level up.
It now runs on every channel a cluster ratio can leave through: the gate report,
the receipt (DOGFOOD_RECEIPT), scripts/dogfood_baseline.py (which holds its own
output to the rule before printing it) and the skill. And the rule is about the
NUMBER, not a phrase: a ratio whose denominator equals the cluster count is a
cluster-level claim whatever words surround it, and must carry a ratio whose
denominator equals the feature count. `N of M` counts, because prose writes it
that way. Both denominators are derived from the ledger; neither is a literal in
a gate file. The phrase forms are kept as additional triggers -- a union can only
make the rule stricter. A line that must state one number alone opts out by
saying so with a reason: `t2-pairing allow (<reason>)`.
Turning it on found four real violations in the skill and the contract, which is
the point.
4. THE BOOTSTRAP DESCRIPTION NAMED THE WRONG CODE PATH
The banner described read_ledger(..., allow_legacy=True) as the active branch.
It is not: resolve_base_ref takes BOOTSTRAP first, so the comparand is this
branch's own HEAD commit, which already carries the 10-column ledger -- the
legacy branch never executes and the per-cluster ratchet IS armed, from HEAD.
The window itself is fine and self-closing. A window whose description does not
match its code is not. The banner now names the path taken and why, and the
report's armed-line states the comparand source and the schema that was read.
REGISTERED MUTATIONS -- 10, each RED with a paired GREEN restore and a
byte-identical no-op discrimination check, engagement proved by md5 delta before
any verdict is read:
M8 relabel a gated feature into the zero-gate cluster -> RED (was GREEN)
M8b declare that move and arm the release -> RED, inherited
M9 state a cluster ratio in the RECEIPT with no feature % -> RED
M9b the same removal in the SKILL -> RED
M10 key on the id with a form the blacklist walked past -> RED (was GREEN)
plus M1/M2/M3/M4/M5/M6/M7 unchanged.
The comparand is still `git show ${BASE_REF}:docs/audits/surface_audit.csv`. No
cluster count, gate count or zero-count is a literal in any gate file.
Refs PMAT-742.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR
Contributor
Author
|
Superseded by #2613, the 0.64.0 integration batch. This PR's commits are merged into Why batched rather than landed individually: one Batching also found four defects that were invisible to every individual PR — most Closing now, deliberately: an open PR that merges first moves #2613's base and forces |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks on #2587 (
feat/apr-dogfood-v3-coverage-gate, basedd2f69c49). Retarget tomainonce that merges.The measurement, recomputed from the CSV
apr-lint-diaghttp-apr-serveapr-core-commands93.0% of gate effort over 26.4% of the surface. 142 gates / 830 features. Nine of fourteen clusters at zero. My reading agrees with the brief on every figure.
Nobody chose that allocation; it accreted. A 56th gate on
apr-lint-diagbuys less than the FIRST gate incontracts-pv(0/76) orhttp-orchestrate-banco(0/95).Replace, not keep-both
docs/audits/surface_audit.csvnow carries 10 columns. The clustered file replaced it rather than landing beside it: two ledgers over one surface is the drift hazard this repo keeps re-finding. It was a superset in shape — same 830 rows, same order — so replacing cost nothing.Three cells disagreed.
apr run --backend {cpu,cuda,wgpu}citedcommands_enum.rs:110in the clustered snapshot and:154in the landed ledger.:154is thebackend:arg;:110is a chat-template arg. The landed value is correct and won. That disagreement, surfacing on the one day both files existed, is the argument.Why the binary is the wrong unit
aprender-orchestrate's 184 features are three unrelated subsystems — 95 Banco routes, a 56-feature agent stack, 17 Pacha secrets commands. A per-binary "≥ 1 gate" floor lets one gate on Pacha make all 184 look touched.The floor keeps the comparand it already had
Every number is derived at run time from
git show origin/main:docs/audits/surface_audit.csv. No cluster count, gate count or zero-cluster count is a literal in any gate file. Whilemainstill carries the 8-column ledger the ratchet prints aSCHEMA UPGRADEbanner rather than passing silently; that branch is self-closing, and a half-migrated comparand (one cluster column, or a blank label) is a hard failure.The three traps, enforced rather than documented
scripts/check_no_cluster_id_keys.shrefuses any contract or gate keying oncluster_id, with a 27-row must-match/must-not-match case table CI runs beside the scan. It found its first real violation in its own mutation harness (sed 's|key: <label>|key: <id>|'is itself a keying line) and its second in a GitHub Actions step name, which sharpened K2 to require the token to be the whole value.enforce_pairing()reads the report back before printing and fails the gate if any line states a cluster fraction with no feature fraction beside it. It also fails on an empty report.quality_1_10is never derived from membership; severity still comes from the 0.63.0 ledger.Sibling sweep (Phase 3)
A defect in cluster X makes X's uncovered members a mandatory sweep list in the same ticket. The prior is measured: 201 findings → 37 root causes, ~5.4 per cause.
Mutations — six, both directions, engagement proved before the verdict
Fixture (
--self-test, all GREEN restores + a byte-identical no-op):Re-run against the real repo, because the fixture's proof does not transfer:
3cadac8a→93a900c6G2.5 per-cluster FAIL: gates in cluster apr-lint-diag is 54, must be >= 55— with G2.3 and G2.4 PASS1033e903→8338f01eG2.5 T2 FAILfa6c5818→7497ec3dapr-dogfood-coverage-v1.yaml:90That real-scope run found a defect the fixture missed: a G2.5 breakage printed as
G2.3 floors FAIL. Fixed in the second commit.pv validate0/0 ·pv auditno findings ·dogfood_baseline.py --checkPASS · reconcile both directions 0 · bashrs ratchet 876 → 138.🤖 Generated with Claude Code
https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR