From ab2ab43a517367f87bdfc338d9bb8d147e5af985 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 11 Oct 2026 16:19:15 +0000 Subject: [PATCH 1/3] test(testing): pin a negative remedy assertion for rule-throw-only-oracle The throw-only-oracle remedy had two positive pins and no negative one. Assert it is not the weak-oracle remedy, and move the detector-findings adopter tally for testing:audit from 7 of 13 to 8 of 13 (convention 3.7.1). Closes #6386 Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01L8KdG7fWg2YrErVYWtohmR --- docs/conventions/detector-findings/CHANGELOG.md | 8 ++++++++ docs/conventions/detector-findings/README.md | 2 +- .../testing/skills/audit/scripts/cant-fail-scan.test.sh | 1 + 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/conventions/detector-findings/CHANGELOG.md b/docs/conventions/detector-findings/CHANGELOG.md index 0a5894c377..3d81739d6a 100644 --- a/docs/conventions/detector-findings/CHANGELOG.md +++ b/docs/conventions/detector-findings/CHANGELOG.md @@ -4,6 +4,14 @@ Notable changes to the detector-findings contract (SemVer). Changing a producer- the coexistence obligations, or an enforceability verdict is a major bump; additive guidance or a new adopter row is a minor bump; docs-only clarification is a patch. +## [3.7.1] - 2026-10-11 + +**Patch, docs-only.** The `testing:audit` adopter row counts remedies pinned 8 of 13: +`rule-throw-only-oracle`'s remedy now carries its negative assertion, that it is not the +`rule-weak-oracle` remedy, so all eight report-only rules are pinned positive and negative in each +scope they fire in. No producer-owned field's rule, coexistence obligation, or enforceability +verdict moves. + ## [3.7.0] - 2026-10-04 **Minor, additive.** One crosswalk row admits `testing:audit`'s new report-only rule: diff --git a/docs/conventions/detector-findings/README.md b/docs/conventions/detector-findings/README.md index 5d495e1604..17fc8e2a7d 100644 --- a/docs/conventions/detector-findings/README.md +++ b/docs/conventions/detector-findings/README.md @@ -732,7 +732,7 @@ the bar in full. |---|---|---| | `mutation-testing:audit` | Conforming, opt-in. Remedies NOT pinned (3.0.0, grandfathered) | The first detector pilot. Persists surviving mutants behind `--persist-findings`; bare invocation still reports and stops. Maps each Phase 4 verdict class to one crosswalk rule and emits `Confidence: high` only. **Its rule selection is a fresh-context reviewer's judgment, not a computation**, so it is the worked case for the fail-safe-toward-emitting criterion rather than an exception to it. It has **two** withholding boundaries and both fall through to the emitting `unclassified` rule: an equivalence verdict that cannot cite its demonstration, and an aridity call whose proposed suppression entry does not bind a node kind from its enumerated vocabulary and name the behavior the suite deliberately leaves unasserted. A wavering judgment can therefore add a row but never silently remove one. **Both bars sit at classification rather than at persist time**, which is what makes the fall-through one answer per survivor instead of a report and a findings file that can disagree, and it means the bar binds a run that never persists, where an unevidenced withholding claim is read by a human rather than by the relay. Omits `tier:`, `## By dimension`, and `## Unparsed` as a detector with no analogue for them; keeps `## Surfaces`, which is the whole payload of a run that examined mutants and found nothing, and where its declined-candidate counts go. Its remediation is off-site, the covering test rather than `Location`, so every row it emits names the target in `Action` and the consumer surfaces rather than applies. It declares no remediation owner and is unaffected by that disposition: its `Auto-applicable` cells keep the forms they already had, `No, ` on its two emitting rows and `Not applicable, no row` on the two that emit none, and off-site is decided first in any case. **Remedy assertions (3.0.0): none.** What its eval `expected_output` pins is the off-site `Action` SHAPE, that the cell names the covering test and states the remediation site is not the row's `Location`. That is a claim about where a fix goes, not about the remedy any rule offers in the scope it fires in, and it is a model-graded expectation rather than a check a run fails. Grandfathered, and the debt is one assertion per emitting rule rather than a redesign. | -| `testing:audit` | Conforming, opt-in. Remedies PARTLY pinned, 7 of 13 (3.0.0; three grandfathered, the two config rules owed in full, `rule-throw-only-oracle` pinned positive only and owed its negative, the other seven report-only rules pinned positive and negative in each scope they fire in) | The first static Tier 1 detector, and the contract's first fully mechanical rule set: selection is a deterministic token/structure scan with no execution, so the fail-safe-toward-emitting criterion holds across both of its layers. Its three original test-body rules carry no withholding verdict at all, and the one uncertain case among them (a deliberate interaction-style test matching `rule-mock-only-oracle`) still emits, with `Confidence` omitted per the high-or-omitted rule. Its two Playwright runner-config rules DO withhold, and every boundary is stated-evidence: `spread-undecidable` requires a depth-1 spread token, `variadic-undecidable` a second `defineConfig` argument, `key-below-top-level` the key read below depth 1, `key-set` the guard read as `true` or as an expression, `retries-not-configured` a complete read of an anchored config object carrying no counted `retries` at all or none but literal `0`s, and `cant-fail-ok:` the annotation itself. Each requires a token present in the file, so nothing is withheld on absence alone, and every decline is counted per rule in `## Surfaces`. A config whose object literal cannot be anchored is judged by neither rule: it is reported in the denominator as enumerated and not examined, the same coverage statement the test-file counts already make about a file of another ecosystem. Persists behind `--persist-findings`; bare invocation reports and stops. Leads every `Finding` cell with the qualified rule id and the fired threshold in the run's own values; `Confidence` is `high` on the three deterministic-defect rules (`rule-zero-assertion`, `rule-recomputed-expectation`, `rule-inert-assertion`) and omitted on `rule-mock-only-oracle`, both config rules and the seven other report-only rules, never `low`. `Location` is repo-relative (computed through git's own prefix under a narrowed scan root) and IS the remediation site, since the repair belongs in the flagged test, or in the flagged config for the two config rules, but every row is judgment repair, the oracle a behavior deserves on the test-body rules and the team's flaky-tolerance or `only`-guard policy on the config rules, so none is auto-applicable. It declares **no** remediation owner, and that is correct rather than an omission: no skill owns choosing the assertion a behavior deserves or the runner policy a team wants, so its cells keep the plain `No, ` form and Step 4's judgment fence surfaces the rows, exactly as before. Omits `tier:`, `## By dimension`, and `## Unparsed`; keeps `## Surfaces` with per-rule declined-candidate counts, stating honestly that the recomputed-expectation rule's candidate assertions are not tallied v1 rather than inventing a number. A deliberate case is declined at selection by an in-file `cant-fail-ok: ` annotation, the rules' stated decline evidence, counted in `## Surfaces`, which is a recorded decision at the site it governs (the test for the body rules, the whole file for a Playwright config, on the repo's incumbent test-gate annotation shape), not a suppression of an emitted finding, so the finding-suppression home is not in play. Its `--check` mode is the fail-closed gate the liveness contract's fail-loud limb asks of a gating form: findings exit 1 (the config rules only under `--strict`, one switch that gates them with `rule-mock-only-oracle`, and a tree of configs with no test file reports them without gating or persisting them); an unread input, a dead engine, or a scan that examined nothing exits 2 rather than passing. **Remedy assertions (3.0.0): 7 of 13.** Seven of the eight report-only rules added later (`rule-inert-assertion`, whose remedy is scoped per language, `rule-constant-restatement`, `rule-source-text-read`, `rule-conditional-assertion`, whose C# remedy adds the skip routes, `rule-recomputed-derived`, `rule-snapshot-only` and `rule-weak-oracle`) are pinned positive and negative in every scope they fire in. The eighth, `rule-throw-only-oracle`, fires only on C# and its remedy is pinned positive there and not negative, so it is counted as owed, one negative assertion short. For the first five, `action_of()` carries five distinct remedy strings and `cant-fail-scan.test.sh` matches none of them as a remedy pinned to the rule and the scope it fires in; its only `Action` assertions are on the consumed table header and on the `1.52` version floor inside the flaky remedy, which is the expected-token match 3.0.0 names as passing on the defect the bar exists for. No negative assertion either, so nothing holds a remedy out of a scope it would not work in, and the `mock-only-oracle` remedy is still the one that most needs it among the three test-body rules, since it is the rule whose defect-hood is uncertain and whose advice is therefore conditional. Those three are grandfathered. The two config rules land after 3.0.0 and are held in full, so their shortfall is a debt recorded here rather than a clause covering them, and it is one assertion per rule per scope rather than a redesign. | +| `testing:audit` | Conforming, opt-in. Remedies PARTLY pinned, 8 of 13 (3.0.0; three grandfathered, the two config rules owed in full, the eight report-only rules pinned positive and negative in each scope they fire in) | The first static Tier 1 detector, and the contract's first fully mechanical rule set: selection is a deterministic token/structure scan with no execution, so the fail-safe-toward-emitting criterion holds across both of its layers. Its three original test-body rules carry no withholding verdict at all, and the one uncertain case among them (a deliberate interaction-style test matching `rule-mock-only-oracle`) still emits, with `Confidence` omitted per the high-or-omitted rule. Its two Playwright runner-config rules DO withhold, and every boundary is stated-evidence: `spread-undecidable` requires a depth-1 spread token, `variadic-undecidable` a second `defineConfig` argument, `key-below-top-level` the key read below depth 1, `key-set` the guard read as `true` or as an expression, `retries-not-configured` a complete read of an anchored config object carrying no counted `retries` at all or none but literal `0`s, and `cant-fail-ok:` the annotation itself. Each requires a token present in the file, so nothing is withheld on absence alone, and every decline is counted per rule in `## Surfaces`. A config whose object literal cannot be anchored is judged by neither rule: it is reported in the denominator as enumerated and not examined, the same coverage statement the test-file counts already make about a file of another ecosystem. Persists behind `--persist-findings`; bare invocation reports and stops. Leads every `Finding` cell with the qualified rule id and the fired threshold in the run's own values; `Confidence` is `high` on the three deterministic-defect rules (`rule-zero-assertion`, `rule-recomputed-expectation`, `rule-inert-assertion`) and omitted on `rule-mock-only-oracle`, both config rules and the seven other report-only rules, never `low`. `Location` is repo-relative (computed through git's own prefix under a narrowed scan root) and IS the remediation site, since the repair belongs in the flagged test, or in the flagged config for the two config rules, but every row is judgment repair, the oracle a behavior deserves on the test-body rules and the team's flaky-tolerance or `only`-guard policy on the config rules, so none is auto-applicable. It declares **no** remediation owner, and that is correct rather than an omission: no skill owns choosing the assertion a behavior deserves or the runner policy a team wants, so its cells keep the plain `No, ` form and Step 4's judgment fence surfaces the rows, exactly as before. Omits `tier:`, `## By dimension`, and `## Unparsed`; keeps `## Surfaces` with per-rule declined-candidate counts, stating honestly that the recomputed-expectation rule's candidate assertions are not tallied v1 rather than inventing a number. A deliberate case is declined at selection by an in-file `cant-fail-ok: ` annotation, the rules' stated decline evidence, counted in `## Surfaces`, which is a recorded decision at the site it governs (the test for the body rules, the whole file for a Playwright config, on the repo's incumbent test-gate annotation shape), not a suppression of an emitted finding, so the finding-suppression home is not in play. Its `--check` mode is the fail-closed gate the liveness contract's fail-loud limb asks of a gating form: findings exit 1 (the config rules only under `--strict`, one switch that gates them with `rule-mock-only-oracle`, and a tree of configs with no test file reports them without gating or persisting them); an unread input, a dead engine, or a scan that examined nothing exits 2 rather than passing. **Remedy assertions (3.0.0): 8 of 13.** The eight report-only rules added later (`rule-inert-assertion`, whose remedy is scoped per language, `rule-constant-restatement`, `rule-source-text-read`, `rule-conditional-assertion`, whose C# remedy adds the skip routes, `rule-recomputed-derived`, `rule-snapshot-only`, `rule-weak-oracle`, and `rule-throw-only-oracle`, which fires only on C# and is held apart from the weak-oracle remedy there) are pinned positive and negative in every scope they fire in. For the first five, `action_of()` carries five distinct remedy strings and `cant-fail-scan.test.sh` matches none of them as a remedy pinned to the rule and the scope it fires in; its only `Action` assertions are on the consumed table header and on the `1.52` version floor inside the flaky remedy, which is the expected-token match 3.0.0 names as passing on the defect the bar exists for. No negative assertion either, so nothing holds a remedy out of a scope it would not work in, and the `mock-only-oracle` remedy is still the one that most needs it among the three test-body rules, since it is the rule whose defect-hood is uncertain and whose advice is therefore conditional. Those three are grandfathered. The two config rules land after 3.0.0 and are held in full, so their shortfall is a debt recorded here rather than a clause covering them, and it is one assertion per rule per scope rather than a redesign. | | `ai-slop:audit` | Conforming, default-on for repo-examining runs. Remedies PARTLY pinned, 2 of 15 (3.0.0, grandfathered) | The first prose detector, and the first with a rule-sourced tier spread (twelve SUGGESTION style rules, three IMPORTANT generation-residue rules, each argued in its crosswalk row). Selection is fully mechanical (byte-sequence, phrase-list, and density scans; no withholding verdict), so the fail-safe criterion is met by construction; its decline evidence is the in-file ignore markers, config path exemptions, and code-fence stripping, counted per rule in `## Surfaces` from the detector's own `Summary` rows. The findings file is model-persisted by the skill (the deterministic `detect.sh` emits a parseable report only), per its `context/persist-findings.md` read of this contract: fetch this contract before the first write and refuse to persist when unreachable. Leads every `Finding` cell with the qualified rule id and the fired condition in the run's own values (the zero-tolerance marker or the density/threshold/hits/words tuple); `Confidence` is `high` on every row. Judgment-rubric findings never enter the file: no crosswalk row, no relay. **The first producer to declare producer-owned remediation**, and the case that section was written from: only `rule-utm-params` is auto-applicable, and its other fourteen rows are contained to `Location` yet safe only under the rewrite discipline in this plugin's own `reference/rewrite-guide.md`. Each of those fourteen rows leads its `Auto-applicable` cell with ``No, remediated by `/ai-slop:audit fix` ``, the code-span form the grammar states, which the relay resolves through the qualified rule id every emitted row already leads its `Finding` cell with, so those rows route to that action instead of the cleanup route's `/simplify`, which reads no findings file and loads no rewrite guide. **The declaration required no change to what this producer emits**, which is the point of siting it in the crosswalk: its emitted `Action` cells already describe the repair and already name the fix action on the rules with nothing more specific to say. After that fix runs the skill writes a `fix-pass-record` naming the findings file it consumed, then re-runs the detector and emits a fresh file for what remains. Omits `tier:`, `## By dimension`, and `## Unparsed`. **Remedy assertions (3.0.0): 2 of 15 rules, across the eight branches of `rule_action()`.** `detect.test.sh` pins `rule-filler-phrases` and `rule-stacked-hedging` by substring; the rest are unasserted, `rule-utm-params` among them. That one matters most of the fifteen: it is this producer's only auto-applicable rule, so its remedy is the only one here a consumer may act on without a human between, and it is the one remedy in the fleet whose wrongness costs an edit rather than a read. No negative assertion on any rule. Grandfathered. | diff --git a/plugins/testing/skills/audit/scripts/cant-fail-scan.test.sh b/plugins/testing/skills/audit/scripts/cant-fail-scan.test.sh index 8440fda914..2cac0ab3f8 100755 --- a/plugins/testing/skills/audit/scripts/cant-fail-scan.test.sh +++ b/plugins/testing/skills/audit/scripts/cant-fail-scan.test.sh @@ -1960,6 +1960,7 @@ done a="$(remedy "$C/cs-xunit/bad/WidgetConstructedNotNullTests.cs" rule-throw-only-oracle)" assert_contains "throw-only remedy asks for what the constructor sets" "$a" "Assert what the constructed value should hold" assert_contains "throw-only remedy names the cant-fail-ok: exemption for a smoke test" "$a" "cant-fail-ok: " +assert_not_contains "throw-only remedy is not the weak-oracle remedy" "$a" "Assert the value the code should produce" # An early return is a conditional-assertion finding, and its C# remedy names a # skip that xUnit v2 lacks as a package rather than assuming Assert.Skip. a="$(remedy "$C/cs-xunit/bad/ConfigEarlyReturnTests.cs" rule-conditional-assertion)" From ad9d2e1c6b4be5a7fbdf1c1c12399f4e9e88ff22 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:11:33 -0400 Subject: [PATCH 2/3] docs(testing): count eight of thirteen pinned remedies everywhere the README cites it Two prose counts in the detector-findings README still said seven of thirteen after the adopter row moved to eight. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01W6SjPvY7DmXBGzboeCq8nj --- docs/conventions/detector-findings/CHANGELOG.md | 3 ++- docs/conventions/detector-findings/README.md | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/conventions/detector-findings/CHANGELOG.md b/docs/conventions/detector-findings/CHANGELOG.md index 211df92983..2e27ae0e6d 100644 --- a/docs/conventions/detector-findings/CHANGELOG.md +++ b/docs/conventions/detector-findings/CHANGELOG.md @@ -6,7 +6,8 @@ adopter row is a minor bump; docs-only clarification is a patch. ## [3.8.1] - 2026-10-11 -**Patch, docs-only.** The `testing:audit` adopter row counts remedies pinned 8 of 13: +**Patch, docs-only.** The `testing:audit` adopter row, and the two counts that cite it, count +remedies pinned 8 of 13: `rule-throw-only-oracle`'s remedy now carries its negative assertion, that it is not the `rule-weak-oracle` remedy, so all eight report-only rules are pinned positive and negative in each scope they fire in. No producer-owned field's rule, coexistence obligation, or enforceability diff --git a/docs/conventions/detector-findings/README.md b/docs/conventions/detector-findings/README.md index 4462f11b37..d3de450c8e 100644 --- a/docs/conventions/detector-findings/README.md +++ b/docs/conventions/detector-findings/README.md @@ -516,7 +516,7 @@ already states of itself: one remedy, pinned in the scopes it is offered in, by the producer runs, a test case, an eval case, or a golden file. What must be checkable is the claim; the shape of the check is the producer's. **That freedom is over the FORM of the check, and is not a claim that a form is already in place.** Two of the six tabled adopters run no assertion reaching a -remedy at all, a third pins seven of its thirteen (its older five only through an `Action` cell's shape), and a fourth pins two of its fifteen, so +remedy at all, a third pins eight of its thirteen (its older five only through an `Action` cell's shape), and a fourth pins two of its fifteen, so for three of six this is an assertion to write rather than a harness to reuse, and the Adopters table below says which is which and by how much. **It is per rule and per scope, never per finding**, which is "Auto-applicability is settled per @@ -727,7 +727,7 @@ grandfather clause in above. Four of the six rows below do not meet that bar today and each says so in its own cell, with the count it falls short by wherever its remedies can be counted, so "Conforming" in this table never quietly means "conforming except the newest obligation". The annotation is per row because the -shortfall is: it runs from none of a producer's remedies pinned to seven of thirteen, and one row meets +shortfall is: it runs from none of a producer's remedies pinned to eight of thirteen, and one row meets the bar in full. | Producer | Status | Notes | From 4182a80dd2d98385524d06c8fa213815e660dc39 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:14:34 -0400 Subject: [PATCH 3/3] chore(testing): add a none changelog fragment for the test-only change Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01W6SjPvY7DmXBGzboeCq8nj --- .../testing/test-6386-throw-only-negative-remedy-28e5f6a7.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 .changes/testing/test-6386-throw-only-negative-remedy-28e5f6a7.md diff --git a/.changes/testing/test-6386-throw-only-negative-remedy-28e5f6a7.md b/.changes/testing/test-6386-throw-only-negative-remedy-28e5f6a7.md new file mode 100644 index 0000000000..7e06742de2 --- /dev/null +++ b/.changes/testing/test-6386-throw-only-negative-remedy-28e5f6a7.md @@ -0,0 +1,5 @@ +--- +bump: none +--- + +Test-only: a negative assertion on the throw-only-oracle remedy; nothing ships.