diff --git a/plugins/testing/.claude-plugin/plugin.json b/plugins/testing/.claude-plugin/plugin.json index 49f8a54e75..2170df6a21 100644 --- a/plugins/testing/.claude-plugin/plugin.json +++ b/plugins/testing/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "testing", - "version": "0.22.13", + "version": "0.22.14", "description": "Test-stage discipline: /testing:plan (coverage gaps, test plans), /testing:write (TDD authoring), /testing:run-e2e (live E2E and smoke checks), /testing:diagnose (failing-test root cause, with a fix-until-green workflow across files), /testing:audit (can't-fail test audit with a fail-closed gate), /testing:cleanup (rewrite, quarantine, or delete low-value tests behind a mutation gate), /testing:setup, and opt-in hooks that scan test files Claude writes and flag edits that weaken tests.", "author": { "name": "Melodic Software", diff --git a/plugins/testing/CHANGELOG.md b/plugins/testing/CHANGELOG.md index 08f1781898..72c26b1443 100644 --- a/plugins/testing/CHANGELOG.md +++ b/plugins/testing/CHANGELOG.md @@ -3,6 +3,16 @@ All notable changes to the `testing` plugin are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning. +## [0.22.14] - 2026-10-04 + +### Changed + +- **Shorter hook text ([#6225](https://github.com/melodic-software/claude-code-plugins/issues/6225)).** The `test-scan` hook lists findings without their thresholds, gives one Action per rule rather than per finding, lists at most 10 findings, and points to `/testing:test-value` once per file and only when there are findings. `test-scan` and `test-weaken` no longer tell Claude to state a reason before continuing. `test-weaken` names at most five changed expected values, and its deny reason under `test-weaken-block: error` is shorter. `cant-fail-scan.sh --brief` prints the hooks' form, and the `rule-zero-assertion` Action is one sentence. + +### Fixed + +- **The task-end test judge no longer floods the end of a task ([#6226](https://github.com/melodic-software/claude-code-plugins/issues/6226)).** Tests left to a background job (past the 10-test cap, late, or held by another job) and tests whose judge run failed are counts in one line, never a list of test names. A run where every verdict is a PASS shows one line, `test judge: N tests PASS.`, with the deferred count in the same line. A blocking Stop shows only its reason, with the findings path relative to the project, and no repeated summary. A forced Stop turn and an earlier session whose verdicts were all PASS say nothing. + ## [0.22.13] - 2026-10-04 ### Changed diff --git a/plugins/testing/README.md b/plugins/testing/README.md index 095f955774..118de44cfb 100644 --- a/plugins/testing/README.md +++ b/plugins/testing/README.md @@ -67,8 +67,8 @@ below): - `test_guards_enabled` (default `false`) turns on two hooks. `test-scan` (PostToolUse) runs the can't-fail scanner on each test file Claude writes or edits and returns the findings as - context. `test-weaken` (PreToolUse) asks Claude for a reason when an edit removes or skips tests - or assertions. + context. `test-weaken` (PreToolUse) tells Claude which tests, assertions or expected values an + edit removes, skips or changes. - `test_judge_enabled` (default `false`, and only effective with `test_guards_enabled`, whose scan records the tests it judges) turns on the task-end test judge, described below. `test_judge_model` (default `sonnet`) and `test_judge_fallback_model` (default `opus`) name the @@ -90,12 +90,13 @@ changed: where did its expected value come from? It answers FLAG (the value rest implementation), PASS or UNKNOWN, quotes its evidence, and proposes a diff for a FLAG. It never applies anything. A background job judges soon after a write; at the end of the task the Stop hook waits for any run still going, judges what is left (10 tests per task end, the rest at the next -one), writes a review-findings file (under `.work/reviews//`), and shows the counts. In an +one), and writes a review-findings file (under `.work/reviews//`). In an interactive session, when a verdict is a FLAG or an UNKNOWN for a reason other than "no repository" -or no judge class, it also asks Claude once to show you each verdict and proposed diff and wait. -When every verdict is a PASS or one of those two UNKNOWN verdicts, there is nothing to decide and the stop -is not blocked; that case, and an unattended session, get the counts and the file only. A session that ended before its verdicts were shown gets them named at -the next session start. The writing agent never supplies the judge's prompt, model or output, and +or no judge class, it asks Claude once to show you each verdict and proposed diff and wait. +Otherwise there is nothing to decide and the stop is not blocked: when every verdict is a PASS +you get one line with the count, and in the other cases, or in an unattended session, the counts +and the file. Tests left for a later task end are counted, never named. A session that ended +before a FLAG or UNKNOWN verdict was shown gets it named at the next session start. The writing agent never supplies the judge's prompt, model or output, and the judge's model class always differs from every model that wrote the tests: when the configured class wrote them, the fallback or the next of `opus`, `sonnet`, `haiku` is used, and when all three wrote them the tests are reported UNKNOWN. `test_judge_effort` has no effect on a judge model that @@ -147,9 +148,9 @@ it is reported UNKNOWN, "no repository". The judge runs from the repository with `]` in the repository's path is escaped with a backslash, so the rule's gitignore pattern names that one directory. A test file outside the repository git names for it (a `core.worktree` set elsewhere) is not judged: that is logged as a -malfunction and the test is named as not judged. A run in which Claude Code denied the judge a +malfunction and the test is counted as not judged. A run in which Claude Code denied the judge a tool call (the result's `permission_denials`) and that gives no test a FLAG or PASS is a -malfunction too: its UNKNOWN verdicts are dropped and those tests are named as not judged, so +malfunction too: its UNKNOWN verdicts are dropped and those tests are counted as not judged, so they do not stop the task. A denial names a tool call, not a test, and one run judges every in-doubt test of the file, so a denied run that gives any FLAG or PASS keeps all its verdicts, UNKNOWN included, and the denial is only logged. A symbolic link inside the repository that points outside diff --git a/plugins/testing/hooks/judge-lib.sh b/plugins/testing/hooks/judge-lib.sh index 60c2783087..8091b23857 100644 --- a/plugins/testing/hooks/judge-lib.sh +++ b/plugins/testing/hooks/judge-lib.sh @@ -648,13 +648,6 @@ judge::run() { judge::release_run "$res" } -# judge::label : ": ", with #n past the first. -judge::label() { - local l="${1##*[/\\]}: $2" - (($3 > 1)) && l+=" #$3" - printf '%s' "$l" -} - # judge::quoted_in_repo ...: true when every quote is verbatim # in a file of the repository, the judge's read scope: git grep over its # tracked and untracked (not ignored) files. One process per quote, and only @@ -827,10 +820,11 @@ judge::self_ignore() { # judge::findings : write one findings file per # repository in the detector-findings shape (FLAG rows under ## Findings; every # verdict, its quoted evidence and proposed diff under ## Verdicts) and set -# FINDINGS to their paths, comma-separated. +# FINDINGS to their paths, comma-separated, and FINDINGS_SHOWN to the same +# paths with each one under the project directory made relative to it. judge::findings() { - local all="$RELAY" repo dir ts path i branch content tmp p - FINDINGS="" + local all="$RELAY" repo dir ts path i branch content tmp p base="${CLAUDE_PROJECT_DIR:-${pcwd:-}}" + FINDINGS="" FINDINGS_SHOWN="" TZ=UTC0 printf -v ts '%(%Y%m%dT%H%M%SZ)T' -1 for repo in ${RELAY_REPOS[@]+"${RELAY_REPOS[@]}"}; do branch="" @@ -900,7 +894,10 @@ judge::findings() { [[ -e "$tmp" ]] && rm -f -- "$tmp" [[ -n "$path" ]] && break done - [[ -n "$path" ]] && FINDINGS+="${FINDINGS:+, }$path" + [[ -n "$path" ]] || continue + FINDINGS+="${FINDINGS:+, }$path" + [[ -n "$base" && "$path" == "${base%/}/"* ]] && path="${path#"${base%/}"/}" + FINDINGS_SHOWN+="${FINDINGS_SHOWN:+, }$path" done } diff --git a/plugins/testing/hooks/test-judge-start.sh b/plugins/testing/hooks/test-judge-start.sh index c2ba01f91c..38dd96d5ed 100755 --- a/plugins/testing/hooks/test-judge-start.sh +++ b/plugins/testing/hooks/test-judge-start.sh @@ -2,7 +2,8 @@ # SessionStart hook: the task-end test judge's catch-up. It names, once, the # ledger verdicts of this project's sessions whose last write is over an hour # old and that were never relayed (the session ended mid-task), as a -# systemMessage with the findings file, then marks them relayed. A session +# systemMessage with the findings file, then marks them relayed. A set that +# is all PASS is marked relayed and not named. A session # active within the hour is left to the Stop hook of its /clear or fork # successor, which this hook marks: source `clear` or `fork` writes # successors// holding the start time. Exits 0 on every path. @@ -52,9 +53,14 @@ for d in "$DATA/verdicts/$PKEY"/*/; do done done ((RELAY_N)) || exit 0 +# All PASS: nothing to act on. Recorded as relayed so it is never named. +if ((RELAY_F + RELAY_U == 0)); then + judge::mark_relayed "${marks[@]}" + exit 0 +fi judge::findings judge::counts judge::mark_relayed "${marks[@]}" -jq -cn --arg m "test judge: an earlier session ended before these verdicts were shown: $COUNTS. Findings: $FINDINGS" \ +jq -cn --arg m "test judge: an earlier session's verdicts: $COUNTS${FINDINGS_SHOWN:+ in $FINDINGS_SHOWN}" \ '{systemMessage: $m}' >&3 diff --git a/plugins/testing/hooks/test-judge-start.test.sh b/plugins/testing/hooks/test-judge-start.test.sh index 72aee4d662..4f3cbb38d5 100755 --- a/plugins/testing/hooks/test-judge-start.test.sh +++ b/plugins/testing/hooks/test-judge-start.test.sh @@ -27,14 +27,23 @@ assert_empty "no ledger: no output" "$out" # An unrelayed verdict from a session idle over an hour is named once. record old w1 "$F" null "" null 0 "$(iso $((now - 7200)))" -ledger old k1 "$F" PASS +# A FLAG with no diff fails validation, so it is named as an UNKNOWN. +ledger old k1 "$F" FLAG start new1 startup -assert_contains "an unrelayed verdict from an old session is named" "$(field .systemMessage)" "1 test (0 FLAG, 1 PASS, 0 UNKNOWN)" -f="$(field .systemMessage | sed -n 's/.*Findings: //p')" -check "with the findings file" '[[ -f "$f" && "$(cat "$f")" == *"type: review-findings"* ]]' +assert_contains "an unrelayed verdict from an old session is named" "$(field .systemMessage)" "test judge: an earlier session's verdicts: 1 test (0 FLAG, 0 PASS, 1 UNKNOWN) in " +f="$(field .systemMessage | sed -n 's/.* in //p')" +check "with the findings file, relative to the project" '[[ "$f" == .work/reviews/* && -f "$REPO/$f" && "$(cat "$REPO/$f")" == *"type: review-findings"* ]]' start new2 startup assert_empty "the notice appears once" "$out" +# An earlier session whose verdicts are all PASS has nothing to act on: it is +# not named, and is marked relayed. +record oldp w1 "$F" null "" null 0 "$(iso $((now - 7200)))" +ledger oldp kp "$F" PASS +start newp startup +assert_empty "all PASS from an old session: not named" "$out" +check "all PASS from an old session: marked relayed" '[[ -e "$DATA/relayed/$PKEY/oldp/kp" ]]' + # A relayed verdict is not named. record old2 w1 "$F" null "" null 0 "$(iso $((now - 7200)))" ledger old2 k2 "$F" FLAG @@ -45,7 +54,7 @@ assert_empty "a relayed verdict is not named" "$out" # One from a session active in the last hour is not named: adoption owns it. record recent w1 "$F" null "" null 0 "$(iso $((now - 600)))" -ledger recent k3 "$F" PASS +ledger recent k3 "$F" FLAG start new4 startup assert_empty "a verdict from a session active in the last hour is not named" "$out" diff --git a/plugins/testing/hooks/test-judge.sh b/plugins/testing/hooks/test-judge.sh index 3d2d439cef..3a6f2a0d44 100755 --- a/plugins/testing/hooks/test-judge.sh +++ b/plugins/testing/hooks/test-judge.sh @@ -3,8 +3,8 @@ # failure: an EXIT trap turns every path into exit 0. # # 1. stop_hook_active (the turn this hook or another Stop hook forced): judge -# nothing, never block; name the tests still being judged. Their verdicts -# wait in the ledger for the next task end. +# nothing, never block, say nothing. Verdicts wait in the ledger for the +# next task end. # 2. Re-derive the in-doubt keys of every file the session (and, for a /clear # or fork successor, its adopted predecessors) wrote. # 3. Keys with a verdict are ready; keys a live background job or lock holds @@ -20,7 +20,8 @@ # 5. Attended (CLAUDE_CODE_SESSION_ATTENDED exactly 1): block once with the # fixed template when the relayed set has a FLAG or an UNKNOWN for a # reason that is not environmental (no repository, no judge class). -# Either way a systemMessage carries the counts and path. +# Otherwise a systemMessage carries the counts and path, or, when every +# verdict is a PASS, one line with the count. # # Opt-in: hooks.json starts it through exec-bash.mjs --require-true # TEST_GUARDS_ENABLED --require-true TEST_JUDGE_ENABLED. See judge-lib.sh. @@ -41,6 +42,8 @@ source "$HOOK_DIR/scanner-run.sh" testing::fields "$INPUT" .session_id .transcript_path .cwd .stop_hook_active || exit 0 SID="${FIELDS[0]}" TPATH="${FIELDS[1]}" pcwd="${FIELDS[2]}" active="${FIELDS[3]}" [[ "$SID" =~ ^[A-Za-z0-9_-]+$ && -n "$TPATH" ]] || exit 0 +# A turn a Stop hook forced: the verdicts wait for the next task end. +[[ "$active" == true ]] && exit 0 testing::data_dir testing::pkey "${CLAUDE_PROJECT_DIR:-$pcwd}" "$TPATH" || exit 0 # The idle path: no write recorded and nothing to adopt. @@ -52,20 +55,15 @@ source "$HOOK_DIR/judge-lib.sh" exec 3>&1 >/dev/null 2>>"$JUDGE_LOG" LATE="$(mktemp -d)" || exit 0 trap 'rm -rf "$LATE"; exit 0' EXIT -emit() { [[ -z "$2" ]] || jq -cn --arg r "$1" --arg m "$2" 'if $r == "" then {} else {decision: "block", reason: $r} end + {systemMessage: $m}' >&3; } -judge::session_set - -# label : ": ", with #n past the first of a name. -label() { - local r="${KR[$1]}" name - name="${r#* }" && name="${name#* }" - judge::label "${KFILE[$1]}" "$name" "${r%% *}" -} -labels() { - local i out="" - for i in "$@"; do out+="${out:+, }$(label "$i")"; done - printf '%s' "$out" +# emit : either may be empty; both empty prints nothing. +emit() { + [[ -n "$1$2" ]] || return 0 + jq -cn --arg r "$1" --arg m "$2" \ + '(if $r == "" then {} else {decision: "block", reason: $r} end) + (if $m == "" then {} else {systemMessage: $m} end)' >&3 } +# plural +plural() { if (($1 == 1)); then printf '%s' "$2"; else printf '%s' "$3"; fi; } +judge::session_set # relay_needs_decision: true when an attended Stop has something to show. A # FLAG, or an UNKNOWN whose reason is not environmental. The two environmental @@ -81,20 +79,7 @@ relay_needs_decision() { <<<"$RELAY" >/dev/null } -if [[ "$active" == true ]]; then - waiting="" - for s in "${SESSIONS[@]}"; do - for p in "$DATA/pending/$PKEY/$s"/*; do - [[ -f "$p" ]] && ! judge::stale "$p" $((JUDGE_DEBOUNCE + JUDGE_STALE + 60)) && - waiting+="${waiting:+, }$(sed -n '2{s|.*/||;p;}' "$p")" - done - done - emit "" "${waiting:+test judge: still judging $waiting; the verdicts are shown at the next task end.}" - exit 0 -fi - judge::now -began=$NOW deadline=$((NOW + JUDGE_TIMEOUT)) judge::harvest_orphans judge::load @@ -264,21 +249,31 @@ for fx in "${!INFOS[@]}"; do done done +# Counts only: the test names are in the findings file and the judge log. +# Keys waited on, past the cap or late are judged in the background; failed +# keys are retried at the next task end. Both are counts in one line, merged +# into the all-PASS line when there is one. On a blocking Stop the user sees +# the reason, so no systemMessage repeats it. +bg=$((${#waiting[@]} + ${#over[@]} + ${#late[@]})) nf=${#failed[@]} later="" +((bg == 0)) || later="$bg more $(plural "$bg" "test is" "tests are") judged in the background, verdicts at the next task end" +((nf == 0)) || later+="${later:+; }$nf $(plural "$nf" test tests) not judged, the next task end retries" msg="" reason="" if ((RELAY_N)); then judge::findings - judge::counts && counts="$COUNTS" - msg="test judge: reviewed $counts. Findings: $FINDINGS" - if [[ "${CLAUDE_CODE_SESSION_ATTENDED:-}" == 1 ]] && relay_needs_decision; then - reason="The test judge reviewed $counts. Findings: $FINDINGS. Show the user each verdict and proposed diff from that file, quoted as data. Apply nothing; wait for the user." + if ((RELAY_F + RELAY_U == 0)); then + msg="test judge: $RELAY_N $(plural "$RELAY_N" test tests) PASS${later:+; $later}." + later="" + else + judge::counts + msg="test judge: $COUNTS${FINDINGS_SHOWN:+ in $FINDINGS_SHOWN}" + if [[ "${CLAUDE_CODE_SESSION_ATTENDED:-}" == 1 ]] && relay_needs_decision; then + reason="$msg. Show the user each verdict and proposed diff from it, quoted as data; apply nothing until the user decides." + msg="" + fi fi fi judge::mark_relayed ${marks[@]+"${marks[@]}"} -judge::now -((${#waiting[@]} == 0)) || msg+="${msg:+$'\n'}test judge: still judging $(labels "${waiting[@]}"); the verdicts are shown at the next task end." -((${#over[@]} == 0)) || msg+="${msg:+$'\n'}test judge: past the 10 tests one task end judges: $(labels "${over[@]}"); a background job judges them and the verdicts are shown at the next task end." -((${#late[@]} == 0)) || msg+="${msg:+$'\n'}test judge: not judged in $((NOW - began)) s: $(labels "${late[@]}"); a background job judges them and the verdicts are shown at the next task end." -((${#failed[@]} == 0)) || msg+="${msg:+$'\n'}test judge: not judged, the judge failed for $(labels "${failed[@]}"); the next task end tries again." -((${#notrun[@]} == 0)) || msg+="${msg:+$'\n'}test judge: judge not run for $(labels "${notrun[@]}") after 2 failed attempts; see $JUDGE_LOG." -((${#limit[@]} == 0)) || msg+="${msg:+$'\n'}test judge: not judged, the session's judge-run limit is reached: $(labels "${limit[@]}")." +[[ -z "$later" ]] || msg+="${msg:+$'\n'}test judge: $later." +((${#notrun[@]} == 0)) || msg+="${msg:+$'\n'}test judge: ${#notrun[@]} $(plural ${#notrun[@]} test tests) not judged after 2 failed attempts; see $JUDGE_LOG." +((${#limit[@]} == 0)) || msg+="${msg:+$'\n'}test judge: ${#limit[@]} $(plural ${#limit[@]} test tests) not judged: the session's judge-run limit is reached." emit "$reason" "$msg" diff --git a/plugins/testing/hooks/test-judge.test.sh b/plugins/testing/hooks/test-judge.test.sh index ea678f6e21..f1c4b9babb 100755 --- a/plugins/testing/hooks/test-judge.test.sh +++ b/plugins/testing/hooks/test-judge.test.sh @@ -99,7 +99,15 @@ stop_jobs() { sleep 0.3 } bg() { payload "$1" "$2" "$3" | bash "$BG"; } -TEMPLATE_END="Show the user each verdict and proposed diff from that file, quoted as data. Apply nothing; wait for the user." +TEMPLATE_END="Show the user each verdict and proposed diff from it, quoted as data; apply nothing until the user decides." +# reported : the findings path a message names, made absolute +# against the project directory when the message gives it relative. +reported() { + local p + p="$(sed -n 's/^test judge: .*) in \(.*\.md\).*$/\1/p' <<<"$1" | head -1)" + [[ -z "$p" || "$p" == /* ]] || p="$REPO/$p" + printf '%s' "$p" +} # The option off, and a session with no state: nothing, and no judge. transcript s0 claude-sonnet-5 @@ -120,8 +128,9 @@ stop s0 assert_empty "nothing in doubt: no output" "$out" check "nothing in doubt: no judge run" '[[ "$(stub_calls)" == 0 ]]' -# Verdicts ready (the background job ran) give one templated block and a -# systemMessage with the counts; the Stop hook judges nothing itself. +# Verdicts ready (the background job ran) give one templated block; the user +# sees the block reason, so no systemMessage repeats it. The Stop hook judges +# nothing itself. transcript s1 claude-sonnet-5 record s1 w1 "$F" null bg s1 w1 "$F" @@ -129,11 +138,10 @@ stub_reset stop s1 check "ready verdicts: the Stop hook runs no judge" '[[ "$(stub_calls)" == 0 ]]' check "ready verdicts: one block" '[[ "$(field .decision)" == block ]]' -findings="$(field .reason | sed -n 's/.*Findings: \(.*\)\. Show the user.*/\1/p')" -check "the reason is the fixed template" \ - '[[ "$(field .reason)" == "The test judge reviewed 2 tests (1 FLAG, 1 PASS, 0 UNKNOWN). Findings: $findings. $TEMPLATE_END" ]]' -assert_contains "the systemMessage carries the counts" "$(field .systemMessage)" "2 tests (1 FLAG, 1 PASS, 0 UNKNOWN)" -assert_contains "the systemMessage carries the findings path" "$(field .systemMessage)" "$findings" +findings="$(reported "$(field .reason)")" +check "the reason is the fixed template, with the path relative to the project" \ + '[[ "$(field .reason)" == "test judge: 2 tests (1 FLAG, 1 PASS, 0 UNKNOWN) in ${findings#"$REPO"/}. $TEMPLATE_END" && "${findings#"$REPO"/}" == .work/* ]]' +check "a blocking Stop carries no systemMessage" '[[ "$out" != *systemMessage* ]]' check "the findings file sits in the branch's review directory" '[[ "$findings" == "$REPO/.work/reviews/feat-judge-test/"*-test-judge.md && -f "$findings" ]]' check "the memory root self-ignores" '[[ "$(cat "$REPO/.work/.gitignore")" == "*" ]]' check "the findings file is a review-findings file for this branch" \ @@ -156,8 +164,8 @@ assert_contains "and the PASS verdict with its evidence" "$(cat "$findings")" "> stop s1 assert_empty "a relayed set is not relayed again" "$out" -# All PASS: the findings file and the systemMessage are written, and Stop is -# not blocked. There is nothing for the user to decide (#6037). This case +# All PASS: the findings file is written, Stop is not blocked, and the user +# gets one line. There is nothing for the user to decide (#6037). This case # stays quiet on purpose and fails against a hook that blocks on every relay. transcript s1p claude-sonnet-5 FP="$REPO/src/allpass.test.ts" @@ -167,10 +175,9 @@ bg s1p w1 "$FP" stub_reset stop s1p check "all PASS: no block" '[[ "$(field .decision)" != block ]]' -assert_contains "all PASS: the systemMessage carries the counts" "$(field .systemMessage)" "1 test (0 FLAG, 1 PASS, 0 UNKNOWN)" -findings="$(field .systemMessage | sed -n 's/.*Findings: //p')" -check "all PASS: the findings file is written" '[[ -n "$findings" && -f "$findings" ]]' -assert_contains "all PASS: the systemMessage carries the findings path" "$(field .systemMessage)" "$findings" +check "all PASS: one line with the count" '[[ "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' +check "all PASS: the findings file is still written" \ + '[[ -n "$(find "$REPO/.work/reviews" -name "*-test-judge*.md" -newer "$TDIR/s1p.jsonl")" ]]' # stop_hook_active: never blocks or judges, even with unrelayed verdicts, # which wait for the next task end. @@ -184,7 +191,7 @@ stop s2 true check "stop_hook_active: no block" '[[ "$(field .decision)" != block ]]' check "stop_hook_active: no judge run" '[[ "$(stub_calls)" == 0 ]]' stop s2 -check "the unrelayed verdict is relayed at the next task end" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test "* ]]' +check "the unrelayed verdict is relayed at the next task end" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' # A key with no job is judged at Stop, under the lock. transcript s3 claude-sonnet-5 @@ -193,7 +200,7 @@ js_file "$H" nojob record s3 w1 "$H" null stub_reset stop s3 -check "a key with no job is judged at Stop" '[[ "$(stub_calls)" == 1 && "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test "* ]]' +check "a key with no job is judged at Stop" '[[ "$(stub_calls)" == 1 && "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' check "and its lock is released" '[[ -z "$(find "$DATA/locks" -type f)" ]]' # A live pending/ job is waited on, not judged twice. @@ -206,7 +213,7 @@ payload s4 w1 "$P" | TEST_JUDGE_DEBOUNCE=2 bash "$BG" & sleep 0.5 stop s4 wait -check "a live pending/ job is waited on" '[[ "$(stub_calls)" == 1 && "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test "* ]]' +check "a live pending/ job is waited on" '[[ "$(stub_calls)" == 1 && "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' # A live lock is waited on too: its holder writes the verdict. transcript s5 claude-sonnet-5 @@ -230,11 +237,12 @@ printf '%s %s %s\n' "$holder" "${HOSTNAME:-localhost}" "$(date +%s)" >"$DATA/loc stub_reset stop s5 wait -check "a live lock is waited on" '[[ "$(stub_calls)" == 0 && "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test (0 FLAG, 1 PASS"* ]]' +check "a live lock is waited on" '[[ "$(stub_calls)" == 0 && "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' -# The 11th key: named, handed to a background job with a pending/ marker; the -# following stop_hook_active Stop does not block; its verdict is relayed at -# the next task end. +# The 11th key: counted, never named, and handed to a background job with a +# pending/ marker; the following stop_hook_active Stop does not block and +# says nothing; its verdict is relayed at the next task end. An all-PASS run +# past the cap is one short line (#6226). transcript s6 claude-sonnet-5 E="$REPO/src/eleven.test.ts" names=() @@ -243,19 +251,35 @@ js_file "$E" "${names[@]}" record s6 w1 "$E" null stub_reset STUB_SLEEP=1 stop s6 -check "ten keys are judged at Stop" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 10 tests"* ]]' -assert_contains "the 11th key is named as waiting" "$(field .systemMessage)" "eleven.test.ts: t11" +check "ten keys are judged at Stop, the 11th counted in the same one line" \ + '[[ "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 10 tests PASS; 1 more test is judged in the background, verdicts at the next task end." ]]' +assert_not_contains "no test is named" "$(field .systemMessage)" "eleven.test.ts" check "the 11th key is handed to a background job with a pending/ marker" '[[ -n "$(find "$DATA/pending/$PKEY/s6" -type f)" ]]' stop s6 true check "the following stop_hook_active Stop does not block" '[[ "$(field .decision)" != block ]]' +assert_empty "and says nothing while the job judges" "$out" for _ in $(seq 1 40); do [[ -z "$(find "$DATA/pending/$PKEY/s6" -type f)" ]] && break sleep 0.25 done stop s6 -check "the 11th verdict is relayed at the next task end" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test "* ]]' +check "the 11th verdict is relayed at the next task end" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' check "the 11th was judged once, by the background job" '[[ "$(stub_calls)" == 2 ]]' +# A blocking Stop with a key past the cap: the reason carries the verdicts, +# and the count line, which the reason does not hold, stays a systemMessage. +transcript s6b claude-sonnet-5 +EB="$REPO/src/elevenflag.test.ts" +js_file "$EB" "a flag" t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 +record s6b w1 "$EB" null +stub_reset +stop s6b +check "blocking Stop past the cap: blocks on the 10 judged" \ + '[[ "$(field .decision)" == block && "$(field .reason)" == "test judge: 10 tests (1 FLAG, 9 PASS, 0 UNKNOWN) in "* ]]' +check "and keeps the count line as its only systemMessage" \ + '[[ "$(field .systemMessage)" == "test judge: 1 more test is judged in the background, verdicts at the next task end." ]]' +stop_jobs s6b + # An 11th key whose job died is judged at the next task end. transcript s7 claude-sonnet-5 cp "$E" "$REPO/src/eleven2.test.ts" @@ -270,10 +294,10 @@ sleep 0.3 stub_reset stop s7 check "an 11th key whose job died is judged at the next task end" \ - '[[ "$(stub_calls)" == 1 && "$(stub_args 1)" == *"block 1 33-35 t11"* && "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test "* ]]' + '[[ "$(stub_calls)" == 1 && "$(stub_args 1)" == *"block 1 33-35 t11"* && "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' -# Time budget exhaustion: a systemMessage naming the tests not judged and the -# time spent, and no block. +# Time budget exhaustion: a systemMessage counting the test not judged, and +# no block. transcript s8 claude-sonnet-5 T="$REPO/src/slow.test.ts" js_file "$T" slow @@ -281,8 +305,8 @@ record s8 w1 "$T" null stub_reset STUB_MODE=hang TEST_JUDGE_TIMEOUT=1 stop s8 check "budget exhaustion: no block" '[[ "$(field .decision)" != block ]]' -assert_contains "budget exhaustion: names the test and the time" "$(field .systemMessage)" "slow.test.ts: slow" -check "budget exhaustion: states the time spent" '[[ "$(field .systemMessage)" =~ not\ judged\ in\ [0-9]+\ s ]]' +check "budget exhaustion: counts the test, names none" \ + '[[ "$(field .systemMessage)" == "test judge: 1 more test is judged in the background, verdicts at the next task end." ]]' check "budget exhaustion: the key goes to a background job, as the overflow does" '[[ -n "$(find "$DATA/pending/$PKEY/s8" -type f)" ]]' stop_jobs s8 sleep 0.5 @@ -308,7 +332,7 @@ STUB_SLEEP=8 TEST_JUDGE_TIMEOUT=4 stop s8b t1=$EPOCHREALTIME elapsed=$(((${t1/./} - ${t0/./}) / 1000)) check "a slot wait does not carry the Stop hook past its bound (${elapsed} ms <= 6000)" '((elapsed <= 6000 && rc == 0))' -assert_contains "the key not judged in time is named" "$(field .systemMessage)" "slotwait.test.ts: slotwait" +assert_contains "the key not judged in time is counted" "$(field .systemMessage)" "1 more test is judged in the background" check "and handed to a background job" '[[ -n "$(find "$DATA/pending/$PKEY/s8b" -type f)" ]]' kill "$holder" 2>/dev/null rm -f "$DATA/slots/"* @@ -339,8 +363,7 @@ elapsed=$(((${t1/./} - ${t0/./}) / 1000)) check "a TERM-deaf judge: the hook returns within the bound plus the KILL grace (${elapsed} ms <= 10000)" '((elapsed <= 10000 && rc == 0))' p="$(find "$DATA/pending/$PKEY/s8c" -type f | head -1)" check "its key gets a pending/ marker and a live background job" '[[ -n "$p" ]] && kill -0 "$(head -1 "$p" | cut -d" " -f1)" 2>/dev/null' -assert_not_contains "the message does not call this Stop's own late run still judging" "$(field .systemMessage)" "still judging" -assert_contains "the message names the key as not judged in time" "$(field .systemMessage)" "termdeaf.test.ts: termdeaf" +assert_contains "the message counts the key not judged in time" "$(field .systemMessage)" "1 more test is judged in the background" stop_jobs s8c tmp_kill TERM "$TMP/term-stub.sh" wait @@ -352,9 +375,10 @@ js_file "$X" failing record s9 w1 "$X" null stub_reset STUB_MODE=fail stop s9 -assert_contains "a first failure is named as not judged" "$(field .systemMessage)" "not judged" +check "a first failure is counted as retried, not as a background job" '[[ "$(field .systemMessage)" == "test judge: 1 test not judged, the next task end retries." ]]' STUB_MODE=fail stop s9 -assert_contains "2 failed attempts give judge not run" "$(field .systemMessage)" "judge not run for failing.test.ts: failing" +check "2 failed attempts: counted as not judged, with the log, no name" \ + '[[ "$(field .systemMessage)" == "test judge: 1 test not judged after 2 failed attempts; see $DATA/test-judge.log." ]]' STUB_MODE=fail stop s9 check "no third attempt" '[[ "$(stub_calls)" == 2 ]]' @@ -368,7 +392,7 @@ for mode in badquote otherfile; do STUB_MODE=$mode stop "v$mode" check "$mode: relayed as UNKNOWN" '[[ "$(field .reason)" == *"(0 FLAG, 0 PASS, 1 UNKNOWN)"* ]]' done -f="$(field .reason | sed -n 's/.*Findings: \(.*\)\. Show the user.*/\1/p')" +f="$(reported "$(field .reason)")" assert_contains "the reason for a diff touching another file is recorded" "$(cat "$f")" "the proposed diff touches another file" check "a FLAG that fails validation reaches no findings row" '[[ -z "$(rows "$f")" ]]' @@ -471,14 +495,15 @@ check "no judge class: no block" '[[ "$(field .decision)" != block ]]' assert_contains "no judge class: the systemMessage carries the counts" "$(field .systemMessage)" "(0 FLAG, 0 PASS, 1 UNKNOWN)" # Unattended (CLAUDE_CODE_SESSION_ATTENDED not exactly 1): no block, the -# systemMessage and the findings file only. +# systemMessage with the counts and the findings file only. transcript u1 claude-sonnet-5 U="$REPO/src/unattended.test.ts" -js_file "$U" unattended +js_file "$U" "unattended flag" record u1 w1 "$U" null out="$(payload u1 stop "" '{"hook_event_name": "Stop"}' | CLAUDE_CODE_SESSION_ATTENDED=0 bash "$HOOK" 2>/dev/null)" check "unattended: no block" '[[ "$(field .decision)" != block ]]' -assert_contains "unattended: the systemMessage carries the counts" "$(field .systemMessage)" "reviewed 1 test (0 FLAG, 1 PASS, 0 UNKNOWN)" +check "unattended: the systemMessage carries the counts and the relative path" \ + '[[ "$(field .systemMessage)" == "test judge: 1 test (1 FLAG, 0 PASS, 0 UNKNOWN) in .work/reviews/feat-judge-test/"*-test-judge*.md ]]' # A malformed state file is skipped; scanner exit 2 and a crash end in exit 0. transcript z1 claude-sonnet-5 @@ -487,7 +512,7 @@ js_file "$Z" sturdy record z1 w1 "$Z" null printf '{not json' >"$DATA/sessions/$PKEY/z1/broken.json" stop z1 -check "a malformed state file is skipped" '((rc == 0)) && [[ "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test "* ]]' +check "a malformed state file is skipped" '((rc == 0)) && [[ "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' transcript z2 claude-sonnet-5 record z2 w1 "$Z" null printf '#!/usr/bin/env bash\nexit 2\n' >"$TMP/scanner-2.sh" @@ -522,7 +547,7 @@ jq -cn --arg t "$TDIR/sbash.jsonl" --arg c "$REPO" --arg f "$BF" '{hook_event_na bash "$HOOK_DIR/test-scan-bash.sh" >/dev/null 2>&1 stub_reset stop sbash -check "a Bash-written test file is judged at the Stop" '[[ "$(stub_calls)" == 1 && "$(stub_args 1)" == *"block 1 3-5 bashmade"* && "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test "* ]]' +check "a Bash-written test file is judged at the Stop" '[[ "$(stub_calls)" == 1 && "$(stub_args 1)" == *"block 1 3-5 bashmade"* && "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' # The Stop reserves each run as the background jobs do: with a session limit # of 1 and three files to judge, one run starts and the rest are named. @@ -554,7 +579,7 @@ jq -cn --arg r '{"verdicts": [{"name": "orphan", "ordinal": 1, "verdict": "PASS" '{type: "result", subtype: "success", is_error: false, result: $r}' >"$d/.run-1-1" stub_reset stop h1 -check "an orphaned raw run is harvested, not judged again" '[[ "$(stub_calls)" == 0 && "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test (0 FLAG, 1 PASS"* ]]' +check "an orphaned raw run is harvested, not judged again" '[[ "$(stub_calls)" == 0 && "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' check "and the raw file is removed" '[[ -z "$(find "$d" -name ".run-*")" ]]' # Successors: a /clear or fork successor adopts the sessions whose last write @@ -579,7 +604,7 @@ start ca clear >/dev/null stub_reset stop ca check "after a clear marker, the predecessor's in-doubt block is judged" '[[ "$(stub_calls)" == 1 && "$(stub_args 1)" == *"block 1 6-8 unjudged"* ]]' -check "and its unrelayed verdict relayed with it" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 2 tests"* ]]' +check "and its unrelayed verdict relayed with it" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 2 tests PASS." ]]' transcript cb claude-sonnet-5 start cb clear >/dev/null stub_reset @@ -609,22 +634,22 @@ js_file "$S1" sibling record sib w1 "$S1" null "" null 0 "$(iso $((now + 120)))" bg sib w1 "$S1" stop cc -check "a sibling whose last write postdates the marker is not adopted" '[[ "$out" != *sibling* && "$(field .reason)" != *"reviewed"*"sibling"* ]]' -check "while the marker adopts pb, within the hour" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test "* ]]' +check "a sibling whose last write postdates the marker is not adopted" '[[ "$out" != *sibling* ]]' +check "while the marker adopts pb, within the hour" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' # A session whose last write is over an hour before the marker is not adopted; # the SessionStart catch-up names its verdicts instead. transcript po claude-sonnet-5 O1="$REPO/src/old.test.ts" -js_file "$O1" old +js_file "$O1" "old flag" record po w1 "$O1" null "" null 0 "$(iso $((now - 7200)))" bg po w1 "$O1" transcript cd claude-sonnet-5 out="$(start cd clear)" assert_contains "the successor's SessionStart catch-up names its verdicts instead" "$(field .systemMessage)" \ - "1 test (0 FLAG, 1 PASS, 0 UNKNOWN)" + "1 test (1 FLAG, 0 PASS, 0 UNKNOWN) in " record cd w1 "$REPO/src/add.test.ts" "[]" stop cd -check "and its Stop does not adopt that session" '[[ "$out" != *reviewed* ]]' +check "and its Stop does not adopt that session" '[[ "$out" != *"test judge"* ]]' transcript ce claude-sonnet-5 out="$(start ce startup)" assert_empty "the catch-up names them once" "$out" @@ -634,7 +659,7 @@ transcript st claude-sonnet-5 record st w1 "$A1" "$(blocks judged:1:3:5)" bg st w1 "$A1" stop st -check "a startup session relays its own verdict for block text another session relayed" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test "* ]]' +check "a startup session relays its own verdict for block text another session relayed" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' # The re-derive is cached by the file's content: a Stop over an unchanged # file with ready verdicts runs no scanner; a changed file, a changed config @@ -652,13 +677,13 @@ record cache w1 "$CF" null TEST_SCAN_SCANNER="$TMP/count-scan.sh" bg cache w1 "$CF" rm -f "$TMP/scans" TEST_SCAN_SCANNER="$TMP/count-scan.sh" stop cache -check "an unchanged file with ready verdicts: the Stop runs no scanner" '[[ "$(scans)" == 0 && "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 2 tests"* ]]' +check "an unchanged file with ready verdicts: the Stop runs no scanner" '[[ "$(scans)" == 0 && "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 2 tests PASS." ]]' js_file "$CF" one two three record cache w2 "$CF" "$(blocks three:1:9:11)" stub_reset TEST_SCAN_SCANNER="$TMP/count-scan.sh" stop cache check "a changed file is scanned again and its new block judged" \ - '[[ "$(scans)" == 1 && "$(stub_args 1)" == *"block 1 9-11 three"* && "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test "* ]]' + '[[ "$(scans)" == 1 && "$(stub_args 1)" == *"block 1 9-11 three"* && "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' rm -f "$TMP/scans" TEST_SCAN_SCANNER="$TMP/count-scan.sh" stop cache check "and is not scanned a third time while unchanged" '[[ "$(scans)" == 0 ]]' @@ -717,7 +742,7 @@ out="$(wpay st1 "" '{"hook_event_name": "Stop", "stop_hook_active": true}' | win check "Windows: stop_hook_active true does not block" '[[ "$(field .decision)" != block ]]' out="$(wpay st2 "" '{"hook_event_name": "Stop", "stop_hook_active": false}' | win bash "$HOOK" 2>/dev/null)" check "Windows: the Stop finds test-scan's project key and relays both verdicts, the FLAG intact" \ - '[[ "$(field .reason)" == *"reviewed 2 tests (1 FLAG, 1 PASS, 0 UNKNOWN)"* ]]' + '[[ "$(field .reason)" == *"test judge: 2 tests (1 FLAG, 1 PASS, 0 UNKNOWN) in "* ]]' check "Windows: the relay markers sit under test-scan's project key" '[[ "$(find "$DATA/relayed/$WPK/wsid" -type f | wc -l)" == 2 ]]' # CRLF alone, with Linux paths: stop_hook_active must still read as true, so a @@ -730,7 +755,7 @@ bg crlf w1 "$CR1" out="$(payload crlf stop "" '{"hook_event_name": "Stop", "stop_hook_active": true}' | PATH="$WIN_JQ:$PATH" TESTING_OSTYPE=msys bash "$HOOK" 2>/dev/null)" check "CRLF jq: stop_hook_active true does not block" '[[ "$(field .decision)" != block ]]' out="$(payload crlf stop "" '{"hook_event_name": "Stop"}' | PATH="$WIN_JQ:$PATH" TESTING_OSTYPE=msys bash "$HOOK" 2>/dev/null)" -check "CRLF jq: the next task end relays the verdict" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test (0 FLAG, 1 PASS"* ]]' +check "CRLF jq: the next task end relays the verdict" '[[ "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' # Under Git Bash one file is C:\x in a payload, C:/x from git and /c/x from # MSYS, with any case; a diff path from git must match the payload's file, and @@ -767,15 +792,15 @@ out="$(jq -cn --arg t "$WT" --arg c "$WC" '{hook_event_name: "SessionStart", ses check "Windows: SessionStart writes the successor marker under the same project key" '[[ -f "$DATA/successors/$WPK/wsucc" ]]' # Security review of the findings write and the relayed text. -# sec_stop : one ready verdict, then a Stop; sets SF, the +# sec_stop : one FLAG verdict, then a Stop; sets SF, the # findings path the Stop reports. sec_stop() { transcript "$1" claude-sonnet-5 local f="$REPO/src/$2" - js_file "$f" "sec$1" + js_file "$f" "flag sec$1" record "$1" w1 "$f" null stop "$1" - SF="$(field .systemMessage | sed -n 's/.*Findings: //p' | head -1)" + SF="$(reported "$(field .reason)")" } # 1. A symlink under the memory root must not carry the findings write, or the # self-ignoring .gitignore, out of the checkout. @@ -839,7 +864,7 @@ stub_reset stop sec6 check "a test file in no repository: no judge run, UNKNOWN 'no repository', no block" \ '[[ "$(stub_calls)" == 0 && "$(field .decision)" != block && "$(field .systemMessage)" == *"(0 FLAG, 0 PASS, 1 UNKNOWN)"* && "$(cat "$DATA/verdicts/$PKEY/sec6/"*.json)" == *"no repository"* ]]' -sf="$(field .systemMessage | sed -n 's/.*Findings: //p')" +sf="$(reported "$(field .systemMessage)")" check "a test file in no repository: the findings file is written" '[[ -n "$sf" && -f "$sf" ]]' # The repository is the test file's own, whatever the record says: the @@ -850,7 +875,7 @@ WTB="$TMP/wt-b" git -C "$REPO" worktree add -q -b feat/wt-b "$WTB" 2>/dev/null WTOP="$(git -C "$WTB" rev-parse --show-toplevel)" mkdir -p "$WTB/src" -js_file "$WTB/src/linked.test.ts" linked +js_file "$WTB/src/linked.test.ts" "linked flag" transcript wt1 claude-sonnet-5 record wt1 w1 "$WTB/src/linked.test.ts" null stub_reset @@ -858,7 +883,7 @@ stop wt1 check "a linked-worktree test file recorded under the main checkout is judged from the worktree" \ '[[ "$(stub_calls)" == 1 && "$(cut -d" " -f1 "$STUB_DIR"/call-*.env)" == "$WTOP" ]]' check "with the worktree's absolute Read rule" '[[ "$(stub_args 1 | sed -n "/^--allowedTools$/{n;p;}")" == "Read(/$WTOP/**)" ]]' -wf="$(field .systemMessage | sed -n 's/.*Findings: //p')" +wf="$(reported "$(field .reason)")" check "its findings land under the worktree's review directory" '[[ "$wf" == "$WTOP/.work/reviews/feat-wt-b/"*-test-judge.md && -f "$wf" ]]' check "its verdict names the worktree as its repository" '[[ "$(jq -r .repo "$DATA/verdicts/$PKEY/wt1/"*.json)" == "$WTOP" ]]' # A file in no repository, recorded under the hook's: not judged, and quiet, @@ -885,8 +910,8 @@ stub_reset stop cw1 check "a test file outside the repository git names for it: no judge run and no verdict" '[[ "$(stub_calls)" == 0 && -z "$(verdict_files cw1)" ]]' check "that is logged as a malfunction" 'grep -qF "malfunction: the test file is outside the repository git names for it, $CW/elsewhere" "$DATA/test-judge.log"' -check "it does not block, and the test is named as not judged" \ - '[[ "$(field .decision)" != block && "$(field .systemMessage)" == *"not judged, the judge failed for cw.test.ts: cwout"* ]]' +check "it does not block, and the test is counted for a later task end" \ + '[[ "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test not judged, the next task end retries." ]]' # A run Claude Code denied a tool call (the result's permission_denials) that # gives no test a FLAG or PASS is a malfunction: its UNKNOWN verdicts are not @@ -900,9 +925,23 @@ stub_reset STUB_MODE=denied stop dn1 check "a denied run with only UNKNOWN verdicts: one run, and no verdict" '[[ "$(stub_calls)" == 1 && -z "$(verdict_files dn1)" ]]' check "it does not block Stop" '[[ "$(field .decision)" != block ]]' -assert_contains "its tests are named as not judged" "$(field .systemMessage)" "not judged, the judge failed for denied.test.ts: deny me, denied.test.ts: deny too" +check "its tests are counted as retried at the next task end" '[[ "$(field .systemMessage)" == "test judge: 2 tests not judged, the next task end retries." ]]' check "the denial is logged as a malfunction" \ 'grep -qF "malfunction: judge run on $DN: the judge was denied Read and gave no test a FLAG or PASS" "$DATA/test-judge.log"' +# PASS, background and failed keys in one all-PASS line: a failed key is +# counted apart, since the next task end retries it and no background job +# does. Files sort by path, so a-denied runs first (1 key, failed), then 9 of +# z-mixed's 11 keys (PASS), and 2 go to the background. +transcript mx1 claude-sonnet-5 +js_file "$REPO/src/a-denied.test.ts" "deny alone" +js_file "$REPO/src/z-mixed.test.ts" "${names[@]}" +record mx1 w1 "$REPO/src/a-denied.test.ts" null +record mx1 w2 "$REPO/src/z-mixed.test.ts" null +stub_reset +STUB_MODE=denied stop mx1 +check "PASS, background and failed keys: one line, failed keys counted apart" \ + '[[ "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 9 tests PASS; 2 more tests are judged in the background, verdicts at the next task end; 1 test not judged, the next task end retries." ]]' +stop_jobs mx1 # A denial names a tool call, not a block, and one run judges every block of # the file: when the run gives any FLAG or PASS, its UNKNOWN verdicts stand, # so one denied read cannot mute the other blocks. @@ -914,7 +953,7 @@ stub_reset STUB_MODE=denied stop dn3 check "a denied run with a PASS keeps both UNKNOWN verdicts and the PASS" \ '[[ "$(stub_calls)" == 1 && "$(verdict_of dn3 "deny one")" == *"\"verdict\":\"UNKNOWN\""* && "$(verdict_of dn3 "deny two")" == *"\"verdict\":\"UNKNOWN\""* && "$(verdict_of dn3 keeps)" == *"\"verdict\":\"PASS\""* ]]' -check "its UNKNOWN verdicts block Stop" '[[ "$(field .decision)" == block && "$(field .reason)" == *"reviewed 3 tests (0 FLAG, 1 PASS, 2 UNKNOWN)"* ]]' +check "its UNKNOWN verdicts block Stop" '[[ "$(field .decision)" == block && "$(field .reason)" == *"test judge: 3 tests (0 FLAG, 1 PASS, 2 UNKNOWN) in "* ]]' check "the denial is logged, not as a malfunction" \ 'grep -qF "judge run on $DM was denied Read; it gave a FLAG or PASS, so its UNKNOWN verdicts stand" "$DATA/test-judge.log" && ! grep -qF "malfunction: judge run on $DM" "$DATA/test-judge.log"' # The signal is the denial Claude Code records, not the judge's prose: an @@ -1062,7 +1101,7 @@ record sec7 w1 "$REPO/src/sec7.test.ts" null # shellcheck disable=SC2016 # the expansion is the attack, kept literal out="$(payload sec7 stop "" '{"hook_event_name": "Stop"}' | TEST_JUDGE_TIMEOUT='a[$(touch '"$TMP"'/pwned5)]' \ TEST_JUDGE_DEBOUNCE='b[$(touch '"$TMP"'/pwned5b)]' bash "$HOOK" 2>/dev/null)" -check "a timeout or debounce value is not evaluated as arithmetic" '[[ ! -e "$TMP/pwned5" && ! -e "$TMP/pwned5b" && "$(field .decision)" != block && "$(field .systemMessage)" == *"reviewed 1 test "* ]]' +check "a timeout or debounce value is not evaluated as arithmetic" '[[ ! -e "$TMP/pwned5" && ! -e "$TMP/pwned5b" && "$(field .decision)" != block && "$(field .systemMessage)" == "test judge: 1 test PASS." ]]' check "no real claude was ever called" '[[ ! -e "$TMP/real-claude-called" ]]' echo "$SKIPS skipped (no native symlinks)" diff --git a/plugins/testing/hooks/test-scan.sh b/plugins/testing/hooks/test-scan.sh index 2edafe119b..1871ee19a4 100755 --- a/plugins/testing/hooks/test-scan.sh +++ b/plugins/testing/hooks/test-scan.sh @@ -140,7 +140,7 @@ fi # ponytail: patch line numbers go stale if another PostToolUse hook reformats # the file before this read; the scan then scopes to shifted blocks. SCANNER="${TEST_SCAN_SCANNER:-$HOOK_DIR/../skills/audit/scripts/cant-fail-scan.sh}" -testing::run_scanner "${TEST_SCAN_TIMEOUT:-8}" "$out_file" --file "$FILE" "${scope[@]}" --blocks +testing::run_scanner "${TEST_SCAN_TIMEOUT:-8}" "$out_file" --file "$FILE" "${scope[@]}" --blocks --brief rc=$SCAN_RC if ((rc != 0)); then @@ -167,7 +167,7 @@ if grep -q '^ test files: 0 examined' "$out_file"; then fi state_write blocks -findings="$(grep '^finding \[' "$out_file")" +findings="$(sed -n 's/^finding \[/[/p' "$out_file")" ctx="" FINDINGS_JSON='[]' if [[ -n "$findings" ]]; then @@ -180,16 +180,19 @@ if [[ -n "$findings" ]]; then grep -qv -e rule-constant-restatement -e rule-source-text-read <<<"$findings" || lead="has tests that fail on harmless changes (change detectors):" fi - hook::findings_to ctx "testing: $FILE_BASE $lead" "$findings" FINDINGS_JSON - if [[ "$findings" == *rule-recomputed-expectation* || "$findings" == *rule-constant-restatement* || "$findings" == *rule-recomputed-derived* ]]; then - ctx+=$'\n'"Before you continue, state where the expected value in each flagged assertion comes from (a spec, a bug report, a hand-computed literal). If it comes from running the code under test, replace it with a value worked out independently." - fi + hook::findings_to ctx "testing: $FILE_BASE $lead" "$findings" FINDINGS_JSON \ + --max 10 --more "/testing:audit lists the rest" + # One Action per rule, after the findings it covers. + actions="$(sed -n 's/^action \[\([^]]*\)\] /Action [\1]: /p' "$out_file")" + [[ -z "$actions" ]] || ctx+=$'\n'"$actions" + # The skill pointer once per (session, agent, file), and only beside findings. + # CLAUDE_PLUGIN_DATA is set for the consumer settings entry, which has none, + # so both entries share one latch. + CLAUDE_PLUGIN_DATA="$DATA" hook::once_per_file testing-note "$INPUT" "$FILE" && + ctx+=$'\n'"/testing:test-value covers where expected values come from." fi -key="$(printf '%s|%s|%s' "$session" "$agent" "$FILE" | cksum)" -if mark "$DATA/marks/note-${key%% *}"; then - ctx+="${ctx:+$'\n'}Tests here should fail when the behavior they cover breaks. Load the testing:test-value skill for where expected values must come from and what makes a test worth keeping." -fi +# --max bounds the line count, not line length: a finding carries its test name. +((${#ctx} < 10000)) || ctx="${ctx:0:9800}"$'\n'"(truncated; /testing:audit lists the rest)" -((${#ctx} < 10000)) || ctx="${ctx:0:9800}"$'\n'"(truncated; run /testing:audit for the full list)" hook::finish --context "$ctx" ok findings array "$FINDINGS_JSON" diff --git a/plugins/testing/hooks/test-scan.test.sh b/plugins/testing/hooks/test-scan.test.sh index 100bd203b3..03894fa76a 100755 --- a/plugins/testing/hooks/test-scan.test.sh +++ b/plugins/testing/hooks/test-scan.test.sh @@ -5,7 +5,7 @@ # pipes hand-built PostToolUse payloads through the hook. Covers the option # gate (through the exec-form launcher), findings on a create, the once-per- # file-per-agent rules note, Edit scoping to the changed block, the scanner -# timeout, gitignored paths, the doubtful-hit prompt (recomputed and derived +# timeout, gitignored paths, one Action per rule (recomputed and derived # expectations, constant restatements), the lead for findings of tests that # can fail, the marker prune, the per-call dedup that keeps two # overlapping `if` rows from reporting twice, and the per-write session state @@ -163,6 +163,41 @@ assert_not_contains "(c) same session and agent: no second note" "$out" "testing run Write "$REPO/src/sum.test.ts" s1 agent-2 assert_contains "(c) a different agent_id gets the note" "$out" "testing:test-value" +# (c2) a clean first write of a file gets no note: it has nothing to act on. +printf '%s\n' "import { test, expect } from 'vitest';" "test('caps', () => {" " expect(cap(60)).toBe(50);" "});" >"$REPO/src/quiet.test.ts" +run Write "$REPO/src/quiet.test.ts" s-quiet +assert_empty "(c2) clean first write: no context at all" "$out" + +# (c3) two findings of one rule: both listed, the Action once, no threshold. +printf '%s\n' "import { test } from 'vitest';" "test('a', () => {" " sum(1, 2);" "});" \ + "test('b', () => {" " sum(2, 2);" "});" >"$REPO/src/twice.test.ts" +run Write "$REPO/src/twice.test.ts" s-twice +ctx3="$(jq -r '.hookSpecificOutput.additionalContext' <<<"$out")" +if [[ "$(grep -c '^ \[rule-zero-assertion\] ' <<<"$ctx3")" == 2 ]]; then ok "(c3) both findings listed"; else fail "(c3) both findings listed (got: $ctx3)"; fi +if [[ "$(grep -c '^Action \[rule-zero-assertion\]: ' <<<"$ctx3")" == 1 ]]; then ok "(c3) one Action for the rule"; else fail "(c3) one Action for the rule (got: $ctx3)"; fi +assert_not_contains "(c3) no threshold in the context" "$ctx3" "threshold:" + +# (c4) twelve findings: ten listed, then one line counting the rest. +{ + echo "import { test } from 'vitest';" + for k in $(seq 1 12); do printf '%s\n' "test('t$k', () => {" " sum($k, 0);" "});"; done +} >"$REPO/src/many.test.ts" +run Write "$REPO/src/many.test.ts" s-many +ctx4="$(jq -r '.hookSpecificOutput.additionalContext' <<<"$out")" +if [[ "$(grep -c '^ \[rule-zero-assertion\] ' <<<"$ctx4")" == 10 ]]; then ok "(c4) ten findings listed"; else fail "(c4) ten findings listed (got: $ctx4)"; fi +if [[ "$(grep -cxF ' ... and 2 more (/testing:audit lists the rest)' <<<"$ctx4")" == 1 ]]; then ok "(c4) the rest counted, with the audit pointer"; else fail "(c4) the rest counted, with the audit pointer (got: $ctx4)"; fi + +# (c5) ten findings with 2 KB test names still fit the 10000-character context cap. +long="$(printf 'n%.0s' $(seq 1 2000))" +{ + echo "import { test } from 'vitest';" + for k in $(seq 1 10); do printf '%s\n' "test('$long$k', () => {" " sum($k, 0);" "});"; done +} >"$REPO/src/long.test.ts" +run Write "$REPO/src/long.test.ts" s-long +ctx5="$(jq -r '.hookSpecificOutput.additionalContext' <<<"$out")" +if ((${#ctx5} < 10000)); then ok "(c5) context stays under 10000 characters"; else fail "(c5) context is ${#ctx5} characters"; fi +assert_contains "(c5) a cut context says where the rest is" "$ctx5" "(truncated; /testing:audit lists the rest)" + # (d) an Edit touching only the good block leaves the bad block silent. EDIT_GOOD='{"structuredPatch":[{"oldStart":9,"oldLines":1,"newStart":9,"newLines":1, "lines":["- expect(sum(1, 2)).toBe(4);","+ expect(sum(1, 2)).toBe(3);"]}]}' @@ -212,16 +247,17 @@ fi run Write "$REPO/scratch/ignored.test.ts" assert_empty "(f) gitignored path: no output" "$out" -# (g) a recomputed expectation asks where the expected value comes from. +# (g) a recomputed expectation carries its Action and no order. run Write "$REPO/src/again.test.ts" assert_contains "(g) names rule-recomputed-expectation" "$out" "rule-recomputed-expectation" -assert_contains "(g) asks for the expected value's source" "$out" "where the expected value" +assert_contains "(g) carries the rule's Action" "$out" "Action [rule-recomputed-expectation]: State the expected value independently" +assert_not_contains "(g) gives no order to state a source" "$out" "Before you continue" -# (h) a constant restatement carries the same prompt, under a change-detector +# (h) a constant restatement carries its Action, under a change-detector # lead rather than the can't-fail one. run Write "$REPO/src/limit.test.ts" assert_contains "(h) names rule-constant-restatement" "$out" "rule-constant-restatement" -assert_contains "(h) asks for the expected value's source" "$out" "where the expected value" +assert_contains "(h) carries the rule's Action" "$out" "Action [rule-constant-restatement]: " assert_contains "(h) leads with change detectors" "$out" "fail on harmless changes" assert_not_contains "(h) does not call a change detector a test that cannot fail" "$out" "tests that cannot fail" @@ -245,10 +281,10 @@ else fi if [[ -f "$CLAUDE_PLUGIN_DATA/marks/call-call-1" ]]; then ok "prune: a fresh marker stays"; else fail "prune: a fresh marker stays"; fi -# (i) a recomputed-derived expectation carries the doubtful-hit prompt. +# (i) a recomputed-derived expectation carries its Action. run Write "$REPO/src/derived.test.ts" assert_contains "(i) names rule-recomputed-derived" "$out" "rule-recomputed-derived" -assert_contains "(i) asks for the expected value's source" "$out" "where the expected value" +assert_contains "(i) carries the rule's Action" "$out" "Action [rule-recomputed-derived]: " assert_contains "(i) leads with tests that check little" "$out" "tests that check little" assert_not_contains "(i) does not call a derived expectation a test that cannot fail" "$out" "tests that cannot fail" @@ -256,7 +292,7 @@ assert_not_contains "(i) does not call a derived expectation a test that cannot run Write "$REPO/src/weak.test.ts" assert_contains "(j) names rule-weak-oracle" "$out" "rule-weak-oracle" assert_not_contains "(j) does not call a weak oracle a test that cannot fail" "$out" "tests that cannot fail" -assert_not_contains "(j) a weak oracle carries no doubtful-hit prompt" "$out" "where the expected value" +assert_contains "(j) carries the weak-oracle Action" "$out" "Action [rule-weak-oracle]: " # Two overlapping `if` rows run the hook twice for one call; only one reports. run Write "$REPO/src/sum.test.ts" s1 "" dup-call diff --git a/plugins/testing/hooks/test-weaken.sh b/plugins/testing/hooks/test-weaken.sh index bd6fa309a8..2f0f11fae8 100755 --- a/plugins/testing/hooks/test-weaken.sh +++ b/plugins/testing/hooks/test-weaken.sh @@ -114,12 +114,15 @@ END { if (rt > 0) { block = 1; out = out "\n- removed " rt " test block(s):"; names(1, 2, "test") } if (sk > 0) { block = 1; out = out "\n- added " sk " skip marker(s):"; names(2, 1, "skip") } if (ra > 0) { out = out "\n- removed " ra " assertion token(s):"; names(1, 2, "assertion") } - for (i = 1; i <= N[1]; i++) { + # At most 5 changed values, as names() lists at most 5 per kind. + for (i = 1; i <= N[1] && nc <= 5; i++) { a = PA[1, i]; e = PE[1, i] if (P[1, a, e] <= P[2, a, e]) continue for (j = 1; j <= N[2]; j++) - if (PA[2, j] == a && PE[2, j] != e && P[2, a, PE[2, j]] > P[1, a, PE[2, j]]) + if (PA[2, j] == a && PE[2, j] != e && P[2, a, PE[2, j]] > P[1, a, PE[2, j]]) { + if (++nc > 5) { out = out "\n- ..."; break } out = out "\n- changed the expected value of " a " from " e " to " PE[2, j] + } } if (out != "") printf "%s\t%s%s\n", block ? "block" : "advise", level, out }' "$work/out")" @@ -133,7 +136,7 @@ lead="testing: $FILE_BASE: this $tool weakens its tests:"$'\n'"$body" # grep -c prints 0 and exits 1 on no match; the count is what matters. if [[ "$head" == block$'\t'error ]] && (($(grep -c 'test-change:[[:space:]]*[^[:space:]]' "$work/new") <= $(grep -c 'test-change:[[:space:]]*[^[:space:]]' "$old"))); then - reason="$lead"$'\n'"rules.test-weaken-block is error in the testing config, so a skipped or removed test needs a stated reason. If the change is deliberate, retry the same edit with a comment in the edited text that reads test-change: , naming why the test goes (a removed feature, a tracked flaky test). Never skip or delete a test to make failing code pass." + reason="$lead"$'\n'"Denied: rules.test-weaken-block is error. If the change is deliberate, retry the edit with a test-change: comment naming why the test goes; otherwise fix the code, not the test." esc="" hook::json_escape_to esc "$reason" if [[ -n "${start:-}" ]] && hook::telemetry_enabled; then @@ -145,5 +148,4 @@ if [[ "$head" == block$'\t'error ]] && exit 0 fi -ctx="$lead"$'\n'"Before you continue, state the reason for each change (a changed requirement, a wrong expected value, a test of removed code). Never weaken a test to make failing code pass: fix the code instead." -hook::finish --context "$ctx" ok findings array '[]' +hook::finish --context "$lead" ok findings array '[]' diff --git a/plugins/testing/hooks/test-weaken.test.sh b/plugins/testing/hooks/test-weaken.test.sh index a14fe82b44..49a7bb194d 100755 --- a/plugins/testing/hooks/test-weaken.test.sh +++ b/plugins/testing/hooks/test-weaken.test.sh @@ -127,7 +127,7 @@ run Edit "$REPO/src/sum.test.ts" "$DROP_EXPECT" if [[ $rc -eq 0 ]]; then ok "(a) exits 0"; else fail "(a) exit $rc"; fi assert_contains "(a) goes back through additionalContext" "$out" '"additionalContext"' assert_contains "(a) names the removed assertion" "$out" "expect(sum(2, 2)).toBe(4);" -assert_contains "(a) asks for the reason" "$out" "reason" +assert_not_contains "(a) states the facts, gives no order" "$out" "Before you continue" assert_no_decision "(a) no permissionDecision field while advisory" "$out" if jq -e . <<<"$out" >/dev/null 2>&1; then ok "(a) one JSON document"; else fail "(a) not JSON: $out"; fi @@ -151,7 +151,13 @@ assert_contains "suite skip: test.describe.skip is named as an added skip" "$out block_on run Edit "$REPO/src/sum.test.ts" "$ADD_SKIP" assert_deny "(b) block rule on: an added skip is denied" "$out" -assert_contains "(b) the deny reason names the marker" "$out" "test-change: " +reason="$(jq -r '.hookSpecificOutput.permissionDecisionReason' <<<"$out")" +deny_tail="Denied: rules.test-weaken-block is error. If the change is deliberate, retry the edit with a test-change: comment naming why the test goes; otherwise fix the code, not the test." +if [[ "$reason" == "testing: sum.test.ts: this Edit weakens its tests:"$'\n'*$'\n'"$deny_tail" ]]; then + ok "(b) the deny reason is the lead, the change, then the short directive" +else + fail "(b) the deny reason is the lead, the change, then the short directive (got: $reason)" +fi run Edit "$REPO/src/sum.test.ts" "$ADD_SKIP_MARKED" assert_no_decision "(b) the marked edit has no decision" "$out" assert_contains "(b) the marked edit still gets context" "$out" '"additionalContext"' @@ -175,6 +181,18 @@ run Edit "$REPO/src/sum.bats" "$(edit_input ' [ "$output" = "3" ]' ' [ "$outpu assert_contains "bash-bats: a changed literal in a [ ] comparison is named" "$out" 'from \"3\" to \"4\"' run Edit "$REPO/src/sum_test.go" "$(edit_input $'\tif got != 3 {' $'\tif got != 3 && ok {')" assert_not_contains "go-testing: a compound condition is not read as a changed literal" "$out" "changed the expected" +# Seven changed literals: five are named, then one "..." line. +old7="" new7="" +for k in 1 2 3 4 5 6 7; do + old7+=" expect(sum($k, 0)).toBe($k);"$'\n' + new7+=" expect(sum($k, 0)).toBe($((k + 10)));"$'\n' +done +printf "import { test, expect } from 'vitest';\n\ntest('ids', () => {\n%s});\n" "$old7" >"$REPO/src/seven.test.ts" +run Edit "$REPO/src/seven.test.ts" "$(edit_input "$old7" "$new7")" +ctx7="$(jq -r '.hookSpecificOutput.additionalContext' <<<"$out")" +if [[ "$(grep -c '^- changed the expected value of ' <<<"$ctx7")" == 5 ]]; then ok "changed literals: five named"; else fail "changed literals: five named (got: $ctx7)"; fi +assert_contains "changed literals: the rest elided" "$ctx7" $'\n- ...' +assert_not_contains "changed literals: the sixth is not named" "$ctx7" "sum(6, 0)" # A removed test block is denied with the rule on. run Write "$REPO/src/sum.test.ts" "$(jq -cn --arg c "$(sed '/^test(.adds zero/,$d' "$REPO/src/sum.test.ts")" '{content: $c}')" diff --git a/plugins/testing/skills/audit/scripts/cant-fail-scan.sh b/plugins/testing/skills/audit/scripts/cant-fail-scan.sh index e7719cd214..c3f7198ffa 100755 --- a/plugins/testing/skills/audit/scripts/cant-fail-scan.sh +++ b/plugins/testing/skills/audit/scripts/cant-fail-scan.sh @@ -117,6 +117,9 @@ # ` per examined test block (in --lines scope); # the ordinal tells same-named blocks apart and is the block's # identity with its name, since line numbers drift +# --brief with the default mode: finding lines with no threshold or +# Action, then one `action [rule-] ` line per +# distinct Action; the form the hooks pass to the agent # --help # # Scan-root resolution: $CANT_FAIL_SCAN_ROOT (sanctioned operator lever, not a @@ -143,7 +146,7 @@ usage() { cat <<'EOF' cant-fail-scan.sh — detect tests that cannot fail. -Usage: cant-fail-scan.sh [--file [--lines ]] [--blocks | --check [--strict] | --findings | --count | --help] +Usage: cant-fail-scan.sh [--file [--lines ]] [[--blocks] [--brief] | --check [--strict] | --findings | --count | --help] cant-fail-scan.sh --file --inventory [--inventory ...] (no arg) print one finding line per detection, then the coverage block; exit 0 (2 on scan gap) @@ -160,6 +163,8 @@ Usage: cant-fail-scan.sh [--file [--lines ]] [--blocks | --check [- (a list like 12,20-24), for an edit hook scoped to what the edit changed --blocks also print `block :- ` per examined test block (within --lines); the ordinal tells same-named blocks apart + --brief finding lines with no threshold or Action, then one `action [rule-] ` + line per distinct Action --inventory with --file, repeatable: no rules; for the n-th , one record per line `ntest|assertion|skip`, one per equality with a @@ -187,6 +192,7 @@ FILE="" LINES="" INV_TEXTS=() list_blocks=0 +brief=0 blk_lines=() while [[ $# -gt 0 ]]; do case "$1" in @@ -195,6 +201,7 @@ while [[ $# -gt 0 ]]; do exit 0 ;; --blocks) list_blocks=1 ;; + --brief) brief=1 ;; --check) mode="check" ;; --findings) mode="findings" ;; --count) mode="count" ;; @@ -254,8 +261,8 @@ LOADER="$SCRIPT_DIR/adapter-load.awk" require_readable "$LOADER" 'adapter loader' ADAPTER_DIR="$SCRIPT_DIR/../adapters" -if ((list_blocks)) && [[ "$mode" != report ]]; then - printf 'ERROR: --blocks lists blocks in the report mode only\n' >&2 +if ((list_blocks || brief)) && [[ "$mode" != report ]]; then + printf 'ERROR: --blocks and --brief apply to the report mode only\n' >&2 exit 2 fi if [[ (-n "$LINES" || "$mode" == inventory) && -z "$FILE" ]]; then @@ -904,7 +911,7 @@ action_of() { case "$1" in inert-assertion | constant-restatement | source-text-read | conditional-assertion | recomputed-derived | snapshot-only | weak-oracle) return 0 ;; zero-assertion) - printf 'Repair, not pruning: add an assertion on the observable behavior this test exercises; today it passes vacuously and its coverage claim is false.' + printf 'Add an assertion on the observable behavior this test exercises.' ;; recomputed-expectation) printf 'State the expected value independently (a literal or precomputed constant) instead of recomputing it with the same expression, so the assertion can discriminate. A determinism contract, f(x) == f(x) on purpose, records that with a cant-fail-ok: annotation.' @@ -1137,7 +1144,21 @@ emit_findings_file() { } print_findings_lines() { - local i + local i a + if ((brief)); then + # One Action per distinct rule and text: inert-assertion's differs by language. + local -A seen=() + for i in ${f_rule[@]+"${!f_rule[@]}"}; do + printf 'finding [rule-%s] %s: %s\n' "${f_rule[$i]}" "${f_loc[$i]}" "${f_detail[$i]}" + done + for i in ${f_rule[@]+"${!f_rule[@]}"}; do + a="$(action_of "${f_rule[$i]}" "${f_lang[$i]}")" + [[ -n "$a" && -z "${seen["${f_rule[$i]}|$a"]:-}" ]] || continue + seen["${f_rule[$i]}|$a"]=1 + printf 'action [rule-%s] %s\n' "${f_rule[$i]}" "$a" + done + return 0 + fi for i in ${f_rule[@]+"${!f_rule[@]}"}; do printf 'finding [%s] %s: %s (%s). Action: %s\n' \ "$(rule_id "${f_rule[$i]}")" "${f_loc[$i]}" "${f_detail[$i]}" \ diff --git a/plugins/testing/skills/audit/scripts/cant-fail-scan.test.sh b/plugins/testing/skills/audit/scripts/cant-fail-scan.test.sh index 51cdb86880..e058505ed5 100755 --- a/plugins/testing/skills/audit/scripts/cant-fail-scan.test.sh +++ b/plugins/testing/skills/audit/scripts/cant-fail-scan.test.sh @@ -810,7 +810,18 @@ assert_contains "--blocks keeps the findings" "$out" "cant-fail-js.test.js:11: t assert_matches "--blocks lists blocks beside findings" "$out" '^block plugins/testing/skills/audit/evals/fixtures/positive/cant-fail-js\.test\.js:11-13 1 adds numbers$' run_file --file "$B/dup.test.ts" --blocks --check assert_exit "--blocks works only in the report mode (exit 2)" 2 "$rc" -assert_contains "--blocks outside the report mode says so" "$out" "--blocks lists blocks in the report mode only" +assert_contains "--blocks outside the report mode says so" "$out" "--blocks and --brief apply to the report mode only" + +# --brief: the hooks' form. Three recomputed-expectation findings carry one Action. +run_file --file "$FIX/positive/cant-fail-js.test.js" --brief +assert_exit "--brief completes (exit 0)" 0 "$rc" +assert_finding_count "--brief keeps every finding" 5 +assert_matches "--brief finding line has no threshold or Action" "$out" \ + "^finding \[rule-zero-assertion\] plugins/testing/skills/audit/evals/fixtures/positive/cant-fail-js\.test\.js:11: test 'adds numbers' has 0 assertion tokens$" +if [[ "$(count_lines "$out" '^action \[rule-recomputed-expectation\] ')" == 1 ]]; then pass "--brief: one Action for three findings of a rule"; else fail "--brief: one Action for three findings of a rule" "$out"; fi +if [[ "$(count_lines "$out" '^action \[')" == 3 ]]; then pass "--brief: one Action per distinct rule"; else fail "--brief: one Action per distinct rule" "$out"; fi +run_file --file "$FIX/positive/cant-fail-js.test.js" --brief --check +assert_exit "--brief works only in the report mode (exit 2)" 2 "$rc" printf '%s\n' "public class T {" " [Fact]" " public void Adds()" " {" " Assert.Equal(3, Sum(1, 2));" " }" \ " [Fact]" " public void Subs()" " => Assert.Equal(1, Sub(3, 2));" "}" >"$B/BlocksTests.cs" @@ -1472,6 +1483,10 @@ remedy() { printf '%s\n' "$out" | sed -n "s|^finding \[testing/audit/$2\].*Action: ||p" | head -1 } C="$TMP_ROOT/corpus" +a="$(remedy "$FIX/positive/cant-fail-js.test.js" rule-zero-assertion)" +check_eq() { if [[ "$2" == "$3" ]]; then pass "$1"; else fail "$1" "got: $2"; fi; } +check_eq "zero-assertion remedy is one sentence, no rationale tail" "$a" \ + "Add an assertion on the observable behavior this test exercises." a="$(remedy "$C/js-playwright/bad/playwright-saved-unawaited.spec.ts" rule-inert-assertion)" assert_contains "inert remedy (js) says to await the matcher" "$a" "await (or return) the async matcher" assert_not_contains "inert remedy (js) offers no Python tuple advice" "$a" "tuple" diff --git a/plugins/testing/skills/setup/SKILL.md b/plugins/testing/skills/setup/SKILL.md index 7e81d6060f..4f18655986 100644 --- a/plugins/testing/skills/setup/SKILL.md +++ b/plugins/testing/skills/setup/SKILL.md @@ -54,8 +54,8 @@ rules: ``` The opt-in `test-weaken` hook runs before a Write or Edit to a test file and names what it removes: -test blocks, assertions, a changed expected value, or an added skip. By default it only asks the -agent for its reason. With `test-weaken-block: error`, an added skip or a removed test block is +test blocks, assertions, a changed expected value, or an added skip. By default it only tells the +agent. With `test-weaken-block: error`, an added skip or a removed test block is denied until the edit carries a `test-change: ` comment; removed assertions and changed expected values are never denied. The agent can write that marker itself: it makes the reason visible to reviewers, it does not prove the reason.