From 2869c9f5ecef688d7aaeb7728eba11c92c3f982e Mon Sep 17 00:00:00 2001 From: Dan Mordechay Date: Sun, 18 Oct 2026 20:08:32 +0300 Subject: [PATCH 1/5] ldd: scope the R9 gate to the review and resolve doc symbols in one pass A scoped review ran the R9 gate over the whole repository and then dropped most of its lines. On the wekapp monorepo that took about 90 minutes for a 6-file diff. The same review now takes 48 seconds. - check-repo-brain.sh takes --scope and --words. A scoped run checks only the docs in scope: the touched docs, the docs that name a scope file, and the docs that cite a symbol the diff removes. It checks code edges of the scope files only and skips Q3. With no doc in scope it skips the doc checks. - Symbol resolution reads the repository once into a word set. In a git work tree it lists files through git ls-files (tracked plus untracked, never ignored). Elsewhere find prunes bazel-*, node_modules and other build outputs. A multi-pattern grep -F -w was the slow step. - The declaration set is built only when a token needs it. - ldd-scope.sh writes docs.txt (the touched docs) and test-re.txt (the --test-re pattern). ldd-detect.sh reads test-re.txt, and a --glob or --test-re that differs from the bundle is exit 2. - ldd-detect.sh exits 2 when the scope has files but none matches the source glob, instead of an all-zero table. - ldd-detect.sh --gate-budget stops the gate as a process group on expiry and counts its questions "inconclusive", never 0. - The python and generic adapters pass find(1) pruning as an array. The unquoted $(py_prune) expanded ./.git/* in a git repository, so the gate found no code and skipped every code<->docs check. Co-Authored-By: Claude Opus 5.5 --- core/scripts/check-repo-brain.sh | 225 +++++++++++++---- core/scripts/check-repo-brain_test.sh | 69 +++++ core/scripts/ldd-detect.sh | 141 +++++++++-- core/scripts/ldd-detect_test.sh | 76 +++++- core/scripts/ldd-scope.sh | 9 + core/skills/pre-commit-review/SKILL.md | 15 +- docs/token-budget.md | 10 +- .../scripts/check-repo-brain.sh | 225 +++++++++++++---- .../scripts/check-repo-brain_test.sh | 69 +++++ .../scripts/ldd-detect.sh | 141 +++++++++-- .../scripts/ldd-detect_test.sh | 76 +++++- .../scripts/ldd-scope.sh | 9 + .../skills/pre-commit-review/SKILL.md | 15 +- lang/generic/scripts/repo-brain-adapter.sh | 14 +- .../scripts/check-repo-brain_test.sh | 69 +++++ lang/python/scripts/repo-brain-adapter.sh | 14 +- .../scripts/check-repo-brain.sh | 239 ++++++++++++++---- .../scripts/check-repo-brain_test.sh | 69 +++++ .../scripts/ldd-detect.sh | 141 +++++++++-- .../scripts/ldd-detect_test.sh | 76 +++++- .../scripts/ldd-scope.sh | 9 + .../skills/pre-commit-review/SKILL.md | 15 +- .../scripts/check-repo-brain.sh | 239 ++++++++++++++---- .../scripts/check-repo-brain_test.sh | 69 +++++ .../scripts/ldd-detect.sh | 141 +++++++++-- .../scripts/ldd-detect_test.sh | 76 +++++- .../scripts/ldd-scope.sh | 9 + .../skills/pre-commit-review/SKILL.md | 15 +- .../scripts/check-repo-brain.sh | 225 +++++++++++++---- .../scripts/check-repo-brain_test.sh | 69 +++++ .../scripts/ldd-detect.sh | 141 +++++++++-- .../scripts/ldd-detect_test.sh | 76 +++++- .../scripts/ldd-scope.sh | 9 + .../skills/pre-commit-review/SKILL.md | 15 +- 34 files changed, 2388 insertions(+), 422 deletions(-) diff --git a/core/scripts/check-repo-brain.sh b/core/scripts/check-repo-brain.sh index 02793f71..8aca0820 100755 --- a/core/scripts/check-repo-brain.sh +++ b/core/scripts/check-repo-brain.sh @@ -8,10 +8,22 @@ # in-repo copy). Installed into target repos by the documentation skill's # BOOTSTRAP pass (/wire-repo-brain). # -# Usage: bash scripts/check-repo-brain.sh [--fix] [repo-root] (default: cwd) +# Usage: bash scripts/check-repo-brain.sh [--fix] [--scope [--words ]] [repo-root] +# (repo-root default: cwd) # CI: one line — bash scripts/check-repo-brain.sh # --fix: rewrite drifted index lines from each target doc's `description` # (the one mechanical repair; everything else stays report-only) +# --scope a review's scope: one path per line, relative to repo-root +# (code files and docs, deleted ones included). The gate then checks +# only the docs in scope: the listed docs, the docs whose text names a +# listed file's basename, and the docs that cite a word of --words. +# Code edges are checked for the listed code files only, Q3 is skipped +# (root wiring is the repository's state, not a diff's), and with no +# doc in scope the doc checks are skipped. One stdout line says which: +# "check-repo-brain: scope — doc(s) to check" or +# "check-repo-brain: scope — no doc in scope; doc checks skipped". +# --words identifiers the scoped change removes, one per line; a doc +# that cites one of them in backticks is in scope # # Doc roots are discovered at the repo root AND at every sub-project (a # directory holding the language's project marker — see the adapter below), @@ -58,8 +70,11 @@ # against its declaring package or receiver — never against a same-named # member elsewhere. A token missing from the set still resolves when it # appears as a whole word in any non-markdown repo file (config keys, -# alert names, test helpers). A `pkg.Sym` whose package is not declared in -# this repo is external (stdlib, dependencies) and exempt. +# alert names, test helpers): in a git work tree the tracked and +# untracked-but-not-ignored files; elsewhere every file outside the +# dependency and build-output directories (bazel-*, node_modules, …). +# A `pkg.Sym` whose package is not declared in this repo is external +# (stdlib, dependencies) and exempt. # - lines carrying the ⚠️ stale flag or a *(planned)* marker are exempt from # symbol resolution and the description copy check (R9 Q2/Q7 exemptions); # the file:line ban has no exemption beyond URL spans, fenced code blocks, @@ -72,22 +87,37 @@ # 2 usage error, or an internal scanner failure (a failed scan is # inconclusive, never silently clean) # -# Uses only POSIX-portable tools: find, grep, sed, awk, head, sort, wc. No jq/python. +# Uses only POSIX-portable tools: find, grep, sed, awk, head, sort, wc, tr, xargs, +# plus git when the repository is a git work tree. No jq/python. # awk programs avoid interval expressions ({m,n}) — mawk, Debian's default, rejects them. set -u FIX=0 -if [[ "${1:-}" == "--fix" ]]; then - FIX=1 - shift -fi +SCOPE_LIST="" +WORDS_LIST="" +abs_list() { # -> absolute path of an existing file, or exit 2 + [[ -f "$1" ]] || { echo "check-repo-brain: no such list: $1" >&2; exit 2; } + printf '%s/%s\n' "$(cd "$(dirname "$1")" && pwd)" "$(basename "$1")" +} +while (( $# > 0 )); do + case "$1" in + --fix) FIX=1; shift ;; + --scope) SCOPE_LIST=$(abs_list "${2:-}") || exit 2; shift 2 ;; + --words) WORDS_LIST=$(abs_list "${2:-}") || exit 2; shift 2 ;; + --*) echo "check-repo-brain: unknown option: $1" >&2; exit 2 ;; + *) break ;; + esac +done +[[ -n "$WORDS_LIST" && -z "$SCOPE_LIST" ]] && { echo "check-repo-brain: --words needs --scope" >&2; exit 2; } REPO_ROOT="${1:-$(pwd)}" if [[ ! -d "$REPO_ROOT" ]]; then echo "check-repo-brain: not a directory: $REPO_ROOT" >&2 exit 2 fi cd "$REPO_ROOT" || exit 2 +SCOPED=0 +[[ -n "$SCOPE_LIST" ]] && SCOPED=1 {{include "scripts/repo-brain-adapter.sh"}} @@ -204,22 +234,65 @@ lifecycle_stale() { # -> reason when frontmatter marks the doc stale ('' have_code=0 lang_has_code && have_code=1 -# ---------- declaration set: built once, queried per token ---------- -DECLS="" PKGS="" -if (( have_code )); then - DECLS=$(mktemp) PKGS=$(mktemp) DECL_ALL=$(mktemp) - trap 'rm -f "$DECLS" "$PKGS"' EXIT - if ! ( set -o pipefail; lang_declarations | sort -u > "$DECL_ALL" ); then +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +# ---------- declaration set: built once, on the first token to resolve ---------- +DECLS="$WORK/decls" PKGS="$WORK/pkgs" +decls_built=0 +build_declarations() { + (( decls_built )) && return 0 + decls_built=1 + if ! ( set -o pipefail; lang_declarations | sort -u > "$WORK/decl-all" ); then echo "check-repo-brain: internal scanner error — the declaration scan failed; treating the run as inconclusive" >&2 exit 2 fi - grep '^pkg:' "$DECL_ALL" | sed 's/^pkg://' > "$PKGS" - grep -v '^pkg:' "$DECL_ALL" > "$DECLS" - rm -f "$DECL_ALL" -fi + grep '^pkg:' "$WORK/decl-all" | sed 's/^pkg://' > "$PKGS" + grep -v '^pkg:' "$WORK/decl-all" > "$DECLS" + rm -f "$WORK/decl-all" +} is_repo_pkg() { grep -qxF "$1" "$PKGS" 2>/dev/null; } +# repo_files — the repository's non-markdown files, NUL-separated. A git work +# tree lists them through git (tracked files plus untracked ones git does not +# ignore), which never walks build outputs or symlinked caches; elsewhere find +# skips the directories that hold dependencies and build outputs. +repo_files() { + if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then + git ls-files -z --cached --others --exclude-standard -- . \ + ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**' 2>/dev/null && return 0 + fi + find . \( -name .git -o -name vendor -o -name node_modules -o -name 'bazel-*' -o -name .venv \ + -o -name venv -o -name __pycache__ -o -name .tox -o -name dist -o -name target \) -prune \ + -o -type f ! -name '*.md' -print0 +} + +# resolve_words — the tokens that occur as a whole word in any +# non-markdown repository file, one per line. The repository is read once +# into words (runs of [A-Za-z0-9_.]); a dotted token such as `Type.Method` +# matches any run of whole dot-separated parts, as grep -w would match it. +# One pass over the text is linear in its size, where a fixed-string grep for +# hundreds of tokens is not. +resolve_words() { + repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.' '\n' \ + | LC_ALL=C awk -v list="$1" ' + BEGIN { while ((getline t < list) > 0) if (t != "") want[t] = 1 } + seen[$0]++ { next } + { + n = split($0, p, ".") + for (i = 1; i <= n; i++) { + s = p[i] + if (s in want) { print s; delete want[s] } + for (j = i + 1; j <= n; j++) { + s = s "." p[j] + if (s in want) { print s; delete want[s] } + } + } + }' + return 0 +} + # One fence-aware awk pass per bundle extracts everything Q2's doc scan needs: # P — a file:line citation outside fences/URLs/globs # S — a backticked symbol-shaped token to resolve @@ -258,8 +331,53 @@ FNR == 1 { fence = 0 } } ' -# ---------- Q2: code→docs edges (repo-wide; resolved from repo root, then the -# citing file's own sub-project) ---------- +# ---------- the scope: which docs a scoped run checks ---------- +# A doc is in scope when the scope lists it, when its text names the basename +# of a listed file as a whole word (a link to a touched doc or code file), or +# when it cites in backticks a word the change removes (a symbol the diff +# renames or deletes). +# Every list below holds relative paths as find(1) prints them under a root. +SCOPE_DOCS="$WORK/scope-docs" +SCOPE_PATHS="$WORK/scope-paths" +if (( SCOPED )); then + sed -e 's|^\./||' -e '/^$/d' "$SCOPE_LIST" | LC_ALL=C sort -u > "$SCOPE_PATHS" + for r in "${ROOTS[@]}"; do find "$r" -type f -name '*.md'; done | LC_ALL=C sort -u > "$WORK/all-docs" + { + grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" + sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$WORK/basenames" + if [[ -s "$WORK/basenames" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$WORK/basenames" 2>/dev/null + fi + if [[ -n "$WORDS_LIST" && -s "$WORDS_LIST" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" \ + | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ + -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" "$DOCSCAN_AWK" \ + | awk -F'\t' 'NR == FNR { w[$0] = 1; next } + $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in w) || (m in w)) print $2 }' "$WORDS_LIST" - + fi + } | LC_ALL=C sort -u > "$SCOPE_DOCS" + if [[ -s "$SCOPE_DOCS" ]]; then + echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" + else + echo "check-repo-brain: scope — no doc in scope; doc checks skipped" + fi +fi + +# bundle_docs [name] — the docs a run checks under one root: every +# *.md (or every file of that name), limited to the scope on a scoped run +bundle_docs() { + local name="${2:-*.md}" + if (( SCOPED )); then + find "$1" -type f -name "$name" | grep -xF -f "$SCOPE_DOCS" + else + find "$1" -type f -name "$name" + fi + return 0 +} + +# ---------- Q2: code→docs edges (repo-wide, or the scope's code files on a +# scoped run; resolved from repo root, then the citing file's own +# sub-project) ---------- docroot_for_file() { # -> docroot of the longest matching project dir local f="${1#./}" best="" i for i in "${!PROJS[@]}"; do @@ -283,7 +401,13 @@ if (( have_code )); then CUR_DOCROOT=$(docroot_for_file "$file") fail "[Q2] $file:$line — code edge points at missing $target" fi - done < <(lang_code_edges) + done < <(if (( SCOPED )); then + lang_code_edges | awk -v list="$SCOPE_PATHS" ' + BEGIN { while ((getline l < list) > 0) s[l] = 1 } + { f = substr($0, 1, index($0, ":") - 1); sub(/^\.\//, "", f); if (f in s) print }' + else + lang_code_edges + fi) fi # ---------- per-bundle checks ---------- @@ -293,6 +417,8 @@ check_bundle() { # local root_index="$docroot/index.md" local root_index_c root_index_c=$(canon "$root_index") + # a scoped run skips a bundle that holds no doc in scope + (( SCOPED )) && [[ -z "$(bundle_docs "$docroot")" ]] && return 0 # --- Q1: transitive reachability from the bundle's root index --- local reachable="" visited="" queue=("$root_index") @@ -321,7 +447,7 @@ check_bundle() { # case "$reachable" in *"$c"$'\n'*) ;; *) fail "[Q1] $doc — orphan: not reachable from $root_index" ;; esac - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q2: every doc link resolves (a missing parent directory is just as # broken as a missing file — only URLs/anchors are exempt) --- @@ -336,13 +462,13 @@ check_bundle() { # fail "[Q2] $md — link target does not exist: $t" fi done < <(grep -oE '\]\([^)]+\)' "$md" 2>/dev/null) - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q2: doc scan — file:line ban + docs→code symbol resolution. # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). --- local scan; scan=$(mktemp) if ! ( set -o pipefail - find "$docroot" -type f -name '*.md' -print0 \ + bundle_docs "$docroot" | tr '\n' '\0' \ | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" \ "$DOCSCAN_AWK" > "$scan" ); then @@ -357,6 +483,7 @@ check_bundle() { # if (( have_code )) && grep -q $'^S\t' "$scan"; then local toks check members unres bad toks=$(mktemp) check=$(mktemp) members=$(mktemp) unres=$(mktemp) bad=$(mktemp) + build_declarations grep $'^S\t' "$scan" | cut -f4 | sort -u > "$toks" # tokens to resolve (external pkg.Sym exempt). A qualified token resolves # as the PAIR itself — the declaration set carries pkg.Ident / Type.Method @@ -374,12 +501,11 @@ check_bundle() { # esac done < "$toks" > "$check" cut -f2 "$check" | sort -u > "$members" - grep -vxF -f "$DECLS" "$members" > "$unres" || true + grep -vxF -f "$DECLS" "$members" | grep -v '^$' > "$unres" || true if [[ -s "$unres" ]]; then - # ONE repo-wide word grep for the whole unresolved batch + # ONE pass over the repository for the whole unresolved batch local found; found=$(mktemp) - grep -rIhoFw --exclude-dir=vendor --exclude-dir=.git --exclude='*.md' \ - -f "$unres" . 2>/dev/null | sort -u > "$found" + resolve_words "$unres" | sort -u > "$found" grep -vxF -f "$found" "$unres" > "$bad" || true rm -f "$found" fi @@ -397,24 +523,27 @@ check_bundle() { # fi rm -f "$scan" - # --- Q3: root wiring (exact path; sub-roots may ride the repo-root index) --- - local rel="$docroot" - [[ "$proj" != "." ]] && rel="${docroot#$proj/}" - local claude="CLAUDE.md" agents="AGENTS.md" - [[ "$proj" != "." ]] && { claude="$proj/CLAUDE.md"; agents="$proj/AGENTS.md"; } - local wired_claude=0 wired_agents=0 - [[ -f "$claude" ]] && grep -qF -- "$rel/index.md" "$claude" && wired_claude=1 - [[ -f "$agents" ]] && grep -qF -- "$rel/index.md" "$agents" && wired_agents=1 - if (( ! wired_claude && ! wired_agents )); then - local via_root=0 - if [[ "$proj" != "." && -n "$ROOT_BUNDLE" ]]; then - grep -rqF -- "$docroot/index.md" "$ROOT_BUNDLE" --include='*.md' 2>/dev/null && via_root=1 - fi - if (( ! via_root )); then - fail "[Q3] $proj — neither $claude nor $agents references $rel/index.md" + # --- Q3: root wiring (exact path; sub-roots may ride the repo-root index); + # a scoped run leaves it to a whole-repository run --- + if (( ! SCOPED )); then + local rel="$docroot" + [[ "$proj" != "." ]] && rel="${docroot#$proj/}" + local claude="CLAUDE.md" agents="AGENTS.md" + [[ "$proj" != "." ]] && { claude="$proj/CLAUDE.md"; agents="$proj/AGENTS.md"; } + local wired_claude=0 wired_agents=0 + [[ -f "$claude" ]] && grep -qF -- "$rel/index.md" "$claude" && wired_claude=1 + [[ -f "$agents" ]] && grep -qF -- "$rel/index.md" "$agents" && wired_agents=1 + if (( ! wired_claude && ! wired_agents )); then + local via_root=0 + if [[ "$proj" != "." && -n "$ROOT_BUNDLE" ]]; then + grep -rqF -- "$docroot/index.md" "$ROOT_BUNDLE" --include='*.md' 2>/dev/null && via_root=1 + fi + if (( ! via_root )); then + fail "[Q3] $proj — neither $claude nor $agents references $rel/index.md" + fi + elif (( ! wired_agents )); then + note "[Q3] $agents lacks the $rel/index.md routing reference — AGENTS.md-reading tools start blind" fi - elif (( ! wired_agents )); then - note "[Q3] $agents lacks the $rel/index.md routing reference — AGENTS.md-reading tools start blind" fi # --- Q7: bundle contract --- @@ -479,7 +608,7 @@ check_bundle() { # else fail "[Q7] $md — frontmatter missing 'description:'" fi - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q7: drift check — index line text == target's description (⚠️ exempt; # --fix rewrites drifted lines after the read loop, never during it) --- @@ -517,12 +646,12 @@ check_bundle() { # echo " fixed: $idx:$n — index line rewritten from its target's description" done <<< "$fixes" fi - done < <(find "$docroot" -type f -name 'index.md') + done < <(bundle_docs "$docroot" index.md) # --- Q7: no log.md --- while IFS= read -r lg; do fail "[Q7] $lg — log.md is reserved for change history; docs describe current behavior" - done < <(find "$docroot" -type f -name 'log.md') + done < <(bundle_docs "$docroot" log.md) } for i in "${!PROJS[@]}"; do diff --git a/core/scripts/check-repo-brain_test.sh b/core/scripts/check-repo-brain_test.sh index 30fadd5f..14a9b66b 100755 --- a/core/scripts/check-repo-brain_test.sh +++ b/core/scripts/check-repo-brain_test.sh @@ -545,6 +545,75 @@ run_gate expect_exit 0 && expect_not "does not resolve" && expect_has "check-repo-brain: clean" && ok finish +# --- scoped runs (--scope, --words) and the symbol resolver --- +LISTS=$(mktemp -d) # the scope and word lists live outside the repository they describe +begin "--scope: a code-only scope that no doc cites skips the doc checks" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here." +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "scope — no doc in scope; doc checks skipped" && expect_not "orphan" && ok +finish + +begin "--scope: a listed doc is checked, an unlisted orphan and the root wiring are not" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here."; mk_doc other "another unlinked doc" "Nothing." +rm "$REPO/CLAUDE.md" "$REPO/AGENTS.md" +printf '%s\n' "$FX_CODE_FILE" docs/orphan.md > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] docs/orphan.md — orphan" \ + && expect_not "docs/other.md" && expect_not "[Q3]" && ok +finish + +begin "--scope: a doc that names a scope file is checked" +mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../$FX_CODE_FILE)." +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +finish + +begin "--scope --words: a doc that cites a removed symbol is checked" +mk_conformant +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +append_index "- [ghost.md](ghost.md) — a ghost symbol" +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "no doc in scope" \ + && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ + && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok +finish + +begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" +mk_conformant +printf 'build/\n' > "$REPO/.gitignore" +mkdir -p "$REPO/build"; printf 'IgnoredOnlyName\n' > "$REPO/build/gen.yaml" +printf 'TrackedName\n' > "$REPO/tracked.yaml" +mk_doc names "alert names" "Fires \`TrackedName\`, \`UntrackedName\` and \`IgnoredOnlyName\`." +append_index "- [names.md](names.md) — alert names" +(cd "$REPO" && git init -q && git add -A && git -c user.name=t -c user.email=t@t commit -qm init) +printf 'UntrackedName\n' > "$REPO/untracked.yaml" +run_gate +expect_exit 1 && expect_has "\`IgnoredOnlyName\` does not resolve" \ + && expect_not "\`TrackedName\` does not resolve" && expect_not "\`UntrackedName\` does not resolve" && ok +finish + +begin "Q2 outside git a token does not resolve from a bazel-* output directory" +mk_conformant +mkdir -p "$REPO/bazel-out"; printf 'BazelOnlyName\n' > "$REPO/bazel-out/gen.yaml" +mk_doc bazel "bazel names" "Fires \`BazelOnlyName\`." +append_index "- [bazel.md](bazel.md) — bazel names" +run_gate +expect_exit 1 && expect_has "\`BazelOnlyName\` does not resolve" && ok +finish + +begin "Q2 a dotted token resolves on whole dot-separated words only" +mk_conformant +printf 'limits:\n key: Cfg.MaxRetries.value\n other: Cfg.MaxRetriesX\n' > "$REPO/limits.yaml" +mk_doc dotted "dotted names" "Reads \`Cfg.MaxRetries\`; \`Cfg.MaxRetri\` is a typo." +append_index "- [dotted.md](dotted.md) — dotted names" +run_gate +expect_exit 1 && expect_has "\`Cfg.MaxRetri\` does not resolve" && expect_not "\`Cfg.MaxRetries\` does not resolve" && ok +finish +rm -rf "$LISTS" + } # run_row_cases for ROW in $FX_ROWS; do diff --git a/core/scripts/ldd-detect.sh b/core/scripts/ldd-detect.sh index 25647f5b..41f089cf 100755 --- a/core/scripts/ldd-detect.sh +++ b/core/scripts/ldd-detect.sh @@ -25,7 +25,14 @@ # --plan print the parsed detect lines and exit; nothing runs # --glob the scope is one group of a language the table # does not know, matched by this glob -# --test-re test-file pattern over relative paths, same case +# --test-re test-file pattern over relative paths, same case. +# A bundle carries the pattern ldd-scope.sh used +# (test-re.txt), and a custom group its glob +# (language.txt): the bundle's values win, and a +# --glob or --test-re that differs is a usage error +# --gate-budget wall-clock seconds the R9 gate may run (default: no +# limit); on expiry the gate is stopped and its +# questions are counted "inconclusive", never 0 # # The scope's language is the bundle's language.txt; without it, the files in # files.txt tell it by extension, and a list that spans several languages is @@ -39,10 +46,14 @@ # Detect-path: `` over the scope's relative paths (layer directories, # role-named packages); a hit's line is 0 # Detect-gate: Q the R9 gate's [Q] report lines, from one run of -# check-repo-brain.sh beside this script over --root; on a -# scoped bundle (no dirs.txt) only the lines whose file is in -# the scope — the gate's repository-wide findings belong to -# a whole-repository review +# check-repo-brain.sh beside this script over --root. On a +# scoped bundle (no dirs.txt) the gate runs with --scope: it +# checks only the bundle's docs (docs.txt), the docs that +# link to a scope file and the docs that cite a symbol the +# diff removes, the code edges of the scope's code files, +# and no root wiring; with no such doc its doc checks are +# skipped. The repository-wide findings belong to a +# whole-repository review # Detect: judgment no mechanical lead; the hunter runs the prose # A question with no detect line, two of them, or a malformed one is an error: # the generator's lint-core keeps the rule files honest, and this script refuses @@ -53,15 +64,18 @@ # the overflow row of a capped question: file "-", line 0, # excerpt "+ more hit(s) not listed" # hits-all.tsv the same columns, every hit, no cap and no overflow row -# counts.tsv rule question kind hits (judgment rows carry "-") +# counts.tsv rule question kind hits (judgment rows carry "-", gate +# rows "inconclusive" when the gate ran out of --gate-budget) # the last row is SUPPRESS - grep : suppression directives # on the diff's added lines when the bundle has a diff.patch, # else over every scope file # Two runs over one tree write byte-identical counts.tsv. # # Exit codes: 0 tables written · 2 usage error, a detect line that does not -# parse, a pattern grep rejects, or a gate run that failed -# (inconclusive, never silently clean) +# parse, a pattern grep rejects, a gate run that failed +# (inconclusive, never silently clean), or a non-empty scope in +# which no file matches the source glob (a glob mismatch, never +# an empty table) # # Uses only POSIX-portable tools: grep, sed, awk, sort, wc, head. GNU grep's # \b and \s are assumed, as the rules already assume them. @@ -77,6 +91,7 @@ CAP=40 PLAN=0 OPT_GLOB="" OPT_TEST_RE="" +GATE_BUDGET=0 BUNDLE="" usage() { sed -n '2,/^$/p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; } @@ -94,12 +109,14 @@ while (( $# > 0 )); do --plan) PLAN=1; shift ;; --glob) value "$1" $#; OPT_GLOB="$2"; shift 2 ;; --test-re) value "$1" $#; OPT_TEST_RE="$2"; shift 2 ;; + --gate-budget) value "$1" $#; GATE_BUDGET="$2"; shift 2 ;; -h|--help) usage ;; --*) die "unknown option: $1" ;; *) [[ -n "$BUNDLE" ]] && die "one bundle directory, got also: $1"; BUNDLE="$1"; shift ;; esac done [[ "$CAP" =~ ^[0-9]+$ ]] || die "--cap takes a number, got: $CAP" +[[ "$GATE_BUDGET" =~ ^[0-9]+$ ]] || die "--gate-budget takes a number of seconds, got: $GATE_BUDGET" [[ -d "$RULES_DIR" ]] || die "no rules directory: $RULES_DIR" RULES_DIR=$(cd "$RULES_DIR" && pwd) if (( ! PLAN )); then @@ -136,6 +153,7 @@ if (( ! PLAN )) && [[ -f "$BUNDLE/groups.txt" ]]; then fi args=(--root "$ROOT" --cap "$CAP") [[ -n "$OPT_TEST_RE" ]] && args+=(--test-re "$OPT_TEST_RE") + (( GATE_BUDGET > 0 )) && args+=(--gate-budget "$GATE_BUDGET") bash "$script" "${args[@]}" "$bundle" || { failed=1; echo "$SCRIPT_NAME: the $id group's detection pass failed; it is inconclusive" >&2; } echo done < "$BUNDLE/groups.txt" @@ -150,7 +168,9 @@ LANG_ID="" if (( ! PLAN )); then if [[ -f "$BUNDLE/language.txt" ]]; then IFS=$'\t' read -r LANG_ID glob < "$BUNDLE/language.txt" - [[ "$LANG_ID" == "custom" && -z "$OPT_GLOB" ]] && OPT_GLOB="$glob" + [[ "$LANG_ID" == "custom" && -n "$OPT_GLOB" && "$OPT_GLOB" != "$glob" ]] \ + && die "--glob '$OPT_GLOB' differs from the bundle's glob '$glob' (ldd-scope.sh wrote it; drop --glob)" + [[ "$LANG_ID" == "custom" ]] && OPT_GLOB="$glob" else ids="" while IFS= read -r line; do @@ -165,6 +185,14 @@ if (( ! PLAN )); then [[ "$ids" == *" "* ]] && die "the scope list spans several languages ($ids); ldd-scope.sh writes one bundle per language" LANG_ID="$ids" fi + # the test-file pattern ldd-scope.sh used, so the two scripts cannot disagree + # on what a test file is + if [[ -f "$BUNDLE/test-re.txt" ]]; then + b_test_re=$(cat "$BUNDLE/test-re.txt") + [[ -n "$OPT_TEST_RE" && "$OPT_TEST_RE" != "$b_test_re" ]] \ + && die "--test-re '$OPT_TEST_RE' differs from the bundle's '$b_test_re' (ldd-scope.sh wrote it; drop --test-re)" + OPT_TEST_RE="$b_test_re" + fi lang_configure "$LANG_ID" || die "this plugin has no language block for $LANG_ID" fi @@ -210,7 +238,7 @@ END { if (insec) flush() } ' plan_file=$(mktemp) -trap 'rm -f "$plan_file" "${tmp_hits:-}" "${tmp_hits_all:-}" "${tmp_counts:-}" "${gate_out:-}"' EXIT +trap 'rm -f "$plan_file" "${tmp_hits:-}" "${tmp_hits_all:-}" "${tmp_counts:-}" "${gate_out:-}" "${gate_scope:-}" "${gate_words:-}"' EXIT rule_files=$(find "$RULES_DIR" -maxdepth 1 -name 'R*.md' | awk -F/ '{ n = $NF; sub(/^R/, "", n); sub(/-.*$/, "", n); print n "\t" $0 }' | sort -n | cut -f2) [[ -n "$rule_files" ]] || die "no rule files (R*.md) under $RULES_DIR" while IFS= read -r f; do @@ -236,11 +264,13 @@ fi SRC_FILES=() TEST_FILES=() ALL_FILES=() +n_present=0 while IFS= read -r line; do f="${line%% (not bundled:*}" f="${f#./}" [[ -z "$f" ]] && continue [[ -f "$f" ]] || continue + n_present=$((n_present + 1)) [[ "$(lang_of_path "$f")" == "$LANG_ID" ]] || continue printf '%s\n' "$f" | grep -qE -- "$LANG_EXCLUDE_RE" && continue ALL_FILES+=("$f") @@ -248,6 +278,12 @@ done < <(LC_ALL=C sort -u "$FILES_LIST") for f in ${ALL_FILES[@]+"${ALL_FILES[@]}"}; do if lang_is_test "$f"; then TEST_FILES+=("$f"); else SRC_FILES+=("$f"); fi done +# A scope whose files exist but none of which is a source file is a glob +# mismatch between the scope and this run: an all-zero table would read as a +# clean review of nothing. +if (( ${#ALL_FILES[@]} == 0 && n_present > 0 )); then + die "0 source + 0 test files: none of the $n_present file(s) in $FILES_LIST is a $LANG_ID source file (pass the --glob ldd-scope.sh used)" +fi tmp_hits=$(mktemp) tmp_hits_all=$(mktemp) @@ -361,24 +397,86 @@ run_path() { # rm -f "$raw" "$err" } -# The R9 gate runs once, lazily; its [Q] lines are split by question. +# gate_scope_lists — what a scoped gate run checks: +# every path of the scope (deleted ones too: a doc can still link to them) and +# of docs.txt; and the identifiers the diff removes — on its removed lines and +# on none of its added ones — the symbols a renamed or deleted declaration +# takes away from the docs that cite them. Comment lines (a line that opens +# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped. +gate_scope_lists() { + local scope="$1" words="$2" line f + { + while IFS= read -r line; do + f="${line%% (not bundled:*}"; f="${f#./}" + [[ -n "$f" ]] && printf '%s\n' "$f" + done < "$FILES_LIST" + [[ -f "$BUNDLE/docs.txt" ]] && cat "$BUNDLE/docs.txt" + } | LC_ALL=C sort -u > "$scope" + : > "$words" + [[ -s "$BUNDLE/diff.patch" ]] || return 0 + grep -vE '^(\+\+\+|---) ' "$BUNDLE/diff.patch" | grep -E '^[-+]' \ + | LC_ALL=C awk ' + { + s = substr($0, 1, 1); t = substr($0, 2) + if (t ~ /^[ \t]*(\/\/|#|\/\*|\*|--|""")/) next + while (match(t, /[A-Za-z_][A-Za-z0-9_]*/)) { + w = substr(t, RSTART, RLENGTH); t = substr(t, RSTART + RLENGTH) + if (s == "+") add[w] = 1; else del[w] = 1 + } + } + END { for (w in del) if (!(w in add)) print w }' | LC_ALL=C sort > "$words" +} + +# The R9 gate runs once, lazily; its [Q] lines are split by question. With +# --gate-budget the gate runs as its own process group, so expiry stops every +# process it started, not only its shell. +GATE_STATE="whole repository" run_gate_once() { [[ -n "$gate_out" ]] && return 0 gate_out=$(mktemp) - local gate="$SCRIPT_DIR/check-repo-brain.sh" code + local gate="$SCRIPT_DIR/check-repo-brain.sh" code=0 start pid scope_line + local -a args=() [[ -f "$gate" ]] || die "Detect-gate needs $gate beside this script" - bash "$gate" "$ROOT" > "$gate_out" 2>&1 - code=$? + if [[ ! -f "$BUNDLE/dirs.txt" ]]; then + gate_scope=$(mktemp); gate_words=$(mktemp) + gate_scope_lists "$gate_scope" "$gate_words" + args=(--scope "$gate_scope" --words "$gate_words") + fi + if (( GATE_BUDGET == 0 )); then + bash "$gate" ${args[@]+"${args[@]}"} "$ROOT" > "$gate_out" 2>&1 || code=$? + else + start=$SECONDS + set -m + bash "$gate" ${args[@]+"${args[@]}"} "$ROOT" > "$gate_out" 2>&1 & + pid=$! + set +m + while kill -0 "$pid" 2>/dev/null; do + if (( SECONDS - start >= GATE_BUDGET )); then + kill -TERM -- "-$pid" 2>/dev/null + wait "$pid" 2>/dev/null + GATE_STATE="inconclusive — stopped after the ${GATE_BUDGET}s --gate-budget" + return 0 + fi + sleep 0.2 + done + wait "$pid" || code=$? + fi if (( code == 2 )); then head -5 "$gate_out" >&2 die "the R9 gate failed to run (exit 2); its questions are inconclusive" fi + scope_line=$(grep -m1 '^check-repo-brain: scope — ' "$gate_out") + [[ -n "$scope_line" ]] && GATE_STATE="scoped, ${scope_line#check-repo-brain: scope — }" return 0 } run_gate() { # local rule="$1" q="$2" qn="$3" raw tab run_gate_once + if [[ "$GATE_STATE" == inconclusive* ]]; then + printf '%s\t%s\tgate\tinconclusive\n' "$rule" "$q" >> "$tmp_counts" + return 0 + fi raw=$(mktemp) tab=$(printf '\t') # "[Qn] [:line] — — see " becomes @@ -396,15 +494,9 @@ run_gate() { # if (match(loc, /:[0-9]+$/)) { file = substr(loc, 1, RSTART - 1); line = substr(loc, RSTART + 1) + 0 } print file "\t" line "\t" rest }' > "$raw" - # A scoped review owns only the gate lines about its own files: an orphan - # doc, an unwired root or another file's broken edge is the repository's - # state, not the diff's, and is not a lead for this review's hunter. - if [[ ! -f "$BUNDLE/dirs.txt" ]] && [[ -s "$raw" ]]; then - scoped=$(mktemp); scope_list=$(mktemp) - printf '%s\n' ${ALL_FILES[@]+"${ALL_FILES[@]}"} > "$scope_list" - awk -F'\t' 'NR == FNR { in_scope[$0] = 1; next } ($1 in in_scope)' "$scope_list" "$raw" > "$scoped" - mv "$scoped" "$raw"; rm -f "$scope_list" - fi + # A scoped review owns only the gate lines about its own files and the docs + # that cite them; the gate's --scope run reports nothing else (an orphan doc + # elsewhere or an unwired root is the repository's state, not the diff's). record "$rule" "$q" gate "$raw" 0 rm -f "$raw" } @@ -454,6 +546,7 @@ n_gate=$(awk -F'\t' '$3 == "gate"' "$plan_file" | wc -l | tr -d ' ') n_judg=$(awk -F'\t' '$3 == "judgment"' "$plan_file" | wc -l | tr -d ' ') n_q=$(wc -l < "$plan_file" | tr -d ' ') echo "$SCRIPT_NAME: $n_q questions ($n_grep grep, $n_path path, $n_gate gate, $n_judg judgment) over ${#SRC_FILES[@]} source + ${#TEST_FILES[@]} test files → $BUNDLE/hits.tsv, counts.tsv" +(( n_gate > 0 )) && [[ -n "$gate_out" ]] && echo "R9 gate: $GATE_STATE" echo awk -F'\t' 'BEGIN { printf "%-9s %-4s %-9s %s\n", "rule", "q", "kind", "hits" } { q = ($2 == "-") ? "-" : "Q" $2; printf "%-9s %-4s %-9s %s\n", $1, q, $3, $4 }' "$BUNDLE/counts.tsv" @@ -473,12 +566,14 @@ awk -F'\t' ' f = fam[$1]; if (f == "") f = "other" nq[f]++ if ($3 == "judgment") { nj[f]++; next } + if ($4 == "inconclusive") { ni[f]++; next } if ($4 + 0 > 0) { hits[f] += $4; nh[f]++; rh[f, $1] += $4 } } END { for (i = 1; i <= 4; i++) { f = order[i] printf "%-15s %5d hit(s) in %d of %d questions, %d judgment", f, hits[f] + 0, nh[f] + 0, nq[f] + 0, nj[f] + 0 + if (ni[f] + 0 > 0) printf ", %d inconclusive", ni[f] per = ""; n = split(rules[f], rs, " ") for (j = 1; j <= n; j++) if (rh[f, rs[j]] + 0 > 0) per = per (per == "" ? "" : " · ") rs[j] " " rh[f, rs[j]] if (per != "") printf " — %s", per diff --git a/core/scripts/ldd-detect_test.sh b/core/scripts/ldd-detect_test.sh index cd70ffbd..dad58dc6 100644 --- a/core/scripts/ldd-detect_test.sh +++ b/core/scripts/ldd-detect_test.sh @@ -49,6 +49,11 @@ run_detect() { # [options...] — runs the script over $REPO with $BUNDLE OUT=$(cd "$REPO" && bash "$DETECT" "$@" --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" } +run_detect_bundle() { # [options...] — runs the script over $REPO with that bundle + local b="$1"; shift + OUT=$(cd "$REPO" && bash "$DETECT" "$@" --root . "$b" 2>"$BUNDLE/.stderr"); CODE=$? + ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +} run_scope() { # [args...] — runs ldd-scope.sh over $REPO into $BUNDLE/scope-out rm -rf "$BUNDLE/scope-out" OUT=$(cd "$REPO" && bash "$SCOPE_SH" --out "$BUNDLE/scope-out" "$@" 2>"$BUNDLE/.stderr"); CODE=$? @@ -75,8 +80,9 @@ expect_not() { return 0 } expect_quiet() { [[ -z "$ERR" ]] || { bad "stderr not empty"; return 1; }; } -count_of() { # — the hits cell of counts.tsv, "-" for judgment - awk -F'\t' -v r="$1" -v q="$2" '$1 == r && $2 == q { print $4 }' "$BUNDLE/counts.tsv" +count_of() { # — the hits cell of counts.tsv (COUNTS_FILE, default the + # case bundle's), "-" for judgment + awk -F'\t' -v r="$1" -v q="$2" '$1 == r && $2 == q { print $4 }' "${COUNTS_FILE:-$BUNDLE/counts.tsv}" } expect_count() { # =n> local got; got=$(count_of "$1" "$2") @@ -216,6 +222,72 @@ expect_exit 0 && expect_count SUPPRESS - 1 \ && grep -q "^SUPPRESS - grep $FX_SRC_FILE 2 " "$BUNDLE/hits.tsv" && ok finish +# --- the bundle ldd-scope.sh writes, read by detection --- +SO="" # the scope bundle of the case: $BUNDLE/scope-out + +begin "a code-only scoped diff skips the R9 gate's doc checks" +mk_repo; git_commit_repo +printf '\n' >> "$REPO/$FX_SRC_FILE" # a code-only change +(cd "$REPO" && git -c user.name=t -c user.email=t@t commit -qam change) +run_scope --base HEAD~1; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" +expect_exit 0 && expect_has "R9 gate: scoped, no doc in scope; doc checks skipped" \ + && COUNTS_FILE="$SO/counts.tsv" expect_count R9 1 0 && ok +finish + +begin "a scoped diff that touches a doc gets that doc checked by the R9 gate" +mk_repo; git_commit_repo +printf '\n' >> "$REPO/$FX_SRC_FILE" # a code-only change +printf '\nMore text.\n' >> "$REPO/docs/orphan.md" +(cd "$REPO" && git -c user.name=t -c user.email=t@t commit -qam change) +run_scope --base HEAD~1; SO="$BUNDLE/scope-out" +expect_exit 0 && { grep -qx 'docs/orphan.md' "$SO/docs.txt" || bad "docs.txt lacks the touched doc"; } \ + && grep -qx 'docs/orphan.md' "$SO/docs.txt" \ + && { run_detect_bundle "$SO"; expect_exit 0; } \ + && expect_has "R9 gate: scoped, 1 doc(s) to check" \ + && COUNTS_FILE="$SO/counts.tsv" expect_count R9 1 1 \ + && { grep -q "^R9 1 gate docs/orphan.md " "$SO/hits.tsv" || bad "no R9 Q1 hit row for docs/orphan.md"; } \ + && grep -q "^R9 1 gate docs/orphan.md " "$SO/hits.tsv" && ok +finish + +begin "detection reads the source glob from the bundle ldd-scope.sh wrote" +mk_repo; mkdir -p "$REPO/lib"; printf 'thing\n' > "$REPO/lib/thing.zz"; git_commit_repo +run_scope --glob '*.zz' lib/thing.zz; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" +expect_exit 0 && expect_has "over 1 source + 0 test files" && ok +finish + +begin "a --glob that differs from the bundle's is a usage error (exit 2)" +mk_repo; mkdir -p "$REPO/lib"; printf 'thing\n' > "$REPO/lib/thing.zz"; git_commit_repo +run_scope --glob '*.zz' lib/thing.zz; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" --glob '*.yy' +expect_exit 2 && expect_has "differs from the bundle's glob" && ok +finish + +begin "a scope whose files match no source glob is an error, not an all-zero table (exit 2)" +mk_repo; printf 'notes\n' > "$REPO/notes.txt"; printf 'notes.txt\n' > "$BUNDLE/files.txt" +printf 'custom\t*.zz\n' > "$BUNDLE/language.txt" +run_detect +expect_exit 2 && expect_has "0 source + 0 test files" && ok +finish + +begin "--gate-budget stops a slow gate and its children; the gate questions are inconclusive" +mk_repo; : > "$BUNDLE/dirs.txt" +FAKE=$(mktemp -d) +cp "$DETECT" "$FAKE/ldd-detect.sh" +printf '#!/usr/bin/env bash\nsleep 37\necho never\n' > "$FAKE/check-repo-brain.sh" +t0=$SECONDS +OUT=$(cd "$REPO" && bash "$FAKE/ldd-detect.sh" --rules "$RULES" --gate-budget 1 --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? +ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +took=$((SECONDS - t0)) +expect_exit 0 && { (( took < 15 )) || bad "the run took ${took}s with a 1s gate budget"; } && (( took < 15 )) \ + && expect_has "R9 gate: inconclusive" && expect_count R9 1 inconclusive && expect_has "inconclusive" \ + && { ! pgrep -f 'sleep 37' >/dev/null || bad "the gate's child process outlived the budget"; } \ + && ! pgrep -f 'sleep 37' >/dev/null && ok +pkill -f 'sleep 37' 2>/dev/null +rm -rf "$FAKE" +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/core/scripts/ldd-scope.sh b/core/scripts/ldd-scope.sh index 17acc261..8f9063f3 100755 --- a/core/scripts/ldd-scope.sh +++ b/core/scripts/ldd-scope.sh @@ -46,6 +46,11 @@ # # The bundle: # language.txt the group's language id; "custom TAB " under --glob +# test-re.txt the --test-re pattern, when one was passed, so ldd-detect.sh +# tells test files the same way +# docs.txt on a scoped rung, the markdown files the rung touches, one +# relative path per line, deleted ones included: the docs the +# R9 gate checks in full; --all writes none # files.txt the scope, one relative path per line. A file left out of # scope/ keeps its line with the reason: (not bundled: deleted), # (not bundled: binary), (not bundled: lines) over the @@ -226,6 +231,10 @@ write_bundle() { mkdir -p "$OUT/scope" if [[ "$GROUP_ID" == "custom" ]]; then printf 'custom\t%s\n' "$OPT_GLOB" > "$OUT/language.txt" else printf '%s\n' "$GROUP_ID" > "$OUT/language.txt"; fi + [[ -n "$OPT_TEST_RE" ]] && printf '%s\n' "$OPT_TEST_RE" > "$OUT/test-re.txt" + if [[ "$RUNG" != "all" ]]; then + sed 's|^\./||' "$CANDIDATES" | grep -E '\.md$' | LC_ALL=C sort -u > "$OUT/docs.txt" + fi # not_bundled_reason — prints the reason, or nothing when it is bundled not_bundled_reason() { diff --git a/core/skills/pre-commit-review/SKILL.md b/core/skills/pre-commit-review/SKILL.md index 3579d0ac..b06d7571 100644 --- a/core/skills/pre-commit-review/SKILL.md +++ b/core/skills/pre-commit-review/SKILL.md @@ -80,11 +80,15 @@ writes `hits.tsv`, `hits-all.tsv` and `counts.tsv` into the bundle, and prints t question only a hunter's reading answers — then one line per rule family. That table, about 300 tokens, is the pre-filter: the parent prints no rule file, composes no detection command, and never reads a Falsifying-questions section. On a scoped review the R9 -gate's rows are the scope's own files' — an orphan doc or an unwired root is the -repository's state, for a `--all` review or BROADER CONTEXT, never this diff's -finding. When the +gate checks only the docs the diff touches, the docs that link to a scope file and the +docs that cite a symbol the diff removes, and its rows are about those docs and the +scope's own files — an orphan doc elsewhere or an unwired root is the repository's +state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line +`R9 gate: …` under the summary says which docs the gate checked. A gate row whose +hits cell is `inconclusive` means the gate ran out of its `--gate-budget`: it is not +a `0`, and the documentation hunter answers that question from the prose. When the detection script exits 2 instead — a pattern its grep rejects, the R9 gate failing to -run — the review did not run: report its message, spawn nothing, and never render a +run, a scope in which no file matches the source glob — the review did not run: report its message, spawn nothing, and never render a rule as `skipped` or compose the pass by hand. A rule family none of whose rules has a hit — `0` on every row of theirs that is not `judgment` — is skipped: no hunter for it. @@ -108,7 +112,8 @@ zero prior use; the fix is a discussion or a separate PR, never silent inclusion The bundle exists (step 1 wrote it: `files.txt`, `diff.patch` on a scoped review, `scope/.txt` per bundled file with the file's own line numbers, `dirs.txt` on -`--all`, `comments.txt`, `hits.tsv`, `hits-all.tsv`, `counts.tsv`); nothing is +`--all`, `docs.txt` with the touched docs on a scoped review, `test-re.txt` with the +test-file pattern when one was passed, `comments.txt`, `hits.tsv`, `hits-all.tsv`, `counts.tsv`); nothing is written here. One Bash command, only when the scope is not empty, lists every rule path the prompts will carry and prints `sed -n '/^## Waiting for agents/,/^## The merged report/p' diff --git a/docs/token-budget.md b/docs/token-budget.md index 0d70a4bb..abc68644 100644 --- a/docs/token-budget.md +++ b/docs/token-budget.md @@ -375,7 +375,7 @@ defaults carries one **detect line** beside its prose, of one of four kinds. |---|---| | `Detect-grep: PATTERN [files=src\|test\|all] [exclude-path=ERE,…] [context=n]` — the pattern an ERE in backticks | `grep -nHE` over the scope's source files — the non-test ones unless `files=` says otherwise, minus paths matching an `exclude-path` entry; `context` appends the next n lines to each hit's excerpt | | `Detect-path: PATTERN` | the pattern over the scope's relative paths: layer directories, role-named packages | -| `Detect-gate: Q` | the R9 gate's `[Q]` report lines, from one run of `check-repo-brain.sh` over the repository | +| `Detect-gate: Q` | the R9 gate's `[Q]` report lines, from one run of `check-repo-brain.sh` over the repository, or over the scope's docs on a scoped review; `inconclusive` when the run outlives `--gate-budget` | | `Detect: judgment` | no mechanical lead; the hunter runs the prose, as it does for a question whose only procedure depends on an earlier hit (R1's second question) or on a read | A pattern is a literal: the placeholder forms the prose used to carry — ` [--words ]] [repo-root] +# (repo-root default: cwd) # CI: one line — bash scripts/check-repo-brain.sh # --fix: rewrite drifted index lines from each target doc's `description` # (the one mechanical repair; everything else stays report-only) +# --scope a review's scope: one path per line, relative to repo-root +# (code files and docs, deleted ones included). The gate then checks +# only the docs in scope: the listed docs, the docs whose text names a +# listed file's basename, and the docs that cite a word of --words. +# Code edges are checked for the listed code files only, Q3 is skipped +# (root wiring is the repository's state, not a diff's), and with no +# doc in scope the doc checks are skipped. One stdout line says which: +# "check-repo-brain: scope — doc(s) to check" or +# "check-repo-brain: scope — no doc in scope; doc checks skipped". +# --words identifiers the scoped change removes, one per line; a doc +# that cites one of them in backticks is in scope # # Doc roots are discovered at the repo root AND at every sub-project (a # directory holding the language's project marker — see the adapter below), @@ -58,8 +70,11 @@ # against its declaring package or receiver — never against a same-named # member elsewhere. A token missing from the set still resolves when it # appears as a whole word in any non-markdown repo file (config keys, -# alert names, test helpers). A `pkg.Sym` whose package is not declared in -# this repo is external (stdlib, dependencies) and exempt. +# alert names, test helpers): in a git work tree the tracked and +# untracked-but-not-ignored files; elsewhere every file outside the +# dependency and build-output directories (bazel-*, node_modules, …). +# A `pkg.Sym` whose package is not declared in this repo is external +# (stdlib, dependencies) and exempt. # - lines carrying the ⚠️ stale flag or a *(planned)* marker are exempt from # symbol resolution and the description copy check (R9 Q2/Q7 exemptions); # the file:line ban has no exemption beyond URL spans, fenced code blocks, @@ -72,22 +87,37 @@ # 2 usage error, or an internal scanner failure (a failed scan is # inconclusive, never silently clean) # -# Uses only POSIX-portable tools: find, grep, sed, awk, head, sort, wc. No jq/python. +# Uses only POSIX-portable tools: find, grep, sed, awk, head, sort, wc, tr, xargs, +# plus git when the repository is a git work tree. No jq/python. # awk programs avoid interval expressions ({m,n}) — mawk, Debian's default, rejects them. set -u FIX=0 -if [[ "${1:-}" == "--fix" ]]; then - FIX=1 - shift -fi +SCOPE_LIST="" +WORDS_LIST="" +abs_list() { # -> absolute path of an existing file, or exit 2 + [[ -f "$1" ]] || { echo "check-repo-brain: no such list: $1" >&2; exit 2; } + printf '%s/%s\n' "$(cd "$(dirname "$1")" && pwd)" "$(basename "$1")" +} +while (( $# > 0 )); do + case "$1" in + --fix) FIX=1; shift ;; + --scope) SCOPE_LIST=$(abs_list "${2:-}") || exit 2; shift 2 ;; + --words) WORDS_LIST=$(abs_list "${2:-}") || exit 2; shift 2 ;; + --*) echo "check-repo-brain: unknown option: $1" >&2; exit 2 ;; + *) break ;; + esac +done +[[ -n "$WORDS_LIST" && -z "$SCOPE_LIST" ]] && { echo "check-repo-brain: --words needs --scope" >&2; exit 2; } REPO_ROOT="${1:-$(pwd)}" if [[ ! -d "$REPO_ROOT" ]]; then echo "check-repo-brain: not a directory: $REPO_ROOT" >&2 exit 2 fi cd "$REPO_ROOT" || exit 2 +SCOPED=0 +[[ -n "$SCOPE_LIST" ]] && SCOPED=1 # ======================= language adapter: go ======================= # Everything language-specific lives between these two marker comments. The @@ -315,22 +345,65 @@ lifecycle_stale() { # -> reason when frontmatter marks the doc stale ('' have_code=0 lang_has_code && have_code=1 -# ---------- declaration set: built once, queried per token ---------- -DECLS="" PKGS="" -if (( have_code )); then - DECLS=$(mktemp) PKGS=$(mktemp) DECL_ALL=$(mktemp) - trap 'rm -f "$DECLS" "$PKGS"' EXIT - if ! ( set -o pipefail; lang_declarations | sort -u > "$DECL_ALL" ); then +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +# ---------- declaration set: built once, on the first token to resolve ---------- +DECLS="$WORK/decls" PKGS="$WORK/pkgs" +decls_built=0 +build_declarations() { + (( decls_built )) && return 0 + decls_built=1 + if ! ( set -o pipefail; lang_declarations | sort -u > "$WORK/decl-all" ); then echo "check-repo-brain: internal scanner error — the declaration scan failed; treating the run as inconclusive" >&2 exit 2 fi - grep '^pkg:' "$DECL_ALL" | sed 's/^pkg://' > "$PKGS" - grep -v '^pkg:' "$DECL_ALL" > "$DECLS" - rm -f "$DECL_ALL" -fi + grep '^pkg:' "$WORK/decl-all" | sed 's/^pkg://' > "$PKGS" + grep -v '^pkg:' "$WORK/decl-all" > "$DECLS" + rm -f "$WORK/decl-all" +} is_repo_pkg() { grep -qxF "$1" "$PKGS" 2>/dev/null; } +# repo_files — the repository's non-markdown files, NUL-separated. A git work +# tree lists them through git (tracked files plus untracked ones git does not +# ignore), which never walks build outputs or symlinked caches; elsewhere find +# skips the directories that hold dependencies and build outputs. +repo_files() { + if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then + git ls-files -z --cached --others --exclude-standard -- . \ + ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**' 2>/dev/null && return 0 + fi + find . \( -name .git -o -name vendor -o -name node_modules -o -name 'bazel-*' -o -name .venv \ + -o -name venv -o -name __pycache__ -o -name .tox -o -name dist -o -name target \) -prune \ + -o -type f ! -name '*.md' -print0 +} + +# resolve_words — the tokens that occur as a whole word in any +# non-markdown repository file, one per line. The repository is read once +# into words (runs of [A-Za-z0-9_.]); a dotted token such as `Type.Method` +# matches any run of whole dot-separated parts, as grep -w would match it. +# One pass over the text is linear in its size, where a fixed-string grep for +# hundreds of tokens is not. +resolve_words() { + repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.' '\n' \ + | LC_ALL=C awk -v list="$1" ' + BEGIN { while ((getline t < list) > 0) if (t != "") want[t] = 1 } + seen[$0]++ { next } + { + n = split($0, p, ".") + for (i = 1; i <= n; i++) { + s = p[i] + if (s in want) { print s; delete want[s] } + for (j = i + 1; j <= n; j++) { + s = s "." p[j] + if (s in want) { print s; delete want[s] } + } + } + }' + return 0 +} + # One fence-aware awk pass per bundle extracts everything Q2's doc scan needs: # P — a file:line citation outside fences/URLs/globs # S — a backticked symbol-shaped token to resolve @@ -369,8 +442,53 @@ FNR == 1 { fence = 0 } } ' -# ---------- Q2: code→docs edges (repo-wide; resolved from repo root, then the -# citing file's own sub-project) ---------- +# ---------- the scope: which docs a scoped run checks ---------- +# A doc is in scope when the scope lists it, when its text names the basename +# of a listed file as a whole word (a link to a touched doc or code file), or +# when it cites in backticks a word the change removes (a symbol the diff +# renames or deletes). +# Every list below holds relative paths as find(1) prints them under a root. +SCOPE_DOCS="$WORK/scope-docs" +SCOPE_PATHS="$WORK/scope-paths" +if (( SCOPED )); then + sed -e 's|^\./||' -e '/^$/d' "$SCOPE_LIST" | LC_ALL=C sort -u > "$SCOPE_PATHS" + for r in "${ROOTS[@]}"; do find "$r" -type f -name '*.md'; done | LC_ALL=C sort -u > "$WORK/all-docs" + { + grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" + sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$WORK/basenames" + if [[ -s "$WORK/basenames" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$WORK/basenames" 2>/dev/null + fi + if [[ -n "$WORDS_LIST" && -s "$WORDS_LIST" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" \ + | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ + -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" "$DOCSCAN_AWK" \ + | awk -F'\t' 'NR == FNR { w[$0] = 1; next } + $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in w) || (m in w)) print $2 }' "$WORDS_LIST" - + fi + } | LC_ALL=C sort -u > "$SCOPE_DOCS" + if [[ -s "$SCOPE_DOCS" ]]; then + echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" + else + echo "check-repo-brain: scope — no doc in scope; doc checks skipped" + fi +fi + +# bundle_docs [name] — the docs a run checks under one root: every +# *.md (or every file of that name), limited to the scope on a scoped run +bundle_docs() { + local name="${2:-*.md}" + if (( SCOPED )); then + find "$1" -type f -name "$name" | grep -xF -f "$SCOPE_DOCS" + else + find "$1" -type f -name "$name" + fi + return 0 +} + +# ---------- Q2: code→docs edges (repo-wide, or the scope's code files on a +# scoped run; resolved from repo root, then the citing file's own +# sub-project) ---------- docroot_for_file() { # -> docroot of the longest matching project dir local f="${1#./}" best="" i for i in "${!PROJS[@]}"; do @@ -394,7 +512,13 @@ if (( have_code )); then CUR_DOCROOT=$(docroot_for_file "$file") fail "[Q2] $file:$line — code edge points at missing $target" fi - done < <(lang_code_edges) + done < <(if (( SCOPED )); then + lang_code_edges | awk -v list="$SCOPE_PATHS" ' + BEGIN { while ((getline l < list) > 0) s[l] = 1 } + { f = substr($0, 1, index($0, ":") - 1); sub(/^\.\//, "", f); if (f in s) print }' + else + lang_code_edges + fi) fi # ---------- per-bundle checks ---------- @@ -404,6 +528,8 @@ check_bundle() { # local root_index="$docroot/index.md" local root_index_c root_index_c=$(canon "$root_index") + # a scoped run skips a bundle that holds no doc in scope + (( SCOPED )) && [[ -z "$(bundle_docs "$docroot")" ]] && return 0 # --- Q1: transitive reachability from the bundle's root index --- local reachable="" visited="" queue=("$root_index") @@ -432,7 +558,7 @@ check_bundle() { # case "$reachable" in *"$c"$'\n'*) ;; *) fail "[Q1] $doc — orphan: not reachable from $root_index" ;; esac - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q2: every doc link resolves (a missing parent directory is just as # broken as a missing file — only URLs/anchors are exempt) --- @@ -447,13 +573,13 @@ check_bundle() { # fail "[Q2] $md — link target does not exist: $t" fi done < <(grep -oE '\]\([^)]+\)' "$md" 2>/dev/null) - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q2: doc scan — file:line ban + docs→code symbol resolution. # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). --- local scan; scan=$(mktemp) if ! ( set -o pipefail - find "$docroot" -type f -name '*.md' -print0 \ + bundle_docs "$docroot" | tr '\n' '\0' \ | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" \ "$DOCSCAN_AWK" > "$scan" ); then @@ -468,6 +594,7 @@ check_bundle() { # if (( have_code )) && grep -q $'^S\t' "$scan"; then local toks check members unres bad toks=$(mktemp) check=$(mktemp) members=$(mktemp) unres=$(mktemp) bad=$(mktemp) + build_declarations grep $'^S\t' "$scan" | cut -f4 | sort -u > "$toks" # tokens to resolve (external pkg.Sym exempt). A qualified token resolves # as the PAIR itself — the declaration set carries pkg.Ident / Type.Method @@ -485,12 +612,11 @@ check_bundle() { # esac done < "$toks" > "$check" cut -f2 "$check" | sort -u > "$members" - grep -vxF -f "$DECLS" "$members" > "$unres" || true + grep -vxF -f "$DECLS" "$members" | grep -v '^$' > "$unres" || true if [[ -s "$unres" ]]; then - # ONE repo-wide word grep for the whole unresolved batch + # ONE pass over the repository for the whole unresolved batch local found; found=$(mktemp) - grep -rIhoFw --exclude-dir=vendor --exclude-dir=.git --exclude='*.md' \ - -f "$unres" . 2>/dev/null | sort -u > "$found" + resolve_words "$unres" | sort -u > "$found" grep -vxF -f "$found" "$unres" > "$bad" || true rm -f "$found" fi @@ -508,24 +634,27 @@ check_bundle() { # fi rm -f "$scan" - # --- Q3: root wiring (exact path; sub-roots may ride the repo-root index) --- - local rel="$docroot" - [[ "$proj" != "." ]] && rel="${docroot#$proj/}" - local claude="CLAUDE.md" agents="AGENTS.md" - [[ "$proj" != "." ]] && { claude="$proj/CLAUDE.md"; agents="$proj/AGENTS.md"; } - local wired_claude=0 wired_agents=0 - [[ -f "$claude" ]] && grep -qF -- "$rel/index.md" "$claude" && wired_claude=1 - [[ -f "$agents" ]] && grep -qF -- "$rel/index.md" "$agents" && wired_agents=1 - if (( ! wired_claude && ! wired_agents )); then - local via_root=0 - if [[ "$proj" != "." && -n "$ROOT_BUNDLE" ]]; then - grep -rqF -- "$docroot/index.md" "$ROOT_BUNDLE" --include='*.md' 2>/dev/null && via_root=1 - fi - if (( ! via_root )); then - fail "[Q3] $proj — neither $claude nor $agents references $rel/index.md" + # --- Q3: root wiring (exact path; sub-roots may ride the repo-root index); + # a scoped run leaves it to a whole-repository run --- + if (( ! SCOPED )); then + local rel="$docroot" + [[ "$proj" != "." ]] && rel="${docroot#$proj/}" + local claude="CLAUDE.md" agents="AGENTS.md" + [[ "$proj" != "." ]] && { claude="$proj/CLAUDE.md"; agents="$proj/AGENTS.md"; } + local wired_claude=0 wired_agents=0 + [[ -f "$claude" ]] && grep -qF -- "$rel/index.md" "$claude" && wired_claude=1 + [[ -f "$agents" ]] && grep -qF -- "$rel/index.md" "$agents" && wired_agents=1 + if (( ! wired_claude && ! wired_agents )); then + local via_root=0 + if [[ "$proj" != "." && -n "$ROOT_BUNDLE" ]]; then + grep -rqF -- "$docroot/index.md" "$ROOT_BUNDLE" --include='*.md' 2>/dev/null && via_root=1 + fi + if (( ! via_root )); then + fail "[Q3] $proj — neither $claude nor $agents references $rel/index.md" + fi + elif (( ! wired_agents )); then + note "[Q3] $agents lacks the $rel/index.md routing reference — AGENTS.md-reading tools start blind" fi - elif (( ! wired_agents )); then - note "[Q3] $agents lacks the $rel/index.md routing reference — AGENTS.md-reading tools start blind" fi # --- Q7: bundle contract --- @@ -590,7 +719,7 @@ check_bundle() { # else fail "[Q7] $md — frontmatter missing 'description:'" fi - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q7: drift check — index line text == target's description (⚠️ exempt; # --fix rewrites drifted lines after the read loop, never during it) --- @@ -628,12 +757,12 @@ check_bundle() { # echo " fixed: $idx:$n — index line rewritten from its target's description" done <<< "$fixes" fi - done < <(find "$docroot" -type f -name 'index.md') + done < <(bundle_docs "$docroot" index.md) # --- Q7: no log.md --- while IFS= read -r lg; do fail "[Q7] $lg — log.md is reserved for change history; docs describe current behavior" - done < <(find "$docroot" -type f -name 'log.md') + done < <(bundle_docs "$docroot" log.md) } for i in "${!PROJS[@]}"; do diff --git a/go-linter-driven-development/scripts/check-repo-brain_test.sh b/go-linter-driven-development/scripts/check-repo-brain_test.sh index 59aeac72..93297cf5 100644 --- a/go-linter-driven-development/scripts/check-repo-brain_test.sh +++ b/go-linter-driven-development/scripts/check-repo-brain_test.sh @@ -591,6 +591,75 @@ run_gate expect_exit 0 && expect_not "does not resolve" && expect_has "check-repo-brain: clean" && ok finish +# --- scoped runs (--scope, --words) and the symbol resolver --- +LISTS=$(mktemp -d) # the scope and word lists live outside the repository they describe +begin "--scope: a code-only scope that no doc cites skips the doc checks" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here." +printf '%s\n' "retry/policy.go" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "scope — no doc in scope; doc checks skipped" && expect_not "orphan" && ok +finish + +begin "--scope: a listed doc is checked, an unlisted orphan and the root wiring are not" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here."; mk_doc other "another unlinked doc" "Nothing." +rm "$REPO/CLAUDE.md" "$REPO/AGENTS.md" +printf '%s\n' "retry/policy.go" docs/orphan.md > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] docs/orphan.md — orphan" \ + && expect_not "docs/other.md" && expect_not "[Q3]" && ok +finish + +begin "--scope: a doc that names a scope file is checked" +mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../retry/policy.go)." +printf '%s\n' "retry/policy.go" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +finish + +begin "--scope --words: a doc that cites a removed symbol is checked" +mk_conformant +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +append_index "- [ghost.md](ghost.md) — a ghost symbol" +printf '%s\n' "retry/policy.go" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "no doc in scope" \ + && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ + && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok +finish + +begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" +mk_conformant +printf 'build/\n' > "$REPO/.gitignore" +mkdir -p "$REPO/build"; printf 'IgnoredOnlyName\n' > "$REPO/build/gen.yaml" +printf 'TrackedName\n' > "$REPO/tracked.yaml" +mk_doc names "alert names" "Fires \`TrackedName\`, \`UntrackedName\` and \`IgnoredOnlyName\`." +append_index "- [names.md](names.md) — alert names" +(cd "$REPO" && git init -q && git add -A && git -c user.name=t -c user.email=t@t commit -qm init) +printf 'UntrackedName\n' > "$REPO/untracked.yaml" +run_gate +expect_exit 1 && expect_has "\`IgnoredOnlyName\` does not resolve" \ + && expect_not "\`TrackedName\` does not resolve" && expect_not "\`UntrackedName\` does not resolve" && ok +finish + +begin "Q2 outside git a token does not resolve from a bazel-* output directory" +mk_conformant +mkdir -p "$REPO/bazel-out"; printf 'BazelOnlyName\n' > "$REPO/bazel-out/gen.yaml" +mk_doc bazel "bazel names" "Fires \`BazelOnlyName\`." +append_index "- [bazel.md](bazel.md) — bazel names" +run_gate +expect_exit 1 && expect_has "\`BazelOnlyName\` does not resolve" && ok +finish + +begin "Q2 a dotted token resolves on whole dot-separated words only" +mk_conformant +printf 'limits:\n key: Cfg.MaxRetries.value\n other: Cfg.MaxRetriesX\n' > "$REPO/limits.yaml" +mk_doc dotted "dotted names" "Reads \`Cfg.MaxRetries\`; \`Cfg.MaxRetri\` is a typo." +append_index "- [dotted.md](dotted.md) — dotted names" +run_gate +expect_exit 1 && expect_has "\`Cfg.MaxRetri\` does not resolve" && expect_not "\`Cfg.MaxRetries\` does not resolve" && ok +finish +rm -rf "$LISTS" + # ========================= summary ========================= echo if [[ -n "$GATE_REF" ]]; then diff --git a/go-linter-driven-development/scripts/ldd-detect.sh b/go-linter-driven-development/scripts/ldd-detect.sh index b4ecc2be..702f67b4 100755 --- a/go-linter-driven-development/scripts/ldd-detect.sh +++ b/go-linter-driven-development/scripts/ldd-detect.sh @@ -25,7 +25,14 @@ # --plan print the parsed detect lines and exit; nothing runs # --glob the scope is one group of a language the table # does not know, matched by this glob -# --test-re test-file pattern over relative paths, same case +# --test-re test-file pattern over relative paths, same case. +# A bundle carries the pattern ldd-scope.sh used +# (test-re.txt), and a custom group its glob +# (language.txt): the bundle's values win, and a +# --glob or --test-re that differs is a usage error +# --gate-budget wall-clock seconds the R9 gate may run (default: no +# limit); on expiry the gate is stopped and its +# questions are counted "inconclusive", never 0 # # The scope's language is the bundle's language.txt; without it, the files in # files.txt tell it by extension, and a list that spans several languages is @@ -39,10 +46,14 @@ # Detect-path: `` over the scope's relative paths (layer directories, # role-named packages); a hit's line is 0 # Detect-gate: Q the R9 gate's [Q] report lines, from one run of -# check-repo-brain.sh beside this script over --root; on a -# scoped bundle (no dirs.txt) only the lines whose file is in -# the scope — the gate's repository-wide findings belong to -# a whole-repository review +# check-repo-brain.sh beside this script over --root. On a +# scoped bundle (no dirs.txt) the gate runs with --scope: it +# checks only the bundle's docs (docs.txt), the docs that +# link to a scope file and the docs that cite a symbol the +# diff removes, the code edges of the scope's code files, +# and no root wiring; with no such doc its doc checks are +# skipped. The repository-wide findings belong to a +# whole-repository review # Detect: judgment no mechanical lead; the hunter runs the prose # A question with no detect line, two of them, or a malformed one is an error: # the generator's lint-core keeps the rule files honest, and this script refuses @@ -53,15 +64,18 @@ # the overflow row of a capped question: file "-", line 0, # excerpt "+ more hit(s) not listed" # hits-all.tsv the same columns, every hit, no cap and no overflow row -# counts.tsv rule question kind hits (judgment rows carry "-") +# counts.tsv rule question kind hits (judgment rows carry "-", gate +# rows "inconclusive" when the gate ran out of --gate-budget) # the last row is SUPPRESS - grep : suppression directives # on the diff's added lines when the bundle has a diff.patch, # else over every scope file # Two runs over one tree write byte-identical counts.tsv. # # Exit codes: 0 tables written · 2 usage error, a detect line that does not -# parse, a pattern grep rejects, or a gate run that failed -# (inconclusive, never silently clean) +# parse, a pattern grep rejects, a gate run that failed +# (inconclusive, never silently clean), or a non-empty scope in +# which no file matches the source glob (a glob mismatch, never +# an empty table) # # Uses only POSIX-portable tools: grep, sed, awk, sort, wc, head. GNU grep's # \b and \s are assumed, as the rules already assume them. @@ -77,6 +91,7 @@ CAP=40 PLAN=0 OPT_GLOB="" OPT_TEST_RE="" +GATE_BUDGET=0 BUNDLE="" usage() { sed -n '2,/^$/p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; } @@ -94,12 +109,14 @@ while (( $# > 0 )); do --plan) PLAN=1; shift ;; --glob) value "$1" $#; OPT_GLOB="$2"; shift 2 ;; --test-re) value "$1" $#; OPT_TEST_RE="$2"; shift 2 ;; + --gate-budget) value "$1" $#; GATE_BUDGET="$2"; shift 2 ;; -h|--help) usage ;; --*) die "unknown option: $1" ;; *) [[ -n "$BUNDLE" ]] && die "one bundle directory, got also: $1"; BUNDLE="$1"; shift ;; esac done [[ "$CAP" =~ ^[0-9]+$ ]] || die "--cap takes a number, got: $CAP" +[[ "$GATE_BUDGET" =~ ^[0-9]+$ ]] || die "--gate-budget takes a number of seconds, got: $GATE_BUDGET" [[ -d "$RULES_DIR" ]] || die "no rules directory: $RULES_DIR" RULES_DIR=$(cd "$RULES_DIR" && pwd) if (( ! PLAN )); then @@ -242,6 +259,7 @@ if (( ! PLAN )) && [[ -f "$BUNDLE/groups.txt" ]]; then fi args=(--root "$ROOT" --cap "$CAP") [[ -n "$OPT_TEST_RE" ]] && args+=(--test-re "$OPT_TEST_RE") + (( GATE_BUDGET > 0 )) && args+=(--gate-budget "$GATE_BUDGET") bash "$script" "${args[@]}" "$bundle" || { failed=1; echo "$SCRIPT_NAME: the $id group's detection pass failed; it is inconclusive" >&2; } echo done < "$BUNDLE/groups.txt" @@ -256,7 +274,9 @@ LANG_ID="" if (( ! PLAN )); then if [[ -f "$BUNDLE/language.txt" ]]; then IFS=$'\t' read -r LANG_ID glob < "$BUNDLE/language.txt" - [[ "$LANG_ID" == "custom" && -z "$OPT_GLOB" ]] && OPT_GLOB="$glob" + [[ "$LANG_ID" == "custom" && -n "$OPT_GLOB" && "$OPT_GLOB" != "$glob" ]] \ + && die "--glob '$OPT_GLOB' differs from the bundle's glob '$glob' (ldd-scope.sh wrote it; drop --glob)" + [[ "$LANG_ID" == "custom" ]] && OPT_GLOB="$glob" else ids="" while IFS= read -r line; do @@ -271,6 +291,14 @@ if (( ! PLAN )); then [[ "$ids" == *" "* ]] && die "the scope list spans several languages ($ids); ldd-scope.sh writes one bundle per language" LANG_ID="$ids" fi + # the test-file pattern ldd-scope.sh used, so the two scripts cannot disagree + # on what a test file is + if [[ -f "$BUNDLE/test-re.txt" ]]; then + b_test_re=$(cat "$BUNDLE/test-re.txt") + [[ -n "$OPT_TEST_RE" && "$OPT_TEST_RE" != "$b_test_re" ]] \ + && die "--test-re '$OPT_TEST_RE' differs from the bundle's '$b_test_re' (ldd-scope.sh wrote it; drop --test-re)" + OPT_TEST_RE="$b_test_re" + fi lang_configure "$LANG_ID" || die "this plugin has no language block for $LANG_ID" fi @@ -316,7 +344,7 @@ END { if (insec) flush() } ' plan_file=$(mktemp) -trap 'rm -f "$plan_file" "${tmp_hits:-}" "${tmp_hits_all:-}" "${tmp_counts:-}" "${gate_out:-}"' EXIT +trap 'rm -f "$plan_file" "${tmp_hits:-}" "${tmp_hits_all:-}" "${tmp_counts:-}" "${gate_out:-}" "${gate_scope:-}" "${gate_words:-}"' EXIT rule_files=$(find "$RULES_DIR" -maxdepth 1 -name 'R*.md' | awk -F/ '{ n = $NF; sub(/^R/, "", n); sub(/-.*$/, "", n); print n "\t" $0 }' | sort -n | cut -f2) [[ -n "$rule_files" ]] || die "no rule files (R*.md) under $RULES_DIR" while IFS= read -r f; do @@ -342,11 +370,13 @@ fi SRC_FILES=() TEST_FILES=() ALL_FILES=() +n_present=0 while IFS= read -r line; do f="${line%% (not bundled:*}" f="${f#./}" [[ -z "$f" ]] && continue [[ -f "$f" ]] || continue + n_present=$((n_present + 1)) [[ "$(lang_of_path "$f")" == "$LANG_ID" ]] || continue printf '%s\n' "$f" | grep -qE -- "$LANG_EXCLUDE_RE" && continue ALL_FILES+=("$f") @@ -354,6 +384,12 @@ done < <(LC_ALL=C sort -u "$FILES_LIST") for f in ${ALL_FILES[@]+"${ALL_FILES[@]}"}; do if lang_is_test "$f"; then TEST_FILES+=("$f"); else SRC_FILES+=("$f"); fi done +# A scope whose files exist but none of which is a source file is a glob +# mismatch between the scope and this run: an all-zero table would read as a +# clean review of nothing. +if (( ${#ALL_FILES[@]} == 0 && n_present > 0 )); then + die "0 source + 0 test files: none of the $n_present file(s) in $FILES_LIST is a $LANG_ID source file (pass the --glob ldd-scope.sh used)" +fi tmp_hits=$(mktemp) tmp_hits_all=$(mktemp) @@ -467,24 +503,86 @@ run_path() { # rm -f "$raw" "$err" } -# The R9 gate runs once, lazily; its [Q] lines are split by question. +# gate_scope_lists — what a scoped gate run checks: +# every path of the scope (deleted ones too: a doc can still link to them) and +# of docs.txt; and the identifiers the diff removes — on its removed lines and +# on none of its added ones — the symbols a renamed or deleted declaration +# takes away from the docs that cite them. Comment lines (a line that opens +# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped. +gate_scope_lists() { + local scope="$1" words="$2" line f + { + while IFS= read -r line; do + f="${line%% (not bundled:*}"; f="${f#./}" + [[ -n "$f" ]] && printf '%s\n' "$f" + done < "$FILES_LIST" + [[ -f "$BUNDLE/docs.txt" ]] && cat "$BUNDLE/docs.txt" + } | LC_ALL=C sort -u > "$scope" + : > "$words" + [[ -s "$BUNDLE/diff.patch" ]] || return 0 + grep -vE '^(\+\+\+|---) ' "$BUNDLE/diff.patch" | grep -E '^[-+]' \ + | LC_ALL=C awk ' + { + s = substr($0, 1, 1); t = substr($0, 2) + if (t ~ /^[ \t]*(\/\/|#|\/\*|\*|--|""")/) next + while (match(t, /[A-Za-z_][A-Za-z0-9_]*/)) { + w = substr(t, RSTART, RLENGTH); t = substr(t, RSTART + RLENGTH) + if (s == "+") add[w] = 1; else del[w] = 1 + } + } + END { for (w in del) if (!(w in add)) print w }' | LC_ALL=C sort > "$words" +} + +# The R9 gate runs once, lazily; its [Q] lines are split by question. With +# --gate-budget the gate runs as its own process group, so expiry stops every +# process it started, not only its shell. +GATE_STATE="whole repository" run_gate_once() { [[ -n "$gate_out" ]] && return 0 gate_out=$(mktemp) - local gate="$SCRIPT_DIR/check-repo-brain.sh" code + local gate="$SCRIPT_DIR/check-repo-brain.sh" code=0 start pid scope_line + local -a args=() [[ -f "$gate" ]] || die "Detect-gate needs $gate beside this script" - bash "$gate" "$ROOT" > "$gate_out" 2>&1 - code=$? + if [[ ! -f "$BUNDLE/dirs.txt" ]]; then + gate_scope=$(mktemp); gate_words=$(mktemp) + gate_scope_lists "$gate_scope" "$gate_words" + args=(--scope "$gate_scope" --words "$gate_words") + fi + if (( GATE_BUDGET == 0 )); then + bash "$gate" ${args[@]+"${args[@]}"} "$ROOT" > "$gate_out" 2>&1 || code=$? + else + start=$SECONDS + set -m + bash "$gate" ${args[@]+"${args[@]}"} "$ROOT" > "$gate_out" 2>&1 & + pid=$! + set +m + while kill -0 "$pid" 2>/dev/null; do + if (( SECONDS - start >= GATE_BUDGET )); then + kill -TERM -- "-$pid" 2>/dev/null + wait "$pid" 2>/dev/null + GATE_STATE="inconclusive — stopped after the ${GATE_BUDGET}s --gate-budget" + return 0 + fi + sleep 0.2 + done + wait "$pid" || code=$? + fi if (( code == 2 )); then head -5 "$gate_out" >&2 die "the R9 gate failed to run (exit 2); its questions are inconclusive" fi + scope_line=$(grep -m1 '^check-repo-brain: scope — ' "$gate_out") + [[ -n "$scope_line" ]] && GATE_STATE="scoped, ${scope_line#check-repo-brain: scope — }" return 0 } run_gate() { # local rule="$1" q="$2" qn="$3" raw tab run_gate_once + if [[ "$GATE_STATE" == inconclusive* ]]; then + printf '%s\t%s\tgate\tinconclusive\n' "$rule" "$q" >> "$tmp_counts" + return 0 + fi raw=$(mktemp) tab=$(printf '\t') # "[Qn] [:line] — — see " becomes @@ -502,15 +600,9 @@ run_gate() { # if (match(loc, /:[0-9]+$/)) { file = substr(loc, 1, RSTART - 1); line = substr(loc, RSTART + 1) + 0 } print file "\t" line "\t" rest }' > "$raw" - # A scoped review owns only the gate lines about its own files: an orphan - # doc, an unwired root or another file's broken edge is the repository's - # state, not the diff's, and is not a lead for this review's hunter. - if [[ ! -f "$BUNDLE/dirs.txt" ]] && [[ -s "$raw" ]]; then - scoped=$(mktemp); scope_list=$(mktemp) - printf '%s\n' ${ALL_FILES[@]+"${ALL_FILES[@]}"} > "$scope_list" - awk -F'\t' 'NR == FNR { in_scope[$0] = 1; next } ($1 in in_scope)' "$scope_list" "$raw" > "$scoped" - mv "$scoped" "$raw"; rm -f "$scope_list" - fi + # A scoped review owns only the gate lines about its own files and the docs + # that cite them; the gate's --scope run reports nothing else (an orphan doc + # elsewhere or an unwired root is the repository's state, not the diff's). record "$rule" "$q" gate "$raw" 0 rm -f "$raw" } @@ -560,6 +652,7 @@ n_gate=$(awk -F'\t' '$3 == "gate"' "$plan_file" | wc -l | tr -d ' ') n_judg=$(awk -F'\t' '$3 == "judgment"' "$plan_file" | wc -l | tr -d ' ') n_q=$(wc -l < "$plan_file" | tr -d ' ') echo "$SCRIPT_NAME: $n_q questions ($n_grep grep, $n_path path, $n_gate gate, $n_judg judgment) over ${#SRC_FILES[@]} source + ${#TEST_FILES[@]} test files → $BUNDLE/hits.tsv, counts.tsv" +(( n_gate > 0 )) && [[ -n "$gate_out" ]] && echo "R9 gate: $GATE_STATE" echo awk -F'\t' 'BEGIN { printf "%-9s %-4s %-9s %s\n", "rule", "q", "kind", "hits" } { q = ($2 == "-") ? "-" : "Q" $2; printf "%-9s %-4s %-9s %s\n", $1, q, $3, $4 }' "$BUNDLE/counts.tsv" @@ -579,12 +672,14 @@ awk -F'\t' ' f = fam[$1]; if (f == "") f = "other" nq[f]++ if ($3 == "judgment") { nj[f]++; next } + if ($4 == "inconclusive") { ni[f]++; next } if ($4 + 0 > 0) { hits[f] += $4; nh[f]++; rh[f, $1] += $4 } } END { for (i = 1; i <= 4; i++) { f = order[i] printf "%-15s %5d hit(s) in %d of %d questions, %d judgment", f, hits[f] + 0, nh[f] + 0, nq[f] + 0, nj[f] + 0 + if (ni[f] + 0 > 0) printf ", %d inconclusive", ni[f] per = ""; n = split(rules[f], rs, " ") for (j = 1; j <= n; j++) if (rh[f, rs[j]] + 0 > 0) per = per (per == "" ? "" : " · ") rs[j] " " rh[f, rs[j]] if (per != "") printf " — %s", per diff --git a/go-linter-driven-development/scripts/ldd-detect_test.sh b/go-linter-driven-development/scripts/ldd-detect_test.sh index 7d7a85e1..5c201080 100644 --- a/go-linter-driven-development/scripts/ldd-detect_test.sh +++ b/go-linter-driven-development/scripts/ldd-detect_test.sh @@ -49,6 +49,11 @@ run_detect() { # [options...] — runs the script over $REPO with $BUNDLE OUT=$(cd "$REPO" && bash "$DETECT" "$@" --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" } +run_detect_bundle() { # [options...] — runs the script over $REPO with that bundle + local b="$1"; shift + OUT=$(cd "$REPO" && bash "$DETECT" "$@" --root . "$b" 2>"$BUNDLE/.stderr"); CODE=$? + ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +} run_scope() { # [args...] — runs ldd-scope.sh over $REPO into $BUNDLE/scope-out rm -rf "$BUNDLE/scope-out" OUT=$(cd "$REPO" && bash "$SCOPE_SH" --out "$BUNDLE/scope-out" "$@" 2>"$BUNDLE/.stderr"); CODE=$? @@ -75,8 +80,9 @@ expect_not() { return 0 } expect_quiet() { [[ -z "$ERR" ]] || { bad "stderr not empty"; return 1; }; } -count_of() { # — the hits cell of counts.tsv, "-" for judgment - awk -F'\t' -v r="$1" -v q="$2" '$1 == r && $2 == q { print $4 }' "$BUNDLE/counts.tsv" +count_of() { # — the hits cell of counts.tsv (COUNTS_FILE, default the + # case bundle's), "-" for judgment + awk -F'\t' -v r="$1" -v q="$2" '$1 == r && $2 == q { print $4 }' "${COUNTS_FILE:-$BUNDLE/counts.tsv}" } expect_count() { # =n> local got; got=$(count_of "$1" "$2") @@ -479,6 +485,72 @@ expect_exit 0 && expect_count SUPPRESS - 1 \ && grep -q "^SUPPRESS - grep $FX_SRC_FILE 2 " "$BUNDLE/hits.tsv" && ok finish +# --- the bundle ldd-scope.sh writes, read by detection --- +SO="" # the scope bundle of the case: $BUNDLE/scope-out + +begin "a code-only scoped diff skips the R9 gate's doc checks" +mk_repo; git_commit_repo +printf '\n' >> "$REPO/$FX_SRC_FILE" # a code-only change +(cd "$REPO" && git -c user.name=t -c user.email=t@t commit -qam change) +run_scope --base HEAD~1; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" +expect_exit 0 && expect_has "R9 gate: scoped, no doc in scope; doc checks skipped" \ + && COUNTS_FILE="$SO/counts.tsv" expect_count R9 1 0 && ok +finish + +begin "a scoped diff that touches a doc gets that doc checked by the R9 gate" +mk_repo; git_commit_repo +printf '\n' >> "$REPO/$FX_SRC_FILE" # a code-only change +printf '\nMore text.\n' >> "$REPO/docs/orphan.md" +(cd "$REPO" && git -c user.name=t -c user.email=t@t commit -qam change) +run_scope --base HEAD~1; SO="$BUNDLE/scope-out" +expect_exit 0 && { grep -qx 'docs/orphan.md' "$SO/docs.txt" || bad "docs.txt lacks the touched doc"; } \ + && grep -qx 'docs/orphan.md' "$SO/docs.txt" \ + && { run_detect_bundle "$SO"; expect_exit 0; } \ + && expect_has "R9 gate: scoped, 1 doc(s) to check" \ + && COUNTS_FILE="$SO/counts.tsv" expect_count R9 1 1 \ + && { grep -q "^R9 1 gate docs/orphan.md " "$SO/hits.tsv" || bad "no R9 Q1 hit row for docs/orphan.md"; } \ + && grep -q "^R9 1 gate docs/orphan.md " "$SO/hits.tsv" && ok +finish + +begin "detection reads the source glob from the bundle ldd-scope.sh wrote" +mk_repo; mkdir -p "$REPO/lib"; printf 'thing\n' > "$REPO/lib/thing.zz"; git_commit_repo +run_scope --glob '*.zz' lib/thing.zz; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" +expect_exit 0 && expect_has "over 1 source + 0 test files" && ok +finish + +begin "a --glob that differs from the bundle's is a usage error (exit 2)" +mk_repo; mkdir -p "$REPO/lib"; printf 'thing\n' > "$REPO/lib/thing.zz"; git_commit_repo +run_scope --glob '*.zz' lib/thing.zz; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" --glob '*.yy' +expect_exit 2 && expect_has "differs from the bundle's glob" && ok +finish + +begin "a scope whose files match no source glob is an error, not an all-zero table (exit 2)" +mk_repo; printf 'notes\n' > "$REPO/notes.txt"; printf 'notes.txt\n' > "$BUNDLE/files.txt" +printf 'custom\t*.zz\n' > "$BUNDLE/language.txt" +run_detect +expect_exit 2 && expect_has "0 source + 0 test files" && ok +finish + +begin "--gate-budget stops a slow gate and its children; the gate questions are inconclusive" +mk_repo; : > "$BUNDLE/dirs.txt" +FAKE=$(mktemp -d) +cp "$DETECT" "$FAKE/ldd-detect.sh" +printf '#!/usr/bin/env bash\nsleep 37\necho never\n' > "$FAKE/check-repo-brain.sh" +t0=$SECONDS +OUT=$(cd "$REPO" && bash "$FAKE/ldd-detect.sh" --rules "$RULES" --gate-budget 1 --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? +ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +took=$((SECONDS - t0)) +expect_exit 0 && { (( took < 15 )) || bad "the run took ${took}s with a 1s gate budget"; } && (( took < 15 )) \ + && expect_has "R9 gate: inconclusive" && expect_count R9 1 inconclusive && expect_has "inconclusive" \ + && { ! pgrep -f 'sleep 37' >/dev/null || bad "the gate's child process outlived the budget"; } \ + && ! pgrep -f 'sleep 37' >/dev/null && ok +pkill -f 'sleep 37' 2>/dev/null +rm -rf "$FAKE" +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/go-linter-driven-development/scripts/ldd-scope.sh b/go-linter-driven-development/scripts/ldd-scope.sh index 8b0e0e3e..8a99047d 100755 --- a/go-linter-driven-development/scripts/ldd-scope.sh +++ b/go-linter-driven-development/scripts/ldd-scope.sh @@ -46,6 +46,11 @@ # # The bundle: # language.txt the group's language id; "custom TAB " under --glob +# test-re.txt the --test-re pattern, when one was passed, so ldd-detect.sh +# tells test files the same way +# docs.txt on a scoped rung, the markdown files the rung touches, one +# relative path per line, deleted ones included: the docs the +# R9 gate checks in full; --all writes none # files.txt the scope, one relative path per line. A file left out of # scope/ keeps its line with the reason: (not bundled: deleted), # (not bundled: binary), (not bundled: lines) over the @@ -332,6 +337,10 @@ write_bundle() { mkdir -p "$OUT/scope" if [[ "$GROUP_ID" == "custom" ]]; then printf 'custom\t%s\n' "$OPT_GLOB" > "$OUT/language.txt" else printf '%s\n' "$GROUP_ID" > "$OUT/language.txt"; fi + [[ -n "$OPT_TEST_RE" ]] && printf '%s\n' "$OPT_TEST_RE" > "$OUT/test-re.txt" + if [[ "$RUNG" != "all" ]]; then + sed 's|^\./||' "$CANDIDATES" | grep -E '\.md$' | LC_ALL=C sort -u > "$OUT/docs.txt" + fi # not_bundled_reason — prints the reason, or nothing when it is bundled not_bundled_reason() { diff --git a/go-linter-driven-development/skills/pre-commit-review/SKILL.md b/go-linter-driven-development/skills/pre-commit-review/SKILL.md index 5da72c90..93bcf50d 100644 --- a/go-linter-driven-development/skills/pre-commit-review/SKILL.md +++ b/go-linter-driven-development/skills/pre-commit-review/SKILL.md @@ -80,11 +80,15 @@ writes `hits.tsv`, `hits-all.tsv` and `counts.tsv` into the bundle, and prints t question only a hunter's reading answers — then one line per rule family. That table, about 300 tokens, is the pre-filter: the parent prints no rule file, composes no detection command, and never reads a Falsifying-questions section. On a scoped review the R9 -gate's rows are the scope's own files' — an orphan doc or an unwired root is the -repository's state, for a `--all` review or BROADER CONTEXT, never this diff's -finding. When the +gate checks only the docs the diff touches, the docs that link to a scope file and the +docs that cite a symbol the diff removes, and its rows are about those docs and the +scope's own files — an orphan doc elsewhere or an unwired root is the repository's +state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line +`R9 gate: …` under the summary says which docs the gate checked. A gate row whose +hits cell is `inconclusive` means the gate ran out of its `--gate-budget`: it is not +a `0`, and the documentation hunter answers that question from the prose. When the detection script exits 2 instead — a pattern its grep rejects, the R9 gate failing to -run — the review did not run: report its message, spawn nothing, and never render a +run, a scope in which no file matches the source glob — the review did not run: report its message, spawn nothing, and never render a rule as `skipped` or compose the pass by hand. A rule family none of whose rules has a hit — `0` on every row of theirs that is not `judgment` — is skipped: no hunter for it. @@ -110,7 +114,8 @@ zero prior use; the fix is a discussion or a separate PR, never silent inclusion The bundle exists (step 1 wrote it: `files.txt`, `diff.patch` on a scoped review, `scope/.txt` per bundled file with the file's own line numbers, `dirs.txt` on -`--all`, `comments.txt`, `hits.tsv`, `hits-all.tsv`, `counts.tsv`); nothing is +`--all`, `docs.txt` with the touched docs on a scoped review, `test-re.txt` with the +test-file pattern when one was passed, `comments.txt`, `hits.tsv`, `hits-all.tsv`, `counts.tsv`); nothing is written here. One Bash command, only when the scope is not empty, lists every rule path the prompts will carry and prints `sed -n '/^## Waiting for agents/,/^## The merged report/p' diff --git a/lang/generic/scripts/repo-brain-adapter.sh b/lang/generic/scripts/repo-brain-adapter.sh index dfd10deb..0c2f6ce8 100644 --- a/lang/generic/scripts/repo-brain-adapter.sh +++ b/lang/generic/scripts/repo-brain-adapter.sh @@ -147,10 +147,10 @@ LANG_FILE_RE='\\.py(:[0-9]+)?$' LANG_SYMBOL_RE='^([A-Z][A-Za-z0-9_]*|[a-z_][a-z0-9_]*_[a-z0-9_]*)$' LANG_QUALIFIED_RE='^[A-Za-z_][A-Za-z0-9_]*\\.[A-Za-z_][A-Za-z0-9_]*$' -py_prune() { # find(1) pruning shared by the three walkers - printf '%s\n' -not -path '*/.venv/*' -not -path '*/venv/*' -not -path '*/node_modules/*' \ - -not -path '*/__pycache__/*' -not -path '*/site-packages/*' -not -path './.git/*' -} +# find(1) pruning shared by the three walkers; an array, so a pattern such as +# ./.git/* reaches find as it is and never expands against the tree +PY_PRUNE=(-not -path '*/.venv/*' -not -path '*/venv/*' -not -path '*/node_modules/*' + -not -path '*/__pycache__/*' -not -path '*/site-packages/*' -not -path './.git/*') lang_project_dirs() { local m p @@ -158,11 +158,11 @@ lang_project_dirs() { p=$(dirname "$m"); p="${p#./}" [[ "$p" == "." || -z "$p" ]] && continue printf '%s\n' "$p" - done < <(find . -name "$LANG_PROJECT_MARKER" $(py_prune) 2>/dev/null | sort) + done < <(find . -name "$LANG_PROJECT_MARKER" "${PY_PRUNE[@]}" 2>/dev/null | sort) } lang_has_code() { - find . -name "$LANG_CODE_GLOB" $(py_prune) -print -quit 2>/dev/null | grep -q . + find . -name "$LANG_CODE_GLOB" "${PY_PRUNE[@]}" -print -quit 2>/dev/null | grep -q . } LANG_DECL_AWK=' @@ -213,7 +213,7 @@ FNR == 1 { ' lang_declarations() { - find . -name "$LANG_CODE_GLOB" $(py_prune) -print0 2>/dev/null \ + find . -name "$LANG_CODE_GLOB" "${PY_PRUNE[@]}" -print0 2>/dev/null \ | xargs -0 awk "$LANG_DECL_AWK" } diff --git a/lang/go/overrides/scripts/check-repo-brain_test.sh b/lang/go/overrides/scripts/check-repo-brain_test.sh index 59aeac72..93297cf5 100644 --- a/lang/go/overrides/scripts/check-repo-brain_test.sh +++ b/lang/go/overrides/scripts/check-repo-brain_test.sh @@ -591,6 +591,75 @@ run_gate expect_exit 0 && expect_not "does not resolve" && expect_has "check-repo-brain: clean" && ok finish +# --- scoped runs (--scope, --words) and the symbol resolver --- +LISTS=$(mktemp -d) # the scope and word lists live outside the repository they describe +begin "--scope: a code-only scope that no doc cites skips the doc checks" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here." +printf '%s\n' "retry/policy.go" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "scope — no doc in scope; doc checks skipped" && expect_not "orphan" && ok +finish + +begin "--scope: a listed doc is checked, an unlisted orphan and the root wiring are not" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here."; mk_doc other "another unlinked doc" "Nothing." +rm "$REPO/CLAUDE.md" "$REPO/AGENTS.md" +printf '%s\n' "retry/policy.go" docs/orphan.md > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] docs/orphan.md — orphan" \ + && expect_not "docs/other.md" && expect_not "[Q3]" && ok +finish + +begin "--scope: a doc that names a scope file is checked" +mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../retry/policy.go)." +printf '%s\n' "retry/policy.go" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +finish + +begin "--scope --words: a doc that cites a removed symbol is checked" +mk_conformant +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +append_index "- [ghost.md](ghost.md) — a ghost symbol" +printf '%s\n' "retry/policy.go" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "no doc in scope" \ + && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ + && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok +finish + +begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" +mk_conformant +printf 'build/\n' > "$REPO/.gitignore" +mkdir -p "$REPO/build"; printf 'IgnoredOnlyName\n' > "$REPO/build/gen.yaml" +printf 'TrackedName\n' > "$REPO/tracked.yaml" +mk_doc names "alert names" "Fires \`TrackedName\`, \`UntrackedName\` and \`IgnoredOnlyName\`." +append_index "- [names.md](names.md) — alert names" +(cd "$REPO" && git init -q && git add -A && git -c user.name=t -c user.email=t@t commit -qm init) +printf 'UntrackedName\n' > "$REPO/untracked.yaml" +run_gate +expect_exit 1 && expect_has "\`IgnoredOnlyName\` does not resolve" \ + && expect_not "\`TrackedName\` does not resolve" && expect_not "\`UntrackedName\` does not resolve" && ok +finish + +begin "Q2 outside git a token does not resolve from a bazel-* output directory" +mk_conformant +mkdir -p "$REPO/bazel-out"; printf 'BazelOnlyName\n' > "$REPO/bazel-out/gen.yaml" +mk_doc bazel "bazel names" "Fires \`BazelOnlyName\`." +append_index "- [bazel.md](bazel.md) — bazel names" +run_gate +expect_exit 1 && expect_has "\`BazelOnlyName\` does not resolve" && ok +finish + +begin "Q2 a dotted token resolves on whole dot-separated words only" +mk_conformant +printf 'limits:\n key: Cfg.MaxRetries.value\n other: Cfg.MaxRetriesX\n' > "$REPO/limits.yaml" +mk_doc dotted "dotted names" "Reads \`Cfg.MaxRetries\`; \`Cfg.MaxRetri\` is a typo." +append_index "- [dotted.md](dotted.md) — dotted names" +run_gate +expect_exit 1 && expect_has "\`Cfg.MaxRetri\` does not resolve" && expect_not "\`Cfg.MaxRetries\` does not resolve" && ok +finish +rm -rf "$LISTS" + # ========================= summary ========================= echo if [[ -n "$GATE_REF" ]]; then diff --git a/lang/python/scripts/repo-brain-adapter.sh b/lang/python/scripts/repo-brain-adapter.sh index e13f3abc..0a86f887 100644 --- a/lang/python/scripts/repo-brain-adapter.sh +++ b/lang/python/scripts/repo-brain-adapter.sh @@ -35,10 +35,10 @@ LANG_FILE_RE='\\.py(:[0-9]+)?$' LANG_SYMBOL_RE='^([A-Z][A-Za-z0-9_]*|[a-z_][a-z0-9_]*_[a-z0-9_]*)$' LANG_QUALIFIED_RE='^[A-Za-z_][A-Za-z0-9_]*\\.[A-Za-z_][A-Za-z0-9_]*$' -py_prune() { # find(1) pruning shared by the three walkers - printf '%s\n' -not -path '*/.venv/*' -not -path '*/venv/*' -not -path '*/node_modules/*' \ - -not -path '*/__pycache__/*' -not -path '*/site-packages/*' -not -path './.git/*' -} +# find(1) pruning shared by the three walkers; an array, so a pattern such as +# ./.git/* reaches find as it is and never expands against the tree +PY_PRUNE=(-not -path '*/.venv/*' -not -path '*/venv/*' -not -path '*/node_modules/*' + -not -path '*/__pycache__/*' -not -path '*/site-packages/*' -not -path './.git/*') lang_project_dirs() { local m p @@ -46,11 +46,11 @@ lang_project_dirs() { p=$(dirname "$m"); p="${p#./}" [[ "$p" == "." || -z "$p" ]] && continue printf '%s\n' "$p" - done < <(find . -name "$LANG_PROJECT_MARKER" $(py_prune) 2>/dev/null | sort) + done < <(find . -name "$LANG_PROJECT_MARKER" "${PY_PRUNE[@]}" 2>/dev/null | sort) } lang_has_code() { - find . -name "$LANG_CODE_GLOB" $(py_prune) -print -quit 2>/dev/null | grep -q . + find . -name "$LANG_CODE_GLOB" "${PY_PRUNE[@]}" -print -quit 2>/dev/null | grep -q . } LANG_DECL_AWK=' @@ -101,7 +101,7 @@ FNR == 1 { ' lang_declarations() { - find . -name "$LANG_CODE_GLOB" $(py_prune) -print0 2>/dev/null \ + find . -name "$LANG_CODE_GLOB" "${PY_PRUNE[@]}" -print0 2>/dev/null \ | xargs -0 awk "$LANG_DECL_AWK" } diff --git a/linter-driven-development/scripts/check-repo-brain.sh b/linter-driven-development/scripts/check-repo-brain.sh index c7c76ff7..2d470690 100755 --- a/linter-driven-development/scripts/check-repo-brain.sh +++ b/linter-driven-development/scripts/check-repo-brain.sh @@ -8,10 +8,22 @@ # in-repo copy). Installed into target repos by the documentation skill's # BOOTSTRAP pass (/wire-repo-brain). # -# Usage: bash scripts/check-repo-brain.sh [--fix] [repo-root] (default: cwd) +# Usage: bash scripts/check-repo-brain.sh [--fix] [--scope [--words ]] [repo-root] +# (repo-root default: cwd) # CI: one line — bash scripts/check-repo-brain.sh # --fix: rewrite drifted index lines from each target doc's `description` # (the one mechanical repair; everything else stays report-only) +# --scope a review's scope: one path per line, relative to repo-root +# (code files and docs, deleted ones included). The gate then checks +# only the docs in scope: the listed docs, the docs whose text names a +# listed file's basename, and the docs that cite a word of --words. +# Code edges are checked for the listed code files only, Q3 is skipped +# (root wiring is the repository's state, not a diff's), and with no +# doc in scope the doc checks are skipped. One stdout line says which: +# "check-repo-brain: scope — doc(s) to check" or +# "check-repo-brain: scope — no doc in scope; doc checks skipped". +# --words identifiers the scoped change removes, one per line; a doc +# that cites one of them in backticks is in scope # # Doc roots are discovered at the repo root AND at every sub-project (a # directory holding the language's project marker — see the adapter below), @@ -58,8 +70,11 @@ # against its declaring package or receiver — never against a same-named # member elsewhere. A token missing from the set still resolves when it # appears as a whole word in any non-markdown repo file (config keys, -# alert names, test helpers). A `pkg.Sym` whose package is not declared in -# this repo is external (stdlib, dependencies) and exempt. +# alert names, test helpers): in a git work tree the tracked and +# untracked-but-not-ignored files; elsewhere every file outside the +# dependency and build-output directories (bazel-*, node_modules, …). +# A `pkg.Sym` whose package is not declared in this repo is external +# (stdlib, dependencies) and exempt. # - lines carrying the ⚠️ stale flag or a *(planned)* marker are exempt from # symbol resolution and the description copy check (R9 Q2/Q7 exemptions); # the file:line ban has no exemption beyond URL spans, fenced code blocks, @@ -72,22 +87,37 @@ # 2 usage error, or an internal scanner failure (a failed scan is # inconclusive, never silently clean) # -# Uses only POSIX-portable tools: find, grep, sed, awk, head, sort, wc. No jq/python. +# Uses only POSIX-portable tools: find, grep, sed, awk, head, sort, wc, tr, xargs, +# plus git when the repository is a git work tree. No jq/python. # awk programs avoid interval expressions ({m,n}) — mawk, Debian's default, rejects them. set -u FIX=0 -if [[ "${1:-}" == "--fix" ]]; then - FIX=1 - shift -fi +SCOPE_LIST="" +WORDS_LIST="" +abs_list() { # -> absolute path of an existing file, or exit 2 + [[ -f "$1" ]] || { echo "check-repo-brain: no such list: $1" >&2; exit 2; } + printf '%s/%s\n' "$(cd "$(dirname "$1")" && pwd)" "$(basename "$1")" +} +while (( $# > 0 )); do + case "$1" in + --fix) FIX=1; shift ;; + --scope) SCOPE_LIST=$(abs_list "${2:-}") || exit 2; shift 2 ;; + --words) WORDS_LIST=$(abs_list "${2:-}") || exit 2; shift 2 ;; + --*) echo "check-repo-brain: unknown option: $1" >&2; exit 2 ;; + *) break ;; + esac +done +[[ -n "$WORDS_LIST" && -z "$SCOPE_LIST" ]] && { echo "check-repo-brain: --words needs --scope" >&2; exit 2; } REPO_ROOT="${1:-$(pwd)}" if [[ ! -d "$REPO_ROOT" ]]; then echo "check-repo-brain: not a directory: $REPO_ROOT" >&2 exit 2 fi cd "$REPO_ROOT" || exit 2 +SCOPED=0 +[[ -n "$SCOPE_LIST" ]] && SCOPED=1 # ==================== language adapter: detected ==================== # Everything language-specific lives between these two marker comments. The @@ -238,10 +268,10 @@ LANG_FILE_RE='\\.py(:[0-9]+)?$' LANG_SYMBOL_RE='^([A-Z][A-Za-z0-9_]*|[a-z_][a-z0-9_]*_[a-z0-9_]*)$' LANG_QUALIFIED_RE='^[A-Za-z_][A-Za-z0-9_]*\\.[A-Za-z_][A-Za-z0-9_]*$' -py_prune() { # find(1) pruning shared by the three walkers - printf '%s\n' -not -path '*/.venv/*' -not -path '*/venv/*' -not -path '*/node_modules/*' \ - -not -path '*/__pycache__/*' -not -path '*/site-packages/*' -not -path './.git/*' -} +# find(1) pruning shared by the three walkers; an array, so a pattern such as +# ./.git/* reaches find as it is and never expands against the tree +PY_PRUNE=(-not -path '*/.venv/*' -not -path '*/venv/*' -not -path '*/node_modules/*' + -not -path '*/__pycache__/*' -not -path '*/site-packages/*' -not -path './.git/*') lang_project_dirs() { local m p @@ -249,11 +279,11 @@ lang_project_dirs() { p=$(dirname "$m"); p="${p#./}" [[ "$p" == "." || -z "$p" ]] && continue printf '%s\n' "$p" - done < <(find . -name "$LANG_PROJECT_MARKER" $(py_prune) 2>/dev/null | sort) + done < <(find . -name "$LANG_PROJECT_MARKER" "${PY_PRUNE[@]}" 2>/dev/null | sort) } lang_has_code() { - find . -name "$LANG_CODE_GLOB" $(py_prune) -print -quit 2>/dev/null | grep -q . + find . -name "$LANG_CODE_GLOB" "${PY_PRUNE[@]}" -print -quit 2>/dev/null | grep -q . } LANG_DECL_AWK=' @@ -304,7 +334,7 @@ FNR == 1 { ' lang_declarations() { - find . -name "$LANG_CODE_GLOB" $(py_prune) -print0 2>/dev/null \ + find . -name "$LANG_CODE_GLOB" "${PY_PRUNE[@]}" -print0 2>/dev/null \ | xargs -0 awk "$LANG_DECL_AWK" } @@ -447,22 +477,65 @@ lifecycle_stale() { # -> reason when frontmatter marks the doc stale ('' have_code=0 lang_has_code && have_code=1 -# ---------- declaration set: built once, queried per token ---------- -DECLS="" PKGS="" -if (( have_code )); then - DECLS=$(mktemp) PKGS=$(mktemp) DECL_ALL=$(mktemp) - trap 'rm -f "$DECLS" "$PKGS"' EXIT - if ! ( set -o pipefail; lang_declarations | sort -u > "$DECL_ALL" ); then +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +# ---------- declaration set: built once, on the first token to resolve ---------- +DECLS="$WORK/decls" PKGS="$WORK/pkgs" +decls_built=0 +build_declarations() { + (( decls_built )) && return 0 + decls_built=1 + if ! ( set -o pipefail; lang_declarations | sort -u > "$WORK/decl-all" ); then echo "check-repo-brain: internal scanner error — the declaration scan failed; treating the run as inconclusive" >&2 exit 2 fi - grep '^pkg:' "$DECL_ALL" | sed 's/^pkg://' > "$PKGS" - grep -v '^pkg:' "$DECL_ALL" > "$DECLS" - rm -f "$DECL_ALL" -fi + grep '^pkg:' "$WORK/decl-all" | sed 's/^pkg://' > "$PKGS" + grep -v '^pkg:' "$WORK/decl-all" > "$DECLS" + rm -f "$WORK/decl-all" +} is_repo_pkg() { grep -qxF "$1" "$PKGS" 2>/dev/null; } +# repo_files — the repository's non-markdown files, NUL-separated. A git work +# tree lists them through git (tracked files plus untracked ones git does not +# ignore), which never walks build outputs or symlinked caches; elsewhere find +# skips the directories that hold dependencies and build outputs. +repo_files() { + if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then + git ls-files -z --cached --others --exclude-standard -- . \ + ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**' 2>/dev/null && return 0 + fi + find . \( -name .git -o -name vendor -o -name node_modules -o -name 'bazel-*' -o -name .venv \ + -o -name venv -o -name __pycache__ -o -name .tox -o -name dist -o -name target \) -prune \ + -o -type f ! -name '*.md' -print0 +} + +# resolve_words — the tokens that occur as a whole word in any +# non-markdown repository file, one per line. The repository is read once +# into words (runs of [A-Za-z0-9_.]); a dotted token such as `Type.Method` +# matches any run of whole dot-separated parts, as grep -w would match it. +# One pass over the text is linear in its size, where a fixed-string grep for +# hundreds of tokens is not. +resolve_words() { + repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.' '\n' \ + | LC_ALL=C awk -v list="$1" ' + BEGIN { while ((getline t < list) > 0) if (t != "") want[t] = 1 } + seen[$0]++ { next } + { + n = split($0, p, ".") + for (i = 1; i <= n; i++) { + s = p[i] + if (s in want) { print s; delete want[s] } + for (j = i + 1; j <= n; j++) { + s = s "." p[j] + if (s in want) { print s; delete want[s] } + } + } + }' + return 0 +} + # One fence-aware awk pass per bundle extracts everything Q2's doc scan needs: # P — a file:line citation outside fences/URLs/globs # S — a backticked symbol-shaped token to resolve @@ -501,8 +574,53 @@ FNR == 1 { fence = 0 } } ' -# ---------- Q2: code→docs edges (repo-wide; resolved from repo root, then the -# citing file's own sub-project) ---------- +# ---------- the scope: which docs a scoped run checks ---------- +# A doc is in scope when the scope lists it, when its text names the basename +# of a listed file as a whole word (a link to a touched doc or code file), or +# when it cites in backticks a word the change removes (a symbol the diff +# renames or deletes). +# Every list below holds relative paths as find(1) prints them under a root. +SCOPE_DOCS="$WORK/scope-docs" +SCOPE_PATHS="$WORK/scope-paths" +if (( SCOPED )); then + sed -e 's|^\./||' -e '/^$/d' "$SCOPE_LIST" | LC_ALL=C sort -u > "$SCOPE_PATHS" + for r in "${ROOTS[@]}"; do find "$r" -type f -name '*.md'; done | LC_ALL=C sort -u > "$WORK/all-docs" + { + grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" + sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$WORK/basenames" + if [[ -s "$WORK/basenames" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$WORK/basenames" 2>/dev/null + fi + if [[ -n "$WORDS_LIST" && -s "$WORDS_LIST" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" \ + | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ + -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" "$DOCSCAN_AWK" \ + | awk -F'\t' 'NR == FNR { w[$0] = 1; next } + $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in w) || (m in w)) print $2 }' "$WORDS_LIST" - + fi + } | LC_ALL=C sort -u > "$SCOPE_DOCS" + if [[ -s "$SCOPE_DOCS" ]]; then + echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" + else + echo "check-repo-brain: scope — no doc in scope; doc checks skipped" + fi +fi + +# bundle_docs [name] — the docs a run checks under one root: every +# *.md (or every file of that name), limited to the scope on a scoped run +bundle_docs() { + local name="${2:-*.md}" + if (( SCOPED )); then + find "$1" -type f -name "$name" | grep -xF -f "$SCOPE_DOCS" + else + find "$1" -type f -name "$name" + fi + return 0 +} + +# ---------- Q2: code→docs edges (repo-wide, or the scope's code files on a +# scoped run; resolved from repo root, then the citing file's own +# sub-project) ---------- docroot_for_file() { # -> docroot of the longest matching project dir local f="${1#./}" best="" i for i in "${!PROJS[@]}"; do @@ -526,7 +644,13 @@ if (( have_code )); then CUR_DOCROOT=$(docroot_for_file "$file") fail "[Q2] $file:$line — code edge points at missing $target" fi - done < <(lang_code_edges) + done < <(if (( SCOPED )); then + lang_code_edges | awk -v list="$SCOPE_PATHS" ' + BEGIN { while ((getline l < list) > 0) s[l] = 1 } + { f = substr($0, 1, index($0, ":") - 1); sub(/^\.\//, "", f); if (f in s) print }' + else + lang_code_edges + fi) fi # ---------- per-bundle checks ---------- @@ -536,6 +660,8 @@ check_bundle() { # local root_index="$docroot/index.md" local root_index_c root_index_c=$(canon "$root_index") + # a scoped run skips a bundle that holds no doc in scope + (( SCOPED )) && [[ -z "$(bundle_docs "$docroot")" ]] && return 0 # --- Q1: transitive reachability from the bundle's root index --- local reachable="" visited="" queue=("$root_index") @@ -564,7 +690,7 @@ check_bundle() { # case "$reachable" in *"$c"$'\n'*) ;; *) fail "[Q1] $doc — orphan: not reachable from $root_index" ;; esac - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q2: every doc link resolves (a missing parent directory is just as # broken as a missing file — only URLs/anchors are exempt) --- @@ -579,13 +705,13 @@ check_bundle() { # fail "[Q2] $md — link target does not exist: $t" fi done < <(grep -oE '\]\([^)]+\)' "$md" 2>/dev/null) - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q2: doc scan — file:line ban + docs→code symbol resolution. # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). --- local scan; scan=$(mktemp) if ! ( set -o pipefail - find "$docroot" -type f -name '*.md' -print0 \ + bundle_docs "$docroot" | tr '\n' '\0' \ | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" \ "$DOCSCAN_AWK" > "$scan" ); then @@ -600,6 +726,7 @@ check_bundle() { # if (( have_code )) && grep -q $'^S\t' "$scan"; then local toks check members unres bad toks=$(mktemp) check=$(mktemp) members=$(mktemp) unres=$(mktemp) bad=$(mktemp) + build_declarations grep $'^S\t' "$scan" | cut -f4 | sort -u > "$toks" # tokens to resolve (external pkg.Sym exempt). A qualified token resolves # as the PAIR itself — the declaration set carries pkg.Ident / Type.Method @@ -617,12 +744,11 @@ check_bundle() { # esac done < "$toks" > "$check" cut -f2 "$check" | sort -u > "$members" - grep -vxF -f "$DECLS" "$members" > "$unres" || true + grep -vxF -f "$DECLS" "$members" | grep -v '^$' > "$unres" || true if [[ -s "$unres" ]]; then - # ONE repo-wide word grep for the whole unresolved batch + # ONE pass over the repository for the whole unresolved batch local found; found=$(mktemp) - grep -rIhoFw --exclude-dir=vendor --exclude-dir=.git --exclude='*.md' \ - -f "$unres" . 2>/dev/null | sort -u > "$found" + resolve_words "$unres" | sort -u > "$found" grep -vxF -f "$found" "$unres" > "$bad" || true rm -f "$found" fi @@ -640,24 +766,27 @@ check_bundle() { # fi rm -f "$scan" - # --- Q3: root wiring (exact path; sub-roots may ride the repo-root index) --- - local rel="$docroot" - [[ "$proj" != "." ]] && rel="${docroot#$proj/}" - local claude="CLAUDE.md" agents="AGENTS.md" - [[ "$proj" != "." ]] && { claude="$proj/CLAUDE.md"; agents="$proj/AGENTS.md"; } - local wired_claude=0 wired_agents=0 - [[ -f "$claude" ]] && grep -qF -- "$rel/index.md" "$claude" && wired_claude=1 - [[ -f "$agents" ]] && grep -qF -- "$rel/index.md" "$agents" && wired_agents=1 - if (( ! wired_claude && ! wired_agents )); then - local via_root=0 - if [[ "$proj" != "." && -n "$ROOT_BUNDLE" ]]; then - grep -rqF -- "$docroot/index.md" "$ROOT_BUNDLE" --include='*.md' 2>/dev/null && via_root=1 - fi - if (( ! via_root )); then - fail "[Q3] $proj — neither $claude nor $agents references $rel/index.md" + # --- Q3: root wiring (exact path; sub-roots may ride the repo-root index); + # a scoped run leaves it to a whole-repository run --- + if (( ! SCOPED )); then + local rel="$docroot" + [[ "$proj" != "." ]] && rel="${docroot#$proj/}" + local claude="CLAUDE.md" agents="AGENTS.md" + [[ "$proj" != "." ]] && { claude="$proj/CLAUDE.md"; agents="$proj/AGENTS.md"; } + local wired_claude=0 wired_agents=0 + [[ -f "$claude" ]] && grep -qF -- "$rel/index.md" "$claude" && wired_claude=1 + [[ -f "$agents" ]] && grep -qF -- "$rel/index.md" "$agents" && wired_agents=1 + if (( ! wired_claude && ! wired_agents )); then + local via_root=0 + if [[ "$proj" != "." && -n "$ROOT_BUNDLE" ]]; then + grep -rqF -- "$docroot/index.md" "$ROOT_BUNDLE" --include='*.md' 2>/dev/null && via_root=1 + fi + if (( ! via_root )); then + fail "[Q3] $proj — neither $claude nor $agents references $rel/index.md" + fi + elif (( ! wired_agents )); then + note "[Q3] $agents lacks the $rel/index.md routing reference — AGENTS.md-reading tools start blind" fi - elif (( ! wired_agents )); then - note "[Q3] $agents lacks the $rel/index.md routing reference — AGENTS.md-reading tools start blind" fi # --- Q7: bundle contract --- @@ -722,7 +851,7 @@ check_bundle() { # else fail "[Q7] $md — frontmatter missing 'description:'" fi - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q7: drift check — index line text == target's description (⚠️ exempt; # --fix rewrites drifted lines after the read loop, never during it) --- @@ -760,12 +889,12 @@ check_bundle() { # echo " fixed: $idx:$n — index line rewritten from its target's description" done <<< "$fixes" fi - done < <(find "$docroot" -type f -name 'index.md') + done < <(bundle_docs "$docroot" index.md) # --- Q7: no log.md --- while IFS= read -r lg; do fail "[Q7] $lg — log.md is reserved for change history; docs describe current behavior" - done < <(find "$docroot" -type f -name 'log.md') + done < <(bundle_docs "$docroot" log.md) } for i in "${!PROJS[@]}"; do diff --git a/linter-driven-development/scripts/check-repo-brain_test.sh b/linter-driven-development/scripts/check-repo-brain_test.sh index effe8d1c..fc37f32a 100755 --- a/linter-driven-development/scripts/check-repo-brain_test.sh +++ b/linter-driven-development/scripts/check-repo-brain_test.sh @@ -769,6 +769,75 @@ run_gate expect_exit 0 && expect_not "does not resolve" && expect_has "check-repo-brain: clean" && ok finish +# --- scoped runs (--scope, --words) and the symbol resolver --- +LISTS=$(mktemp -d) # the scope and word lists live outside the repository they describe +begin "--scope: a code-only scope that no doc cites skips the doc checks" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here." +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "scope — no doc in scope; doc checks skipped" && expect_not "orphan" && ok +finish + +begin "--scope: a listed doc is checked, an unlisted orphan and the root wiring are not" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here."; mk_doc other "another unlinked doc" "Nothing." +rm "$REPO/CLAUDE.md" "$REPO/AGENTS.md" +printf '%s\n' "$FX_CODE_FILE" docs/orphan.md > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] docs/orphan.md — orphan" \ + && expect_not "docs/other.md" && expect_not "[Q3]" && ok +finish + +begin "--scope: a doc that names a scope file is checked" +mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../$FX_CODE_FILE)." +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +finish + +begin "--scope --words: a doc that cites a removed symbol is checked" +mk_conformant +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +append_index "- [ghost.md](ghost.md) — a ghost symbol" +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "no doc in scope" \ + && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ + && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok +finish + +begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" +mk_conformant +printf 'build/\n' > "$REPO/.gitignore" +mkdir -p "$REPO/build"; printf 'IgnoredOnlyName\n' > "$REPO/build/gen.yaml" +printf 'TrackedName\n' > "$REPO/tracked.yaml" +mk_doc names "alert names" "Fires \`TrackedName\`, \`UntrackedName\` and \`IgnoredOnlyName\`." +append_index "- [names.md](names.md) — alert names" +(cd "$REPO" && git init -q && git add -A && git -c user.name=t -c user.email=t@t commit -qm init) +printf 'UntrackedName\n' > "$REPO/untracked.yaml" +run_gate +expect_exit 1 && expect_has "\`IgnoredOnlyName\` does not resolve" \ + && expect_not "\`TrackedName\` does not resolve" && expect_not "\`UntrackedName\` does not resolve" && ok +finish + +begin "Q2 outside git a token does not resolve from a bazel-* output directory" +mk_conformant +mkdir -p "$REPO/bazel-out"; printf 'BazelOnlyName\n' > "$REPO/bazel-out/gen.yaml" +mk_doc bazel "bazel names" "Fires \`BazelOnlyName\`." +append_index "- [bazel.md](bazel.md) — bazel names" +run_gate +expect_exit 1 && expect_has "\`BazelOnlyName\` does not resolve" && ok +finish + +begin "Q2 a dotted token resolves on whole dot-separated words only" +mk_conformant +printf 'limits:\n key: Cfg.MaxRetries.value\n other: Cfg.MaxRetriesX\n' > "$REPO/limits.yaml" +mk_doc dotted "dotted names" "Reads \`Cfg.MaxRetries\`; \`Cfg.MaxRetri\` is a typo." +append_index "- [dotted.md](dotted.md) — dotted names" +run_gate +expect_exit 1 && expect_has "\`Cfg.MaxRetri\` does not resolve" && expect_not "\`Cfg.MaxRetries\` does not resolve" && ok +finish +rm -rf "$LISTS" + } # run_row_cases for ROW in $FX_ROWS; do diff --git a/linter-driven-development/scripts/ldd-detect.sh b/linter-driven-development/scripts/ldd-detect.sh index 69f08363..40e37670 100755 --- a/linter-driven-development/scripts/ldd-detect.sh +++ b/linter-driven-development/scripts/ldd-detect.sh @@ -25,7 +25,14 @@ # --plan print the parsed detect lines and exit; nothing runs # --glob the scope is one group of a language the table # does not know, matched by this glob -# --test-re test-file pattern over relative paths, same case +# --test-re test-file pattern over relative paths, same case. +# A bundle carries the pattern ldd-scope.sh used +# (test-re.txt), and a custom group its glob +# (language.txt): the bundle's values win, and a +# --glob or --test-re that differs is a usage error +# --gate-budget wall-clock seconds the R9 gate may run (default: no +# limit); on expiry the gate is stopped and its +# questions are counted "inconclusive", never 0 # # The scope's language is the bundle's language.txt; without it, the files in # files.txt tell it by extension, and a list that spans several languages is @@ -39,10 +46,14 @@ # Detect-path: `` over the scope's relative paths (layer directories, # role-named packages); a hit's line is 0 # Detect-gate: Q the R9 gate's [Q] report lines, from one run of -# check-repo-brain.sh beside this script over --root; on a -# scoped bundle (no dirs.txt) only the lines whose file is in -# the scope — the gate's repository-wide findings belong to -# a whole-repository review +# check-repo-brain.sh beside this script over --root. On a +# scoped bundle (no dirs.txt) the gate runs with --scope: it +# checks only the bundle's docs (docs.txt), the docs that +# link to a scope file and the docs that cite a symbol the +# diff removes, the code edges of the scope's code files, +# and no root wiring; with no such doc its doc checks are +# skipped. The repository-wide findings belong to a +# whole-repository review # Detect: judgment no mechanical lead; the hunter runs the prose # A question with no detect line, two of them, or a malformed one is an error: # the generator's lint-core keeps the rule files honest, and this script refuses @@ -53,15 +64,18 @@ # the overflow row of a capped question: file "-", line 0, # excerpt "+ more hit(s) not listed" # hits-all.tsv the same columns, every hit, no cap and no overflow row -# counts.tsv rule question kind hits (judgment rows carry "-") +# counts.tsv rule question kind hits (judgment rows carry "-", gate +# rows "inconclusive" when the gate ran out of --gate-budget) # the last row is SUPPRESS - grep : suppression directives # on the diff's added lines when the bundle has a diff.patch, # else over every scope file # Two runs over one tree write byte-identical counts.tsv. # # Exit codes: 0 tables written · 2 usage error, a detect line that does not -# parse, a pattern grep rejects, or a gate run that failed -# (inconclusive, never silently clean) +# parse, a pattern grep rejects, a gate run that failed +# (inconclusive, never silently clean), or a non-empty scope in +# which no file matches the source glob (a glob mismatch, never +# an empty table) # # Uses only POSIX-portable tools: grep, sed, awk, sort, wc, head. GNU grep's # \b and \s are assumed, as the rules already assume them. @@ -77,6 +91,7 @@ CAP=40 PLAN=0 OPT_GLOB="" OPT_TEST_RE="" +GATE_BUDGET=0 BUNDLE="" usage() { sed -n '2,/^$/p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; } @@ -94,12 +109,14 @@ while (( $# > 0 )); do --plan) PLAN=1; shift ;; --glob) value "$1" $#; OPT_GLOB="$2"; shift 2 ;; --test-re) value "$1" $#; OPT_TEST_RE="$2"; shift 2 ;; + --gate-budget) value "$1" $#; GATE_BUDGET="$2"; shift 2 ;; -h|--help) usage ;; --*) die "unknown option: $1" ;; *) [[ -n "$BUNDLE" ]] && die "one bundle directory, got also: $1"; BUNDLE="$1"; shift ;; esac done [[ "$CAP" =~ ^[0-9]+$ ]] || die "--cap takes a number, got: $CAP" +[[ "$GATE_BUDGET" =~ ^[0-9]+$ ]] || die "--gate-budget takes a number of seconds, got: $GATE_BUDGET" [[ -d "$RULES_DIR" ]] || die "no rules directory: $RULES_DIR" RULES_DIR=$(cd "$RULES_DIR" && pwd) if (( ! PLAN )); then @@ -358,6 +375,7 @@ if (( ! PLAN )) && [[ -f "$BUNDLE/groups.txt" ]]; then fi args=(--root "$ROOT" --cap "$CAP") [[ -n "$OPT_TEST_RE" ]] && args+=(--test-re "$OPT_TEST_RE") + (( GATE_BUDGET > 0 )) && args+=(--gate-budget "$GATE_BUDGET") bash "$script" "${args[@]}" "$bundle" || { failed=1; echo "$SCRIPT_NAME: the $id group's detection pass failed; it is inconclusive" >&2; } echo done < "$BUNDLE/groups.txt" @@ -372,7 +390,9 @@ LANG_ID="" if (( ! PLAN )); then if [[ -f "$BUNDLE/language.txt" ]]; then IFS=$'\t' read -r LANG_ID glob < "$BUNDLE/language.txt" - [[ "$LANG_ID" == "custom" && -z "$OPT_GLOB" ]] && OPT_GLOB="$glob" + [[ "$LANG_ID" == "custom" && -n "$OPT_GLOB" && "$OPT_GLOB" != "$glob" ]] \ + && die "--glob '$OPT_GLOB' differs from the bundle's glob '$glob' (ldd-scope.sh wrote it; drop --glob)" + [[ "$LANG_ID" == "custom" ]] && OPT_GLOB="$glob" else ids="" while IFS= read -r line; do @@ -387,6 +407,14 @@ if (( ! PLAN )); then [[ "$ids" == *" "* ]] && die "the scope list spans several languages ($ids); ldd-scope.sh writes one bundle per language" LANG_ID="$ids" fi + # the test-file pattern ldd-scope.sh used, so the two scripts cannot disagree + # on what a test file is + if [[ -f "$BUNDLE/test-re.txt" ]]; then + b_test_re=$(cat "$BUNDLE/test-re.txt") + [[ -n "$OPT_TEST_RE" && "$OPT_TEST_RE" != "$b_test_re" ]] \ + && die "--test-re '$OPT_TEST_RE' differs from the bundle's '$b_test_re' (ldd-scope.sh wrote it; drop --test-re)" + OPT_TEST_RE="$b_test_re" + fi lang_configure "$LANG_ID" || die "this plugin has no language block for $LANG_ID" fi @@ -432,7 +460,7 @@ END { if (insec) flush() } ' plan_file=$(mktemp) -trap 'rm -f "$plan_file" "${tmp_hits:-}" "${tmp_hits_all:-}" "${tmp_counts:-}" "${gate_out:-}"' EXIT +trap 'rm -f "$plan_file" "${tmp_hits:-}" "${tmp_hits_all:-}" "${tmp_counts:-}" "${gate_out:-}" "${gate_scope:-}" "${gate_words:-}"' EXIT rule_files=$(find "$RULES_DIR" -maxdepth 1 -name 'R*.md' | awk -F/ '{ n = $NF; sub(/^R/, "", n); sub(/-.*$/, "", n); print n "\t" $0 }' | sort -n | cut -f2) [[ -n "$rule_files" ]] || die "no rule files (R*.md) under $RULES_DIR" while IFS= read -r f; do @@ -458,11 +486,13 @@ fi SRC_FILES=() TEST_FILES=() ALL_FILES=() +n_present=0 while IFS= read -r line; do f="${line%% (not bundled:*}" f="${f#./}" [[ -z "$f" ]] && continue [[ -f "$f" ]] || continue + n_present=$((n_present + 1)) [[ "$(lang_of_path "$f")" == "$LANG_ID" ]] || continue printf '%s\n' "$f" | grep -qE -- "$LANG_EXCLUDE_RE" && continue ALL_FILES+=("$f") @@ -470,6 +500,12 @@ done < <(LC_ALL=C sort -u "$FILES_LIST") for f in ${ALL_FILES[@]+"${ALL_FILES[@]}"}; do if lang_is_test "$f"; then TEST_FILES+=("$f"); else SRC_FILES+=("$f"); fi done +# A scope whose files exist but none of which is a source file is a glob +# mismatch between the scope and this run: an all-zero table would read as a +# clean review of nothing. +if (( ${#ALL_FILES[@]} == 0 && n_present > 0 )); then + die "0 source + 0 test files: none of the $n_present file(s) in $FILES_LIST is a $LANG_ID source file (pass the --glob ldd-scope.sh used)" +fi tmp_hits=$(mktemp) tmp_hits_all=$(mktemp) @@ -583,24 +619,86 @@ run_path() { # rm -f "$raw" "$err" } -# The R9 gate runs once, lazily; its [Q] lines are split by question. +# gate_scope_lists — what a scoped gate run checks: +# every path of the scope (deleted ones too: a doc can still link to them) and +# of docs.txt; and the identifiers the diff removes — on its removed lines and +# on none of its added ones — the symbols a renamed or deleted declaration +# takes away from the docs that cite them. Comment lines (a line that opens +# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped. +gate_scope_lists() { + local scope="$1" words="$2" line f + { + while IFS= read -r line; do + f="${line%% (not bundled:*}"; f="${f#./}" + [[ -n "$f" ]] && printf '%s\n' "$f" + done < "$FILES_LIST" + [[ -f "$BUNDLE/docs.txt" ]] && cat "$BUNDLE/docs.txt" + } | LC_ALL=C sort -u > "$scope" + : > "$words" + [[ -s "$BUNDLE/diff.patch" ]] || return 0 + grep -vE '^(\+\+\+|---) ' "$BUNDLE/diff.patch" | grep -E '^[-+]' \ + | LC_ALL=C awk ' + { + s = substr($0, 1, 1); t = substr($0, 2) + if (t ~ /^[ \t]*(\/\/|#|\/\*|\*|--|""")/) next + while (match(t, /[A-Za-z_][A-Za-z0-9_]*/)) { + w = substr(t, RSTART, RLENGTH); t = substr(t, RSTART + RLENGTH) + if (s == "+") add[w] = 1; else del[w] = 1 + } + } + END { for (w in del) if (!(w in add)) print w }' | LC_ALL=C sort > "$words" +} + +# The R9 gate runs once, lazily; its [Q] lines are split by question. With +# --gate-budget the gate runs as its own process group, so expiry stops every +# process it started, not only its shell. +GATE_STATE="whole repository" run_gate_once() { [[ -n "$gate_out" ]] && return 0 gate_out=$(mktemp) - local gate="$SCRIPT_DIR/check-repo-brain.sh" code + local gate="$SCRIPT_DIR/check-repo-brain.sh" code=0 start pid scope_line + local -a args=() [[ -f "$gate" ]] || die "Detect-gate needs $gate beside this script" - bash "$gate" "$ROOT" > "$gate_out" 2>&1 - code=$? + if [[ ! -f "$BUNDLE/dirs.txt" ]]; then + gate_scope=$(mktemp); gate_words=$(mktemp) + gate_scope_lists "$gate_scope" "$gate_words" + args=(--scope "$gate_scope" --words "$gate_words") + fi + if (( GATE_BUDGET == 0 )); then + bash "$gate" ${args[@]+"${args[@]}"} "$ROOT" > "$gate_out" 2>&1 || code=$? + else + start=$SECONDS + set -m + bash "$gate" ${args[@]+"${args[@]}"} "$ROOT" > "$gate_out" 2>&1 & + pid=$! + set +m + while kill -0 "$pid" 2>/dev/null; do + if (( SECONDS - start >= GATE_BUDGET )); then + kill -TERM -- "-$pid" 2>/dev/null + wait "$pid" 2>/dev/null + GATE_STATE="inconclusive — stopped after the ${GATE_BUDGET}s --gate-budget" + return 0 + fi + sleep 0.2 + done + wait "$pid" || code=$? + fi if (( code == 2 )); then head -5 "$gate_out" >&2 die "the R9 gate failed to run (exit 2); its questions are inconclusive" fi + scope_line=$(grep -m1 '^check-repo-brain: scope — ' "$gate_out") + [[ -n "$scope_line" ]] && GATE_STATE="scoped, ${scope_line#check-repo-brain: scope — }" return 0 } run_gate() { # local rule="$1" q="$2" qn="$3" raw tab run_gate_once + if [[ "$GATE_STATE" == inconclusive* ]]; then + printf '%s\t%s\tgate\tinconclusive\n' "$rule" "$q" >> "$tmp_counts" + return 0 + fi raw=$(mktemp) tab=$(printf '\t') # "[Qn] [:line] — — see " becomes @@ -618,15 +716,9 @@ run_gate() { # if (match(loc, /:[0-9]+$/)) { file = substr(loc, 1, RSTART - 1); line = substr(loc, RSTART + 1) + 0 } print file "\t" line "\t" rest }' > "$raw" - # A scoped review owns only the gate lines about its own files: an orphan - # doc, an unwired root or another file's broken edge is the repository's - # state, not the diff's, and is not a lead for this review's hunter. - if [[ ! -f "$BUNDLE/dirs.txt" ]] && [[ -s "$raw" ]]; then - scoped=$(mktemp); scope_list=$(mktemp) - printf '%s\n' ${ALL_FILES[@]+"${ALL_FILES[@]}"} > "$scope_list" - awk -F'\t' 'NR == FNR { in_scope[$0] = 1; next } ($1 in in_scope)' "$scope_list" "$raw" > "$scoped" - mv "$scoped" "$raw"; rm -f "$scope_list" - fi + # A scoped review owns only the gate lines about its own files and the docs + # that cite them; the gate's --scope run reports nothing else (an orphan doc + # elsewhere or an unwired root is the repository's state, not the diff's). record "$rule" "$q" gate "$raw" 0 rm -f "$raw" } @@ -676,6 +768,7 @@ n_gate=$(awk -F'\t' '$3 == "gate"' "$plan_file" | wc -l | tr -d ' ') n_judg=$(awk -F'\t' '$3 == "judgment"' "$plan_file" | wc -l | tr -d ' ') n_q=$(wc -l < "$plan_file" | tr -d ' ') echo "$SCRIPT_NAME: $n_q questions ($n_grep grep, $n_path path, $n_gate gate, $n_judg judgment) over ${#SRC_FILES[@]} source + ${#TEST_FILES[@]} test files → $BUNDLE/hits.tsv, counts.tsv" +(( n_gate > 0 )) && [[ -n "$gate_out" ]] && echo "R9 gate: $GATE_STATE" echo awk -F'\t' 'BEGIN { printf "%-9s %-4s %-9s %s\n", "rule", "q", "kind", "hits" } { q = ($2 == "-") ? "-" : "Q" $2; printf "%-9s %-4s %-9s %s\n", $1, q, $3, $4 }' "$BUNDLE/counts.tsv" @@ -695,12 +788,14 @@ awk -F'\t' ' f = fam[$1]; if (f == "") f = "other" nq[f]++ if ($3 == "judgment") { nj[f]++; next } + if ($4 == "inconclusive") { ni[f]++; next } if ($4 + 0 > 0) { hits[f] += $4; nh[f]++; rh[f, $1] += $4 } } END { for (i = 1; i <= 4; i++) { f = order[i] printf "%-15s %5d hit(s) in %d of %d questions, %d judgment", f, hits[f] + 0, nh[f] + 0, nq[f] + 0, nj[f] + 0 + if (ni[f] + 0 > 0) printf ", %d inconclusive", ni[f] per = ""; n = split(rules[f], rs, " ") for (j = 1; j <= n; j++) if (rh[f, rs[j]] + 0 > 0) per = per (per == "" ? "" : " · ") rs[j] " " rh[f, rs[j]] if (per != "") printf " — %s", per diff --git a/linter-driven-development/scripts/ldd-detect_test.sh b/linter-driven-development/scripts/ldd-detect_test.sh index fe0cfc71..ff83999c 100644 --- a/linter-driven-development/scripts/ldd-detect_test.sh +++ b/linter-driven-development/scripts/ldd-detect_test.sh @@ -49,6 +49,11 @@ run_detect() { # [options...] — runs the script over $REPO with $BUNDLE OUT=$(cd "$REPO" && bash "$DETECT" "$@" --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" } +run_detect_bundle() { # [options...] — runs the script over $REPO with that bundle + local b="$1"; shift + OUT=$(cd "$REPO" && bash "$DETECT" "$@" --root . "$b" 2>"$BUNDLE/.stderr"); CODE=$? + ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +} run_scope() { # [args...] — runs ldd-scope.sh over $REPO into $BUNDLE/scope-out rm -rf "$BUNDLE/scope-out" OUT=$(cd "$REPO" && bash "$SCOPE_SH" --out "$BUNDLE/scope-out" "$@" 2>"$BUNDLE/.stderr"); CODE=$? @@ -75,8 +80,9 @@ expect_not() { return 0 } expect_quiet() { [[ -z "$ERR" ]] || { bad "stderr not empty"; return 1; }; } -count_of() { # — the hits cell of counts.tsv, "-" for judgment - awk -F'\t' -v r="$1" -v q="$2" '$1 == r && $2 == q { print $4 }' "$BUNDLE/counts.tsv" +count_of() { # — the hits cell of counts.tsv (COUNTS_FILE, default the + # case bundle's), "-" for judgment + awk -F'\t' -v r="$1" -v q="$2" '$1 == r && $2 == q { print $4 }' "${COUNTS_FILE:-$BUNDLE/counts.tsv}" } expect_count() { # =n> local got; got=$(count_of "$1" "$2") @@ -545,6 +551,72 @@ expect_exit 0 && expect_count SUPPRESS - 1 \ && grep -q "^SUPPRESS - grep $FX_SRC_FILE 2 " "$BUNDLE/hits.tsv" && ok finish +# --- the bundle ldd-scope.sh writes, read by detection --- +SO="" # the scope bundle of the case: $BUNDLE/scope-out + +begin "a code-only scoped diff skips the R9 gate's doc checks" +mk_repo; git_commit_repo +printf '\n' >> "$REPO/$FX_SRC_FILE" # a code-only change +(cd "$REPO" && git -c user.name=t -c user.email=t@t commit -qam change) +run_scope --base HEAD~1; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" +expect_exit 0 && expect_has "R9 gate: scoped, no doc in scope; doc checks skipped" \ + && COUNTS_FILE="$SO/counts.tsv" expect_count R9 1 0 && ok +finish + +begin "a scoped diff that touches a doc gets that doc checked by the R9 gate" +mk_repo; git_commit_repo +printf '\n' >> "$REPO/$FX_SRC_FILE" # a code-only change +printf '\nMore text.\n' >> "$REPO/docs/orphan.md" +(cd "$REPO" && git -c user.name=t -c user.email=t@t commit -qam change) +run_scope --base HEAD~1; SO="$BUNDLE/scope-out" +expect_exit 0 && { grep -qx 'docs/orphan.md' "$SO/docs.txt" || bad "docs.txt lacks the touched doc"; } \ + && grep -qx 'docs/orphan.md' "$SO/docs.txt" \ + && { run_detect_bundle "$SO"; expect_exit 0; } \ + && expect_has "R9 gate: scoped, 1 doc(s) to check" \ + && COUNTS_FILE="$SO/counts.tsv" expect_count R9 1 1 \ + && { grep -q "^R9 1 gate docs/orphan.md " "$SO/hits.tsv" || bad "no R9 Q1 hit row for docs/orphan.md"; } \ + && grep -q "^R9 1 gate docs/orphan.md " "$SO/hits.tsv" && ok +finish + +begin "detection reads the source glob from the bundle ldd-scope.sh wrote" +mk_repo; mkdir -p "$REPO/lib"; printf 'thing\n' > "$REPO/lib/thing.zz"; git_commit_repo +run_scope --glob '*.zz' lib/thing.zz; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" +expect_exit 0 && expect_has "over 1 source + 0 test files" && ok +finish + +begin "a --glob that differs from the bundle's is a usage error (exit 2)" +mk_repo; mkdir -p "$REPO/lib"; printf 'thing\n' > "$REPO/lib/thing.zz"; git_commit_repo +run_scope --glob '*.zz' lib/thing.zz; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" --glob '*.yy' +expect_exit 2 && expect_has "differs from the bundle's glob" && ok +finish + +begin "a scope whose files match no source glob is an error, not an all-zero table (exit 2)" +mk_repo; printf 'notes\n' > "$REPO/notes.txt"; printf 'notes.txt\n' > "$BUNDLE/files.txt" +printf 'custom\t*.zz\n' > "$BUNDLE/language.txt" +run_detect +expect_exit 2 && expect_has "0 source + 0 test files" && ok +finish + +begin "--gate-budget stops a slow gate and its children; the gate questions are inconclusive" +mk_repo; : > "$BUNDLE/dirs.txt" +FAKE=$(mktemp -d) +cp "$DETECT" "$FAKE/ldd-detect.sh" +printf '#!/usr/bin/env bash\nsleep 37\necho never\n' > "$FAKE/check-repo-brain.sh" +t0=$SECONDS +OUT=$(cd "$REPO" && bash "$FAKE/ldd-detect.sh" --rules "$RULES" --gate-budget 1 --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? +ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +took=$((SECONDS - t0)) +expect_exit 0 && { (( took < 15 )) || bad "the run took ${took}s with a 1s gate budget"; } && (( took < 15 )) \ + && expect_has "R9 gate: inconclusive" && expect_count R9 1 inconclusive && expect_has "inconclusive" \ + && { ! pgrep -f 'sleep 37' >/dev/null || bad "the gate's child process outlived the budget"; } \ + && ! pgrep -f 'sleep 37' >/dev/null && ok +pkill -f 'sleep 37' 2>/dev/null +rm -rf "$FAKE" +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/linter-driven-development/scripts/ldd-scope.sh b/linter-driven-development/scripts/ldd-scope.sh index 55b12f89..e0cd0995 100755 --- a/linter-driven-development/scripts/ldd-scope.sh +++ b/linter-driven-development/scripts/ldd-scope.sh @@ -46,6 +46,11 @@ # # The bundle: # language.txt the group's language id; "custom TAB " under --glob +# test-re.txt the --test-re pattern, when one was passed, so ldd-detect.sh +# tells test files the same way +# docs.txt on a scoped rung, the markdown files the rung touches, one +# relative path per line, deleted ones included: the docs the +# R9 gate checks in full; --all writes none # files.txt the scope, one relative path per line. A file left out of # scope/ keeps its line with the reason: (not bundled: deleted), # (not bundled: binary), (not bundled: lines) over the @@ -448,6 +453,10 @@ write_bundle() { mkdir -p "$OUT/scope" if [[ "$GROUP_ID" == "custom" ]]; then printf 'custom\t%s\n' "$OPT_GLOB" > "$OUT/language.txt" else printf '%s\n' "$GROUP_ID" > "$OUT/language.txt"; fi + [[ -n "$OPT_TEST_RE" ]] && printf '%s\n' "$OPT_TEST_RE" > "$OUT/test-re.txt" + if [[ "$RUNG" != "all" ]]; then + sed 's|^\./||' "$CANDIDATES" | grep -E '\.md$' | LC_ALL=C sort -u > "$OUT/docs.txt" + fi # not_bundled_reason — prints the reason, or nothing when it is bundled not_bundled_reason() { diff --git a/linter-driven-development/skills/pre-commit-review/SKILL.md b/linter-driven-development/skills/pre-commit-review/SKILL.md index b434e9e2..6453c1c5 100644 --- a/linter-driven-development/skills/pre-commit-review/SKILL.md +++ b/linter-driven-development/skills/pre-commit-review/SKILL.md @@ -80,11 +80,15 @@ writes `hits.tsv`, `hits-all.tsv` and `counts.tsv` into the bundle, and prints t question only a hunter's reading answers — then one line per rule family. That table, about 300 tokens, is the pre-filter: the parent prints no rule file, composes no detection command, and never reads a Falsifying-questions section. On a scoped review the R9 -gate's rows are the scope's own files' — an orphan doc or an unwired root is the -repository's state, for a `--all` review or BROADER CONTEXT, never this diff's -finding. When the +gate checks only the docs the diff touches, the docs that link to a scope file and the +docs that cite a symbol the diff removes, and its rows are about those docs and the +scope's own files — an orphan doc elsewhere or an unwired root is the repository's +state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line +`R9 gate: …` under the summary says which docs the gate checked. A gate row whose +hits cell is `inconclusive` means the gate ran out of its `--gate-budget`: it is not +a `0`, and the documentation hunter answers that question from the prose. When the detection script exits 2 instead — a pattern its grep rejects, the R9 gate failing to -run — the review did not run: report its message, spawn nothing, and never render a +run, a scope in which no file matches the source glob — the review did not run: report its message, spawn nothing, and never render a rule as `skipped` or compose the pass by hand. A rule family none of whose rules has a hit — `0` on every row of theirs that is not `judgment` — is skipped: no hunter for it. @@ -112,7 +116,8 @@ zero prior use; the fix is a discussion or a separate PR, never silent inclusion The bundle exists (step 1 wrote it: `files.txt`, `diff.patch` on a scoped review, `scope/.txt` per bundled file with the file's own line numbers, `dirs.txt` on -`--all`, `comments.txt`, `hits.tsv`, `hits-all.tsv`, `counts.tsv`); nothing is +`--all`, `docs.txt` with the touched docs on a scoped review, `test-re.txt` with the +test-file pattern when one was passed, `comments.txt`, `hits.tsv`, `hits-all.tsv`, `counts.tsv`); nothing is written here. One Bash command, only when the scope is not empty, lists every rule path the prompts will carry and prints `sed -n '/^## Waiting for agents/,/^## The merged report/p' diff --git a/python-linter-driven-development/scripts/check-repo-brain.sh b/python-linter-driven-development/scripts/check-repo-brain.sh index 76b5ce80..8f70611d 100755 --- a/python-linter-driven-development/scripts/check-repo-brain.sh +++ b/python-linter-driven-development/scripts/check-repo-brain.sh @@ -8,10 +8,22 @@ # in-repo copy). Installed into target repos by the documentation skill's # BOOTSTRAP pass (/wire-repo-brain). # -# Usage: bash scripts/check-repo-brain.sh [--fix] [repo-root] (default: cwd) +# Usage: bash scripts/check-repo-brain.sh [--fix] [--scope [--words ]] [repo-root] +# (repo-root default: cwd) # CI: one line — bash scripts/check-repo-brain.sh # --fix: rewrite drifted index lines from each target doc's `description` # (the one mechanical repair; everything else stays report-only) +# --scope a review's scope: one path per line, relative to repo-root +# (code files and docs, deleted ones included). The gate then checks +# only the docs in scope: the listed docs, the docs whose text names a +# listed file's basename, and the docs that cite a word of --words. +# Code edges are checked for the listed code files only, Q3 is skipped +# (root wiring is the repository's state, not a diff's), and with no +# doc in scope the doc checks are skipped. One stdout line says which: +# "check-repo-brain: scope — doc(s) to check" or +# "check-repo-brain: scope — no doc in scope; doc checks skipped". +# --words identifiers the scoped change removes, one per line; a doc +# that cites one of them in backticks is in scope # # Doc roots are discovered at the repo root AND at every sub-project (a # directory holding the language's project marker — see the adapter below), @@ -58,8 +70,11 @@ # against its declaring package or receiver — never against a same-named # member elsewhere. A token missing from the set still resolves when it # appears as a whole word in any non-markdown repo file (config keys, -# alert names, test helpers). A `pkg.Sym` whose package is not declared in -# this repo is external (stdlib, dependencies) and exempt. +# alert names, test helpers): in a git work tree the tracked and +# untracked-but-not-ignored files; elsewhere every file outside the +# dependency and build-output directories (bazel-*, node_modules, …). +# A `pkg.Sym` whose package is not declared in this repo is external +# (stdlib, dependencies) and exempt. # - lines carrying the ⚠️ stale flag or a *(planned)* marker are exempt from # symbol resolution and the description copy check (R9 Q2/Q7 exemptions); # the file:line ban has no exemption beyond URL spans, fenced code blocks, @@ -72,22 +87,37 @@ # 2 usage error, or an internal scanner failure (a failed scan is # inconclusive, never silently clean) # -# Uses only POSIX-portable tools: find, grep, sed, awk, head, sort, wc. No jq/python. +# Uses only POSIX-portable tools: find, grep, sed, awk, head, sort, wc, tr, xargs, +# plus git when the repository is a git work tree. No jq/python. # awk programs avoid interval expressions ({m,n}) — mawk, Debian's default, rejects them. set -u FIX=0 -if [[ "${1:-}" == "--fix" ]]; then - FIX=1 - shift -fi +SCOPE_LIST="" +WORDS_LIST="" +abs_list() { # -> absolute path of an existing file, or exit 2 + [[ -f "$1" ]] || { echo "check-repo-brain: no such list: $1" >&2; exit 2; } + printf '%s/%s\n' "$(cd "$(dirname "$1")" && pwd)" "$(basename "$1")" +} +while (( $# > 0 )); do + case "$1" in + --fix) FIX=1; shift ;; + --scope) SCOPE_LIST=$(abs_list "${2:-}") || exit 2; shift 2 ;; + --words) WORDS_LIST=$(abs_list "${2:-}") || exit 2; shift 2 ;; + --*) echo "check-repo-brain: unknown option: $1" >&2; exit 2 ;; + *) break ;; + esac +done +[[ -n "$WORDS_LIST" && -z "$SCOPE_LIST" ]] && { echo "check-repo-brain: --words needs --scope" >&2; exit 2; } REPO_ROOT="${1:-$(pwd)}" if [[ ! -d "$REPO_ROOT" ]]; then echo "check-repo-brain: not a directory: $REPO_ROOT" >&2 exit 2 fi cd "$REPO_ROOT" || exit 2 +SCOPED=0 +[[ -n "$SCOPE_LIST" ]] && SCOPED=1 # ===================== language adapter: python ===================== # Everything language-specific lives between these two marker comments. The @@ -126,10 +156,10 @@ LANG_FILE_RE='\\.py(:[0-9]+)?$' LANG_SYMBOL_RE='^([A-Z][A-Za-z0-9_]*|[a-z_][a-z0-9_]*_[a-z0-9_]*)$' LANG_QUALIFIED_RE='^[A-Za-z_][A-Za-z0-9_]*\\.[A-Za-z_][A-Za-z0-9_]*$' -py_prune() { # find(1) pruning shared by the three walkers - printf '%s\n' -not -path '*/.venv/*' -not -path '*/venv/*' -not -path '*/node_modules/*' \ - -not -path '*/__pycache__/*' -not -path '*/site-packages/*' -not -path './.git/*' -} +# find(1) pruning shared by the three walkers; an array, so a pattern such as +# ./.git/* reaches find as it is and never expands against the tree +PY_PRUNE=(-not -path '*/.venv/*' -not -path '*/venv/*' -not -path '*/node_modules/*' + -not -path '*/__pycache__/*' -not -path '*/site-packages/*' -not -path './.git/*') lang_project_dirs() { local m p @@ -137,11 +167,11 @@ lang_project_dirs() { p=$(dirname "$m"); p="${p#./}" [[ "$p" == "." || -z "$p" ]] && continue printf '%s\n' "$p" - done < <(find . -name "$LANG_PROJECT_MARKER" $(py_prune) 2>/dev/null | sort) + done < <(find . -name "$LANG_PROJECT_MARKER" "${PY_PRUNE[@]}" 2>/dev/null | sort) } lang_has_code() { - find . -name "$LANG_CODE_GLOB" $(py_prune) -print -quit 2>/dev/null | grep -q . + find . -name "$LANG_CODE_GLOB" "${PY_PRUNE[@]}" -print -quit 2>/dev/null | grep -q . } LANG_DECL_AWK=' @@ -192,7 +222,7 @@ FNR == 1 { ' lang_declarations() { - find . -name "$LANG_CODE_GLOB" $(py_prune) -print0 2>/dev/null \ + find . -name "$LANG_CODE_GLOB" "${PY_PRUNE[@]}" -print0 2>/dev/null \ | xargs -0 awk "$LANG_DECL_AWK" } @@ -316,22 +346,65 @@ lifecycle_stale() { # -> reason when frontmatter marks the doc stale ('' have_code=0 lang_has_code && have_code=1 -# ---------- declaration set: built once, queried per token ---------- -DECLS="" PKGS="" -if (( have_code )); then - DECLS=$(mktemp) PKGS=$(mktemp) DECL_ALL=$(mktemp) - trap 'rm -f "$DECLS" "$PKGS"' EXIT - if ! ( set -o pipefail; lang_declarations | sort -u > "$DECL_ALL" ); then +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +# ---------- declaration set: built once, on the first token to resolve ---------- +DECLS="$WORK/decls" PKGS="$WORK/pkgs" +decls_built=0 +build_declarations() { + (( decls_built )) && return 0 + decls_built=1 + if ! ( set -o pipefail; lang_declarations | sort -u > "$WORK/decl-all" ); then echo "check-repo-brain: internal scanner error — the declaration scan failed; treating the run as inconclusive" >&2 exit 2 fi - grep '^pkg:' "$DECL_ALL" | sed 's/^pkg://' > "$PKGS" - grep -v '^pkg:' "$DECL_ALL" > "$DECLS" - rm -f "$DECL_ALL" -fi + grep '^pkg:' "$WORK/decl-all" | sed 's/^pkg://' > "$PKGS" + grep -v '^pkg:' "$WORK/decl-all" > "$DECLS" + rm -f "$WORK/decl-all" +} is_repo_pkg() { grep -qxF "$1" "$PKGS" 2>/dev/null; } +# repo_files — the repository's non-markdown files, NUL-separated. A git work +# tree lists them through git (tracked files plus untracked ones git does not +# ignore), which never walks build outputs or symlinked caches; elsewhere find +# skips the directories that hold dependencies and build outputs. +repo_files() { + if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then + git ls-files -z --cached --others --exclude-standard -- . \ + ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**' 2>/dev/null && return 0 + fi + find . \( -name .git -o -name vendor -o -name node_modules -o -name 'bazel-*' -o -name .venv \ + -o -name venv -o -name __pycache__ -o -name .tox -o -name dist -o -name target \) -prune \ + -o -type f ! -name '*.md' -print0 +} + +# resolve_words — the tokens that occur as a whole word in any +# non-markdown repository file, one per line. The repository is read once +# into words (runs of [A-Za-z0-9_.]); a dotted token such as `Type.Method` +# matches any run of whole dot-separated parts, as grep -w would match it. +# One pass over the text is linear in its size, where a fixed-string grep for +# hundreds of tokens is not. +resolve_words() { + repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.' '\n' \ + | LC_ALL=C awk -v list="$1" ' + BEGIN { while ((getline t < list) > 0) if (t != "") want[t] = 1 } + seen[$0]++ { next } + { + n = split($0, p, ".") + for (i = 1; i <= n; i++) { + s = p[i] + if (s in want) { print s; delete want[s] } + for (j = i + 1; j <= n; j++) { + s = s "." p[j] + if (s in want) { print s; delete want[s] } + } + } + }' + return 0 +} + # One fence-aware awk pass per bundle extracts everything Q2's doc scan needs: # P — a file:line citation outside fences/URLs/globs # S — a backticked symbol-shaped token to resolve @@ -370,8 +443,53 @@ FNR == 1 { fence = 0 } } ' -# ---------- Q2: code→docs edges (repo-wide; resolved from repo root, then the -# citing file's own sub-project) ---------- +# ---------- the scope: which docs a scoped run checks ---------- +# A doc is in scope when the scope lists it, when its text names the basename +# of a listed file as a whole word (a link to a touched doc or code file), or +# when it cites in backticks a word the change removes (a symbol the diff +# renames or deletes). +# Every list below holds relative paths as find(1) prints them under a root. +SCOPE_DOCS="$WORK/scope-docs" +SCOPE_PATHS="$WORK/scope-paths" +if (( SCOPED )); then + sed -e 's|^\./||' -e '/^$/d' "$SCOPE_LIST" | LC_ALL=C sort -u > "$SCOPE_PATHS" + for r in "${ROOTS[@]}"; do find "$r" -type f -name '*.md'; done | LC_ALL=C sort -u > "$WORK/all-docs" + { + grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" + sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$WORK/basenames" + if [[ -s "$WORK/basenames" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$WORK/basenames" 2>/dev/null + fi + if [[ -n "$WORDS_LIST" && -s "$WORDS_LIST" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" \ + | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ + -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" "$DOCSCAN_AWK" \ + | awk -F'\t' 'NR == FNR { w[$0] = 1; next } + $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in w) || (m in w)) print $2 }' "$WORDS_LIST" - + fi + } | LC_ALL=C sort -u > "$SCOPE_DOCS" + if [[ -s "$SCOPE_DOCS" ]]; then + echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" + else + echo "check-repo-brain: scope — no doc in scope; doc checks skipped" + fi +fi + +# bundle_docs [name] — the docs a run checks under one root: every +# *.md (or every file of that name), limited to the scope on a scoped run +bundle_docs() { + local name="${2:-*.md}" + if (( SCOPED )); then + find "$1" -type f -name "$name" | grep -xF -f "$SCOPE_DOCS" + else + find "$1" -type f -name "$name" + fi + return 0 +} + +# ---------- Q2: code→docs edges (repo-wide, or the scope's code files on a +# scoped run; resolved from repo root, then the citing file's own +# sub-project) ---------- docroot_for_file() { # -> docroot of the longest matching project dir local f="${1#./}" best="" i for i in "${!PROJS[@]}"; do @@ -395,7 +513,13 @@ if (( have_code )); then CUR_DOCROOT=$(docroot_for_file "$file") fail "[Q2] $file:$line — code edge points at missing $target" fi - done < <(lang_code_edges) + done < <(if (( SCOPED )); then + lang_code_edges | awk -v list="$SCOPE_PATHS" ' + BEGIN { while ((getline l < list) > 0) s[l] = 1 } + { f = substr($0, 1, index($0, ":") - 1); sub(/^\.\//, "", f); if (f in s) print }' + else + lang_code_edges + fi) fi # ---------- per-bundle checks ---------- @@ -405,6 +529,8 @@ check_bundle() { # local root_index="$docroot/index.md" local root_index_c root_index_c=$(canon "$root_index") + # a scoped run skips a bundle that holds no doc in scope + (( SCOPED )) && [[ -z "$(bundle_docs "$docroot")" ]] && return 0 # --- Q1: transitive reachability from the bundle's root index --- local reachable="" visited="" queue=("$root_index") @@ -433,7 +559,7 @@ check_bundle() { # case "$reachable" in *"$c"$'\n'*) ;; *) fail "[Q1] $doc — orphan: not reachable from $root_index" ;; esac - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q2: every doc link resolves (a missing parent directory is just as # broken as a missing file — only URLs/anchors are exempt) --- @@ -448,13 +574,13 @@ check_bundle() { # fail "[Q2] $md — link target does not exist: $t" fi done < <(grep -oE '\]\([^)]+\)' "$md" 2>/dev/null) - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q2: doc scan — file:line ban + docs→code symbol resolution. # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). --- local scan; scan=$(mktemp) if ! ( set -o pipefail - find "$docroot" -type f -name '*.md' -print0 \ + bundle_docs "$docroot" | tr '\n' '\0' \ | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" \ "$DOCSCAN_AWK" > "$scan" ); then @@ -469,6 +595,7 @@ check_bundle() { # if (( have_code )) && grep -q $'^S\t' "$scan"; then local toks check members unres bad toks=$(mktemp) check=$(mktemp) members=$(mktemp) unres=$(mktemp) bad=$(mktemp) + build_declarations grep $'^S\t' "$scan" | cut -f4 | sort -u > "$toks" # tokens to resolve (external pkg.Sym exempt). A qualified token resolves # as the PAIR itself — the declaration set carries pkg.Ident / Type.Method @@ -486,12 +613,11 @@ check_bundle() { # esac done < "$toks" > "$check" cut -f2 "$check" | sort -u > "$members" - grep -vxF -f "$DECLS" "$members" > "$unres" || true + grep -vxF -f "$DECLS" "$members" | grep -v '^$' > "$unres" || true if [[ -s "$unres" ]]; then - # ONE repo-wide word grep for the whole unresolved batch + # ONE pass over the repository for the whole unresolved batch local found; found=$(mktemp) - grep -rIhoFw --exclude-dir=vendor --exclude-dir=.git --exclude='*.md' \ - -f "$unres" . 2>/dev/null | sort -u > "$found" + resolve_words "$unres" | sort -u > "$found" grep -vxF -f "$found" "$unres" > "$bad" || true rm -f "$found" fi @@ -509,24 +635,27 @@ check_bundle() { # fi rm -f "$scan" - # --- Q3: root wiring (exact path; sub-roots may ride the repo-root index) --- - local rel="$docroot" - [[ "$proj" != "." ]] && rel="${docroot#$proj/}" - local claude="CLAUDE.md" agents="AGENTS.md" - [[ "$proj" != "." ]] && { claude="$proj/CLAUDE.md"; agents="$proj/AGENTS.md"; } - local wired_claude=0 wired_agents=0 - [[ -f "$claude" ]] && grep -qF -- "$rel/index.md" "$claude" && wired_claude=1 - [[ -f "$agents" ]] && grep -qF -- "$rel/index.md" "$agents" && wired_agents=1 - if (( ! wired_claude && ! wired_agents )); then - local via_root=0 - if [[ "$proj" != "." && -n "$ROOT_BUNDLE" ]]; then - grep -rqF -- "$docroot/index.md" "$ROOT_BUNDLE" --include='*.md' 2>/dev/null && via_root=1 - fi - if (( ! via_root )); then - fail "[Q3] $proj — neither $claude nor $agents references $rel/index.md" + # --- Q3: root wiring (exact path; sub-roots may ride the repo-root index); + # a scoped run leaves it to a whole-repository run --- + if (( ! SCOPED )); then + local rel="$docroot" + [[ "$proj" != "." ]] && rel="${docroot#$proj/}" + local claude="CLAUDE.md" agents="AGENTS.md" + [[ "$proj" != "." ]] && { claude="$proj/CLAUDE.md"; agents="$proj/AGENTS.md"; } + local wired_claude=0 wired_agents=0 + [[ -f "$claude" ]] && grep -qF -- "$rel/index.md" "$claude" && wired_claude=1 + [[ -f "$agents" ]] && grep -qF -- "$rel/index.md" "$agents" && wired_agents=1 + if (( ! wired_claude && ! wired_agents )); then + local via_root=0 + if [[ "$proj" != "." && -n "$ROOT_BUNDLE" ]]; then + grep -rqF -- "$docroot/index.md" "$ROOT_BUNDLE" --include='*.md' 2>/dev/null && via_root=1 + fi + if (( ! via_root )); then + fail "[Q3] $proj — neither $claude nor $agents references $rel/index.md" + fi + elif (( ! wired_agents )); then + note "[Q3] $agents lacks the $rel/index.md routing reference — AGENTS.md-reading tools start blind" fi - elif (( ! wired_agents )); then - note "[Q3] $agents lacks the $rel/index.md routing reference — AGENTS.md-reading tools start blind" fi # --- Q7: bundle contract --- @@ -591,7 +720,7 @@ check_bundle() { # else fail "[Q7] $md — frontmatter missing 'description:'" fi - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q7: drift check — index line text == target's description (⚠️ exempt; # --fix rewrites drifted lines after the read loop, never during it) --- @@ -629,12 +758,12 @@ check_bundle() { # echo " fixed: $idx:$n — index line rewritten from its target's description" done <<< "$fixes" fi - done < <(find "$docroot" -type f -name 'index.md') + done < <(bundle_docs "$docroot" index.md) # --- Q7: no log.md --- while IFS= read -r lg; do fail "[Q7] $lg — log.md is reserved for change history; docs describe current behavior" - done < <(find "$docroot" -type f -name 'log.md') + done < <(bundle_docs "$docroot" log.md) } for i in "${!PROJS[@]}"; do diff --git a/python-linter-driven-development/scripts/check-repo-brain_test.sh b/python-linter-driven-development/scripts/check-repo-brain_test.sh index fde446ed..3aedc568 100755 --- a/python-linter-driven-development/scripts/check-repo-brain_test.sh +++ b/python-linter-driven-development/scripts/check-repo-brain_test.sh @@ -642,6 +642,75 @@ run_gate expect_exit 0 && expect_not "does not resolve" && expect_has "check-repo-brain: clean" && ok finish +# --- scoped runs (--scope, --words) and the symbol resolver --- +LISTS=$(mktemp -d) # the scope and word lists live outside the repository they describe +begin "--scope: a code-only scope that no doc cites skips the doc checks" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here." +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "scope — no doc in scope; doc checks skipped" && expect_not "orphan" && ok +finish + +begin "--scope: a listed doc is checked, an unlisted orphan and the root wiring are not" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here."; mk_doc other "another unlinked doc" "Nothing." +rm "$REPO/CLAUDE.md" "$REPO/AGENTS.md" +printf '%s\n' "$FX_CODE_FILE" docs/orphan.md > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] docs/orphan.md — orphan" \ + && expect_not "docs/other.md" && expect_not "[Q3]" && ok +finish + +begin "--scope: a doc that names a scope file is checked" +mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../$FX_CODE_FILE)." +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +finish + +begin "--scope --words: a doc that cites a removed symbol is checked" +mk_conformant +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +append_index "- [ghost.md](ghost.md) — a ghost symbol" +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "no doc in scope" \ + && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ + && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok +finish + +begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" +mk_conformant +printf 'build/\n' > "$REPO/.gitignore" +mkdir -p "$REPO/build"; printf 'IgnoredOnlyName\n' > "$REPO/build/gen.yaml" +printf 'TrackedName\n' > "$REPO/tracked.yaml" +mk_doc names "alert names" "Fires \`TrackedName\`, \`UntrackedName\` and \`IgnoredOnlyName\`." +append_index "- [names.md](names.md) — alert names" +(cd "$REPO" && git init -q && git add -A && git -c user.name=t -c user.email=t@t commit -qm init) +printf 'UntrackedName\n' > "$REPO/untracked.yaml" +run_gate +expect_exit 1 && expect_has "\`IgnoredOnlyName\` does not resolve" \ + && expect_not "\`TrackedName\` does not resolve" && expect_not "\`UntrackedName\` does not resolve" && ok +finish + +begin "Q2 outside git a token does not resolve from a bazel-* output directory" +mk_conformant +mkdir -p "$REPO/bazel-out"; printf 'BazelOnlyName\n' > "$REPO/bazel-out/gen.yaml" +mk_doc bazel "bazel names" "Fires \`BazelOnlyName\`." +append_index "- [bazel.md](bazel.md) — bazel names" +run_gate +expect_exit 1 && expect_has "\`BazelOnlyName\` does not resolve" && ok +finish + +begin "Q2 a dotted token resolves on whole dot-separated words only" +mk_conformant +printf 'limits:\n key: Cfg.MaxRetries.value\n other: Cfg.MaxRetriesX\n' > "$REPO/limits.yaml" +mk_doc dotted "dotted names" "Reads \`Cfg.MaxRetries\`; \`Cfg.MaxRetri\` is a typo." +append_index "- [dotted.md](dotted.md) — dotted names" +run_gate +expect_exit 1 && expect_has "\`Cfg.MaxRetri\` does not resolve" && expect_not "\`Cfg.MaxRetries\` does not resolve" && ok +finish +rm -rf "$LISTS" + } # run_row_cases for ROW in $FX_ROWS; do diff --git a/python-linter-driven-development/scripts/ldd-detect.sh b/python-linter-driven-development/scripts/ldd-detect.sh index 2d704b37..9ade67ba 100755 --- a/python-linter-driven-development/scripts/ldd-detect.sh +++ b/python-linter-driven-development/scripts/ldd-detect.sh @@ -25,7 +25,14 @@ # --plan print the parsed detect lines and exit; nothing runs # --glob the scope is one group of a language the table # does not know, matched by this glob -# --test-re test-file pattern over relative paths, same case +# --test-re test-file pattern over relative paths, same case. +# A bundle carries the pattern ldd-scope.sh used +# (test-re.txt), and a custom group its glob +# (language.txt): the bundle's values win, and a +# --glob or --test-re that differs is a usage error +# --gate-budget wall-clock seconds the R9 gate may run (default: no +# limit); on expiry the gate is stopped and its +# questions are counted "inconclusive", never 0 # # The scope's language is the bundle's language.txt; without it, the files in # files.txt tell it by extension, and a list that spans several languages is @@ -39,10 +46,14 @@ # Detect-path: `` over the scope's relative paths (layer directories, # role-named packages); a hit's line is 0 # Detect-gate: Q the R9 gate's [Q] report lines, from one run of -# check-repo-brain.sh beside this script over --root; on a -# scoped bundle (no dirs.txt) only the lines whose file is in -# the scope — the gate's repository-wide findings belong to -# a whole-repository review +# check-repo-brain.sh beside this script over --root. On a +# scoped bundle (no dirs.txt) the gate runs with --scope: it +# checks only the bundle's docs (docs.txt), the docs that +# link to a scope file and the docs that cite a symbol the +# diff removes, the code edges of the scope's code files, +# and no root wiring; with no such doc its doc checks are +# skipped. The repository-wide findings belong to a +# whole-repository review # Detect: judgment no mechanical lead; the hunter runs the prose # A question with no detect line, two of them, or a malformed one is an error: # the generator's lint-core keeps the rule files honest, and this script refuses @@ -53,15 +64,18 @@ # the overflow row of a capped question: file "-", line 0, # excerpt "+ more hit(s) not listed" # hits-all.tsv the same columns, every hit, no cap and no overflow row -# counts.tsv rule question kind hits (judgment rows carry "-") +# counts.tsv rule question kind hits (judgment rows carry "-", gate +# rows "inconclusive" when the gate ran out of --gate-budget) # the last row is SUPPRESS - grep : suppression directives # on the diff's added lines when the bundle has a diff.patch, # else over every scope file # Two runs over one tree write byte-identical counts.tsv. # # Exit codes: 0 tables written · 2 usage error, a detect line that does not -# parse, a pattern grep rejects, or a gate run that failed -# (inconclusive, never silently clean) +# parse, a pattern grep rejects, a gate run that failed +# (inconclusive, never silently clean), or a non-empty scope in +# which no file matches the source glob (a glob mismatch, never +# an empty table) # # Uses only POSIX-portable tools: grep, sed, awk, sort, wc, head. GNU grep's # \b and \s are assumed, as the rules already assume them. @@ -77,6 +91,7 @@ CAP=40 PLAN=0 OPT_GLOB="" OPT_TEST_RE="" +GATE_BUDGET=0 BUNDLE="" usage() { sed -n '2,/^$/p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; } @@ -94,12 +109,14 @@ while (( $# > 0 )); do --plan) PLAN=1; shift ;; --glob) value "$1" $#; OPT_GLOB="$2"; shift 2 ;; --test-re) value "$1" $#; OPT_TEST_RE="$2"; shift 2 ;; + --gate-budget) value "$1" $#; GATE_BUDGET="$2"; shift 2 ;; -h|--help) usage ;; --*) die "unknown option: $1" ;; *) [[ -n "$BUNDLE" ]] && die "one bundle directory, got also: $1"; BUNDLE="$1"; shift ;; esac done [[ "$CAP" =~ ^[0-9]+$ ]] || die "--cap takes a number, got: $CAP" +[[ "$GATE_BUDGET" =~ ^[0-9]+$ ]] || die "--gate-budget takes a number of seconds, got: $GATE_BUDGET" [[ -d "$RULES_DIR" ]] || die "no rules directory: $RULES_DIR" RULES_DIR=$(cd "$RULES_DIR" && pwd) if (( ! PLAN )); then @@ -245,6 +262,7 @@ if (( ! PLAN )) && [[ -f "$BUNDLE/groups.txt" ]]; then fi args=(--root "$ROOT" --cap "$CAP") [[ -n "$OPT_TEST_RE" ]] && args+=(--test-re "$OPT_TEST_RE") + (( GATE_BUDGET > 0 )) && args+=(--gate-budget "$GATE_BUDGET") bash "$script" "${args[@]}" "$bundle" || { failed=1; echo "$SCRIPT_NAME: the $id group's detection pass failed; it is inconclusive" >&2; } echo done < "$BUNDLE/groups.txt" @@ -259,7 +277,9 @@ LANG_ID="" if (( ! PLAN )); then if [[ -f "$BUNDLE/language.txt" ]]; then IFS=$'\t' read -r LANG_ID glob < "$BUNDLE/language.txt" - [[ "$LANG_ID" == "custom" && -z "$OPT_GLOB" ]] && OPT_GLOB="$glob" + [[ "$LANG_ID" == "custom" && -n "$OPT_GLOB" && "$OPT_GLOB" != "$glob" ]] \ + && die "--glob '$OPT_GLOB' differs from the bundle's glob '$glob' (ldd-scope.sh wrote it; drop --glob)" + [[ "$LANG_ID" == "custom" ]] && OPT_GLOB="$glob" else ids="" while IFS= read -r line; do @@ -274,6 +294,14 @@ if (( ! PLAN )); then [[ "$ids" == *" "* ]] && die "the scope list spans several languages ($ids); ldd-scope.sh writes one bundle per language" LANG_ID="$ids" fi + # the test-file pattern ldd-scope.sh used, so the two scripts cannot disagree + # on what a test file is + if [[ -f "$BUNDLE/test-re.txt" ]]; then + b_test_re=$(cat "$BUNDLE/test-re.txt") + [[ -n "$OPT_TEST_RE" && "$OPT_TEST_RE" != "$b_test_re" ]] \ + && die "--test-re '$OPT_TEST_RE' differs from the bundle's '$b_test_re' (ldd-scope.sh wrote it; drop --test-re)" + OPT_TEST_RE="$b_test_re" + fi lang_configure "$LANG_ID" || die "this plugin has no language block for $LANG_ID" fi @@ -319,7 +347,7 @@ END { if (insec) flush() } ' plan_file=$(mktemp) -trap 'rm -f "$plan_file" "${tmp_hits:-}" "${tmp_hits_all:-}" "${tmp_counts:-}" "${gate_out:-}"' EXIT +trap 'rm -f "$plan_file" "${tmp_hits:-}" "${tmp_hits_all:-}" "${tmp_counts:-}" "${gate_out:-}" "${gate_scope:-}" "${gate_words:-}"' EXIT rule_files=$(find "$RULES_DIR" -maxdepth 1 -name 'R*.md' | awk -F/ '{ n = $NF; sub(/^R/, "", n); sub(/-.*$/, "", n); print n "\t" $0 }' | sort -n | cut -f2) [[ -n "$rule_files" ]] || die "no rule files (R*.md) under $RULES_DIR" while IFS= read -r f; do @@ -345,11 +373,13 @@ fi SRC_FILES=() TEST_FILES=() ALL_FILES=() +n_present=0 while IFS= read -r line; do f="${line%% (not bundled:*}" f="${f#./}" [[ -z "$f" ]] && continue [[ -f "$f" ]] || continue + n_present=$((n_present + 1)) [[ "$(lang_of_path "$f")" == "$LANG_ID" ]] || continue printf '%s\n' "$f" | grep -qE -- "$LANG_EXCLUDE_RE" && continue ALL_FILES+=("$f") @@ -357,6 +387,12 @@ done < <(LC_ALL=C sort -u "$FILES_LIST") for f in ${ALL_FILES[@]+"${ALL_FILES[@]}"}; do if lang_is_test "$f"; then TEST_FILES+=("$f"); else SRC_FILES+=("$f"); fi done +# A scope whose files exist but none of which is a source file is a glob +# mismatch between the scope and this run: an all-zero table would read as a +# clean review of nothing. +if (( ${#ALL_FILES[@]} == 0 && n_present > 0 )); then + die "0 source + 0 test files: none of the $n_present file(s) in $FILES_LIST is a $LANG_ID source file (pass the --glob ldd-scope.sh used)" +fi tmp_hits=$(mktemp) tmp_hits_all=$(mktemp) @@ -470,24 +506,86 @@ run_path() { # rm -f "$raw" "$err" } -# The R9 gate runs once, lazily; its [Q] lines are split by question. +# gate_scope_lists — what a scoped gate run checks: +# every path of the scope (deleted ones too: a doc can still link to them) and +# of docs.txt; and the identifiers the diff removes — on its removed lines and +# on none of its added ones — the symbols a renamed or deleted declaration +# takes away from the docs that cite them. Comment lines (a line that opens +# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped. +gate_scope_lists() { + local scope="$1" words="$2" line f + { + while IFS= read -r line; do + f="${line%% (not bundled:*}"; f="${f#./}" + [[ -n "$f" ]] && printf '%s\n' "$f" + done < "$FILES_LIST" + [[ -f "$BUNDLE/docs.txt" ]] && cat "$BUNDLE/docs.txt" + } | LC_ALL=C sort -u > "$scope" + : > "$words" + [[ -s "$BUNDLE/diff.patch" ]] || return 0 + grep -vE '^(\+\+\+|---) ' "$BUNDLE/diff.patch" | grep -E '^[-+]' \ + | LC_ALL=C awk ' + { + s = substr($0, 1, 1); t = substr($0, 2) + if (t ~ /^[ \t]*(\/\/|#|\/\*|\*|--|""")/) next + while (match(t, /[A-Za-z_][A-Za-z0-9_]*/)) { + w = substr(t, RSTART, RLENGTH); t = substr(t, RSTART + RLENGTH) + if (s == "+") add[w] = 1; else del[w] = 1 + } + } + END { for (w in del) if (!(w in add)) print w }' | LC_ALL=C sort > "$words" +} + +# The R9 gate runs once, lazily; its [Q] lines are split by question. With +# --gate-budget the gate runs as its own process group, so expiry stops every +# process it started, not only its shell. +GATE_STATE="whole repository" run_gate_once() { [[ -n "$gate_out" ]] && return 0 gate_out=$(mktemp) - local gate="$SCRIPT_DIR/check-repo-brain.sh" code + local gate="$SCRIPT_DIR/check-repo-brain.sh" code=0 start pid scope_line + local -a args=() [[ -f "$gate" ]] || die "Detect-gate needs $gate beside this script" - bash "$gate" "$ROOT" > "$gate_out" 2>&1 - code=$? + if [[ ! -f "$BUNDLE/dirs.txt" ]]; then + gate_scope=$(mktemp); gate_words=$(mktemp) + gate_scope_lists "$gate_scope" "$gate_words" + args=(--scope "$gate_scope" --words "$gate_words") + fi + if (( GATE_BUDGET == 0 )); then + bash "$gate" ${args[@]+"${args[@]}"} "$ROOT" > "$gate_out" 2>&1 || code=$? + else + start=$SECONDS + set -m + bash "$gate" ${args[@]+"${args[@]}"} "$ROOT" > "$gate_out" 2>&1 & + pid=$! + set +m + while kill -0 "$pid" 2>/dev/null; do + if (( SECONDS - start >= GATE_BUDGET )); then + kill -TERM -- "-$pid" 2>/dev/null + wait "$pid" 2>/dev/null + GATE_STATE="inconclusive — stopped after the ${GATE_BUDGET}s --gate-budget" + return 0 + fi + sleep 0.2 + done + wait "$pid" || code=$? + fi if (( code == 2 )); then head -5 "$gate_out" >&2 die "the R9 gate failed to run (exit 2); its questions are inconclusive" fi + scope_line=$(grep -m1 '^check-repo-brain: scope — ' "$gate_out") + [[ -n "$scope_line" ]] && GATE_STATE="scoped, ${scope_line#check-repo-brain: scope — }" return 0 } run_gate() { # local rule="$1" q="$2" qn="$3" raw tab run_gate_once + if [[ "$GATE_STATE" == inconclusive* ]]; then + printf '%s\t%s\tgate\tinconclusive\n' "$rule" "$q" >> "$tmp_counts" + return 0 + fi raw=$(mktemp) tab=$(printf '\t') # "[Qn] [:line] — — see " becomes @@ -505,15 +603,9 @@ run_gate() { # if (match(loc, /:[0-9]+$/)) { file = substr(loc, 1, RSTART - 1); line = substr(loc, RSTART + 1) + 0 } print file "\t" line "\t" rest }' > "$raw" - # A scoped review owns only the gate lines about its own files: an orphan - # doc, an unwired root or another file's broken edge is the repository's - # state, not the diff's, and is not a lead for this review's hunter. - if [[ ! -f "$BUNDLE/dirs.txt" ]] && [[ -s "$raw" ]]; then - scoped=$(mktemp); scope_list=$(mktemp) - printf '%s\n' ${ALL_FILES[@]+"${ALL_FILES[@]}"} > "$scope_list" - awk -F'\t' 'NR == FNR { in_scope[$0] = 1; next } ($1 in in_scope)' "$scope_list" "$raw" > "$scoped" - mv "$scoped" "$raw"; rm -f "$scope_list" - fi + # A scoped review owns only the gate lines about its own files and the docs + # that cite them; the gate's --scope run reports nothing else (an orphan doc + # elsewhere or an unwired root is the repository's state, not the diff's). record "$rule" "$q" gate "$raw" 0 rm -f "$raw" } @@ -563,6 +655,7 @@ n_gate=$(awk -F'\t' '$3 == "gate"' "$plan_file" | wc -l | tr -d ' ') n_judg=$(awk -F'\t' '$3 == "judgment"' "$plan_file" | wc -l | tr -d ' ') n_q=$(wc -l < "$plan_file" | tr -d ' ') echo "$SCRIPT_NAME: $n_q questions ($n_grep grep, $n_path path, $n_gate gate, $n_judg judgment) over ${#SRC_FILES[@]} source + ${#TEST_FILES[@]} test files → $BUNDLE/hits.tsv, counts.tsv" +(( n_gate > 0 )) && [[ -n "$gate_out" ]] && echo "R9 gate: $GATE_STATE" echo awk -F'\t' 'BEGIN { printf "%-9s %-4s %-9s %s\n", "rule", "q", "kind", "hits" } { q = ($2 == "-") ? "-" : "Q" $2; printf "%-9s %-4s %-9s %s\n", $1, q, $3, $4 }' "$BUNDLE/counts.tsv" @@ -582,12 +675,14 @@ awk -F'\t' ' f = fam[$1]; if (f == "") f = "other" nq[f]++ if ($3 == "judgment") { nj[f]++; next } + if ($4 == "inconclusive") { ni[f]++; next } if ($4 + 0 > 0) { hits[f] += $4; nh[f]++; rh[f, $1] += $4 } } END { for (i = 1; i <= 4; i++) { f = order[i] printf "%-15s %5d hit(s) in %d of %d questions, %d judgment", f, hits[f] + 0, nh[f] + 0, nq[f] + 0, nj[f] + 0 + if (ni[f] + 0 > 0) printf ", %d inconclusive", ni[f] per = ""; n = split(rules[f], rs, " ") for (j = 1; j <= n; j++) if (rh[f, rs[j]] + 0 > 0) per = per (per == "" ? "" : " · ") rs[j] " " rh[f, rs[j]] if (per != "") printf " — %s", per diff --git a/python-linter-driven-development/scripts/ldd-detect_test.sh b/python-linter-driven-development/scripts/ldd-detect_test.sh index 739b3717..7c5b335c 100644 --- a/python-linter-driven-development/scripts/ldd-detect_test.sh +++ b/python-linter-driven-development/scripts/ldd-detect_test.sh @@ -49,6 +49,11 @@ run_detect() { # [options...] — runs the script over $REPO with $BUNDLE OUT=$(cd "$REPO" && bash "$DETECT" "$@" --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" } +run_detect_bundle() { # [options...] — runs the script over $REPO with that bundle + local b="$1"; shift + OUT=$(cd "$REPO" && bash "$DETECT" "$@" --root . "$b" 2>"$BUNDLE/.stderr"); CODE=$? + ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +} run_scope() { # [args...] — runs ldd-scope.sh over $REPO into $BUNDLE/scope-out rm -rf "$BUNDLE/scope-out" OUT=$(cd "$REPO" && bash "$SCOPE_SH" --out "$BUNDLE/scope-out" "$@" 2>"$BUNDLE/.stderr"); CODE=$? @@ -75,8 +80,9 @@ expect_not() { return 0 } expect_quiet() { [[ -z "$ERR" ]] || { bad "stderr not empty"; return 1; }; } -count_of() { # — the hits cell of counts.tsv, "-" for judgment - awk -F'\t' -v r="$1" -v q="$2" '$1 == r && $2 == q { print $4 }' "$BUNDLE/counts.tsv" +count_of() { # — the hits cell of counts.tsv (COUNTS_FILE, default the + # case bundle's), "-" for judgment + awk -F'\t' -v r="$1" -v q="$2" '$1 == r && $2 == q { print $4 }' "${COUNTS_FILE:-$BUNDLE/counts.tsv}" } expect_count() { # =n> local got; got=$(count_of "$1" "$2") @@ -472,6 +478,72 @@ expect_exit 0 && expect_count SUPPRESS - 1 \ && grep -q "^SUPPRESS - grep $FX_SRC_FILE 2 " "$BUNDLE/hits.tsv" && ok finish +# --- the bundle ldd-scope.sh writes, read by detection --- +SO="" # the scope bundle of the case: $BUNDLE/scope-out + +begin "a code-only scoped diff skips the R9 gate's doc checks" +mk_repo; git_commit_repo +printf '\n' >> "$REPO/$FX_SRC_FILE" # a code-only change +(cd "$REPO" && git -c user.name=t -c user.email=t@t commit -qam change) +run_scope --base HEAD~1; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" +expect_exit 0 && expect_has "R9 gate: scoped, no doc in scope; doc checks skipped" \ + && COUNTS_FILE="$SO/counts.tsv" expect_count R9 1 0 && ok +finish + +begin "a scoped diff that touches a doc gets that doc checked by the R9 gate" +mk_repo; git_commit_repo +printf '\n' >> "$REPO/$FX_SRC_FILE" # a code-only change +printf '\nMore text.\n' >> "$REPO/docs/orphan.md" +(cd "$REPO" && git -c user.name=t -c user.email=t@t commit -qam change) +run_scope --base HEAD~1; SO="$BUNDLE/scope-out" +expect_exit 0 && { grep -qx 'docs/orphan.md' "$SO/docs.txt" || bad "docs.txt lacks the touched doc"; } \ + && grep -qx 'docs/orphan.md' "$SO/docs.txt" \ + && { run_detect_bundle "$SO"; expect_exit 0; } \ + && expect_has "R9 gate: scoped, 1 doc(s) to check" \ + && COUNTS_FILE="$SO/counts.tsv" expect_count R9 1 1 \ + && { grep -q "^R9 1 gate docs/orphan.md " "$SO/hits.tsv" || bad "no R9 Q1 hit row for docs/orphan.md"; } \ + && grep -q "^R9 1 gate docs/orphan.md " "$SO/hits.tsv" && ok +finish + +begin "detection reads the source glob from the bundle ldd-scope.sh wrote" +mk_repo; mkdir -p "$REPO/lib"; printf 'thing\n' > "$REPO/lib/thing.zz"; git_commit_repo +run_scope --glob '*.zz' lib/thing.zz; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" +expect_exit 0 && expect_has "over 1 source + 0 test files" && ok +finish + +begin "a --glob that differs from the bundle's is a usage error (exit 2)" +mk_repo; mkdir -p "$REPO/lib"; printf 'thing\n' > "$REPO/lib/thing.zz"; git_commit_repo +run_scope --glob '*.zz' lib/thing.zz; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" --glob '*.yy' +expect_exit 2 && expect_has "differs from the bundle's glob" && ok +finish + +begin "a scope whose files match no source glob is an error, not an all-zero table (exit 2)" +mk_repo; printf 'notes\n' > "$REPO/notes.txt"; printf 'notes.txt\n' > "$BUNDLE/files.txt" +printf 'custom\t*.zz\n' > "$BUNDLE/language.txt" +run_detect +expect_exit 2 && expect_has "0 source + 0 test files" && ok +finish + +begin "--gate-budget stops a slow gate and its children; the gate questions are inconclusive" +mk_repo; : > "$BUNDLE/dirs.txt" +FAKE=$(mktemp -d) +cp "$DETECT" "$FAKE/ldd-detect.sh" +printf '#!/usr/bin/env bash\nsleep 37\necho never\n' > "$FAKE/check-repo-brain.sh" +t0=$SECONDS +OUT=$(cd "$REPO" && bash "$FAKE/ldd-detect.sh" --rules "$RULES" --gate-budget 1 --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? +ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +took=$((SECONDS - t0)) +expect_exit 0 && { (( took < 15 )) || bad "the run took ${took}s with a 1s gate budget"; } && (( took < 15 )) \ + && expect_has "R9 gate: inconclusive" && expect_count R9 1 inconclusive && expect_has "inconclusive" \ + && { ! pgrep -f 'sleep 37' >/dev/null || bad "the gate's child process outlived the budget"; } \ + && ! pgrep -f 'sleep 37' >/dev/null && ok +pkill -f 'sleep 37' 2>/dev/null +rm -rf "$FAKE" +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/python-linter-driven-development/scripts/ldd-scope.sh b/python-linter-driven-development/scripts/ldd-scope.sh index c0717d4c..1aa3e227 100755 --- a/python-linter-driven-development/scripts/ldd-scope.sh +++ b/python-linter-driven-development/scripts/ldd-scope.sh @@ -46,6 +46,11 @@ # # The bundle: # language.txt the group's language id; "custom TAB " under --glob +# test-re.txt the --test-re pattern, when one was passed, so ldd-detect.sh +# tells test files the same way +# docs.txt on a scoped rung, the markdown files the rung touches, one +# relative path per line, deleted ones included: the docs the +# R9 gate checks in full; --all writes none # files.txt the scope, one relative path per line. A file left out of # scope/ keeps its line with the reason: (not bundled: deleted), # (not bundled: binary), (not bundled: lines) over the @@ -335,6 +340,10 @@ write_bundle() { mkdir -p "$OUT/scope" if [[ "$GROUP_ID" == "custom" ]]; then printf 'custom\t%s\n' "$OPT_GLOB" > "$OUT/language.txt" else printf '%s\n' "$GROUP_ID" > "$OUT/language.txt"; fi + [[ -n "$OPT_TEST_RE" ]] && printf '%s\n' "$OPT_TEST_RE" > "$OUT/test-re.txt" + if [[ "$RUNG" != "all" ]]; then + sed 's|^\./||' "$CANDIDATES" | grep -E '\.md$' | LC_ALL=C sort -u > "$OUT/docs.txt" + fi # not_bundled_reason — prints the reason, or nothing when it is bundled not_bundled_reason() { diff --git a/python-linter-driven-development/skills/pre-commit-review/SKILL.md b/python-linter-driven-development/skills/pre-commit-review/SKILL.md index c69e3c0d..624592bd 100644 --- a/python-linter-driven-development/skills/pre-commit-review/SKILL.md +++ b/python-linter-driven-development/skills/pre-commit-review/SKILL.md @@ -80,11 +80,15 @@ writes `hits.tsv`, `hits-all.tsv` and `counts.tsv` into the bundle, and prints t question only a hunter's reading answers — then one line per rule family. That table, about 300 tokens, is the pre-filter: the parent prints no rule file, composes no detection command, and never reads a Falsifying-questions section. On a scoped review the R9 -gate's rows are the scope's own files' — an orphan doc or an unwired root is the -repository's state, for a `--all` review or BROADER CONTEXT, never this diff's -finding. When the +gate checks only the docs the diff touches, the docs that link to a scope file and the +docs that cite a symbol the diff removes, and its rows are about those docs and the +scope's own files — an orphan doc elsewhere or an unwired root is the repository's +state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line +`R9 gate: …` under the summary says which docs the gate checked. A gate row whose +hits cell is `inconclusive` means the gate ran out of its `--gate-budget`: it is not +a `0`, and the documentation hunter answers that question from the prose. When the detection script exits 2 instead — a pattern its grep rejects, the R9 gate failing to -run — the review did not run: report its message, spawn nothing, and never render a +run, a scope in which no file matches the source glob — the review did not run: report its message, spawn nothing, and never render a rule as `skipped` or compose the pass by hand. A rule family none of whose rules has a hit — `0` on every row of theirs that is not `judgment` — is skipped: no hunter for it. @@ -114,7 +118,8 @@ zero prior use; the fix is a discussion or a separate PR, never silent inclusion The bundle exists (step 1 wrote it: `files.txt`, `diff.patch` on a scoped review, `scope/.txt` per bundled file with the file's own line numbers, `dirs.txt` on -`--all`, `comments.txt`, `hits.tsv`, `hits-all.tsv`, `counts.tsv`); nothing is +`--all`, `docs.txt` with the touched docs on a scoped review, `test-re.txt` with the +test-file pattern when one was passed, `comments.txt`, `hits.tsv`, `hits-all.tsv`, `counts.tsv`); nothing is written here. One Bash command, only when the scope is not empty, lists every rule path the prompts will carry and prints `sed -n '/^## Waiting for agents/,/^## The merged report/p' diff --git a/ts-react-linter-driven-development/scripts/check-repo-brain.sh b/ts-react-linter-driven-development/scripts/check-repo-brain.sh index 15f55e27..d28c9384 100755 --- a/ts-react-linter-driven-development/scripts/check-repo-brain.sh +++ b/ts-react-linter-driven-development/scripts/check-repo-brain.sh @@ -8,10 +8,22 @@ # in-repo copy). Installed into target repos by the documentation skill's # BOOTSTRAP pass (/wire-repo-brain). # -# Usage: bash scripts/check-repo-brain.sh [--fix] [repo-root] (default: cwd) +# Usage: bash scripts/check-repo-brain.sh [--fix] [--scope [--words ]] [repo-root] +# (repo-root default: cwd) # CI: one line — bash scripts/check-repo-brain.sh # --fix: rewrite drifted index lines from each target doc's `description` # (the one mechanical repair; everything else stays report-only) +# --scope a review's scope: one path per line, relative to repo-root +# (code files and docs, deleted ones included). The gate then checks +# only the docs in scope: the listed docs, the docs whose text names a +# listed file's basename, and the docs that cite a word of --words. +# Code edges are checked for the listed code files only, Q3 is skipped +# (root wiring is the repository's state, not a diff's), and with no +# doc in scope the doc checks are skipped. One stdout line says which: +# "check-repo-brain: scope — doc(s) to check" or +# "check-repo-brain: scope — no doc in scope; doc checks skipped". +# --words identifiers the scoped change removes, one per line; a doc +# that cites one of them in backticks is in scope # # Doc roots are discovered at the repo root AND at every sub-project (a # directory holding the language's project marker — see the adapter below), @@ -58,8 +70,11 @@ # against its declaring package or receiver — never against a same-named # member elsewhere. A token missing from the set still resolves when it # appears as a whole word in any non-markdown repo file (config keys, -# alert names, test helpers). A `pkg.Sym` whose package is not declared in -# this repo is external (stdlib, dependencies) and exempt. +# alert names, test helpers): in a git work tree the tracked and +# untracked-but-not-ignored files; elsewhere every file outside the +# dependency and build-output directories (bazel-*, node_modules, …). +# A `pkg.Sym` whose package is not declared in this repo is external +# (stdlib, dependencies) and exempt. # - lines carrying the ⚠️ stale flag or a *(planned)* marker are exempt from # symbol resolution and the description copy check (R9 Q2/Q7 exemptions); # the file:line ban has no exemption beyond URL spans, fenced code blocks, @@ -72,22 +87,37 @@ # 2 usage error, or an internal scanner failure (a failed scan is # inconclusive, never silently clean) # -# Uses only POSIX-portable tools: find, grep, sed, awk, head, sort, wc. No jq/python. +# Uses only POSIX-portable tools: find, grep, sed, awk, head, sort, wc, tr, xargs, +# plus git when the repository is a git work tree. No jq/python. # awk programs avoid interval expressions ({m,n}) — mawk, Debian's default, rejects them. set -u FIX=0 -if [[ "${1:-}" == "--fix" ]]; then - FIX=1 - shift -fi +SCOPE_LIST="" +WORDS_LIST="" +abs_list() { # -> absolute path of an existing file, or exit 2 + [[ -f "$1" ]] || { echo "check-repo-brain: no such list: $1" >&2; exit 2; } + printf '%s/%s\n' "$(cd "$(dirname "$1")" && pwd)" "$(basename "$1")" +} +while (( $# > 0 )); do + case "$1" in + --fix) FIX=1; shift ;; + --scope) SCOPE_LIST=$(abs_list "${2:-}") || exit 2; shift 2 ;; + --words) WORDS_LIST=$(abs_list "${2:-}") || exit 2; shift 2 ;; + --*) echo "check-repo-brain: unknown option: $1" >&2; exit 2 ;; + *) break ;; + esac +done +[[ -n "$WORDS_LIST" && -z "$SCOPE_LIST" ]] && { echo "check-repo-brain: --words needs --scope" >&2; exit 2; } REPO_ROOT="${1:-$(pwd)}" if [[ ! -d "$REPO_ROOT" ]]; then echo "check-repo-brain: not a directory: $REPO_ROOT" >&2 exit 2 fi cd "$REPO_ROOT" || exit 2 +SCOPED=0 +[[ -n "$SCOPE_LIST" ]] && SCOPED=1 # ==================== language adapter: ts-react ==================== # Everything language-specific lives between these two marker comments. The @@ -411,22 +441,65 @@ lifecycle_stale() { # -> reason when frontmatter marks the doc stale ('' have_code=0 lang_has_code && have_code=1 -# ---------- declaration set: built once, queried per token ---------- -DECLS="" PKGS="" -if (( have_code )); then - DECLS=$(mktemp) PKGS=$(mktemp) DECL_ALL=$(mktemp) - trap 'rm -f "$DECLS" "$PKGS"' EXIT - if ! ( set -o pipefail; lang_declarations | sort -u > "$DECL_ALL" ); then +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"' EXIT + +# ---------- declaration set: built once, on the first token to resolve ---------- +DECLS="$WORK/decls" PKGS="$WORK/pkgs" +decls_built=0 +build_declarations() { + (( decls_built )) && return 0 + decls_built=1 + if ! ( set -o pipefail; lang_declarations | sort -u > "$WORK/decl-all" ); then echo "check-repo-brain: internal scanner error — the declaration scan failed; treating the run as inconclusive" >&2 exit 2 fi - grep '^pkg:' "$DECL_ALL" | sed 's/^pkg://' > "$PKGS" - grep -v '^pkg:' "$DECL_ALL" > "$DECLS" - rm -f "$DECL_ALL" -fi + grep '^pkg:' "$WORK/decl-all" | sed 's/^pkg://' > "$PKGS" + grep -v '^pkg:' "$WORK/decl-all" > "$DECLS" + rm -f "$WORK/decl-all" +} is_repo_pkg() { grep -qxF "$1" "$PKGS" 2>/dev/null; } +# repo_files — the repository's non-markdown files, NUL-separated. A git work +# tree lists them through git (tracked files plus untracked ones git does not +# ignore), which never walks build outputs or symlinked caches; elsewhere find +# skips the directories that hold dependencies and build outputs. +repo_files() { + if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then + git ls-files -z --cached --others --exclude-standard -- . \ + ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**' 2>/dev/null && return 0 + fi + find . \( -name .git -o -name vendor -o -name node_modules -o -name 'bazel-*' -o -name .venv \ + -o -name venv -o -name __pycache__ -o -name .tox -o -name dist -o -name target \) -prune \ + -o -type f ! -name '*.md' -print0 +} + +# resolve_words — the tokens that occur as a whole word in any +# non-markdown repository file, one per line. The repository is read once +# into words (runs of [A-Za-z0-9_.]); a dotted token such as `Type.Method` +# matches any run of whole dot-separated parts, as grep -w would match it. +# One pass over the text is linear in its size, where a fixed-string grep for +# hundreds of tokens is not. +resolve_words() { + repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.' '\n' \ + | LC_ALL=C awk -v list="$1" ' + BEGIN { while ((getline t < list) > 0) if (t != "") want[t] = 1 } + seen[$0]++ { next } + { + n = split($0, p, ".") + for (i = 1; i <= n; i++) { + s = p[i] + if (s in want) { print s; delete want[s] } + for (j = i + 1; j <= n; j++) { + s = s "." p[j] + if (s in want) { print s; delete want[s] } + } + } + }' + return 0 +} + # One fence-aware awk pass per bundle extracts everything Q2's doc scan needs: # P — a file:line citation outside fences/URLs/globs # S — a backticked symbol-shaped token to resolve @@ -465,8 +538,53 @@ FNR == 1 { fence = 0 } } ' -# ---------- Q2: code→docs edges (repo-wide; resolved from repo root, then the -# citing file's own sub-project) ---------- +# ---------- the scope: which docs a scoped run checks ---------- +# A doc is in scope when the scope lists it, when its text names the basename +# of a listed file as a whole word (a link to a touched doc or code file), or +# when it cites in backticks a word the change removes (a symbol the diff +# renames or deletes). +# Every list below holds relative paths as find(1) prints them under a root. +SCOPE_DOCS="$WORK/scope-docs" +SCOPE_PATHS="$WORK/scope-paths" +if (( SCOPED )); then + sed -e 's|^\./||' -e '/^$/d' "$SCOPE_LIST" | LC_ALL=C sort -u > "$SCOPE_PATHS" + for r in "${ROOTS[@]}"; do find "$r" -type f -name '*.md'; done | LC_ALL=C sort -u > "$WORK/all-docs" + { + grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" + sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$WORK/basenames" + if [[ -s "$WORK/basenames" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$WORK/basenames" 2>/dev/null + fi + if [[ -n "$WORDS_LIST" && -s "$WORDS_LIST" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" \ + | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ + -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" "$DOCSCAN_AWK" \ + | awk -F'\t' 'NR == FNR { w[$0] = 1; next } + $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in w) || (m in w)) print $2 }' "$WORDS_LIST" - + fi + } | LC_ALL=C sort -u > "$SCOPE_DOCS" + if [[ -s "$SCOPE_DOCS" ]]; then + echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" + else + echo "check-repo-brain: scope — no doc in scope; doc checks skipped" + fi +fi + +# bundle_docs [name] — the docs a run checks under one root: every +# *.md (or every file of that name), limited to the scope on a scoped run +bundle_docs() { + local name="${2:-*.md}" + if (( SCOPED )); then + find "$1" -type f -name "$name" | grep -xF -f "$SCOPE_DOCS" + else + find "$1" -type f -name "$name" + fi + return 0 +} + +# ---------- Q2: code→docs edges (repo-wide, or the scope's code files on a +# scoped run; resolved from repo root, then the citing file's own +# sub-project) ---------- docroot_for_file() { # -> docroot of the longest matching project dir local f="${1#./}" best="" i for i in "${!PROJS[@]}"; do @@ -490,7 +608,13 @@ if (( have_code )); then CUR_DOCROOT=$(docroot_for_file "$file") fail "[Q2] $file:$line — code edge points at missing $target" fi - done < <(lang_code_edges) + done < <(if (( SCOPED )); then + lang_code_edges | awk -v list="$SCOPE_PATHS" ' + BEGIN { while ((getline l < list) > 0) s[l] = 1 } + { f = substr($0, 1, index($0, ":") - 1); sub(/^\.\//, "", f); if (f in s) print }' + else + lang_code_edges + fi) fi # ---------- per-bundle checks ---------- @@ -500,6 +624,8 @@ check_bundle() { # local root_index="$docroot/index.md" local root_index_c root_index_c=$(canon "$root_index") + # a scoped run skips a bundle that holds no doc in scope + (( SCOPED )) && [[ -z "$(bundle_docs "$docroot")" ]] && return 0 # --- Q1: transitive reachability from the bundle's root index --- local reachable="" visited="" queue=("$root_index") @@ -528,7 +654,7 @@ check_bundle() { # case "$reachable" in *"$c"$'\n'*) ;; *) fail "[Q1] $doc — orphan: not reachable from $root_index" ;; esac - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q2: every doc link resolves (a missing parent directory is just as # broken as a missing file — only URLs/anchors are exempt) --- @@ -543,13 +669,13 @@ check_bundle() { # fail "[Q2] $md — link target does not exist: $t" fi done < <(grep -oE '\]\([^)]+\)' "$md" 2>/dev/null) - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q2: doc scan — file:line ban + docs→code symbol resolution. # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). --- local scan; scan=$(mktemp) if ! ( set -o pipefail - find "$docroot" -type f -name '*.md' -print0 \ + bundle_docs "$docroot" | tr '\n' '\0' \ | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" \ "$DOCSCAN_AWK" > "$scan" ); then @@ -564,6 +690,7 @@ check_bundle() { # if (( have_code )) && grep -q $'^S\t' "$scan"; then local toks check members unres bad toks=$(mktemp) check=$(mktemp) members=$(mktemp) unres=$(mktemp) bad=$(mktemp) + build_declarations grep $'^S\t' "$scan" | cut -f4 | sort -u > "$toks" # tokens to resolve (external pkg.Sym exempt). A qualified token resolves # as the PAIR itself — the declaration set carries pkg.Ident / Type.Method @@ -581,12 +708,11 @@ check_bundle() { # esac done < "$toks" > "$check" cut -f2 "$check" | sort -u > "$members" - grep -vxF -f "$DECLS" "$members" > "$unres" || true + grep -vxF -f "$DECLS" "$members" | grep -v '^$' > "$unres" || true if [[ -s "$unres" ]]; then - # ONE repo-wide word grep for the whole unresolved batch + # ONE pass over the repository for the whole unresolved batch local found; found=$(mktemp) - grep -rIhoFw --exclude-dir=vendor --exclude-dir=.git --exclude='*.md' \ - -f "$unres" . 2>/dev/null | sort -u > "$found" + resolve_words "$unres" | sort -u > "$found" grep -vxF -f "$found" "$unres" > "$bad" || true rm -f "$found" fi @@ -604,24 +730,27 @@ check_bundle() { # fi rm -f "$scan" - # --- Q3: root wiring (exact path; sub-roots may ride the repo-root index) --- - local rel="$docroot" - [[ "$proj" != "." ]] && rel="${docroot#$proj/}" - local claude="CLAUDE.md" agents="AGENTS.md" - [[ "$proj" != "." ]] && { claude="$proj/CLAUDE.md"; agents="$proj/AGENTS.md"; } - local wired_claude=0 wired_agents=0 - [[ -f "$claude" ]] && grep -qF -- "$rel/index.md" "$claude" && wired_claude=1 - [[ -f "$agents" ]] && grep -qF -- "$rel/index.md" "$agents" && wired_agents=1 - if (( ! wired_claude && ! wired_agents )); then - local via_root=0 - if [[ "$proj" != "." && -n "$ROOT_BUNDLE" ]]; then - grep -rqF -- "$docroot/index.md" "$ROOT_BUNDLE" --include='*.md' 2>/dev/null && via_root=1 - fi - if (( ! via_root )); then - fail "[Q3] $proj — neither $claude nor $agents references $rel/index.md" + # --- Q3: root wiring (exact path; sub-roots may ride the repo-root index); + # a scoped run leaves it to a whole-repository run --- + if (( ! SCOPED )); then + local rel="$docroot" + [[ "$proj" != "." ]] && rel="${docroot#$proj/}" + local claude="CLAUDE.md" agents="AGENTS.md" + [[ "$proj" != "." ]] && { claude="$proj/CLAUDE.md"; agents="$proj/AGENTS.md"; } + local wired_claude=0 wired_agents=0 + [[ -f "$claude" ]] && grep -qF -- "$rel/index.md" "$claude" && wired_claude=1 + [[ -f "$agents" ]] && grep -qF -- "$rel/index.md" "$agents" && wired_agents=1 + if (( ! wired_claude && ! wired_agents )); then + local via_root=0 + if [[ "$proj" != "." && -n "$ROOT_BUNDLE" ]]; then + grep -rqF -- "$docroot/index.md" "$ROOT_BUNDLE" --include='*.md' 2>/dev/null && via_root=1 + fi + if (( ! via_root )); then + fail "[Q3] $proj — neither $claude nor $agents references $rel/index.md" + fi + elif (( ! wired_agents )); then + note "[Q3] $agents lacks the $rel/index.md routing reference — AGENTS.md-reading tools start blind" fi - elif (( ! wired_agents )); then - note "[Q3] $agents lacks the $rel/index.md routing reference — AGENTS.md-reading tools start blind" fi # --- Q7: bundle contract --- @@ -686,7 +815,7 @@ check_bundle() { # else fail "[Q7] $md — frontmatter missing 'description:'" fi - done < <(find "$docroot" -type f -name '*.md') + done < <(bundle_docs "$docroot") # --- Q7: drift check — index line text == target's description (⚠️ exempt; # --fix rewrites drifted lines after the read loop, never during it) --- @@ -724,12 +853,12 @@ check_bundle() { # echo " fixed: $idx:$n — index line rewritten from its target's description" done <<< "$fixes" fi - done < <(find "$docroot" -type f -name 'index.md') + done < <(bundle_docs "$docroot" index.md) # --- Q7: no log.md --- while IFS= read -r lg; do fail "[Q7] $lg — log.md is reserved for change history; docs describe current behavior" - done < <(find "$docroot" -type f -name 'log.md') + done < <(bundle_docs "$docroot" log.md) } for i in "${!PROJS[@]}"; do diff --git a/ts-react-linter-driven-development/scripts/check-repo-brain_test.sh b/ts-react-linter-driven-development/scripts/check-repo-brain_test.sh index 13768b68..020d1215 100755 --- a/ts-react-linter-driven-development/scripts/check-repo-brain_test.sh +++ b/ts-react-linter-driven-development/scripts/check-repo-brain_test.sh @@ -653,6 +653,75 @@ run_gate expect_exit 0 && expect_not "does not resolve" && expect_has "check-repo-brain: clean" && ok finish +# --- scoped runs (--scope, --words) and the symbol resolver --- +LISTS=$(mktemp -d) # the scope and word lists live outside the repository they describe +begin "--scope: a code-only scope that no doc cites skips the doc checks" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here." +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "scope — no doc in scope; doc checks skipped" && expect_not "orphan" && ok +finish + +begin "--scope: a listed doc is checked, an unlisted orphan and the root wiring are not" +mk_conformant; mk_doc orphan "an unlinked doc" "Nothing links here."; mk_doc other "another unlinked doc" "Nothing." +rm "$REPO/CLAUDE.md" "$REPO/AGENTS.md" +printf '%s\n' "$FX_CODE_FILE" docs/orphan.md > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] docs/orphan.md — orphan" \ + && expect_not "docs/other.md" && expect_not "[Q3]" && ok +finish + +begin "--scope: a doc that names a scope file is checked" +mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../$FX_CODE_FILE)." +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +finish + +begin "--scope --words: a doc that cites a removed symbol is checked" +mk_conformant +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +append_index "- [ghost.md](ghost.md) — a ghost symbol" +printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" +run_gate --scope "$LISTS/scope.txt" +expect_exit 0 && expect_has "no doc in scope" \ + && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ + && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok +finish + +begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" +mk_conformant +printf 'build/\n' > "$REPO/.gitignore" +mkdir -p "$REPO/build"; printf 'IgnoredOnlyName\n' > "$REPO/build/gen.yaml" +printf 'TrackedName\n' > "$REPO/tracked.yaml" +mk_doc names "alert names" "Fires \`TrackedName\`, \`UntrackedName\` and \`IgnoredOnlyName\`." +append_index "- [names.md](names.md) — alert names" +(cd "$REPO" && git init -q && git add -A && git -c user.name=t -c user.email=t@t commit -qm init) +printf 'UntrackedName\n' > "$REPO/untracked.yaml" +run_gate +expect_exit 1 && expect_has "\`IgnoredOnlyName\` does not resolve" \ + && expect_not "\`TrackedName\` does not resolve" && expect_not "\`UntrackedName\` does not resolve" && ok +finish + +begin "Q2 outside git a token does not resolve from a bazel-* output directory" +mk_conformant +mkdir -p "$REPO/bazel-out"; printf 'BazelOnlyName\n' > "$REPO/bazel-out/gen.yaml" +mk_doc bazel "bazel names" "Fires \`BazelOnlyName\`." +append_index "- [bazel.md](bazel.md) — bazel names" +run_gate +expect_exit 1 && expect_has "\`BazelOnlyName\` does not resolve" && ok +finish + +begin "Q2 a dotted token resolves on whole dot-separated words only" +mk_conformant +printf 'limits:\n key: Cfg.MaxRetries.value\n other: Cfg.MaxRetriesX\n' > "$REPO/limits.yaml" +mk_doc dotted "dotted names" "Reads \`Cfg.MaxRetries\`; \`Cfg.MaxRetri\` is a typo." +append_index "- [dotted.md](dotted.md) — dotted names" +run_gate +expect_exit 1 && expect_has "\`Cfg.MaxRetri\` does not resolve" && expect_not "\`Cfg.MaxRetries\` does not resolve" && ok +finish +rm -rf "$LISTS" + } # run_row_cases for ROW in $FX_ROWS; do diff --git a/ts-react-linter-driven-development/scripts/ldd-detect.sh b/ts-react-linter-driven-development/scripts/ldd-detect.sh index 7e35d2a2..10b8a856 100755 --- a/ts-react-linter-driven-development/scripts/ldd-detect.sh +++ b/ts-react-linter-driven-development/scripts/ldd-detect.sh @@ -25,7 +25,14 @@ # --plan print the parsed detect lines and exit; nothing runs # --glob the scope is one group of a language the table # does not know, matched by this glob -# --test-re test-file pattern over relative paths, same case +# --test-re test-file pattern over relative paths, same case. +# A bundle carries the pattern ldd-scope.sh used +# (test-re.txt), and a custom group its glob +# (language.txt): the bundle's values win, and a +# --glob or --test-re that differs is a usage error +# --gate-budget wall-clock seconds the R9 gate may run (default: no +# limit); on expiry the gate is stopped and its +# questions are counted "inconclusive", never 0 # # The scope's language is the bundle's language.txt; without it, the files in # files.txt tell it by extension, and a list that spans several languages is @@ -39,10 +46,14 @@ # Detect-path: `` over the scope's relative paths (layer directories, # role-named packages); a hit's line is 0 # Detect-gate: Q the R9 gate's [Q] report lines, from one run of -# check-repo-brain.sh beside this script over --root; on a -# scoped bundle (no dirs.txt) only the lines whose file is in -# the scope — the gate's repository-wide findings belong to -# a whole-repository review +# check-repo-brain.sh beside this script over --root. On a +# scoped bundle (no dirs.txt) the gate runs with --scope: it +# checks only the bundle's docs (docs.txt), the docs that +# link to a scope file and the docs that cite a symbol the +# diff removes, the code edges of the scope's code files, +# and no root wiring; with no such doc its doc checks are +# skipped. The repository-wide findings belong to a +# whole-repository review # Detect: judgment no mechanical lead; the hunter runs the prose # A question with no detect line, two of them, or a malformed one is an error: # the generator's lint-core keeps the rule files honest, and this script refuses @@ -53,15 +64,18 @@ # the overflow row of a capped question: file "-", line 0, # excerpt "+ more hit(s) not listed" # hits-all.tsv the same columns, every hit, no cap and no overflow row -# counts.tsv rule question kind hits (judgment rows carry "-") +# counts.tsv rule question kind hits (judgment rows carry "-", gate +# rows "inconclusive" when the gate ran out of --gate-budget) # the last row is SUPPRESS - grep : suppression directives # on the diff's added lines when the bundle has a diff.patch, # else over every scope file # Two runs over one tree write byte-identical counts.tsv. # # Exit codes: 0 tables written · 2 usage error, a detect line that does not -# parse, a pattern grep rejects, or a gate run that failed -# (inconclusive, never silently clean) +# parse, a pattern grep rejects, a gate run that failed +# (inconclusive, never silently clean), or a non-empty scope in +# which no file matches the source glob (a glob mismatch, never +# an empty table) # # Uses only POSIX-portable tools: grep, sed, awk, sort, wc, head. GNU grep's # \b and \s are assumed, as the rules already assume them. @@ -77,6 +91,7 @@ CAP=40 PLAN=0 OPT_GLOB="" OPT_TEST_RE="" +GATE_BUDGET=0 BUNDLE="" usage() { sed -n '2,/^$/p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; } @@ -94,12 +109,14 @@ while (( $# > 0 )); do --plan) PLAN=1; shift ;; --glob) value "$1" $#; OPT_GLOB="$2"; shift 2 ;; --test-re) value "$1" $#; OPT_TEST_RE="$2"; shift 2 ;; + --gate-budget) value "$1" $#; GATE_BUDGET="$2"; shift 2 ;; -h|--help) usage ;; --*) die "unknown option: $1" ;; *) [[ -n "$BUNDLE" ]] && die "one bundle directory, got also: $1"; BUNDLE="$1"; shift ;; esac done [[ "$CAP" =~ ^[0-9]+$ ]] || die "--cap takes a number, got: $CAP" +[[ "$GATE_BUDGET" =~ ^[0-9]+$ ]] || die "--gate-budget takes a number of seconds, got: $GATE_BUDGET" [[ -d "$RULES_DIR" ]] || die "no rules directory: $RULES_DIR" RULES_DIR=$(cd "$RULES_DIR" && pwd) if (( ! PLAN )); then @@ -255,6 +272,7 @@ if (( ! PLAN )) && [[ -f "$BUNDLE/groups.txt" ]]; then fi args=(--root "$ROOT" --cap "$CAP") [[ -n "$OPT_TEST_RE" ]] && args+=(--test-re "$OPT_TEST_RE") + (( GATE_BUDGET > 0 )) && args+=(--gate-budget "$GATE_BUDGET") bash "$script" "${args[@]}" "$bundle" || { failed=1; echo "$SCRIPT_NAME: the $id group's detection pass failed; it is inconclusive" >&2; } echo done < "$BUNDLE/groups.txt" @@ -269,7 +287,9 @@ LANG_ID="" if (( ! PLAN )); then if [[ -f "$BUNDLE/language.txt" ]]; then IFS=$'\t' read -r LANG_ID glob < "$BUNDLE/language.txt" - [[ "$LANG_ID" == "custom" && -z "$OPT_GLOB" ]] && OPT_GLOB="$glob" + [[ "$LANG_ID" == "custom" && -n "$OPT_GLOB" && "$OPT_GLOB" != "$glob" ]] \ + && die "--glob '$OPT_GLOB' differs from the bundle's glob '$glob' (ldd-scope.sh wrote it; drop --glob)" + [[ "$LANG_ID" == "custom" ]] && OPT_GLOB="$glob" else ids="" while IFS= read -r line; do @@ -284,6 +304,14 @@ if (( ! PLAN )); then [[ "$ids" == *" "* ]] && die "the scope list spans several languages ($ids); ldd-scope.sh writes one bundle per language" LANG_ID="$ids" fi + # the test-file pattern ldd-scope.sh used, so the two scripts cannot disagree + # on what a test file is + if [[ -f "$BUNDLE/test-re.txt" ]]; then + b_test_re=$(cat "$BUNDLE/test-re.txt") + [[ -n "$OPT_TEST_RE" && "$OPT_TEST_RE" != "$b_test_re" ]] \ + && die "--test-re '$OPT_TEST_RE' differs from the bundle's '$b_test_re' (ldd-scope.sh wrote it; drop --test-re)" + OPT_TEST_RE="$b_test_re" + fi lang_configure "$LANG_ID" || die "this plugin has no language block for $LANG_ID" fi @@ -329,7 +357,7 @@ END { if (insec) flush() } ' plan_file=$(mktemp) -trap 'rm -f "$plan_file" "${tmp_hits:-}" "${tmp_hits_all:-}" "${tmp_counts:-}" "${gate_out:-}"' EXIT +trap 'rm -f "$plan_file" "${tmp_hits:-}" "${tmp_hits_all:-}" "${tmp_counts:-}" "${gate_out:-}" "${gate_scope:-}" "${gate_words:-}"' EXIT rule_files=$(find "$RULES_DIR" -maxdepth 1 -name 'R*.md' | awk -F/ '{ n = $NF; sub(/^R/, "", n); sub(/-.*$/, "", n); print n "\t" $0 }' | sort -n | cut -f2) [[ -n "$rule_files" ]] || die "no rule files (R*.md) under $RULES_DIR" while IFS= read -r f; do @@ -355,11 +383,13 @@ fi SRC_FILES=() TEST_FILES=() ALL_FILES=() +n_present=0 while IFS= read -r line; do f="${line%% (not bundled:*}" f="${f#./}" [[ -z "$f" ]] && continue [[ -f "$f" ]] || continue + n_present=$((n_present + 1)) [[ "$(lang_of_path "$f")" == "$LANG_ID" ]] || continue printf '%s\n' "$f" | grep -qE -- "$LANG_EXCLUDE_RE" && continue ALL_FILES+=("$f") @@ -367,6 +397,12 @@ done < <(LC_ALL=C sort -u "$FILES_LIST") for f in ${ALL_FILES[@]+"${ALL_FILES[@]}"}; do if lang_is_test "$f"; then TEST_FILES+=("$f"); else SRC_FILES+=("$f"); fi done +# A scope whose files exist but none of which is a source file is a glob +# mismatch between the scope and this run: an all-zero table would read as a +# clean review of nothing. +if (( ${#ALL_FILES[@]} == 0 && n_present > 0 )); then + die "0 source + 0 test files: none of the $n_present file(s) in $FILES_LIST is a $LANG_ID source file (pass the --glob ldd-scope.sh used)" +fi tmp_hits=$(mktemp) tmp_hits_all=$(mktemp) @@ -480,24 +516,86 @@ run_path() { # rm -f "$raw" "$err" } -# The R9 gate runs once, lazily; its [Q] lines are split by question. +# gate_scope_lists — what a scoped gate run checks: +# every path of the scope (deleted ones too: a doc can still link to them) and +# of docs.txt; and the identifiers the diff removes — on its removed lines and +# on none of its added ones — the symbols a renamed or deleted declaration +# takes away from the docs that cite them. Comment lines (a line that opens +# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped. +gate_scope_lists() { + local scope="$1" words="$2" line f + { + while IFS= read -r line; do + f="${line%% (not bundled:*}"; f="${f#./}" + [[ -n "$f" ]] && printf '%s\n' "$f" + done < "$FILES_LIST" + [[ -f "$BUNDLE/docs.txt" ]] && cat "$BUNDLE/docs.txt" + } | LC_ALL=C sort -u > "$scope" + : > "$words" + [[ -s "$BUNDLE/diff.patch" ]] || return 0 + grep -vE '^(\+\+\+|---) ' "$BUNDLE/diff.patch" | grep -E '^[-+]' \ + | LC_ALL=C awk ' + { + s = substr($0, 1, 1); t = substr($0, 2) + if (t ~ /^[ \t]*(\/\/|#|\/\*|\*|--|""")/) next + while (match(t, /[A-Za-z_][A-Za-z0-9_]*/)) { + w = substr(t, RSTART, RLENGTH); t = substr(t, RSTART + RLENGTH) + if (s == "+") add[w] = 1; else del[w] = 1 + } + } + END { for (w in del) if (!(w in add)) print w }' | LC_ALL=C sort > "$words" +} + +# The R9 gate runs once, lazily; its [Q] lines are split by question. With +# --gate-budget the gate runs as its own process group, so expiry stops every +# process it started, not only its shell. +GATE_STATE="whole repository" run_gate_once() { [[ -n "$gate_out" ]] && return 0 gate_out=$(mktemp) - local gate="$SCRIPT_DIR/check-repo-brain.sh" code + local gate="$SCRIPT_DIR/check-repo-brain.sh" code=0 start pid scope_line + local -a args=() [[ -f "$gate" ]] || die "Detect-gate needs $gate beside this script" - bash "$gate" "$ROOT" > "$gate_out" 2>&1 - code=$? + if [[ ! -f "$BUNDLE/dirs.txt" ]]; then + gate_scope=$(mktemp); gate_words=$(mktemp) + gate_scope_lists "$gate_scope" "$gate_words" + args=(--scope "$gate_scope" --words "$gate_words") + fi + if (( GATE_BUDGET == 0 )); then + bash "$gate" ${args[@]+"${args[@]}"} "$ROOT" > "$gate_out" 2>&1 || code=$? + else + start=$SECONDS + set -m + bash "$gate" ${args[@]+"${args[@]}"} "$ROOT" > "$gate_out" 2>&1 & + pid=$! + set +m + while kill -0 "$pid" 2>/dev/null; do + if (( SECONDS - start >= GATE_BUDGET )); then + kill -TERM -- "-$pid" 2>/dev/null + wait "$pid" 2>/dev/null + GATE_STATE="inconclusive — stopped after the ${GATE_BUDGET}s --gate-budget" + return 0 + fi + sleep 0.2 + done + wait "$pid" || code=$? + fi if (( code == 2 )); then head -5 "$gate_out" >&2 die "the R9 gate failed to run (exit 2); its questions are inconclusive" fi + scope_line=$(grep -m1 '^check-repo-brain: scope — ' "$gate_out") + [[ -n "$scope_line" ]] && GATE_STATE="scoped, ${scope_line#check-repo-brain: scope — }" return 0 } run_gate() { # local rule="$1" q="$2" qn="$3" raw tab run_gate_once + if [[ "$GATE_STATE" == inconclusive* ]]; then + printf '%s\t%s\tgate\tinconclusive\n' "$rule" "$q" >> "$tmp_counts" + return 0 + fi raw=$(mktemp) tab=$(printf '\t') # "[Qn] [:line] — — see " becomes @@ -515,15 +613,9 @@ run_gate() { # if (match(loc, /:[0-9]+$/)) { file = substr(loc, 1, RSTART - 1); line = substr(loc, RSTART + 1) + 0 } print file "\t" line "\t" rest }' > "$raw" - # A scoped review owns only the gate lines about its own files: an orphan - # doc, an unwired root or another file's broken edge is the repository's - # state, not the diff's, and is not a lead for this review's hunter. - if [[ ! -f "$BUNDLE/dirs.txt" ]] && [[ -s "$raw" ]]; then - scoped=$(mktemp); scope_list=$(mktemp) - printf '%s\n' ${ALL_FILES[@]+"${ALL_FILES[@]}"} > "$scope_list" - awk -F'\t' 'NR == FNR { in_scope[$0] = 1; next } ($1 in in_scope)' "$scope_list" "$raw" > "$scoped" - mv "$scoped" "$raw"; rm -f "$scope_list" - fi + # A scoped review owns only the gate lines about its own files and the docs + # that cite them; the gate's --scope run reports nothing else (an orphan doc + # elsewhere or an unwired root is the repository's state, not the diff's). record "$rule" "$q" gate "$raw" 0 rm -f "$raw" } @@ -573,6 +665,7 @@ n_gate=$(awk -F'\t' '$3 == "gate"' "$plan_file" | wc -l | tr -d ' ') n_judg=$(awk -F'\t' '$3 == "judgment"' "$plan_file" | wc -l | tr -d ' ') n_q=$(wc -l < "$plan_file" | tr -d ' ') echo "$SCRIPT_NAME: $n_q questions ($n_grep grep, $n_path path, $n_gate gate, $n_judg judgment) over ${#SRC_FILES[@]} source + ${#TEST_FILES[@]} test files → $BUNDLE/hits.tsv, counts.tsv" +(( n_gate > 0 )) && [[ -n "$gate_out" ]] && echo "R9 gate: $GATE_STATE" echo awk -F'\t' 'BEGIN { printf "%-9s %-4s %-9s %s\n", "rule", "q", "kind", "hits" } { q = ($2 == "-") ? "-" : "Q" $2; printf "%-9s %-4s %-9s %s\n", $1, q, $3, $4 }' "$BUNDLE/counts.tsv" @@ -592,12 +685,14 @@ awk -F'\t' ' f = fam[$1]; if (f == "") f = "other" nq[f]++ if ($3 == "judgment") { nj[f]++; next } + if ($4 == "inconclusive") { ni[f]++; next } if ($4 + 0 > 0) { hits[f] += $4; nh[f]++; rh[f, $1] += $4 } } END { for (i = 1; i <= 4; i++) { f = order[i] printf "%-15s %5d hit(s) in %d of %d questions, %d judgment", f, hits[f] + 0, nh[f] + 0, nq[f] + 0, nj[f] + 0 + if (ni[f] + 0 > 0) printf ", %d inconclusive", ni[f] per = ""; n = split(rules[f], rs, " ") for (j = 1; j <= n; j++) if (rh[f, rs[j]] + 0 > 0) per = per (per == "" ? "" : " · ") rs[j] " " rh[f, rs[j]] if (per != "") printf " — %s", per diff --git a/ts-react-linter-driven-development/scripts/ldd-detect_test.sh b/ts-react-linter-driven-development/scripts/ldd-detect_test.sh index 10b19c46..5b7c517a 100644 --- a/ts-react-linter-driven-development/scripts/ldd-detect_test.sh +++ b/ts-react-linter-driven-development/scripts/ldd-detect_test.sh @@ -49,6 +49,11 @@ run_detect() { # [options...] — runs the script over $REPO with $BUNDLE OUT=$(cd "$REPO" && bash "$DETECT" "$@" --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" } +run_detect_bundle() { # [options...] — runs the script over $REPO with that bundle + local b="$1"; shift + OUT=$(cd "$REPO" && bash "$DETECT" "$@" --root . "$b" 2>"$BUNDLE/.stderr"); CODE=$? + ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +} run_scope() { # [args...] — runs ldd-scope.sh over $REPO into $BUNDLE/scope-out rm -rf "$BUNDLE/scope-out" OUT=$(cd "$REPO" && bash "$SCOPE_SH" --out "$BUNDLE/scope-out" "$@" 2>"$BUNDLE/.stderr"); CODE=$? @@ -75,8 +80,9 @@ expect_not() { return 0 } expect_quiet() { [[ -z "$ERR" ]] || { bad "stderr not empty"; return 1; }; } -count_of() { # — the hits cell of counts.tsv, "-" for judgment - awk -F'\t' -v r="$1" -v q="$2" '$1 == r && $2 == q { print $4 }' "$BUNDLE/counts.tsv" +count_of() { # — the hits cell of counts.tsv (COUNTS_FILE, default the + # case bundle's), "-" for judgment + awk -F'\t' -v r="$1" -v q="$2" '$1 == r && $2 == q { print $4 }' "${COUNTS_FILE:-$BUNDLE/counts.tsv}" } expect_count() { # =n> local got; got=$(count_of "$1" "$2") @@ -478,6 +484,72 @@ expect_exit 0 && expect_count SUPPRESS - 1 \ && grep -q "^SUPPRESS - grep $FX_SRC_FILE 2 " "$BUNDLE/hits.tsv" && ok finish +# --- the bundle ldd-scope.sh writes, read by detection --- +SO="" # the scope bundle of the case: $BUNDLE/scope-out + +begin "a code-only scoped diff skips the R9 gate's doc checks" +mk_repo; git_commit_repo +printf '\n' >> "$REPO/$FX_SRC_FILE" # a code-only change +(cd "$REPO" && git -c user.name=t -c user.email=t@t commit -qam change) +run_scope --base HEAD~1; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" +expect_exit 0 && expect_has "R9 gate: scoped, no doc in scope; doc checks skipped" \ + && COUNTS_FILE="$SO/counts.tsv" expect_count R9 1 0 && ok +finish + +begin "a scoped diff that touches a doc gets that doc checked by the R9 gate" +mk_repo; git_commit_repo +printf '\n' >> "$REPO/$FX_SRC_FILE" # a code-only change +printf '\nMore text.\n' >> "$REPO/docs/orphan.md" +(cd "$REPO" && git -c user.name=t -c user.email=t@t commit -qam change) +run_scope --base HEAD~1; SO="$BUNDLE/scope-out" +expect_exit 0 && { grep -qx 'docs/orphan.md' "$SO/docs.txt" || bad "docs.txt lacks the touched doc"; } \ + && grep -qx 'docs/orphan.md' "$SO/docs.txt" \ + && { run_detect_bundle "$SO"; expect_exit 0; } \ + && expect_has "R9 gate: scoped, 1 doc(s) to check" \ + && COUNTS_FILE="$SO/counts.tsv" expect_count R9 1 1 \ + && { grep -q "^R9 1 gate docs/orphan.md " "$SO/hits.tsv" || bad "no R9 Q1 hit row for docs/orphan.md"; } \ + && grep -q "^R9 1 gate docs/orphan.md " "$SO/hits.tsv" && ok +finish + +begin "detection reads the source glob from the bundle ldd-scope.sh wrote" +mk_repo; mkdir -p "$REPO/lib"; printf 'thing\n' > "$REPO/lib/thing.zz"; git_commit_repo +run_scope --glob '*.zz' lib/thing.zz; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" +expect_exit 0 && expect_has "over 1 source + 0 test files" && ok +finish + +begin "a --glob that differs from the bundle's is a usage error (exit 2)" +mk_repo; mkdir -p "$REPO/lib"; printf 'thing\n' > "$REPO/lib/thing.zz"; git_commit_repo +run_scope --glob '*.zz' lib/thing.zz; SO="$BUNDLE/scope-out" +run_detect_bundle "$SO" --glob '*.yy' +expect_exit 2 && expect_has "differs from the bundle's glob" && ok +finish + +begin "a scope whose files match no source glob is an error, not an all-zero table (exit 2)" +mk_repo; printf 'notes\n' > "$REPO/notes.txt"; printf 'notes.txt\n' > "$BUNDLE/files.txt" +printf 'custom\t*.zz\n' > "$BUNDLE/language.txt" +run_detect +expect_exit 2 && expect_has "0 source + 0 test files" && ok +finish + +begin "--gate-budget stops a slow gate and its children; the gate questions are inconclusive" +mk_repo; : > "$BUNDLE/dirs.txt" +FAKE=$(mktemp -d) +cp "$DETECT" "$FAKE/ldd-detect.sh" +printf '#!/usr/bin/env bash\nsleep 37\necho never\n' > "$FAKE/check-repo-brain.sh" +t0=$SECONDS +OUT=$(cd "$REPO" && bash "$FAKE/ldd-detect.sh" --rules "$RULES" --gate-budget 1 --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? +ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +took=$((SECONDS - t0)) +expect_exit 0 && { (( took < 15 )) || bad "the run took ${took}s with a 1s gate budget"; } && (( took < 15 )) \ + && expect_has "R9 gate: inconclusive" && expect_count R9 1 inconclusive && expect_has "inconclusive" \ + && { ! pgrep -f 'sleep 37' >/dev/null || bad "the gate's child process outlived the budget"; } \ + && ! pgrep -f 'sleep 37' >/dev/null && ok +pkill -f 'sleep 37' 2>/dev/null +rm -rf "$FAKE" +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/ts-react-linter-driven-development/scripts/ldd-scope.sh b/ts-react-linter-driven-development/scripts/ldd-scope.sh index 0a64212e..9c8d5832 100755 --- a/ts-react-linter-driven-development/scripts/ldd-scope.sh +++ b/ts-react-linter-driven-development/scripts/ldd-scope.sh @@ -46,6 +46,11 @@ # # The bundle: # language.txt the group's language id; "custom TAB " under --glob +# test-re.txt the --test-re pattern, when one was passed, so ldd-detect.sh +# tells test files the same way +# docs.txt on a scoped rung, the markdown files the rung touches, one +# relative path per line, deleted ones included: the docs the +# R9 gate checks in full; --all writes none # files.txt the scope, one relative path per line. A file left out of # scope/ keeps its line with the reason: (not bundled: deleted), # (not bundled: binary), (not bundled: lines) over the @@ -345,6 +350,10 @@ write_bundle() { mkdir -p "$OUT/scope" if [[ "$GROUP_ID" == "custom" ]]; then printf 'custom\t%s\n' "$OPT_GLOB" > "$OUT/language.txt" else printf '%s\n' "$GROUP_ID" > "$OUT/language.txt"; fi + [[ -n "$OPT_TEST_RE" ]] && printf '%s\n' "$OPT_TEST_RE" > "$OUT/test-re.txt" + if [[ "$RUNG" != "all" ]]; then + sed 's|^\./||' "$CANDIDATES" | grep -E '\.md$' | LC_ALL=C sort -u > "$OUT/docs.txt" + fi # not_bundled_reason — prints the reason, or nothing when it is bundled not_bundled_reason() { diff --git a/ts-react-linter-driven-development/skills/pre-commit-review/SKILL.md b/ts-react-linter-driven-development/skills/pre-commit-review/SKILL.md index d0273495..6fa70cbd 100644 --- a/ts-react-linter-driven-development/skills/pre-commit-review/SKILL.md +++ b/ts-react-linter-driven-development/skills/pre-commit-review/SKILL.md @@ -80,11 +80,15 @@ writes `hits.tsv`, `hits-all.tsv` and `counts.tsv` into the bundle, and prints t question only a hunter's reading answers — then one line per rule family. That table, about 300 tokens, is the pre-filter: the parent prints no rule file, composes no detection command, and never reads a Falsifying-questions section. On a scoped review the R9 -gate's rows are the scope's own files' — an orphan doc or an unwired root is the -repository's state, for a `--all` review or BROADER CONTEXT, never this diff's -finding. When the +gate checks only the docs the diff touches, the docs that link to a scope file and the +docs that cite a symbol the diff removes, and its rows are about those docs and the +scope's own files — an orphan doc elsewhere or an unwired root is the repository's +state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line +`R9 gate: …` under the summary says which docs the gate checked. A gate row whose +hits cell is `inconclusive` means the gate ran out of its `--gate-budget`: it is not +a `0`, and the documentation hunter answers that question from the prose. When the detection script exits 2 instead — a pattern its grep rejects, the R9 gate failing to -run — the review did not run: report its message, spawn nothing, and never render a +run, a scope in which no file matches the source glob — the review did not run: report its message, spawn nothing, and never render a rule as `skipped` or compose the pass by hand. A rule family none of whose rules has a hit — `0` on every row of theirs that is not `judgment` — is skipped: no hunter for it. @@ -114,7 +118,8 @@ zero prior use; the fix is a discussion or a separate PR, never silent inclusion The bundle exists (step 1 wrote it: `files.txt`, `diff.patch` on a scoped review, `scope/.txt` per bundled file with the file's own line numbers, `dirs.txt` on -`--all`, `comments.txt`, `hits.tsv`, `hits-all.tsv`, `counts.tsv`); nothing is +`--all`, `docs.txt` with the touched docs on a scoped review, `test-re.txt` with the +test-file pattern when one was passed, `comments.txt`, `hits.tsv`, `hits-all.tsv`, `counts.tsv`); nothing is written here. One Bash command, only when the scope is not empty, lists every rule path the prompts will carry and prints `sed -n '/^## Waiting for agents/,/^## The merged report/p' From 7b345f440bded1fb1788878b95ee2090c88f102a Mon Sep 17 00:00:00 2001 From: Dan Mordechay Date: Sun, 18 Oct 2026 20:25:51 +0300 Subject: [PATCH 2/5] ldd: a scoped R9 gate reports only what the scope can break in citing docs On the wekapp test diff the scoped gate reported 58 lines. None came from the diff: they were older drift in docs that only name a scope file. The same review now reports 0 R9 lines in 42 seconds. - A doc the diff touches gets every check. A doc that only names a scope file or cites a removed symbol reports only its links into a listed file and its backticked removed symbols. - ldd-detect.sh drops a removed word that the scope files still carry after the change. - The word set reads submodule files too (git ls-files --recurse-submodules), as grep -r did before. - A word with $ (a JavaScript identifier) also offers its $-free parts, so ts-react tokens with $ still resolve. - Tests: a budget the gate meets writes the same counts as no budget; a gate that fails under a budget fails the run. Co-Authored-By: Claude Opus 5.5 --- core/scripts/check-repo-brain.sh | 103 ++++++++++++------ core/scripts/check-repo-brain_test.sh | 14 ++- core/scripts/ldd-detect.sh | 26 +++-- core/scripts/ldd-detect_test.sh | 19 ++++ core/skills/pre-commit-review/SKILL.md | 8 +- docs/token-budget.md | 6 +- .../scripts/check-repo-brain.sh | 103 ++++++++++++------ .../scripts/check-repo-brain_test.sh | 14 ++- .../scripts/ldd-detect.sh | 26 +++-- .../scripts/ldd-detect_test.sh | 19 ++++ .../skills/pre-commit-review/SKILL.md | 8 +- .../scripts/check-repo-brain_test.sh | 14 ++- .../scripts/check-repo-brain.sh | 103 ++++++++++++------ .../scripts/check-repo-brain_test.sh | 14 ++- .../scripts/ldd-detect.sh | 26 +++-- .../scripts/ldd-detect_test.sh | 19 ++++ .../skills/pre-commit-review/SKILL.md | 8 +- .../scripts/check-repo-brain.sh | 103 ++++++++++++------ .../scripts/check-repo-brain_test.sh | 14 ++- .../scripts/ldd-detect.sh | 26 +++-- .../scripts/ldd-detect_test.sh | 19 ++++ .../skills/pre-commit-review/SKILL.md | 8 +- .../scripts/check-repo-brain.sh | 103 ++++++++++++------ .../scripts/check-repo-brain_test.sh | 14 ++- .../scripts/ldd-detect.sh | 26 +++-- .../scripts/ldd-detect_test.sh | 19 ++++ .../skills/pre-commit-review/SKILL.md | 8 +- 27 files changed, 601 insertions(+), 269 deletions(-) diff --git a/core/scripts/check-repo-brain.sh b/core/scripts/check-repo-brain.sh index 8aca0820..6ed7e56f 100755 --- a/core/scripts/check-repo-brain.sh +++ b/core/scripts/check-repo-brain.sh @@ -15,8 +15,10 @@ # (the one mechanical repair; everything else stays report-only) # --scope a review's scope: one path per line, relative to repo-root # (code files and docs, deleted ones included). The gate then checks -# only the docs in scope: the listed docs, the docs whose text names a -# listed file's basename, and the docs that cite a word of --words. +# only the docs in scope. A listed doc gets every check. A doc whose +# text names a listed file's basename, or that cites a word of +# --words, reports only its links into a listed file and its +# backticked --words. # Code edges are checked for the listed code files only, Q3 is skipped # (root wiring is the repository's state, not a diff's), and with no # doc in scope the doc checks are skipped. One stdout line says which: @@ -255,13 +257,16 @@ build_declarations() { is_repo_pkg() { grep -qxF "$1" "$PKGS" 2>/dev/null; } # repo_files — the repository's non-markdown files, NUL-separated. A git work -# tree lists them through git (tracked files plus untracked ones git does not -# ignore), which never walks build outputs or symlinked caches; elsewhere find -# skips the directories that hold dependencies and build outputs. +# tree lists them through git — the tracked files, those of its submodules +# included, then the untracked ones git does not ignore — which never walks +# build outputs or symlinked caches; elsewhere find skips the directories that +# hold dependencies and build outputs. repo_files() { + local -a spec=(-- . ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**') if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then - git ls-files -z --cached --others --exclude-standard -- . \ - ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**' 2>/dev/null && return 0 + { git ls-files -z --cached --recurse-submodules "${spec[@]}" 2>/dev/null \ + || git ls-files -z --cached "${spec[@]}" 2>/dev/null; } \ + && git ls-files -z --others --exclude-standard "${spec[@]}" 2>/dev/null && return 0 fi find . \( -name .git -o -name vendor -o -name node_modules -o -name 'bazel-*' -o -name .venv \ -o -name venv -o -name __pycache__ -o -name .tox -o -name dist -o -name target \) -prune \ @@ -270,12 +275,13 @@ repo_files() { # resolve_words — the tokens that occur as a whole word in any # non-markdown repository file, one per line. The repository is read once -# into words (runs of [A-Za-z0-9_.]); a dotted token such as `Type.Method` -# matches any run of whole dot-separated parts, as grep -w would match it. +# into words (runs of [A-Za-z0-9_.$]); a dotted token such as `Type.Method` +# matches any run of whole dot-separated parts, as grep -w would match it, and +# a word that carries $ (a JavaScript identifier) also offers its $-free parts. # One pass over the text is linear in its size, where a fixed-string grep for # hundreds of tokens is not. resolve_words() { - repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.' '\n' \ + repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.$' '\n' \ | LC_ALL=C awk -v list="$1" ' BEGIN { while ((getline t < list) > 0) if (t != "") want[t] = 1 } seen[$0]++ { next } @@ -289,6 +295,10 @@ resolve_words() { if (s in want) { print s; delete want[s] } } } + if (index($0, "$") > 0) { + n = split($0, p, /[$.]/) + for (i = 1; i <= n; i++) if (p[i] in want) { print p[i]; delete want[p[i]] } + } }' return 0 } @@ -297,7 +307,7 @@ resolve_words() { # P — a file:line citation outside fences/URLs/globs # S — a backticked symbol-shaped token to resolve # Tokens are then resolved as SETS (one grep against the declaration file, one -# repo-wide word grep for the whole unresolved batch) — never per token. +# pass over the repository for the whole unresolved batch) — never per token. # Language-specific shapes arrive as -v variables: file_ext, file_re, sym_re, qual_re. DOCSCAN_AWK=' FNR == 1 { fence = 0 } @@ -331,50 +341,62 @@ FNR == 1 { fence = 0 } } ' -# ---------- the scope: which docs a scoped run checks ---------- -# A doc is in scope when the scope lists it, when its text names the basename -# of a listed file as a whole word (a link to a touched doc or code file), or -# when it cites in backticks a word the change removes (a symbol the diff -# renames or deletes). +# ---------- the scope: which docs a scoped run checks, and how far ---------- +# A doc the scope lists (a doc the diff touches) gets every check. A doc that +# only cites the scope — its text names the basename of a listed file as a +# whole word, or it cites in backticks a word the change removes (a symbol the +# diff renames or deletes) — reports only what the scope can break in it: a +# doc link into a listed file, and the backticked removed words. Its other +# findings are the repository's state, for a whole-repository run. # Every list below holds relative paths as find(1) prints them under a root. -SCOPE_DOCS="$WORK/scope-docs" +SCOPE_DOCS="$WORK/scope-docs" # every doc a scoped run reads +FULL_DOCS="$WORK/full-docs" # the listed docs: every check SCOPE_PATHS="$WORK/scope-paths" +BASENAMES="$WORK/basenames" if (( SCOPED )); then sed -e 's|^\./||' -e '/^$/d' "$SCOPE_LIST" | LC_ALL=C sort -u > "$SCOPE_PATHS" + sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$BASENAMES" for r in "${ROOTS[@]}"; do find "$r" -type f -name '*.md'; done | LC_ALL=C sort -u > "$WORK/all-docs" + grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" > "$FULL_DOCS" { - grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" - sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$WORK/basenames" - if [[ -s "$WORK/basenames" && -s "$WORK/all-docs" ]]; then - tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$WORK/basenames" 2>/dev/null + cat "$FULL_DOCS" + if [[ -s "$BASENAMES" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$BASENAMES" 2>/dev/null fi if [[ -n "$WORDS_LIST" && -s "$WORDS_LIST" && -s "$WORK/all-docs" ]]; then tr '\n' '\0' < "$WORK/all-docs" \ | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" "$DOCSCAN_AWK" \ - | awk -F'\t' 'NR == FNR { w[$0] = 1; next } - $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in w) || (m in w)) print $2 }' "$WORDS_LIST" - + | awk -F'\t' -v list="$WORDS_LIST" ' + BEGIN { while ((getline w < list) > 0) want[w] = 1 } + $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in want) || (m in want)) print $2 }' fi } | LC_ALL=C sort -u > "$SCOPE_DOCS" if [[ -s "$SCOPE_DOCS" ]]; then - echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" + echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" \ + "($(wc -l < "$FULL_DOCS" | tr -d ' ') touched, every check; the rest cite the scope, scope lines only)" else echo "check-repo-brain: scope — no doc in scope; doc checks skipped" fi fi -# bundle_docs [name] — the docs a run checks under one root: every -# *.md (or every file of that name), limited to the scope on a scoped run +# bundle_docs [name] [full] — the docs a run checks under one root: +# every *.md (or every file of that name); on a scoped run only the docs in +# scope, or with "full" only the docs that get every check bundle_docs() { - local name="${2:-*.md}" + local name="${2:-*.md}" list="$SCOPE_DOCS" + [[ "${3:-}" == full ]] && list="$FULL_DOCS" if (( SCOPED )); then - find "$1" -type f -name "$name" | grep -xF -f "$SCOPE_DOCS" + find "$1" -type f -name "$name" | grep -xF -f "$list" else find "$1" -type f -name "$name" fi return 0 } +# full_doc — exit 0 iff the doc gets every check +full_doc() { (( ! SCOPED )) || grep -qxF -- "$1" "$FULL_DOCS"; } + # ---------- Q2: code→docs edges (repo-wide, or the scope's code files on a # scoped run; resolved from repo root, then the citing file's own # sub-project) ---------- @@ -437,7 +459,7 @@ check_bundle() { # [[ "$(basename "$resolved")" == "index.md" ]] && queue+=("$resolved") done < <(grep -oE '\]\([^)]+\)' "$idx" 2>/dev/null) done - if [[ ! -f "$root_index" ]]; then + if [[ ! -f "$root_index" ]] && [[ -n "$(bundle_docs "$docroot" '*.md' full)" ]]; then fail "[Q1] $docroot — no index.md: the bundle has no map" fi while IFS= read -r doc; do @@ -447,14 +469,17 @@ check_bundle() { # case "$reachable" in *"$c"$'\n'*) ;; *) fail "[Q1] $doc — orphan: not reachable from $root_index" ;; esac - done < <(bundle_docs "$docroot") + done < <(bundle_docs "$docroot" '*.md' full) # --- Q2: every doc link resolves (a missing parent directory is just as - # broken as a missing file — only URLs/anchors are exempt) --- + # broken as a missing file — only URLs/anchors are exempt); a doc that only + # cites the scope reports only its links into a listed file --- while IFS= read -r md; do + local full=0; full_doc "$md" && full=1 while IFS= read -r raw; do local t="${raw#](}"; t="${t%)}" [[ "$t" == *.md* ]] || continue + (( full )) || grep -qxF -- "$(basename "${t%%#*}")" "$BASENAMES" || continue local resolved rc=0 resolved=$(resolve_link "$md" "$t" "$docroot") || rc=$? (( rc == 1 )) && continue @@ -465,7 +490,8 @@ check_bundle() { # done < <(bundle_docs "$docroot") # --- Q2: doc scan — file:line ban + docs→code symbol resolution. - # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). --- + # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). A + # doc that only cites the scope keeps only its backticked removed words. --- local scan; scan=$(mktemp) if ! ( set -o pipefail bundle_docs "$docroot" | tr '\n' '\0' \ @@ -476,6 +502,13 @@ check_bundle() { # rm -f "$scan" exit 2 fi + if (( SCOPED )); then + awk -F'\t' -v full="$FULL_DOCS" -v words="${WORDS_LIST:-/dev/null}" ' + BEGIN { while ((getline l < full) > 0) f[l] = 1; while ((getline l < words) > 0) w[l] = 1 } + ($2 in f) { print; next } + $1 == "S" { m = $4; sub(/^.*\./, "", m); if (($4 in w) || (m in w)) print }' "$scan" > "$scan.kept" + mv -f "$scan.kept" "$scan" + fi local f ln while IFS=$'\t' read -r _ f ln; do fail "[Q2] $f:$ln — cites a file path or line number (churn-prone coordinate)" @@ -608,7 +641,7 @@ check_bundle() { # else fail "[Q7] $md — frontmatter missing 'description:'" fi - done < <(bundle_docs "$docroot") + done < <(bundle_docs "$docroot" '*.md' full) # --- Q7: drift check — index line text == target's description (⚠️ exempt; # --fix rewrites drifted lines after the read loop, never during it) --- @@ -646,12 +679,12 @@ check_bundle() { # echo " fixed: $idx:$n — index line rewritten from its target's description" done <<< "$fixes" fi - done < <(bundle_docs "$docroot" index.md) + done < <(bundle_docs "$docroot" index.md full) # --- Q7: no log.md --- while IFS= read -r lg; do fail "[Q7] $lg — log.md is reserved for change history; docs describe current behavior" - done < <(bundle_docs "$docroot" log.md) + done < <(bundle_docs "$docroot" log.md full) } for i in "${!PROJS[@]}"; do diff --git a/core/scripts/check-repo-brain_test.sh b/core/scripts/check-repo-brain_test.sh index 14a9b66b..485e8057 100755 --- a/core/scripts/check-repo-brain_test.sh +++ b/core/scripts/check-repo-brain_test.sh @@ -563,22 +563,24 @@ expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] do && expect_not "docs/other.md" && expect_not "[Q3]" && ok finish -begin "--scope: a doc that names a scope file is checked" -mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../$FX_CODE_FILE)." -printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +begin "--scope: a doc that only names a scope file reports only its links into the scope" +mk_conformant +mk_doc linker "links the scope" "See [the old doc](gone.md) and [a missing one](missing.md); code: \`$FX_CODE_FILE\` at $FX_CODE_FILE:12." +printf '%s\n' "$FX_CODE_FILE" docs/gone.md > "$LISTS/scope.txt" run_gate --scope "$LISTS/scope.txt" -expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +expect_exit 1 && expect_has "0 touched" && expect_has "[Q2] docs/linker.md — link target does not exist: gone.md" \ + && expect_not "missing.md" && expect_not "orphan" && expect_not "file path or line number" && ok finish begin "--scope --words: a doc that cites a removed symbol is checked" mk_conformant -mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\` and \`OtherGhost\`." append_index "- [ghost.md](ghost.md) — a ghost symbol" printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" run_gate --scope "$LISTS/scope.txt" expect_exit 0 && expect_has "no doc in scope" \ && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ - && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok + && expect_has "backticked \`SnapshotRunner\` does not resolve" && expect_not "OtherGhost" && ok finish begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" diff --git a/core/scripts/ldd-detect.sh b/core/scripts/ldd-detect.sh index 41f089cf..b351fd81 100755 --- a/core/scripts/ldd-detect.sh +++ b/core/scripts/ldd-detect.sh @@ -48,11 +48,11 @@ # Detect-gate: Q the R9 gate's [Q] report lines, from one run of # check-repo-brain.sh beside this script over --root. On a # scoped bundle (no dirs.txt) the gate runs with --scope: it -# checks only the bundle's docs (docs.txt), the docs that -# link to a scope file and the docs that cite a symbol the -# diff removes, the code edges of the scope's code files, -# and no root wiring; with no such doc its doc checks are -# skipped. The repository-wide findings belong to a +# checks the bundle's docs (docs.txt) in full, and in the +# docs that name a scope file or cite a symbol the diff +# removes only those links and symbols; it checks the code +# edges of the scope's code files and no root wiring; with +# no such doc its doc checks are skipped. The repository-wide findings belong to a # whole-repository review # Detect: judgment no mechanical lead; the hunter runs the prose # A question with no detect line, two of them, or a malformed one is an error: @@ -402,7 +402,8 @@ run_path() { # # of docs.txt; and the identifiers the diff removes — on its removed lines and # on none of its added ones — the symbols a renamed or deleted declaration # takes away from the docs that cite them. Comment lines (a line that opens -# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped. +# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped, +# and a word the scope's files still carry after the change is not removed. gate_scope_lists() { local scope="$1" words="$2" line f { @@ -424,7 +425,18 @@ gate_scope_lists() { if (s == "+") add[w] = 1; else del[w] = 1 } } - END { for (w in del) if (!(w in add)) print w }' | LC_ALL=C sort > "$words" + END { for (w in del) if (!(w in add)) print w }' \ + | LC_ALL=C sort > "$words.removed" + if (( ${#ALL_FILES[@]} > 0 )); then + cat -- "${ALL_FILES[@]}" | LC_ALL=C tr -cs 'A-Za-z0-9_' '\n' \ + | LC_ALL=C awk -v list="$words.removed" ' + BEGIN { while ((getline w < list) > 0) gone[w] = 1 } + ($0 in gone) { delete gone[$0] } + END { for (w in gone) print w }' | LC_ALL=C sort > "$words" + else + cp "$words.removed" "$words" + fi + rm -f "$words.removed" } # The R9 gate runs once, lazily; its [Q] lines are split by question. With diff --git a/core/scripts/ldd-detect_test.sh b/core/scripts/ldd-detect_test.sh index dad58dc6..e81b79db 100644 --- a/core/scripts/ldd-detect_test.sh +++ b/core/scripts/ldd-detect_test.sh @@ -288,6 +288,25 @@ pkill -f 'sleep 37' 2>/dev/null rm -rf "$FAKE" finish +begin "a --gate-budget the gate finishes within writes the same counts as no budget" +mk_repo; : > "$BUNDLE/dirs.txt"; run_detect +cp "$BUNDLE/counts.tsv" "$BUNDLE/first.tsv" +run_detect --gate-budget 60 +expect_exit 0 && { cmp -s "$BUNDLE/first.tsv" "$BUNDLE/counts.tsv" || bad "counts.tsv differs under a budget the gate meets"; } \ + && cmp -s "$BUNDLE/first.tsv" "$BUNDLE/counts.tsv" && ok +finish + +begin "a gate that fails under --gate-budget fails the run (exit 2)" +mk_repo; : > "$BUNDLE/dirs.txt" +FAKE=$(mktemp -d) +cp "$DETECT" "$FAKE/ldd-detect.sh" +printf '#!/usr/bin/env bash\necho "check-repo-brain: internal scanner error"\nexit 2\n' > "$FAKE/check-repo-brain.sh" +OUT=$(cd "$REPO" && bash "$FAKE/ldd-detect.sh" --rules "$RULES" --gate-budget 30 --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? +ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +expect_exit 2 && expect_has "the R9 gate failed to run" && ok +rm -rf "$FAKE" +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/core/skills/pre-commit-review/SKILL.md b/core/skills/pre-commit-review/SKILL.md index b06d7571..abd49c20 100644 --- a/core/skills/pre-commit-review/SKILL.md +++ b/core/skills/pre-commit-review/SKILL.md @@ -80,10 +80,10 @@ writes `hits.tsv`, `hits-all.tsv` and `counts.tsv` into the bundle, and prints t question only a hunter's reading answers — then one line per rule family. That table, about 300 tokens, is the pre-filter: the parent prints no rule file, composes no detection command, and never reads a Falsifying-questions section. On a scoped review the R9 -gate checks only the docs the diff touches, the docs that link to a scope file and the -docs that cite a symbol the diff removes, and its rows are about those docs and the -scope's own files — an orphan doc elsewhere or an unwired root is the repository's -state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line +gate checks the docs the diff touches in full; in a doc that links to a scope file +or cites a symbol the diff removes it checks only those links and symbols; and it +checks the code edges of the scope's own files — an orphan doc elsewhere, a citing +doc's older drift or an unwired root is the repository's state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line `R9 gate: …` under the summary says which docs the gate checked. A gate row whose hits cell is `inconclusive` means the gate ran out of its `--gate-budget`: it is not a `0`, and the documentation hunter answers that question from the prose. When the diff --git a/docs/token-budget.md b/docs/token-budget.md index abc68644..2af6add1 100644 --- a/docs/token-budget.md +++ b/docs/token-budget.md @@ -424,9 +424,9 @@ argument) is still read wherever its family is. The script writes the hits table twice, capped per question for the hunter's first turn and uncapped as `hits-all.tsv`, so a capped question's remaining hits are one more `awk` and no hunter runs a pattern of its own. On a scoped review the detection script runs the R9 -gate over the scope alone — the docs the diff touches, the docs that link to a scope -file, the docs that cite a symbol the diff removes, and the code edges of the scope's -own files — so repository-wide doc drift does not spawn the documentation hunter on +gate over the scope alone — the docs the diff touches in full, only the scope's links +and removed symbols in the docs that cite the scope, and the code edges of the +scope's own files — so repository-wide doc drift does not spawn the documentation hunter on every file review, and a code-only diff that no doc cites skips the doc checks. Each line of `comments.txt` carries the code line below the comment — the declaration it documents — so the critic settles most comments from the inventory and opens a scope file only for the ones it does not. The `Hunters:` header keeps its shape and its numbers now come diff --git a/go-linter-driven-development/scripts/check-repo-brain.sh b/go-linter-driven-development/scripts/check-repo-brain.sh index 08e01752..943d9d1e 100755 --- a/go-linter-driven-development/scripts/check-repo-brain.sh +++ b/go-linter-driven-development/scripts/check-repo-brain.sh @@ -15,8 +15,10 @@ # (the one mechanical repair; everything else stays report-only) # --scope a review's scope: one path per line, relative to repo-root # (code files and docs, deleted ones included). The gate then checks -# only the docs in scope: the listed docs, the docs whose text names a -# listed file's basename, and the docs that cite a word of --words. +# only the docs in scope. A listed doc gets every check. A doc whose +# text names a listed file's basename, or that cites a word of +# --words, reports only its links into a listed file and its +# backticked --words. # Code edges are checked for the listed code files only, Q3 is skipped # (root wiring is the repository's state, not a diff's), and with no # doc in scope the doc checks are skipped. One stdout line says which: @@ -366,13 +368,16 @@ build_declarations() { is_repo_pkg() { grep -qxF "$1" "$PKGS" 2>/dev/null; } # repo_files — the repository's non-markdown files, NUL-separated. A git work -# tree lists them through git (tracked files plus untracked ones git does not -# ignore), which never walks build outputs or symlinked caches; elsewhere find -# skips the directories that hold dependencies and build outputs. +# tree lists them through git — the tracked files, those of its submodules +# included, then the untracked ones git does not ignore — which never walks +# build outputs or symlinked caches; elsewhere find skips the directories that +# hold dependencies and build outputs. repo_files() { + local -a spec=(-- . ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**') if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then - git ls-files -z --cached --others --exclude-standard -- . \ - ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**' 2>/dev/null && return 0 + { git ls-files -z --cached --recurse-submodules "${spec[@]}" 2>/dev/null \ + || git ls-files -z --cached "${spec[@]}" 2>/dev/null; } \ + && git ls-files -z --others --exclude-standard "${spec[@]}" 2>/dev/null && return 0 fi find . \( -name .git -o -name vendor -o -name node_modules -o -name 'bazel-*' -o -name .venv \ -o -name venv -o -name __pycache__ -o -name .tox -o -name dist -o -name target \) -prune \ @@ -381,12 +386,13 @@ repo_files() { # resolve_words — the tokens that occur as a whole word in any # non-markdown repository file, one per line. The repository is read once -# into words (runs of [A-Za-z0-9_.]); a dotted token such as `Type.Method` -# matches any run of whole dot-separated parts, as grep -w would match it. +# into words (runs of [A-Za-z0-9_.$]); a dotted token such as `Type.Method` +# matches any run of whole dot-separated parts, as grep -w would match it, and +# a word that carries $ (a JavaScript identifier) also offers its $-free parts. # One pass over the text is linear in its size, where a fixed-string grep for # hundreds of tokens is not. resolve_words() { - repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.' '\n' \ + repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.$' '\n' \ | LC_ALL=C awk -v list="$1" ' BEGIN { while ((getline t < list) > 0) if (t != "") want[t] = 1 } seen[$0]++ { next } @@ -400,6 +406,10 @@ resolve_words() { if (s in want) { print s; delete want[s] } } } + if (index($0, "$") > 0) { + n = split($0, p, /[$.]/) + for (i = 1; i <= n; i++) if (p[i] in want) { print p[i]; delete want[p[i]] } + } }' return 0 } @@ -408,7 +418,7 @@ resolve_words() { # P — a file:line citation outside fences/URLs/globs # S — a backticked symbol-shaped token to resolve # Tokens are then resolved as SETS (one grep against the declaration file, one -# repo-wide word grep for the whole unresolved batch) — never per token. +# pass over the repository for the whole unresolved batch) — never per token. # Language-specific shapes arrive as -v variables: file_ext, file_re, sym_re, qual_re. DOCSCAN_AWK=' FNR == 1 { fence = 0 } @@ -442,50 +452,62 @@ FNR == 1 { fence = 0 } } ' -# ---------- the scope: which docs a scoped run checks ---------- -# A doc is in scope when the scope lists it, when its text names the basename -# of a listed file as a whole word (a link to a touched doc or code file), or -# when it cites in backticks a word the change removes (a symbol the diff -# renames or deletes). +# ---------- the scope: which docs a scoped run checks, and how far ---------- +# A doc the scope lists (a doc the diff touches) gets every check. A doc that +# only cites the scope — its text names the basename of a listed file as a +# whole word, or it cites in backticks a word the change removes (a symbol the +# diff renames or deletes) — reports only what the scope can break in it: a +# doc link into a listed file, and the backticked removed words. Its other +# findings are the repository's state, for a whole-repository run. # Every list below holds relative paths as find(1) prints them under a root. -SCOPE_DOCS="$WORK/scope-docs" +SCOPE_DOCS="$WORK/scope-docs" # every doc a scoped run reads +FULL_DOCS="$WORK/full-docs" # the listed docs: every check SCOPE_PATHS="$WORK/scope-paths" +BASENAMES="$WORK/basenames" if (( SCOPED )); then sed -e 's|^\./||' -e '/^$/d' "$SCOPE_LIST" | LC_ALL=C sort -u > "$SCOPE_PATHS" + sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$BASENAMES" for r in "${ROOTS[@]}"; do find "$r" -type f -name '*.md'; done | LC_ALL=C sort -u > "$WORK/all-docs" + grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" > "$FULL_DOCS" { - grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" - sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$WORK/basenames" - if [[ -s "$WORK/basenames" && -s "$WORK/all-docs" ]]; then - tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$WORK/basenames" 2>/dev/null + cat "$FULL_DOCS" + if [[ -s "$BASENAMES" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$BASENAMES" 2>/dev/null fi if [[ -n "$WORDS_LIST" && -s "$WORDS_LIST" && -s "$WORK/all-docs" ]]; then tr '\n' '\0' < "$WORK/all-docs" \ | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" "$DOCSCAN_AWK" \ - | awk -F'\t' 'NR == FNR { w[$0] = 1; next } - $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in w) || (m in w)) print $2 }' "$WORDS_LIST" - + | awk -F'\t' -v list="$WORDS_LIST" ' + BEGIN { while ((getline w < list) > 0) want[w] = 1 } + $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in want) || (m in want)) print $2 }' fi } | LC_ALL=C sort -u > "$SCOPE_DOCS" if [[ -s "$SCOPE_DOCS" ]]; then - echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" + echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" \ + "($(wc -l < "$FULL_DOCS" | tr -d ' ') touched, every check; the rest cite the scope, scope lines only)" else echo "check-repo-brain: scope — no doc in scope; doc checks skipped" fi fi -# bundle_docs [name] — the docs a run checks under one root: every -# *.md (or every file of that name), limited to the scope on a scoped run +# bundle_docs [name] [full] — the docs a run checks under one root: +# every *.md (or every file of that name); on a scoped run only the docs in +# scope, or with "full" only the docs that get every check bundle_docs() { - local name="${2:-*.md}" + local name="${2:-*.md}" list="$SCOPE_DOCS" + [[ "${3:-}" == full ]] && list="$FULL_DOCS" if (( SCOPED )); then - find "$1" -type f -name "$name" | grep -xF -f "$SCOPE_DOCS" + find "$1" -type f -name "$name" | grep -xF -f "$list" else find "$1" -type f -name "$name" fi return 0 } +# full_doc — exit 0 iff the doc gets every check +full_doc() { (( ! SCOPED )) || grep -qxF -- "$1" "$FULL_DOCS"; } + # ---------- Q2: code→docs edges (repo-wide, or the scope's code files on a # scoped run; resolved from repo root, then the citing file's own # sub-project) ---------- @@ -548,7 +570,7 @@ check_bundle() { # [[ "$(basename "$resolved")" == "index.md" ]] && queue+=("$resolved") done < <(grep -oE '\]\([^)]+\)' "$idx" 2>/dev/null) done - if [[ ! -f "$root_index" ]]; then + if [[ ! -f "$root_index" ]] && [[ -n "$(bundle_docs "$docroot" '*.md' full)" ]]; then fail "[Q1] $docroot — no index.md: the bundle has no map" fi while IFS= read -r doc; do @@ -558,14 +580,17 @@ check_bundle() { # case "$reachable" in *"$c"$'\n'*) ;; *) fail "[Q1] $doc — orphan: not reachable from $root_index" ;; esac - done < <(bundle_docs "$docroot") + done < <(bundle_docs "$docroot" '*.md' full) # --- Q2: every doc link resolves (a missing parent directory is just as - # broken as a missing file — only URLs/anchors are exempt) --- + # broken as a missing file — only URLs/anchors are exempt); a doc that only + # cites the scope reports only its links into a listed file --- while IFS= read -r md; do + local full=0; full_doc "$md" && full=1 while IFS= read -r raw; do local t="${raw#](}"; t="${t%)}" [[ "$t" == *.md* ]] || continue + (( full )) || grep -qxF -- "$(basename "${t%%#*}")" "$BASENAMES" || continue local resolved rc=0 resolved=$(resolve_link "$md" "$t" "$docroot") || rc=$? (( rc == 1 )) && continue @@ -576,7 +601,8 @@ check_bundle() { # done < <(bundle_docs "$docroot") # --- Q2: doc scan — file:line ban + docs→code symbol resolution. - # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). --- + # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). A + # doc that only cites the scope keeps only its backticked removed words. --- local scan; scan=$(mktemp) if ! ( set -o pipefail bundle_docs "$docroot" | tr '\n' '\0' \ @@ -587,6 +613,13 @@ check_bundle() { # rm -f "$scan" exit 2 fi + if (( SCOPED )); then + awk -F'\t' -v full="$FULL_DOCS" -v words="${WORDS_LIST:-/dev/null}" ' + BEGIN { while ((getline l < full) > 0) f[l] = 1; while ((getline l < words) > 0) w[l] = 1 } + ($2 in f) { print; next } + $1 == "S" { m = $4; sub(/^.*\./, "", m); if (($4 in w) || (m in w)) print }' "$scan" > "$scan.kept" + mv -f "$scan.kept" "$scan" + fi local f ln while IFS=$'\t' read -r _ f ln; do fail "[Q2] $f:$ln — cites a file path or line number (churn-prone coordinate)" @@ -719,7 +752,7 @@ check_bundle() { # else fail "[Q7] $md — frontmatter missing 'description:'" fi - done < <(bundle_docs "$docroot") + done < <(bundle_docs "$docroot" '*.md' full) # --- Q7: drift check — index line text == target's description (⚠️ exempt; # --fix rewrites drifted lines after the read loop, never during it) --- @@ -757,12 +790,12 @@ check_bundle() { # echo " fixed: $idx:$n — index line rewritten from its target's description" done <<< "$fixes" fi - done < <(bundle_docs "$docroot" index.md) + done < <(bundle_docs "$docroot" index.md full) # --- Q7: no log.md --- while IFS= read -r lg; do fail "[Q7] $lg — log.md is reserved for change history; docs describe current behavior" - done < <(bundle_docs "$docroot" log.md) + done < <(bundle_docs "$docroot" log.md full) } for i in "${!PROJS[@]}"; do diff --git a/go-linter-driven-development/scripts/check-repo-brain_test.sh b/go-linter-driven-development/scripts/check-repo-brain_test.sh index 93297cf5..e5fc9229 100644 --- a/go-linter-driven-development/scripts/check-repo-brain_test.sh +++ b/go-linter-driven-development/scripts/check-repo-brain_test.sh @@ -609,22 +609,24 @@ expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] do && expect_not "docs/other.md" && expect_not "[Q3]" && ok finish -begin "--scope: a doc that names a scope file is checked" -mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../retry/policy.go)." -printf '%s\n' "retry/policy.go" > "$LISTS/scope.txt" +begin "--scope: a doc that only names a scope file reports only its links into the scope" +mk_conformant +mk_doc linker "links the scope" "See [the old doc](gone.md) and [a missing one](missing.md); code: \`retry/policy.go\` at retry/policy.go:12." +printf '%s\n' "retry/policy.go" docs/gone.md > "$LISTS/scope.txt" run_gate --scope "$LISTS/scope.txt" -expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +expect_exit 1 && expect_has "0 touched" && expect_has "[Q2] docs/linker.md — link target does not exist: gone.md" \ + && expect_not "missing.md" && expect_not "orphan" && expect_not "file path or line number" && ok finish begin "--scope --words: a doc that cites a removed symbol is checked" mk_conformant -mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\` and \`OtherGhost\`." append_index "- [ghost.md](ghost.md) — a ghost symbol" printf '%s\n' "retry/policy.go" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" run_gate --scope "$LISTS/scope.txt" expect_exit 0 && expect_has "no doc in scope" \ && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ - && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok + && expect_has "backticked \`SnapshotRunner\` does not resolve" && expect_not "OtherGhost" && ok finish begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" diff --git a/go-linter-driven-development/scripts/ldd-detect.sh b/go-linter-driven-development/scripts/ldd-detect.sh index 702f67b4..23f2112e 100755 --- a/go-linter-driven-development/scripts/ldd-detect.sh +++ b/go-linter-driven-development/scripts/ldd-detect.sh @@ -48,11 +48,11 @@ # Detect-gate: Q the R9 gate's [Q] report lines, from one run of # check-repo-brain.sh beside this script over --root. On a # scoped bundle (no dirs.txt) the gate runs with --scope: it -# checks only the bundle's docs (docs.txt), the docs that -# link to a scope file and the docs that cite a symbol the -# diff removes, the code edges of the scope's code files, -# and no root wiring; with no such doc its doc checks are -# skipped. The repository-wide findings belong to a +# checks the bundle's docs (docs.txt) in full, and in the +# docs that name a scope file or cite a symbol the diff +# removes only those links and symbols; it checks the code +# edges of the scope's code files and no root wiring; with +# no such doc its doc checks are skipped. The repository-wide findings belong to a # whole-repository review # Detect: judgment no mechanical lead; the hunter runs the prose # A question with no detect line, two of them, or a malformed one is an error: @@ -508,7 +508,8 @@ run_path() { # # of docs.txt; and the identifiers the diff removes — on its removed lines and # on none of its added ones — the symbols a renamed or deleted declaration # takes away from the docs that cite them. Comment lines (a line that opens -# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped. +# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped, +# and a word the scope's files still carry after the change is not removed. gate_scope_lists() { local scope="$1" words="$2" line f { @@ -530,7 +531,18 @@ gate_scope_lists() { if (s == "+") add[w] = 1; else del[w] = 1 } } - END { for (w in del) if (!(w in add)) print w }' | LC_ALL=C sort > "$words" + END { for (w in del) if (!(w in add)) print w }' \ + | LC_ALL=C sort > "$words.removed" + if (( ${#ALL_FILES[@]} > 0 )); then + cat -- "${ALL_FILES[@]}" | LC_ALL=C tr -cs 'A-Za-z0-9_' '\n' \ + | LC_ALL=C awk -v list="$words.removed" ' + BEGIN { while ((getline w < list) > 0) gone[w] = 1 } + ($0 in gone) { delete gone[$0] } + END { for (w in gone) print w }' | LC_ALL=C sort > "$words" + else + cp "$words.removed" "$words" + fi + rm -f "$words.removed" } # The R9 gate runs once, lazily; its [Q] lines are split by question. With diff --git a/go-linter-driven-development/scripts/ldd-detect_test.sh b/go-linter-driven-development/scripts/ldd-detect_test.sh index 5c201080..9449be3e 100644 --- a/go-linter-driven-development/scripts/ldd-detect_test.sh +++ b/go-linter-driven-development/scripts/ldd-detect_test.sh @@ -551,6 +551,25 @@ pkill -f 'sleep 37' 2>/dev/null rm -rf "$FAKE" finish +begin "a --gate-budget the gate finishes within writes the same counts as no budget" +mk_repo; : > "$BUNDLE/dirs.txt"; run_detect +cp "$BUNDLE/counts.tsv" "$BUNDLE/first.tsv" +run_detect --gate-budget 60 +expect_exit 0 && { cmp -s "$BUNDLE/first.tsv" "$BUNDLE/counts.tsv" || bad "counts.tsv differs under a budget the gate meets"; } \ + && cmp -s "$BUNDLE/first.tsv" "$BUNDLE/counts.tsv" && ok +finish + +begin "a gate that fails under --gate-budget fails the run (exit 2)" +mk_repo; : > "$BUNDLE/dirs.txt" +FAKE=$(mktemp -d) +cp "$DETECT" "$FAKE/ldd-detect.sh" +printf '#!/usr/bin/env bash\necho "check-repo-brain: internal scanner error"\nexit 2\n' > "$FAKE/check-repo-brain.sh" +OUT=$(cd "$REPO" && bash "$FAKE/ldd-detect.sh" --rules "$RULES" --gate-budget 30 --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? +ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +expect_exit 2 && expect_has "the R9 gate failed to run" && ok +rm -rf "$FAKE" +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/go-linter-driven-development/skills/pre-commit-review/SKILL.md b/go-linter-driven-development/skills/pre-commit-review/SKILL.md index 93bcf50d..61faad3b 100644 --- a/go-linter-driven-development/skills/pre-commit-review/SKILL.md +++ b/go-linter-driven-development/skills/pre-commit-review/SKILL.md @@ -80,10 +80,10 @@ writes `hits.tsv`, `hits-all.tsv` and `counts.tsv` into the bundle, and prints t question only a hunter's reading answers — then one line per rule family. That table, about 300 tokens, is the pre-filter: the parent prints no rule file, composes no detection command, and never reads a Falsifying-questions section. On a scoped review the R9 -gate checks only the docs the diff touches, the docs that link to a scope file and the -docs that cite a symbol the diff removes, and its rows are about those docs and the -scope's own files — an orphan doc elsewhere or an unwired root is the repository's -state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line +gate checks the docs the diff touches in full; in a doc that links to a scope file +or cites a symbol the diff removes it checks only those links and symbols; and it +checks the code edges of the scope's own files — an orphan doc elsewhere, a citing +doc's older drift or an unwired root is the repository's state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line `R9 gate: …` under the summary says which docs the gate checked. A gate row whose hits cell is `inconclusive` means the gate ran out of its `--gate-budget`: it is not a `0`, and the documentation hunter answers that question from the prose. When the diff --git a/lang/go/overrides/scripts/check-repo-brain_test.sh b/lang/go/overrides/scripts/check-repo-brain_test.sh index 93297cf5..e5fc9229 100644 --- a/lang/go/overrides/scripts/check-repo-brain_test.sh +++ b/lang/go/overrides/scripts/check-repo-brain_test.sh @@ -609,22 +609,24 @@ expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] do && expect_not "docs/other.md" && expect_not "[Q3]" && ok finish -begin "--scope: a doc that names a scope file is checked" -mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../retry/policy.go)." -printf '%s\n' "retry/policy.go" > "$LISTS/scope.txt" +begin "--scope: a doc that only names a scope file reports only its links into the scope" +mk_conformant +mk_doc linker "links the scope" "See [the old doc](gone.md) and [a missing one](missing.md); code: \`retry/policy.go\` at retry/policy.go:12." +printf '%s\n' "retry/policy.go" docs/gone.md > "$LISTS/scope.txt" run_gate --scope "$LISTS/scope.txt" -expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +expect_exit 1 && expect_has "0 touched" && expect_has "[Q2] docs/linker.md — link target does not exist: gone.md" \ + && expect_not "missing.md" && expect_not "orphan" && expect_not "file path or line number" && ok finish begin "--scope --words: a doc that cites a removed symbol is checked" mk_conformant -mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\` and \`OtherGhost\`." append_index "- [ghost.md](ghost.md) — a ghost symbol" printf '%s\n' "retry/policy.go" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" run_gate --scope "$LISTS/scope.txt" expect_exit 0 && expect_has "no doc in scope" \ && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ - && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok + && expect_has "backticked \`SnapshotRunner\` does not resolve" && expect_not "OtherGhost" && ok finish begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" diff --git a/linter-driven-development/scripts/check-repo-brain.sh b/linter-driven-development/scripts/check-repo-brain.sh index 2d470690..6e98a847 100755 --- a/linter-driven-development/scripts/check-repo-brain.sh +++ b/linter-driven-development/scripts/check-repo-brain.sh @@ -15,8 +15,10 @@ # (the one mechanical repair; everything else stays report-only) # --scope a review's scope: one path per line, relative to repo-root # (code files and docs, deleted ones included). The gate then checks -# only the docs in scope: the listed docs, the docs whose text names a -# listed file's basename, and the docs that cite a word of --words. +# only the docs in scope. A listed doc gets every check. A doc whose +# text names a listed file's basename, or that cites a word of +# --words, reports only its links into a listed file and its +# backticked --words. # Code edges are checked for the listed code files only, Q3 is skipped # (root wiring is the repository's state, not a diff's), and with no # doc in scope the doc checks are skipped. One stdout line says which: @@ -498,13 +500,16 @@ build_declarations() { is_repo_pkg() { grep -qxF "$1" "$PKGS" 2>/dev/null; } # repo_files — the repository's non-markdown files, NUL-separated. A git work -# tree lists them through git (tracked files plus untracked ones git does not -# ignore), which never walks build outputs or symlinked caches; elsewhere find -# skips the directories that hold dependencies and build outputs. +# tree lists them through git — the tracked files, those of its submodules +# included, then the untracked ones git does not ignore — which never walks +# build outputs or symlinked caches; elsewhere find skips the directories that +# hold dependencies and build outputs. repo_files() { + local -a spec=(-- . ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**') if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then - git ls-files -z --cached --others --exclude-standard -- . \ - ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**' 2>/dev/null && return 0 + { git ls-files -z --cached --recurse-submodules "${spec[@]}" 2>/dev/null \ + || git ls-files -z --cached "${spec[@]}" 2>/dev/null; } \ + && git ls-files -z --others --exclude-standard "${spec[@]}" 2>/dev/null && return 0 fi find . \( -name .git -o -name vendor -o -name node_modules -o -name 'bazel-*' -o -name .venv \ -o -name venv -o -name __pycache__ -o -name .tox -o -name dist -o -name target \) -prune \ @@ -513,12 +518,13 @@ repo_files() { # resolve_words — the tokens that occur as a whole word in any # non-markdown repository file, one per line. The repository is read once -# into words (runs of [A-Za-z0-9_.]); a dotted token such as `Type.Method` -# matches any run of whole dot-separated parts, as grep -w would match it. +# into words (runs of [A-Za-z0-9_.$]); a dotted token such as `Type.Method` +# matches any run of whole dot-separated parts, as grep -w would match it, and +# a word that carries $ (a JavaScript identifier) also offers its $-free parts. # One pass over the text is linear in its size, where a fixed-string grep for # hundreds of tokens is not. resolve_words() { - repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.' '\n' \ + repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.$' '\n' \ | LC_ALL=C awk -v list="$1" ' BEGIN { while ((getline t < list) > 0) if (t != "") want[t] = 1 } seen[$0]++ { next } @@ -532,6 +538,10 @@ resolve_words() { if (s in want) { print s; delete want[s] } } } + if (index($0, "$") > 0) { + n = split($0, p, /[$.]/) + for (i = 1; i <= n; i++) if (p[i] in want) { print p[i]; delete want[p[i]] } + } }' return 0 } @@ -540,7 +550,7 @@ resolve_words() { # P — a file:line citation outside fences/URLs/globs # S — a backticked symbol-shaped token to resolve # Tokens are then resolved as SETS (one grep against the declaration file, one -# repo-wide word grep for the whole unresolved batch) — never per token. +# pass over the repository for the whole unresolved batch) — never per token. # Language-specific shapes arrive as -v variables: file_ext, file_re, sym_re, qual_re. DOCSCAN_AWK=' FNR == 1 { fence = 0 } @@ -574,50 +584,62 @@ FNR == 1 { fence = 0 } } ' -# ---------- the scope: which docs a scoped run checks ---------- -# A doc is in scope when the scope lists it, when its text names the basename -# of a listed file as a whole word (a link to a touched doc or code file), or -# when it cites in backticks a word the change removes (a symbol the diff -# renames or deletes). +# ---------- the scope: which docs a scoped run checks, and how far ---------- +# A doc the scope lists (a doc the diff touches) gets every check. A doc that +# only cites the scope — its text names the basename of a listed file as a +# whole word, or it cites in backticks a word the change removes (a symbol the +# diff renames or deletes) — reports only what the scope can break in it: a +# doc link into a listed file, and the backticked removed words. Its other +# findings are the repository's state, for a whole-repository run. # Every list below holds relative paths as find(1) prints them under a root. -SCOPE_DOCS="$WORK/scope-docs" +SCOPE_DOCS="$WORK/scope-docs" # every doc a scoped run reads +FULL_DOCS="$WORK/full-docs" # the listed docs: every check SCOPE_PATHS="$WORK/scope-paths" +BASENAMES="$WORK/basenames" if (( SCOPED )); then sed -e 's|^\./||' -e '/^$/d' "$SCOPE_LIST" | LC_ALL=C sort -u > "$SCOPE_PATHS" + sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$BASENAMES" for r in "${ROOTS[@]}"; do find "$r" -type f -name '*.md'; done | LC_ALL=C sort -u > "$WORK/all-docs" + grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" > "$FULL_DOCS" { - grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" - sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$WORK/basenames" - if [[ -s "$WORK/basenames" && -s "$WORK/all-docs" ]]; then - tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$WORK/basenames" 2>/dev/null + cat "$FULL_DOCS" + if [[ -s "$BASENAMES" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$BASENAMES" 2>/dev/null fi if [[ -n "$WORDS_LIST" && -s "$WORDS_LIST" && -s "$WORK/all-docs" ]]; then tr '\n' '\0' < "$WORK/all-docs" \ | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" "$DOCSCAN_AWK" \ - | awk -F'\t' 'NR == FNR { w[$0] = 1; next } - $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in w) || (m in w)) print $2 }' "$WORDS_LIST" - + | awk -F'\t' -v list="$WORDS_LIST" ' + BEGIN { while ((getline w < list) > 0) want[w] = 1 } + $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in want) || (m in want)) print $2 }' fi } | LC_ALL=C sort -u > "$SCOPE_DOCS" if [[ -s "$SCOPE_DOCS" ]]; then - echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" + echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" \ + "($(wc -l < "$FULL_DOCS" | tr -d ' ') touched, every check; the rest cite the scope, scope lines only)" else echo "check-repo-brain: scope — no doc in scope; doc checks skipped" fi fi -# bundle_docs [name] — the docs a run checks under one root: every -# *.md (or every file of that name), limited to the scope on a scoped run +# bundle_docs [name] [full] — the docs a run checks under one root: +# every *.md (or every file of that name); on a scoped run only the docs in +# scope, or with "full" only the docs that get every check bundle_docs() { - local name="${2:-*.md}" + local name="${2:-*.md}" list="$SCOPE_DOCS" + [[ "${3:-}" == full ]] && list="$FULL_DOCS" if (( SCOPED )); then - find "$1" -type f -name "$name" | grep -xF -f "$SCOPE_DOCS" + find "$1" -type f -name "$name" | grep -xF -f "$list" else find "$1" -type f -name "$name" fi return 0 } +# full_doc — exit 0 iff the doc gets every check +full_doc() { (( ! SCOPED )) || grep -qxF -- "$1" "$FULL_DOCS"; } + # ---------- Q2: code→docs edges (repo-wide, or the scope's code files on a # scoped run; resolved from repo root, then the citing file's own # sub-project) ---------- @@ -680,7 +702,7 @@ check_bundle() { # [[ "$(basename "$resolved")" == "index.md" ]] && queue+=("$resolved") done < <(grep -oE '\]\([^)]+\)' "$idx" 2>/dev/null) done - if [[ ! -f "$root_index" ]]; then + if [[ ! -f "$root_index" ]] && [[ -n "$(bundle_docs "$docroot" '*.md' full)" ]]; then fail "[Q1] $docroot — no index.md: the bundle has no map" fi while IFS= read -r doc; do @@ -690,14 +712,17 @@ check_bundle() { # case "$reachable" in *"$c"$'\n'*) ;; *) fail "[Q1] $doc — orphan: not reachable from $root_index" ;; esac - done < <(bundle_docs "$docroot") + done < <(bundle_docs "$docroot" '*.md' full) # --- Q2: every doc link resolves (a missing parent directory is just as - # broken as a missing file — only URLs/anchors are exempt) --- + # broken as a missing file — only URLs/anchors are exempt); a doc that only + # cites the scope reports only its links into a listed file --- while IFS= read -r md; do + local full=0; full_doc "$md" && full=1 while IFS= read -r raw; do local t="${raw#](}"; t="${t%)}" [[ "$t" == *.md* ]] || continue + (( full )) || grep -qxF -- "$(basename "${t%%#*}")" "$BASENAMES" || continue local resolved rc=0 resolved=$(resolve_link "$md" "$t" "$docroot") || rc=$? (( rc == 1 )) && continue @@ -708,7 +733,8 @@ check_bundle() { # done < <(bundle_docs "$docroot") # --- Q2: doc scan — file:line ban + docs→code symbol resolution. - # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). --- + # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). A + # doc that only cites the scope keeps only its backticked removed words. --- local scan; scan=$(mktemp) if ! ( set -o pipefail bundle_docs "$docroot" | tr '\n' '\0' \ @@ -719,6 +745,13 @@ check_bundle() { # rm -f "$scan" exit 2 fi + if (( SCOPED )); then + awk -F'\t' -v full="$FULL_DOCS" -v words="${WORDS_LIST:-/dev/null}" ' + BEGIN { while ((getline l < full) > 0) f[l] = 1; while ((getline l < words) > 0) w[l] = 1 } + ($2 in f) { print; next } + $1 == "S" { m = $4; sub(/^.*\./, "", m); if (($4 in w) || (m in w)) print }' "$scan" > "$scan.kept" + mv -f "$scan.kept" "$scan" + fi local f ln while IFS=$'\t' read -r _ f ln; do fail "[Q2] $f:$ln — cites a file path or line number (churn-prone coordinate)" @@ -851,7 +884,7 @@ check_bundle() { # else fail "[Q7] $md — frontmatter missing 'description:'" fi - done < <(bundle_docs "$docroot") + done < <(bundle_docs "$docroot" '*.md' full) # --- Q7: drift check — index line text == target's description (⚠️ exempt; # --fix rewrites drifted lines after the read loop, never during it) --- @@ -889,12 +922,12 @@ check_bundle() { # echo " fixed: $idx:$n — index line rewritten from its target's description" done <<< "$fixes" fi - done < <(bundle_docs "$docroot" index.md) + done < <(bundle_docs "$docroot" index.md full) # --- Q7: no log.md --- while IFS= read -r lg; do fail "[Q7] $lg — log.md is reserved for change history; docs describe current behavior" - done < <(bundle_docs "$docroot" log.md) + done < <(bundle_docs "$docroot" log.md full) } for i in "${!PROJS[@]}"; do diff --git a/linter-driven-development/scripts/check-repo-brain_test.sh b/linter-driven-development/scripts/check-repo-brain_test.sh index fc37f32a..ba7159b4 100755 --- a/linter-driven-development/scripts/check-repo-brain_test.sh +++ b/linter-driven-development/scripts/check-repo-brain_test.sh @@ -787,22 +787,24 @@ expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] do && expect_not "docs/other.md" && expect_not "[Q3]" && ok finish -begin "--scope: a doc that names a scope file is checked" -mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../$FX_CODE_FILE)." -printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +begin "--scope: a doc that only names a scope file reports only its links into the scope" +mk_conformant +mk_doc linker "links the scope" "See [the old doc](gone.md) and [a missing one](missing.md); code: \`$FX_CODE_FILE\` at $FX_CODE_FILE:12." +printf '%s\n' "$FX_CODE_FILE" docs/gone.md > "$LISTS/scope.txt" run_gate --scope "$LISTS/scope.txt" -expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +expect_exit 1 && expect_has "0 touched" && expect_has "[Q2] docs/linker.md — link target does not exist: gone.md" \ + && expect_not "missing.md" && expect_not "orphan" && expect_not "file path or line number" && ok finish begin "--scope --words: a doc that cites a removed symbol is checked" mk_conformant -mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\` and \`OtherGhost\`." append_index "- [ghost.md](ghost.md) — a ghost symbol" printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" run_gate --scope "$LISTS/scope.txt" expect_exit 0 && expect_has "no doc in scope" \ && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ - && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok + && expect_has "backticked \`SnapshotRunner\` does not resolve" && expect_not "OtherGhost" && ok finish begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" diff --git a/linter-driven-development/scripts/ldd-detect.sh b/linter-driven-development/scripts/ldd-detect.sh index 40e37670..eabec814 100755 --- a/linter-driven-development/scripts/ldd-detect.sh +++ b/linter-driven-development/scripts/ldd-detect.sh @@ -48,11 +48,11 @@ # Detect-gate: Q the R9 gate's [Q] report lines, from one run of # check-repo-brain.sh beside this script over --root. On a # scoped bundle (no dirs.txt) the gate runs with --scope: it -# checks only the bundle's docs (docs.txt), the docs that -# link to a scope file and the docs that cite a symbol the -# diff removes, the code edges of the scope's code files, -# and no root wiring; with no such doc its doc checks are -# skipped. The repository-wide findings belong to a +# checks the bundle's docs (docs.txt) in full, and in the +# docs that name a scope file or cite a symbol the diff +# removes only those links and symbols; it checks the code +# edges of the scope's code files and no root wiring; with +# no such doc its doc checks are skipped. The repository-wide findings belong to a # whole-repository review # Detect: judgment no mechanical lead; the hunter runs the prose # A question with no detect line, two of them, or a malformed one is an error: @@ -624,7 +624,8 @@ run_path() { # # of docs.txt; and the identifiers the diff removes — on its removed lines and # on none of its added ones — the symbols a renamed or deleted declaration # takes away from the docs that cite them. Comment lines (a line that opens -# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped. +# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped, +# and a word the scope's files still carry after the change is not removed. gate_scope_lists() { local scope="$1" words="$2" line f { @@ -646,7 +647,18 @@ gate_scope_lists() { if (s == "+") add[w] = 1; else del[w] = 1 } } - END { for (w in del) if (!(w in add)) print w }' | LC_ALL=C sort > "$words" + END { for (w in del) if (!(w in add)) print w }' \ + | LC_ALL=C sort > "$words.removed" + if (( ${#ALL_FILES[@]} > 0 )); then + cat -- "${ALL_FILES[@]}" | LC_ALL=C tr -cs 'A-Za-z0-9_' '\n' \ + | LC_ALL=C awk -v list="$words.removed" ' + BEGIN { while ((getline w < list) > 0) gone[w] = 1 } + ($0 in gone) { delete gone[$0] } + END { for (w in gone) print w }' | LC_ALL=C sort > "$words" + else + cp "$words.removed" "$words" + fi + rm -f "$words.removed" } # The R9 gate runs once, lazily; its [Q] lines are split by question. With diff --git a/linter-driven-development/scripts/ldd-detect_test.sh b/linter-driven-development/scripts/ldd-detect_test.sh index ff83999c..dcb03458 100644 --- a/linter-driven-development/scripts/ldd-detect_test.sh +++ b/linter-driven-development/scripts/ldd-detect_test.sh @@ -617,6 +617,25 @@ pkill -f 'sleep 37' 2>/dev/null rm -rf "$FAKE" finish +begin "a --gate-budget the gate finishes within writes the same counts as no budget" +mk_repo; : > "$BUNDLE/dirs.txt"; run_detect +cp "$BUNDLE/counts.tsv" "$BUNDLE/first.tsv" +run_detect --gate-budget 60 +expect_exit 0 && { cmp -s "$BUNDLE/first.tsv" "$BUNDLE/counts.tsv" || bad "counts.tsv differs under a budget the gate meets"; } \ + && cmp -s "$BUNDLE/first.tsv" "$BUNDLE/counts.tsv" && ok +finish + +begin "a gate that fails under --gate-budget fails the run (exit 2)" +mk_repo; : > "$BUNDLE/dirs.txt" +FAKE=$(mktemp -d) +cp "$DETECT" "$FAKE/ldd-detect.sh" +printf '#!/usr/bin/env bash\necho "check-repo-brain: internal scanner error"\nexit 2\n' > "$FAKE/check-repo-brain.sh" +OUT=$(cd "$REPO" && bash "$FAKE/ldd-detect.sh" --rules "$RULES" --gate-budget 30 --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? +ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +expect_exit 2 && expect_has "the R9 gate failed to run" && ok +rm -rf "$FAKE" +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/linter-driven-development/skills/pre-commit-review/SKILL.md b/linter-driven-development/skills/pre-commit-review/SKILL.md index 6453c1c5..b66997c2 100644 --- a/linter-driven-development/skills/pre-commit-review/SKILL.md +++ b/linter-driven-development/skills/pre-commit-review/SKILL.md @@ -80,10 +80,10 @@ writes `hits.tsv`, `hits-all.tsv` and `counts.tsv` into the bundle, and prints t question only a hunter's reading answers — then one line per rule family. That table, about 300 tokens, is the pre-filter: the parent prints no rule file, composes no detection command, and never reads a Falsifying-questions section. On a scoped review the R9 -gate checks only the docs the diff touches, the docs that link to a scope file and the -docs that cite a symbol the diff removes, and its rows are about those docs and the -scope's own files — an orphan doc elsewhere or an unwired root is the repository's -state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line +gate checks the docs the diff touches in full; in a doc that links to a scope file +or cites a symbol the diff removes it checks only those links and symbols; and it +checks the code edges of the scope's own files — an orphan doc elsewhere, a citing +doc's older drift or an unwired root is the repository's state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line `R9 gate: …` under the summary says which docs the gate checked. A gate row whose hits cell is `inconclusive` means the gate ran out of its `--gate-budget`: it is not a `0`, and the documentation hunter answers that question from the prose. When the diff --git a/python-linter-driven-development/scripts/check-repo-brain.sh b/python-linter-driven-development/scripts/check-repo-brain.sh index 8f70611d..acab2185 100755 --- a/python-linter-driven-development/scripts/check-repo-brain.sh +++ b/python-linter-driven-development/scripts/check-repo-brain.sh @@ -15,8 +15,10 @@ # (the one mechanical repair; everything else stays report-only) # --scope a review's scope: one path per line, relative to repo-root # (code files and docs, deleted ones included). The gate then checks -# only the docs in scope: the listed docs, the docs whose text names a -# listed file's basename, and the docs that cite a word of --words. +# only the docs in scope. A listed doc gets every check. A doc whose +# text names a listed file's basename, or that cites a word of +# --words, reports only its links into a listed file and its +# backticked --words. # Code edges are checked for the listed code files only, Q3 is skipped # (root wiring is the repository's state, not a diff's), and with no # doc in scope the doc checks are skipped. One stdout line says which: @@ -367,13 +369,16 @@ build_declarations() { is_repo_pkg() { grep -qxF "$1" "$PKGS" 2>/dev/null; } # repo_files — the repository's non-markdown files, NUL-separated. A git work -# tree lists them through git (tracked files plus untracked ones git does not -# ignore), which never walks build outputs or symlinked caches; elsewhere find -# skips the directories that hold dependencies and build outputs. +# tree lists them through git — the tracked files, those of its submodules +# included, then the untracked ones git does not ignore — which never walks +# build outputs or symlinked caches; elsewhere find skips the directories that +# hold dependencies and build outputs. repo_files() { + local -a spec=(-- . ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**') if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then - git ls-files -z --cached --others --exclude-standard -- . \ - ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**' 2>/dev/null && return 0 + { git ls-files -z --cached --recurse-submodules "${spec[@]}" 2>/dev/null \ + || git ls-files -z --cached "${spec[@]}" 2>/dev/null; } \ + && git ls-files -z --others --exclude-standard "${spec[@]}" 2>/dev/null && return 0 fi find . \( -name .git -o -name vendor -o -name node_modules -o -name 'bazel-*' -o -name .venv \ -o -name venv -o -name __pycache__ -o -name .tox -o -name dist -o -name target \) -prune \ @@ -382,12 +387,13 @@ repo_files() { # resolve_words — the tokens that occur as a whole word in any # non-markdown repository file, one per line. The repository is read once -# into words (runs of [A-Za-z0-9_.]); a dotted token such as `Type.Method` -# matches any run of whole dot-separated parts, as grep -w would match it. +# into words (runs of [A-Za-z0-9_.$]); a dotted token such as `Type.Method` +# matches any run of whole dot-separated parts, as grep -w would match it, and +# a word that carries $ (a JavaScript identifier) also offers its $-free parts. # One pass over the text is linear in its size, where a fixed-string grep for # hundreds of tokens is not. resolve_words() { - repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.' '\n' \ + repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.$' '\n' \ | LC_ALL=C awk -v list="$1" ' BEGIN { while ((getline t < list) > 0) if (t != "") want[t] = 1 } seen[$0]++ { next } @@ -401,6 +407,10 @@ resolve_words() { if (s in want) { print s; delete want[s] } } } + if (index($0, "$") > 0) { + n = split($0, p, /[$.]/) + for (i = 1; i <= n; i++) if (p[i] in want) { print p[i]; delete want[p[i]] } + } }' return 0 } @@ -409,7 +419,7 @@ resolve_words() { # P — a file:line citation outside fences/URLs/globs # S — a backticked symbol-shaped token to resolve # Tokens are then resolved as SETS (one grep against the declaration file, one -# repo-wide word grep for the whole unresolved batch) — never per token. +# pass over the repository for the whole unresolved batch) — never per token. # Language-specific shapes arrive as -v variables: file_ext, file_re, sym_re, qual_re. DOCSCAN_AWK=' FNR == 1 { fence = 0 } @@ -443,50 +453,62 @@ FNR == 1 { fence = 0 } } ' -# ---------- the scope: which docs a scoped run checks ---------- -# A doc is in scope when the scope lists it, when its text names the basename -# of a listed file as a whole word (a link to a touched doc or code file), or -# when it cites in backticks a word the change removes (a symbol the diff -# renames or deletes). +# ---------- the scope: which docs a scoped run checks, and how far ---------- +# A doc the scope lists (a doc the diff touches) gets every check. A doc that +# only cites the scope — its text names the basename of a listed file as a +# whole word, or it cites in backticks a word the change removes (a symbol the +# diff renames or deletes) — reports only what the scope can break in it: a +# doc link into a listed file, and the backticked removed words. Its other +# findings are the repository's state, for a whole-repository run. # Every list below holds relative paths as find(1) prints them under a root. -SCOPE_DOCS="$WORK/scope-docs" +SCOPE_DOCS="$WORK/scope-docs" # every doc a scoped run reads +FULL_DOCS="$WORK/full-docs" # the listed docs: every check SCOPE_PATHS="$WORK/scope-paths" +BASENAMES="$WORK/basenames" if (( SCOPED )); then sed -e 's|^\./||' -e '/^$/d' "$SCOPE_LIST" | LC_ALL=C sort -u > "$SCOPE_PATHS" + sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$BASENAMES" for r in "${ROOTS[@]}"; do find "$r" -type f -name '*.md'; done | LC_ALL=C sort -u > "$WORK/all-docs" + grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" > "$FULL_DOCS" { - grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" - sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$WORK/basenames" - if [[ -s "$WORK/basenames" && -s "$WORK/all-docs" ]]; then - tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$WORK/basenames" 2>/dev/null + cat "$FULL_DOCS" + if [[ -s "$BASENAMES" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$BASENAMES" 2>/dev/null fi if [[ -n "$WORDS_LIST" && -s "$WORDS_LIST" && -s "$WORK/all-docs" ]]; then tr '\n' '\0' < "$WORK/all-docs" \ | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" "$DOCSCAN_AWK" \ - | awk -F'\t' 'NR == FNR { w[$0] = 1; next } - $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in w) || (m in w)) print $2 }' "$WORDS_LIST" - + | awk -F'\t' -v list="$WORDS_LIST" ' + BEGIN { while ((getline w < list) > 0) want[w] = 1 } + $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in want) || (m in want)) print $2 }' fi } | LC_ALL=C sort -u > "$SCOPE_DOCS" if [[ -s "$SCOPE_DOCS" ]]; then - echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" + echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" \ + "($(wc -l < "$FULL_DOCS" | tr -d ' ') touched, every check; the rest cite the scope, scope lines only)" else echo "check-repo-brain: scope — no doc in scope; doc checks skipped" fi fi -# bundle_docs [name] — the docs a run checks under one root: every -# *.md (or every file of that name), limited to the scope on a scoped run +# bundle_docs [name] [full] — the docs a run checks under one root: +# every *.md (or every file of that name); on a scoped run only the docs in +# scope, or with "full" only the docs that get every check bundle_docs() { - local name="${2:-*.md}" + local name="${2:-*.md}" list="$SCOPE_DOCS" + [[ "${3:-}" == full ]] && list="$FULL_DOCS" if (( SCOPED )); then - find "$1" -type f -name "$name" | grep -xF -f "$SCOPE_DOCS" + find "$1" -type f -name "$name" | grep -xF -f "$list" else find "$1" -type f -name "$name" fi return 0 } +# full_doc — exit 0 iff the doc gets every check +full_doc() { (( ! SCOPED )) || grep -qxF -- "$1" "$FULL_DOCS"; } + # ---------- Q2: code→docs edges (repo-wide, or the scope's code files on a # scoped run; resolved from repo root, then the citing file's own # sub-project) ---------- @@ -549,7 +571,7 @@ check_bundle() { # [[ "$(basename "$resolved")" == "index.md" ]] && queue+=("$resolved") done < <(grep -oE '\]\([^)]+\)' "$idx" 2>/dev/null) done - if [[ ! -f "$root_index" ]]; then + if [[ ! -f "$root_index" ]] && [[ -n "$(bundle_docs "$docroot" '*.md' full)" ]]; then fail "[Q1] $docroot — no index.md: the bundle has no map" fi while IFS= read -r doc; do @@ -559,14 +581,17 @@ check_bundle() { # case "$reachable" in *"$c"$'\n'*) ;; *) fail "[Q1] $doc — orphan: not reachable from $root_index" ;; esac - done < <(bundle_docs "$docroot") + done < <(bundle_docs "$docroot" '*.md' full) # --- Q2: every doc link resolves (a missing parent directory is just as - # broken as a missing file — only URLs/anchors are exempt) --- + # broken as a missing file — only URLs/anchors are exempt); a doc that only + # cites the scope reports only its links into a listed file --- while IFS= read -r md; do + local full=0; full_doc "$md" && full=1 while IFS= read -r raw; do local t="${raw#](}"; t="${t%)}" [[ "$t" == *.md* ]] || continue + (( full )) || grep -qxF -- "$(basename "${t%%#*}")" "$BASENAMES" || continue local resolved rc=0 resolved=$(resolve_link "$md" "$t" "$docroot") || rc=$? (( rc == 1 )) && continue @@ -577,7 +602,8 @@ check_bundle() { # done < <(bundle_docs "$docroot") # --- Q2: doc scan — file:line ban + docs→code symbol resolution. - # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). --- + # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). A + # doc that only cites the scope keeps only its backticked removed words. --- local scan; scan=$(mktemp) if ! ( set -o pipefail bundle_docs "$docroot" | tr '\n' '\0' \ @@ -588,6 +614,13 @@ check_bundle() { # rm -f "$scan" exit 2 fi + if (( SCOPED )); then + awk -F'\t' -v full="$FULL_DOCS" -v words="${WORDS_LIST:-/dev/null}" ' + BEGIN { while ((getline l < full) > 0) f[l] = 1; while ((getline l < words) > 0) w[l] = 1 } + ($2 in f) { print; next } + $1 == "S" { m = $4; sub(/^.*\./, "", m); if (($4 in w) || (m in w)) print }' "$scan" > "$scan.kept" + mv -f "$scan.kept" "$scan" + fi local f ln while IFS=$'\t' read -r _ f ln; do fail "[Q2] $f:$ln — cites a file path or line number (churn-prone coordinate)" @@ -720,7 +753,7 @@ check_bundle() { # else fail "[Q7] $md — frontmatter missing 'description:'" fi - done < <(bundle_docs "$docroot") + done < <(bundle_docs "$docroot" '*.md' full) # --- Q7: drift check — index line text == target's description (⚠️ exempt; # --fix rewrites drifted lines after the read loop, never during it) --- @@ -758,12 +791,12 @@ check_bundle() { # echo " fixed: $idx:$n — index line rewritten from its target's description" done <<< "$fixes" fi - done < <(bundle_docs "$docroot" index.md) + done < <(bundle_docs "$docroot" index.md full) # --- Q7: no log.md --- while IFS= read -r lg; do fail "[Q7] $lg — log.md is reserved for change history; docs describe current behavior" - done < <(bundle_docs "$docroot" log.md) + done < <(bundle_docs "$docroot" log.md full) } for i in "${!PROJS[@]}"; do diff --git a/python-linter-driven-development/scripts/check-repo-brain_test.sh b/python-linter-driven-development/scripts/check-repo-brain_test.sh index 3aedc568..d79f6f8f 100755 --- a/python-linter-driven-development/scripts/check-repo-brain_test.sh +++ b/python-linter-driven-development/scripts/check-repo-brain_test.sh @@ -660,22 +660,24 @@ expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] do && expect_not "docs/other.md" && expect_not "[Q3]" && ok finish -begin "--scope: a doc that names a scope file is checked" -mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../$FX_CODE_FILE)." -printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +begin "--scope: a doc that only names a scope file reports only its links into the scope" +mk_conformant +mk_doc linker "links the scope" "See [the old doc](gone.md) and [a missing one](missing.md); code: \`$FX_CODE_FILE\` at $FX_CODE_FILE:12." +printf '%s\n' "$FX_CODE_FILE" docs/gone.md > "$LISTS/scope.txt" run_gate --scope "$LISTS/scope.txt" -expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +expect_exit 1 && expect_has "0 touched" && expect_has "[Q2] docs/linker.md — link target does not exist: gone.md" \ + && expect_not "missing.md" && expect_not "orphan" && expect_not "file path or line number" && ok finish begin "--scope --words: a doc that cites a removed symbol is checked" mk_conformant -mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\` and \`OtherGhost\`." append_index "- [ghost.md](ghost.md) — a ghost symbol" printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" run_gate --scope "$LISTS/scope.txt" expect_exit 0 && expect_has "no doc in scope" \ && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ - && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok + && expect_has "backticked \`SnapshotRunner\` does not resolve" && expect_not "OtherGhost" && ok finish begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" diff --git a/python-linter-driven-development/scripts/ldd-detect.sh b/python-linter-driven-development/scripts/ldd-detect.sh index 9ade67ba..4a02e193 100755 --- a/python-linter-driven-development/scripts/ldd-detect.sh +++ b/python-linter-driven-development/scripts/ldd-detect.sh @@ -48,11 +48,11 @@ # Detect-gate: Q the R9 gate's [Q] report lines, from one run of # check-repo-brain.sh beside this script over --root. On a # scoped bundle (no dirs.txt) the gate runs with --scope: it -# checks only the bundle's docs (docs.txt), the docs that -# link to a scope file and the docs that cite a symbol the -# diff removes, the code edges of the scope's code files, -# and no root wiring; with no such doc its doc checks are -# skipped. The repository-wide findings belong to a +# checks the bundle's docs (docs.txt) in full, and in the +# docs that name a scope file or cite a symbol the diff +# removes only those links and symbols; it checks the code +# edges of the scope's code files and no root wiring; with +# no such doc its doc checks are skipped. The repository-wide findings belong to a # whole-repository review # Detect: judgment no mechanical lead; the hunter runs the prose # A question with no detect line, two of them, or a malformed one is an error: @@ -511,7 +511,8 @@ run_path() { # # of docs.txt; and the identifiers the diff removes — on its removed lines and # on none of its added ones — the symbols a renamed or deleted declaration # takes away from the docs that cite them. Comment lines (a line that opens -# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped. +# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped, +# and a word the scope's files still carry after the change is not removed. gate_scope_lists() { local scope="$1" words="$2" line f { @@ -533,7 +534,18 @@ gate_scope_lists() { if (s == "+") add[w] = 1; else del[w] = 1 } } - END { for (w in del) if (!(w in add)) print w }' | LC_ALL=C sort > "$words" + END { for (w in del) if (!(w in add)) print w }' \ + | LC_ALL=C sort > "$words.removed" + if (( ${#ALL_FILES[@]} > 0 )); then + cat -- "${ALL_FILES[@]}" | LC_ALL=C tr -cs 'A-Za-z0-9_' '\n' \ + | LC_ALL=C awk -v list="$words.removed" ' + BEGIN { while ((getline w < list) > 0) gone[w] = 1 } + ($0 in gone) { delete gone[$0] } + END { for (w in gone) print w }' | LC_ALL=C sort > "$words" + else + cp "$words.removed" "$words" + fi + rm -f "$words.removed" } # The R9 gate runs once, lazily; its [Q] lines are split by question. With diff --git a/python-linter-driven-development/scripts/ldd-detect_test.sh b/python-linter-driven-development/scripts/ldd-detect_test.sh index 7c5b335c..2a0a8bc3 100644 --- a/python-linter-driven-development/scripts/ldd-detect_test.sh +++ b/python-linter-driven-development/scripts/ldd-detect_test.sh @@ -544,6 +544,25 @@ pkill -f 'sleep 37' 2>/dev/null rm -rf "$FAKE" finish +begin "a --gate-budget the gate finishes within writes the same counts as no budget" +mk_repo; : > "$BUNDLE/dirs.txt"; run_detect +cp "$BUNDLE/counts.tsv" "$BUNDLE/first.tsv" +run_detect --gate-budget 60 +expect_exit 0 && { cmp -s "$BUNDLE/first.tsv" "$BUNDLE/counts.tsv" || bad "counts.tsv differs under a budget the gate meets"; } \ + && cmp -s "$BUNDLE/first.tsv" "$BUNDLE/counts.tsv" && ok +finish + +begin "a gate that fails under --gate-budget fails the run (exit 2)" +mk_repo; : > "$BUNDLE/dirs.txt" +FAKE=$(mktemp -d) +cp "$DETECT" "$FAKE/ldd-detect.sh" +printf '#!/usr/bin/env bash\necho "check-repo-brain: internal scanner error"\nexit 2\n' > "$FAKE/check-repo-brain.sh" +OUT=$(cd "$REPO" && bash "$FAKE/ldd-detect.sh" --rules "$RULES" --gate-budget 30 --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? +ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +expect_exit 2 && expect_has "the R9 gate failed to run" && ok +rm -rf "$FAKE" +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/python-linter-driven-development/skills/pre-commit-review/SKILL.md b/python-linter-driven-development/skills/pre-commit-review/SKILL.md index 624592bd..f0ee9fcf 100644 --- a/python-linter-driven-development/skills/pre-commit-review/SKILL.md +++ b/python-linter-driven-development/skills/pre-commit-review/SKILL.md @@ -80,10 +80,10 @@ writes `hits.tsv`, `hits-all.tsv` and `counts.tsv` into the bundle, and prints t question only a hunter's reading answers — then one line per rule family. That table, about 300 tokens, is the pre-filter: the parent prints no rule file, composes no detection command, and never reads a Falsifying-questions section. On a scoped review the R9 -gate checks only the docs the diff touches, the docs that link to a scope file and the -docs that cite a symbol the diff removes, and its rows are about those docs and the -scope's own files — an orphan doc elsewhere or an unwired root is the repository's -state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line +gate checks the docs the diff touches in full; in a doc that links to a scope file +or cites a symbol the diff removes it checks only those links and symbols; and it +checks the code edges of the scope's own files — an orphan doc elsewhere, a citing +doc's older drift or an unwired root is the repository's state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line `R9 gate: …` under the summary says which docs the gate checked. A gate row whose hits cell is `inconclusive` means the gate ran out of its `--gate-budget`: it is not a `0`, and the documentation hunter answers that question from the prose. When the diff --git a/ts-react-linter-driven-development/scripts/check-repo-brain.sh b/ts-react-linter-driven-development/scripts/check-repo-brain.sh index d28c9384..51eeb3ba 100755 --- a/ts-react-linter-driven-development/scripts/check-repo-brain.sh +++ b/ts-react-linter-driven-development/scripts/check-repo-brain.sh @@ -15,8 +15,10 @@ # (the one mechanical repair; everything else stays report-only) # --scope a review's scope: one path per line, relative to repo-root # (code files and docs, deleted ones included). The gate then checks -# only the docs in scope: the listed docs, the docs whose text names a -# listed file's basename, and the docs that cite a word of --words. +# only the docs in scope. A listed doc gets every check. A doc whose +# text names a listed file's basename, or that cites a word of +# --words, reports only its links into a listed file and its +# backticked --words. # Code edges are checked for the listed code files only, Q3 is skipped # (root wiring is the repository's state, not a diff's), and with no # doc in scope the doc checks are skipped. One stdout line says which: @@ -462,13 +464,16 @@ build_declarations() { is_repo_pkg() { grep -qxF "$1" "$PKGS" 2>/dev/null; } # repo_files — the repository's non-markdown files, NUL-separated. A git work -# tree lists them through git (tracked files plus untracked ones git does not -# ignore), which never walks build outputs or symlinked caches; elsewhere find -# skips the directories that hold dependencies and build outputs. +# tree lists them through git — the tracked files, those of its submodules +# included, then the untracked ones git does not ignore — which never walks +# build outputs or symlinked caches; elsewhere find skips the directories that +# hold dependencies and build outputs. repo_files() { + local -a spec=(-- . ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**') if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then - git ls-files -z --cached --others --exclude-standard -- . \ - ':(exclude,glob)**/*.md' ':(exclude,glob)**/vendor/**' 2>/dev/null && return 0 + { git ls-files -z --cached --recurse-submodules "${spec[@]}" 2>/dev/null \ + || git ls-files -z --cached "${spec[@]}" 2>/dev/null; } \ + && git ls-files -z --others --exclude-standard "${spec[@]}" 2>/dev/null && return 0 fi find . \( -name .git -o -name vendor -o -name node_modules -o -name 'bazel-*' -o -name .venv \ -o -name venv -o -name __pycache__ -o -name .tox -o -name dist -o -name target \) -prune \ @@ -477,12 +482,13 @@ repo_files() { # resolve_words — the tokens that occur as a whole word in any # non-markdown repository file, one per line. The repository is read once -# into words (runs of [A-Za-z0-9_.]); a dotted token such as `Type.Method` -# matches any run of whole dot-separated parts, as grep -w would match it. +# into words (runs of [A-Za-z0-9_.$]); a dotted token such as `Type.Method` +# matches any run of whole dot-separated parts, as grep -w would match it, and +# a word that carries $ (a JavaScript identifier) also offers its $-free parts. # One pass over the text is linear in its size, where a fixed-string grep for # hundreds of tokens is not. resolve_words() { - repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.' '\n' \ + repo_files | xargs -0 cat 2>/dev/null | LC_ALL=C tr -cs 'A-Za-z0-9_.$' '\n' \ | LC_ALL=C awk -v list="$1" ' BEGIN { while ((getline t < list) > 0) if (t != "") want[t] = 1 } seen[$0]++ { next } @@ -496,6 +502,10 @@ resolve_words() { if (s in want) { print s; delete want[s] } } } + if (index($0, "$") > 0) { + n = split($0, p, /[$.]/) + for (i = 1; i <= n; i++) if (p[i] in want) { print p[i]; delete want[p[i]] } + } }' return 0 } @@ -504,7 +514,7 @@ resolve_words() { # P — a file:line citation outside fences/URLs/globs # S — a backticked symbol-shaped token to resolve # Tokens are then resolved as SETS (one grep against the declaration file, one -# repo-wide word grep for the whole unresolved batch) — never per token. +# pass over the repository for the whole unresolved batch) — never per token. # Language-specific shapes arrive as -v variables: file_ext, file_re, sym_re, qual_re. DOCSCAN_AWK=' FNR == 1 { fence = 0 } @@ -538,50 +548,62 @@ FNR == 1 { fence = 0 } } ' -# ---------- the scope: which docs a scoped run checks ---------- -# A doc is in scope when the scope lists it, when its text names the basename -# of a listed file as a whole word (a link to a touched doc or code file), or -# when it cites in backticks a word the change removes (a symbol the diff -# renames or deletes). +# ---------- the scope: which docs a scoped run checks, and how far ---------- +# A doc the scope lists (a doc the diff touches) gets every check. A doc that +# only cites the scope — its text names the basename of a listed file as a +# whole word, or it cites in backticks a word the change removes (a symbol the +# diff renames or deletes) — reports only what the scope can break in it: a +# doc link into a listed file, and the backticked removed words. Its other +# findings are the repository's state, for a whole-repository run. # Every list below holds relative paths as find(1) prints them under a root. -SCOPE_DOCS="$WORK/scope-docs" +SCOPE_DOCS="$WORK/scope-docs" # every doc a scoped run reads +FULL_DOCS="$WORK/full-docs" # the listed docs: every check SCOPE_PATHS="$WORK/scope-paths" +BASENAMES="$WORK/basenames" if (( SCOPED )); then sed -e 's|^\./||' -e '/^$/d' "$SCOPE_LIST" | LC_ALL=C sort -u > "$SCOPE_PATHS" + sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$BASENAMES" for r in "${ROOTS[@]}"; do find "$r" -type f -name '*.md'; done | LC_ALL=C sort -u > "$WORK/all-docs" + grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" > "$FULL_DOCS" { - grep -xF -f "$SCOPE_PATHS" "$WORK/all-docs" - sed 's|.*/||' "$SCOPE_PATHS" | LC_ALL=C sort -u > "$WORK/basenames" - if [[ -s "$WORK/basenames" && -s "$WORK/all-docs" ]]; then - tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$WORK/basenames" 2>/dev/null + cat "$FULL_DOCS" + if [[ -s "$BASENAMES" && -s "$WORK/all-docs" ]]; then + tr '\n' '\0' < "$WORK/all-docs" | xargs -0 grep -lwF -f "$BASENAMES" 2>/dev/null fi if [[ -n "$WORDS_LIST" && -s "$WORDS_LIST" && -s "$WORK/all-docs" ]]; then tr '\n' '\0' < "$WORK/all-docs" \ | xargs -0 awk -v file_ext="$LANG_FILE_EXT" -v file_re="$LANG_FILE_RE" \ -v sym_re="$LANG_SYMBOL_RE" -v qual_re="$LANG_QUALIFIED_RE" "$DOCSCAN_AWK" \ - | awk -F'\t' 'NR == FNR { w[$0] = 1; next } - $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in w) || (m in w)) print $2 }' "$WORDS_LIST" - + | awk -F'\t' -v list="$WORDS_LIST" ' + BEGIN { while ((getline w < list) > 0) want[w] = 1 } + $1 == "S" { t = $4; m = t; sub(/^.*\./, "", m); if ((t in want) || (m in want)) print $2 }' fi } | LC_ALL=C sort -u > "$SCOPE_DOCS" if [[ -s "$SCOPE_DOCS" ]]; then - echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" + echo "check-repo-brain: scope — $(wc -l < "$SCOPE_DOCS" | tr -d ' ') doc(s) to check" \ + "($(wc -l < "$FULL_DOCS" | tr -d ' ') touched, every check; the rest cite the scope, scope lines only)" else echo "check-repo-brain: scope — no doc in scope; doc checks skipped" fi fi -# bundle_docs [name] — the docs a run checks under one root: every -# *.md (or every file of that name), limited to the scope on a scoped run +# bundle_docs [name] [full] — the docs a run checks under one root: +# every *.md (or every file of that name); on a scoped run only the docs in +# scope, or with "full" only the docs that get every check bundle_docs() { - local name="${2:-*.md}" + local name="${2:-*.md}" list="$SCOPE_DOCS" + [[ "${3:-}" == full ]] && list="$FULL_DOCS" if (( SCOPED )); then - find "$1" -type f -name "$name" | grep -xF -f "$SCOPE_DOCS" + find "$1" -type f -name "$name" | grep -xF -f "$list" else find "$1" -type f -name "$name" fi return 0 } +# full_doc — exit 0 iff the doc gets every check +full_doc() { (( ! SCOPED )) || grep -qxF -- "$1" "$FULL_DOCS"; } + # ---------- Q2: code→docs edges (repo-wide, or the scope's code files on a # scoped run; resolved from repo root, then the citing file's own # sub-project) ---------- @@ -644,7 +666,7 @@ check_bundle() { # [[ "$(basename "$resolved")" == "index.md" ]] && queue+=("$resolved") done < <(grep -oE '\]\([^)]+\)' "$idx" 2>/dev/null) done - if [[ ! -f "$root_index" ]]; then + if [[ ! -f "$root_index" ]] && [[ -n "$(bundle_docs "$docroot" '*.md' full)" ]]; then fail "[Q1] $docroot — no index.md: the bundle has no map" fi while IFS= read -r doc; do @@ -654,14 +676,17 @@ check_bundle() { # case "$reachable" in *"$c"$'\n'*) ;; *) fail "[Q1] $doc — orphan: not reachable from $root_index" ;; esac - done < <(bundle_docs "$docroot") + done < <(bundle_docs "$docroot" '*.md' full) # --- Q2: every doc link resolves (a missing parent directory is just as - # broken as a missing file — only URLs/anchors are exempt) --- + # broken as a missing file — only URLs/anchors are exempt); a doc that only + # cites the scope reports only its links into a listed file --- while IFS= read -r md; do + local full=0; full_doc "$md" && full=1 while IFS= read -r raw; do local t="${raw#](}"; t="${t%)}" [[ "$t" == *.md* ]] || continue + (( full )) || grep -qxF -- "$(basename "${t%%#*}")" "$BASENAMES" || continue local resolved rc=0 resolved=$(resolve_link "$md" "$t" "$docroot") || rc=$? (( rc == 1 )) && continue @@ -672,7 +697,8 @@ check_bundle() { # done < <(bundle_docs "$docroot") # --- Q2: doc scan — file:line ban + docs→code symbol resolution. - # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). --- + # One awk pass extracts; resolution is set-based (see DOCSCAN_AWK above). A + # doc that only cites the scope keeps only its backticked removed words. --- local scan; scan=$(mktemp) if ! ( set -o pipefail bundle_docs "$docroot" | tr '\n' '\0' \ @@ -683,6 +709,13 @@ check_bundle() { # rm -f "$scan" exit 2 fi + if (( SCOPED )); then + awk -F'\t' -v full="$FULL_DOCS" -v words="${WORDS_LIST:-/dev/null}" ' + BEGIN { while ((getline l < full) > 0) f[l] = 1; while ((getline l < words) > 0) w[l] = 1 } + ($2 in f) { print; next } + $1 == "S" { m = $4; sub(/^.*\./, "", m); if (($4 in w) || (m in w)) print }' "$scan" > "$scan.kept" + mv -f "$scan.kept" "$scan" + fi local f ln while IFS=$'\t' read -r _ f ln; do fail "[Q2] $f:$ln — cites a file path or line number (churn-prone coordinate)" @@ -815,7 +848,7 @@ check_bundle() { # else fail "[Q7] $md — frontmatter missing 'description:'" fi - done < <(bundle_docs "$docroot") + done < <(bundle_docs "$docroot" '*.md' full) # --- Q7: drift check — index line text == target's description (⚠️ exempt; # --fix rewrites drifted lines after the read loop, never during it) --- @@ -853,12 +886,12 @@ check_bundle() { # echo " fixed: $idx:$n — index line rewritten from its target's description" done <<< "$fixes" fi - done < <(bundle_docs "$docroot" index.md) + done < <(bundle_docs "$docroot" index.md full) # --- Q7: no log.md --- while IFS= read -r lg; do fail "[Q7] $lg — log.md is reserved for change history; docs describe current behavior" - done < <(bundle_docs "$docroot" log.md) + done < <(bundle_docs "$docroot" log.md full) } for i in "${!PROJS[@]}"; do diff --git a/ts-react-linter-driven-development/scripts/check-repo-brain_test.sh b/ts-react-linter-driven-development/scripts/check-repo-brain_test.sh index 020d1215..f58ca116 100755 --- a/ts-react-linter-driven-development/scripts/check-repo-brain_test.sh +++ b/ts-react-linter-driven-development/scripts/check-repo-brain_test.sh @@ -671,22 +671,24 @@ expect_exit 1 && expect_has "scope — 1 doc(s) to check" && expect_has "[Q1] do && expect_not "docs/other.md" && expect_not "[Q3]" && ok finish -begin "--scope: a doc that names a scope file is checked" -mk_conformant; mk_doc linker "links the code" "The code lives in [the policy](../$FX_CODE_FILE)." -printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt" +begin "--scope: a doc that only names a scope file reports only its links into the scope" +mk_conformant +mk_doc linker "links the scope" "See [the old doc](gone.md) and [a missing one](missing.md); code: \`$FX_CODE_FILE\` at $FX_CODE_FILE:12." +printf '%s\n' "$FX_CODE_FILE" docs/gone.md > "$LISTS/scope.txt" run_gate --scope "$LISTS/scope.txt" -expect_exit 1 && expect_has "[Q1] docs/linker.md — orphan" && ok +expect_exit 1 && expect_has "0 touched" && expect_has "[Q2] docs/linker.md — link target does not exist: gone.md" \ + && expect_not "missing.md" && expect_not "orphan" && expect_not "file path or line number" && ok finish begin "--scope --words: a doc that cites a removed symbol is checked" mk_conformant -mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\`." +mk_doc ghost "a ghost symbol" "Uses \`SnapshotRunner\` and \`OtherGhost\`." append_index "- [ghost.md](ghost.md) — a ghost symbol" printf '%s\n' "$FX_CODE_FILE" > "$LISTS/scope.txt"; printf 'SnapshotRunner\n' > "$LISTS/words.txt" run_gate --scope "$LISTS/scope.txt" expect_exit 0 && expect_has "no doc in scope" \ && { run_gate --scope "$LISTS/scope.txt" --words "$LISTS/words.txt"; expect_exit 1; } \ - && expect_has "backticked \`SnapshotRunner\` does not resolve" && ok + && expect_has "backticked \`SnapshotRunner\` does not resolve" && expect_not "OtherGhost" && ok finish begin "Q2 in a git work tree a token resolves from tracked and untracked files, never from ignored ones" diff --git a/ts-react-linter-driven-development/scripts/ldd-detect.sh b/ts-react-linter-driven-development/scripts/ldd-detect.sh index 10b8a856..943e7018 100755 --- a/ts-react-linter-driven-development/scripts/ldd-detect.sh +++ b/ts-react-linter-driven-development/scripts/ldd-detect.sh @@ -48,11 +48,11 @@ # Detect-gate: Q the R9 gate's [Q] report lines, from one run of # check-repo-brain.sh beside this script over --root. On a # scoped bundle (no dirs.txt) the gate runs with --scope: it -# checks only the bundle's docs (docs.txt), the docs that -# link to a scope file and the docs that cite a symbol the -# diff removes, the code edges of the scope's code files, -# and no root wiring; with no such doc its doc checks are -# skipped. The repository-wide findings belong to a +# checks the bundle's docs (docs.txt) in full, and in the +# docs that name a scope file or cite a symbol the diff +# removes only those links and symbols; it checks the code +# edges of the scope's code files and no root wiring; with +# no such doc its doc checks are skipped. The repository-wide findings belong to a # whole-repository review # Detect: judgment no mechanical lead; the hunter runs the prose # A question with no detect line, two of them, or a malformed one is an error: @@ -521,7 +521,8 @@ run_path() { # # of docs.txt; and the identifiers the diff removes — on its removed lines and # on none of its added ones — the symbols a renamed or deleted declaration # takes away from the docs that cite them. Comment lines (a line that opens -# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped. +# with //, #, /*, *, -- or """) carry prose, not symbols, and are skipped, +# and a word the scope's files still carry after the change is not removed. gate_scope_lists() { local scope="$1" words="$2" line f { @@ -543,7 +544,18 @@ gate_scope_lists() { if (s == "+") add[w] = 1; else del[w] = 1 } } - END { for (w in del) if (!(w in add)) print w }' | LC_ALL=C sort > "$words" + END { for (w in del) if (!(w in add)) print w }' \ + | LC_ALL=C sort > "$words.removed" + if (( ${#ALL_FILES[@]} > 0 )); then + cat -- "${ALL_FILES[@]}" | LC_ALL=C tr -cs 'A-Za-z0-9_' '\n' \ + | LC_ALL=C awk -v list="$words.removed" ' + BEGIN { while ((getline w < list) > 0) gone[w] = 1 } + ($0 in gone) { delete gone[$0] } + END { for (w in gone) print w }' | LC_ALL=C sort > "$words" + else + cp "$words.removed" "$words" + fi + rm -f "$words.removed" } # The R9 gate runs once, lazily; its [Q] lines are split by question. With diff --git a/ts-react-linter-driven-development/scripts/ldd-detect_test.sh b/ts-react-linter-driven-development/scripts/ldd-detect_test.sh index 5b7c517a..43945b59 100644 --- a/ts-react-linter-driven-development/scripts/ldd-detect_test.sh +++ b/ts-react-linter-driven-development/scripts/ldd-detect_test.sh @@ -550,6 +550,25 @@ pkill -f 'sleep 37' 2>/dev/null rm -rf "$FAKE" finish +begin "a --gate-budget the gate finishes within writes the same counts as no budget" +mk_repo; : > "$BUNDLE/dirs.txt"; run_detect +cp "$BUNDLE/counts.tsv" "$BUNDLE/first.tsv" +run_detect --gate-budget 60 +expect_exit 0 && { cmp -s "$BUNDLE/first.tsv" "$BUNDLE/counts.tsv" || bad "counts.tsv differs under a budget the gate meets"; } \ + && cmp -s "$BUNDLE/first.tsv" "$BUNDLE/counts.tsv" && ok +finish + +begin "a gate that fails under --gate-budget fails the run (exit 2)" +mk_repo; : > "$BUNDLE/dirs.txt" +FAKE=$(mktemp -d) +cp "$DETECT" "$FAKE/ldd-detect.sh" +printf '#!/usr/bin/env bash\necho "check-repo-brain: internal scanner error"\nexit 2\n' > "$FAKE/check-repo-brain.sh" +OUT=$(cd "$REPO" && bash "$FAKE/ldd-detect.sh" --rules "$RULES" --gate-budget 30 --root . "$BUNDLE" 2>"$BUNDLE/.stderr"); CODE=$? +ERR=$(cat "$BUNDLE/.stderr"); rm -f "$BUNDLE/.stderr" +expect_exit 2 && expect_has "the R9 gate failed to run" && ok +rm -rf "$FAKE" +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/ts-react-linter-driven-development/skills/pre-commit-review/SKILL.md b/ts-react-linter-driven-development/skills/pre-commit-review/SKILL.md index 6fa70cbd..1f5582bd 100644 --- a/ts-react-linter-driven-development/skills/pre-commit-review/SKILL.md +++ b/ts-react-linter-driven-development/skills/pre-commit-review/SKILL.md @@ -80,10 +80,10 @@ writes `hits.tsv`, `hits-all.tsv` and `counts.tsv` into the bundle, and prints t question only a hunter's reading answers — then one line per rule family. That table, about 300 tokens, is the pre-filter: the parent prints no rule file, composes no detection command, and never reads a Falsifying-questions section. On a scoped review the R9 -gate checks only the docs the diff touches, the docs that link to a scope file and the -docs that cite a symbol the diff removes, and its rows are about those docs and the -scope's own files — an orphan doc elsewhere or an unwired root is the repository's -state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line +gate checks the docs the diff touches in full; in a doc that links to a scope file +or cites a symbol the diff removes it checks only those links and symbols; and it +checks the code edges of the scope's own files — an orphan doc elsewhere, a citing +doc's older drift or an unwired root is the repository's state, for a `--all` review or BROADER CONTEXT, never this diff's finding. The line `R9 gate: …` under the summary says which docs the gate checked. A gate row whose hits cell is `inconclusive` means the gate ran out of its `--gate-budget`: it is not a `0`, and the documentation hunter answers that question from the prose. When the From f535da6fe26d7262c450b007db915b95b15bfa5c Mon Sep 17 00:00:00 2001 From: Dan Mordechay Date: Sun, 18 Oct 2026 21:56:36 +0300 Subject: [PATCH 3/5] ldd-scope: a comment marker inside a string literal is not a comment On a D review in wekapp, comments.txt listed trace format strings such as DEBUG!"#CROSSCONTAINER ..." as comments. ldd-scope.sh now counts a comment marker only at the line start or after whitespace, so a # or // inside a string literal is not a comment. Co-Authored-By: Claude Opus 5.5 --- core/scripts/ldd-scope.sh | 8 +++++--- core/scripts/ldd-scope_test.sh | 11 +++++++++++ go-linter-driven-development/scripts/ldd-scope.sh | 8 +++++--- .../scripts/ldd-scope_test.sh | 11 +++++++++++ linter-driven-development/scripts/ldd-scope.sh | 8 +++++--- linter-driven-development/scripts/ldd-scope_test.sh | 11 +++++++++++ python-linter-driven-development/scripts/ldd-scope.sh | 8 +++++--- .../scripts/ldd-scope_test.sh | 11 +++++++++++ .../scripts/ldd-scope.sh | 8 +++++--- .../scripts/ldd-scope_test.sh | 11 +++++++++++ 10 files changed, 80 insertions(+), 15 deletions(-) diff --git a/core/scripts/ldd-scope.sh b/core/scripts/ldd-scope.sh index 8f9063f3..8b14afcc 100755 --- a/core/scripts/ldd-scope.sh +++ b/core/scripts/ldd-scope.sh @@ -281,11 +281,13 @@ write_bundle() { # comments.txt — the critic's inventory: on a scoped rung the diff's added # comment lines plus every comment line of a bundled file the diff does not # touch (untracked, or committed and named on the command line); every comment - # line of the bundle on --all; directives never count. + # line of the bundle on --all; directives never count. A marker counts at the + # line start or after whitespace only, so a # or // inside a string literal + # (a trace format string) is not a comment. # comment_lines_of ... — every comment line of the files, grep -nH shape comment_lines_of() { (( $# > 0 )) || return 0 - LC_ALL=C grep -nHE -e "$LANG_COMMENT_RE" -- "$@" 2>/dev/null | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" + LC_ALL=C grep -nHE -e "(^|[[:space:]])($LANG_COMMENT_RE)" -- "$@" 2>/dev/null | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" } : > "$OUT/comments.txt" if [[ "$RUNG" == "all" ]]; then @@ -299,7 +301,7 @@ write_bundle() { /^\+/ { n++; print f ":" n ":" substr($0, 2); next } /^-/ { next } { n++ } - ' "$OUT/diff.patch" | LC_ALL=C grep -E -- "^[^:]*:[0-9]+:.*($LANG_COMMENT_RE)" | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" >> "$OUT/comments.txt" + ' "$OUT/diff.patch" | LC_ALL=C grep -E -- "^[^:]*:[0-9]+:(.*[[:space:]])?($LANG_COMMENT_RE)" | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" >> "$OUT/comments.txt" fi # a bundled file the diff does not touch has no added lines: every comment # line of it is the critic's (an untracked file, or a committed file named diff --git a/core/scripts/ldd-scope_test.sh b/core/scripts/ldd-scope_test.sh index d138e182..f68cae87 100755 --- a/core/scripts/ldd-scope_test.sh +++ b/core/scripts/ldd-scope_test.sh @@ -215,6 +215,17 @@ else fi finish +begin "a comment marker inside a string literal is not a comment line" +mk_repo; mkdir -p "$REPO/lib" +{ fx_comment_line; printf 'x = "#tag" + "//tag"\n'; } > "$REPO/lib/thing.zz" +git_commit_repo +run_scope --glob '*.zz' lib/thing.zz +expect_exit 0 && { grep -q "added by the test" "$BUNDLE/scope-out/comments.txt" || bad "the real comment line is missing"; } \ + && grep -q "added by the test" "$BUNDLE/scope-out/comments.txt" \ + && { ! grep -q '^lib/thing.zz:2:' "$BUNDLE/scope-out/comments.txt" || bad "the string literal line is listed as a comment"; } \ + && ! grep -q '^lib/thing.zz:2:' "$BUNDLE/scope-out/comments.txt" && ok +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/go-linter-driven-development/scripts/ldd-scope.sh b/go-linter-driven-development/scripts/ldd-scope.sh index 8a99047d..309a45c3 100755 --- a/go-linter-driven-development/scripts/ldd-scope.sh +++ b/go-linter-driven-development/scripts/ldd-scope.sh @@ -387,11 +387,13 @@ write_bundle() { # comments.txt — the critic's inventory: on a scoped rung the diff's added # comment lines plus every comment line of a bundled file the diff does not # touch (untracked, or committed and named on the command line); every comment - # line of the bundle on --all; directives never count. + # line of the bundle on --all; directives never count. A marker counts at the + # line start or after whitespace only, so a # or // inside a string literal + # (a trace format string) is not a comment. # comment_lines_of ... — every comment line of the files, grep -nH shape comment_lines_of() { (( $# > 0 )) || return 0 - LC_ALL=C grep -nHE -e "$LANG_COMMENT_RE" -- "$@" 2>/dev/null | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" + LC_ALL=C grep -nHE -e "(^|[[:space:]])($LANG_COMMENT_RE)" -- "$@" 2>/dev/null | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" } : > "$OUT/comments.txt" if [[ "$RUNG" == "all" ]]; then @@ -405,7 +407,7 @@ write_bundle() { /^\+/ { n++; print f ":" n ":" substr($0, 2); next } /^-/ { next } { n++ } - ' "$OUT/diff.patch" | LC_ALL=C grep -E -- "^[^:]*:[0-9]+:.*($LANG_COMMENT_RE)" | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" >> "$OUT/comments.txt" + ' "$OUT/diff.patch" | LC_ALL=C grep -E -- "^[^:]*:[0-9]+:(.*[[:space:]])?($LANG_COMMENT_RE)" | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" >> "$OUT/comments.txt" fi # a bundled file the diff does not touch has no added lines: every comment # line of it is the critic's (an untracked file, or a committed file named diff --git a/go-linter-driven-development/scripts/ldd-scope_test.sh b/go-linter-driven-development/scripts/ldd-scope_test.sh index fe5d9fc3..b974bd5a 100755 --- a/go-linter-driven-development/scripts/ldd-scope_test.sh +++ b/go-linter-driven-development/scripts/ldd-scope_test.sh @@ -478,6 +478,17 @@ else fi finish +begin "a comment marker inside a string literal is not a comment line" +mk_repo; mkdir -p "$REPO/lib" +{ fx_comment_line; printf 'x = "#tag" + "//tag"\n'; } > "$REPO/lib/thing.zz" +git_commit_repo +run_scope --glob '*.zz' lib/thing.zz +expect_exit 0 && { grep -q "added by the test" "$BUNDLE/scope-out/comments.txt" || bad "the real comment line is missing"; } \ + && grep -q "added by the test" "$BUNDLE/scope-out/comments.txt" \ + && { ! grep -q '^lib/thing.zz:2:' "$BUNDLE/scope-out/comments.txt" || bad "the string literal line is listed as a comment"; } \ + && ! grep -q '^lib/thing.zz:2:' "$BUNDLE/scope-out/comments.txt" && ok +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/linter-driven-development/scripts/ldd-scope.sh b/linter-driven-development/scripts/ldd-scope.sh index e0cd0995..d64de41c 100755 --- a/linter-driven-development/scripts/ldd-scope.sh +++ b/linter-driven-development/scripts/ldd-scope.sh @@ -503,11 +503,13 @@ write_bundle() { # comments.txt — the critic's inventory: on a scoped rung the diff's added # comment lines plus every comment line of a bundled file the diff does not # touch (untracked, or committed and named on the command line); every comment - # line of the bundle on --all; directives never count. + # line of the bundle on --all; directives never count. A marker counts at the + # line start or after whitespace only, so a # or // inside a string literal + # (a trace format string) is not a comment. # comment_lines_of ... — every comment line of the files, grep -nH shape comment_lines_of() { (( $# > 0 )) || return 0 - LC_ALL=C grep -nHE -e "$LANG_COMMENT_RE" -- "$@" 2>/dev/null | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" + LC_ALL=C grep -nHE -e "(^|[[:space:]])($LANG_COMMENT_RE)" -- "$@" 2>/dev/null | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" } : > "$OUT/comments.txt" if [[ "$RUNG" == "all" ]]; then @@ -521,7 +523,7 @@ write_bundle() { /^\+/ { n++; print f ":" n ":" substr($0, 2); next } /^-/ { next } { n++ } - ' "$OUT/diff.patch" | LC_ALL=C grep -E -- "^[^:]*:[0-9]+:.*($LANG_COMMENT_RE)" | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" >> "$OUT/comments.txt" + ' "$OUT/diff.patch" | LC_ALL=C grep -E -- "^[^:]*:[0-9]+:(.*[[:space:]])?($LANG_COMMENT_RE)" | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" >> "$OUT/comments.txt" fi # a bundled file the diff does not touch has no added lines: every comment # line of it is the critic's (an untracked file, or a committed file named diff --git a/linter-driven-development/scripts/ldd-scope_test.sh b/linter-driven-development/scripts/ldd-scope_test.sh index 9ef9d3bd..8e8abce8 100755 --- a/linter-driven-development/scripts/ldd-scope_test.sh +++ b/linter-driven-development/scripts/ldd-scope_test.sh @@ -544,6 +544,17 @@ else fi finish +begin "a comment marker inside a string literal is not a comment line" +mk_repo; mkdir -p "$REPO/lib" +{ fx_comment_line; printf 'x = "#tag" + "//tag"\n'; } > "$REPO/lib/thing.zz" +git_commit_repo +run_scope --glob '*.zz' lib/thing.zz +expect_exit 0 && { grep -q "added by the test" "$BUNDLE/scope-out/comments.txt" || bad "the real comment line is missing"; } \ + && grep -q "added by the test" "$BUNDLE/scope-out/comments.txt" \ + && { ! grep -q '^lib/thing.zz:2:' "$BUNDLE/scope-out/comments.txt" || bad "the string literal line is listed as a comment"; } \ + && ! grep -q '^lib/thing.zz:2:' "$BUNDLE/scope-out/comments.txt" && ok +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/python-linter-driven-development/scripts/ldd-scope.sh b/python-linter-driven-development/scripts/ldd-scope.sh index 1aa3e227..b79364d5 100755 --- a/python-linter-driven-development/scripts/ldd-scope.sh +++ b/python-linter-driven-development/scripts/ldd-scope.sh @@ -390,11 +390,13 @@ write_bundle() { # comments.txt — the critic's inventory: on a scoped rung the diff's added # comment lines plus every comment line of a bundled file the diff does not # touch (untracked, or committed and named on the command line); every comment - # line of the bundle on --all; directives never count. + # line of the bundle on --all; directives never count. A marker counts at the + # line start or after whitespace only, so a # or // inside a string literal + # (a trace format string) is not a comment. # comment_lines_of ... — every comment line of the files, grep -nH shape comment_lines_of() { (( $# > 0 )) || return 0 - LC_ALL=C grep -nHE -e "$LANG_COMMENT_RE" -- "$@" 2>/dev/null | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" + LC_ALL=C grep -nHE -e "(^|[[:space:]])($LANG_COMMENT_RE)" -- "$@" 2>/dev/null | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" } : > "$OUT/comments.txt" if [[ "$RUNG" == "all" ]]; then @@ -408,7 +410,7 @@ write_bundle() { /^\+/ { n++; print f ":" n ":" substr($0, 2); next } /^-/ { next } { n++ } - ' "$OUT/diff.patch" | LC_ALL=C grep -E -- "^[^:]*:[0-9]+:.*($LANG_COMMENT_RE)" | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" >> "$OUT/comments.txt" + ' "$OUT/diff.patch" | LC_ALL=C grep -E -- "^[^:]*:[0-9]+:(.*[[:space:]])?($LANG_COMMENT_RE)" | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" >> "$OUT/comments.txt" fi # a bundled file the diff does not touch has no added lines: every comment # line of it is the critic's (an untracked file, or a committed file named diff --git a/python-linter-driven-development/scripts/ldd-scope_test.sh b/python-linter-driven-development/scripts/ldd-scope_test.sh index fbd7a479..ad12c73b 100755 --- a/python-linter-driven-development/scripts/ldd-scope_test.sh +++ b/python-linter-driven-development/scripts/ldd-scope_test.sh @@ -471,6 +471,17 @@ else fi finish +begin "a comment marker inside a string literal is not a comment line" +mk_repo; mkdir -p "$REPO/lib" +{ fx_comment_line; printf 'x = "#tag" + "//tag"\n'; } > "$REPO/lib/thing.zz" +git_commit_repo +run_scope --glob '*.zz' lib/thing.zz +expect_exit 0 && { grep -q "added by the test" "$BUNDLE/scope-out/comments.txt" || bad "the real comment line is missing"; } \ + && grep -q "added by the test" "$BUNDLE/scope-out/comments.txt" \ + && { ! grep -q '^lib/thing.zz:2:' "$BUNDLE/scope-out/comments.txt" || bad "the string literal line is listed as a comment"; } \ + && ! grep -q '^lib/thing.zz:2:' "$BUNDLE/scope-out/comments.txt" && ok +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development diff --git a/ts-react-linter-driven-development/scripts/ldd-scope.sh b/ts-react-linter-driven-development/scripts/ldd-scope.sh index 9c8d5832..2b7f9243 100755 --- a/ts-react-linter-driven-development/scripts/ldd-scope.sh +++ b/ts-react-linter-driven-development/scripts/ldd-scope.sh @@ -400,11 +400,13 @@ write_bundle() { # comments.txt — the critic's inventory: on a scoped rung the diff's added # comment lines plus every comment line of a bundled file the diff does not # touch (untracked, or committed and named on the command line); every comment - # line of the bundle on --all; directives never count. + # line of the bundle on --all; directives never count. A marker counts at the + # line start or after whitespace only, so a # or // inside a string literal + # (a trace format string) is not a comment. # comment_lines_of ... — every comment line of the files, grep -nH shape comment_lines_of() { (( $# > 0 )) || return 0 - LC_ALL=C grep -nHE -e "$LANG_COMMENT_RE" -- "$@" 2>/dev/null | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" + LC_ALL=C grep -nHE -e "(^|[[:space:]])($LANG_COMMENT_RE)" -- "$@" 2>/dev/null | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" } : > "$OUT/comments.txt" if [[ "$RUNG" == "all" ]]; then @@ -418,7 +420,7 @@ write_bundle() { /^\+/ { n++; print f ":" n ":" substr($0, 2); next } /^-/ { next } { n++ } - ' "$OUT/diff.patch" | LC_ALL=C grep -E -- "^[^:]*:[0-9]+:.*($LANG_COMMENT_RE)" | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" >> "$OUT/comments.txt" + ' "$OUT/diff.patch" | LC_ALL=C grep -E -- "^[^:]*:[0-9]+:(.*[[:space:]])?($LANG_COMMENT_RE)" | LC_ALL=C grep -vE -- "$LANG_DIRECTIVE_RE" >> "$OUT/comments.txt" fi # a bundled file the diff does not touch has no added lines: every comment # line of it is the critic's (an untracked file, or a committed file named diff --git a/ts-react-linter-driven-development/scripts/ldd-scope_test.sh b/ts-react-linter-driven-development/scripts/ldd-scope_test.sh index 76464f40..1f11e2e6 100755 --- a/ts-react-linter-driven-development/scripts/ldd-scope_test.sh +++ b/ts-react-linter-driven-development/scripts/ldd-scope_test.sh @@ -477,6 +477,17 @@ else fi finish +begin "a comment marker inside a string literal is not a comment line" +mk_repo; mkdir -p "$REPO/lib" +{ fx_comment_line; printf 'x = "#tag" + "//tag"\n'; } > "$REPO/lib/thing.zz" +git_commit_repo +run_scope --glob '*.zz' lib/thing.zz +expect_exit 0 && { grep -q "added by the test" "$BUNDLE/scope-out/comments.txt" || bad "the real comment line is missing"; } \ + && grep -q "added by the test" "$BUNDLE/scope-out/comments.txt" \ + && { ! grep -q '^lib/thing.zz:2:' "$BUNDLE/scope-out/comments.txt" || bad "the string literal line is listed as a comment"; } \ + && ! grep -q '^lib/thing.zz:2:' "$BUNDLE/scope-out/comments.txt" && ok +finish + } # run_row_cases write_installed_plugins "$SIBLINGS_FILE" go-linter-driven-development python-linter-driven-development linter-driven-development ts-react-linter-driven-development From 735cc5048db5939bcf2c9d2039e61632e40c669b Mon Sep 17 00:00:00 2001 From: Dan Mordechay Date: Sun, 18 Oct 2026 21:57:54 +0300 Subject: [PATCH 4/5] changelog: the scoped R9 gate, the gate budget and the comment fix Co-Authored-By: Claude Opus 5.5 --- go-linter-driven-development/CHANGELOG.md | 17 ++++++++++++++++ lang/generic/passthrough/CHANGELOG.md | 20 +++++++++++++++++++ lang/go/passthrough/CHANGELOG.md | 17 ++++++++++++++++ lang/python/passthrough/CHANGELOG.md | 20 +++++++++++++++++++ lang/ts-react/passthrough/CHANGELOG.md | 17 ++++++++++++++++ linter-driven-development/CHANGELOG.md | 20 +++++++++++++++++++ python-linter-driven-development/CHANGELOG.md | 20 +++++++++++++++++++ .../CHANGELOG.md | 17 ++++++++++++++++ 8 files changed, 148 insertions(+) diff --git a/go-linter-driven-development/CHANGELOG.md b/go-linter-driven-development/CHANGELOG.md index ef2e41a6..3e4150ea 100644 --- a/go-linter-driven-development/CHANGELOG.md +++ b/go-linter-driven-development/CHANGELOG.md @@ -33,6 +33,23 @@ Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versio gap linter applies: the repository linter's rule where it has one, else 450 code lines per file and 8 and 13 files per directory. The package-size procedure runs the gap linter in place of a hand-written `find` command. +- **A scoped review runs the documentation gate on its scope only.** The gate + checks the docs the diff touches in full. In a doc that names a scope file or + cites a symbol the diff removes, it reports only those links and symbols. A + code-only diff that no doc cites skips the doc checks. On a 6-file diff in a + large monorepo, the detection pass took 46 seconds, down from about 90 minutes. +- **Doc symbols resolve in one pass over the repository.** In a git work tree the + gate reads the tracked files, submodule files and untracked files that git does + not ignore. Outside git it skips `bazel-*`, `node_modules` and other build + output directories. +- **`ldd-detect.sh --gate-budget `** stops a gate that runs too long and + counts its questions `inconclusive`, never 0. +- **The two review scripts cannot disagree.** `ldd-detect.sh` reads the + `--test-re` that `ldd-scope.sh` used. A `--glob` or `--test-re` that differs + from the bundle is a usage error. A scope whose files are not source files of + its language is an error, not a table of zeros. +- **`comments.txt` skips string literals.** A `#` or `//` inside a string literal + is not a comment line. ### Removed diff --git a/lang/generic/passthrough/CHANGELOG.md b/lang/generic/passthrough/CHANGELOG.md index 80a07491..a42f862f 100644 --- a/lang/generic/passthrough/CHANGELOG.md +++ b/lang/generic/passthrough/CHANGELOG.md @@ -33,6 +33,23 @@ Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versio gap linter applies: the repository linter's rule where it has one, else 450 code lines per file and 8 and 13 files per directory. The package-size procedure runs the gap linter in place of a hand-written `find` command. +- **A scoped review runs the documentation gate on its scope only.** The gate + checks the docs the diff touches in full. In a doc that names a scope file or + cites a symbol the diff removes, it reports only those links and symbols. A + code-only diff that no doc cites skips the doc checks. On a 6-file diff in a + large monorepo, the detection pass took 46 seconds, down from about 90 minutes. +- **Doc symbols resolve in one pass over the repository.** In a git work tree the + gate reads the tracked files, submodule files and untracked files that git does + not ignore. Outside git it skips `bazel-*`, `node_modules` and other build + output directories. +- **`ldd-detect.sh --gate-budget `** stops a gate that runs too long and + counts its questions `inconclusive`, never 0. +- **The two review scripts cannot disagree.** `ldd-detect.sh` reads the + `--test-re` that `ldd-scope.sh` used. A `--glob` or `--test-re` that differs + from the bundle is a usage error. A scope whose files are not source files of + its language is an error, not a table of zeros. +- **`comments.txt` skips string literals.** A `#` or `//` inside a string literal + is not a comment line. ### Fixed @@ -44,6 +61,9 @@ Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versio `ldd-detect.sh` or `ldd-scope.sh` got an option such as `--root` or `--base` as its last argument, the script looped forever. It now stops with "