Add test-gap-audit and docs-sync-audit skills - #2951
Merged
aaronpowell merged 3 commits intoSep 7, 2026
Merged
Conversation
Two repo-agnostic review skills that answer questions the existing testing and documentation skills do not. test-gap-audit asks which behaviour is not covered, rather than how to write a test in a given framework. Given no scope it audits the whole repository, inventories the testable surfaces, and reports which routes, services, jobs and contracts have no tests, too few assertions, or only indirect coverage. It bundles coverage_map.py, which detects the test framework and naming convention, then matches every source file to tests by name, by mirrored path, and by what the test files actually import, and ranks the unmatched by risk keyword and size. docs-sync-audit compares what the docs claim against what the code does. It bundles docs_drift.py, which checks documented npm scripts and make targets against the ones that exist, relative Markdown links against the filesystem, and environment variable names in both directions. It also reports a documented setting that is read only inside a module nothing imports, which is configuration that reads as working but cannot take effect. Both are read-only: they report and do not edit unless asked. Both emit the same contract, so a finding always carries a P0-P3 severity and a path:line you can open. Both scripts are Python standard library only, install nothing, and are accelerators rather than requirements, so each skill still works when the script cannot run. The existing testing and docs skills here are framework-specific, which is where most of the value is. These are the repo-agnostic complement: pytest-coverage raises coverage inside a pytest project, and this decides where coverage is missing across a repository regardless of language.
github-actions
Bot
added
new-submission
Contributor
🔒 PR Risk Scan ResultsScanned 5 changed file(s).
|
Contributor
🔍 Vally Lint Results✅ All checks passed
Summary
Full linter output |
Two CI failures on the first push. codespell flagged `testng` and `shouldBe` in coverage_map.py. Both are legitimate identifiers rather than typos: TestNG is the Java test framework the script detects by name, and shouldBe is the Kotlin and Scala assertion method matched by its assertion-detection regex. Added both to ignore-words-list with a comment each, following the convention already used for the other entries. validate-readme failed because I had reverted docs/README.agents.md. `npm start` rewrites a Dynatrace MCP URL there from re-fetched external plugin data, which is unrelated to these skills, so I had excluded it to keep the diff scoped. That was wrong: the check regenerates every generated file and compares, so the commit has to carry whatever the build produces. Restored.
aaronpowell
enabled auto-merge (squash)
September 7, 2026 02:51
aaronpowell
approved these changes
Sep 7, 2026
aaronpowell
pushed a commit
that referenced
this pull request
Sep 7, 2026
…as evidence (#2976) Ports four corrections these two skills received upstream after a second trial run against a real repository. #2951 merged the snapshot taken before them. The bundled scripts are already identical to their upstream versions and are not touched here. 1. Neither skill told the agent that text read out of the audited repository is data rather than instruction. These skills exist to read untrusted repositories, so a README, a code comment, a commit message or a dependency manifest reached the model with no framing -- and a line claiming a file is approved, or telling the audit to skip a module, reads exactly like a guardrail. Both skills now carry the rule and report such text as a finding instead of following it. 2. The citation rule allowed anchors to land beside the symbol rather than on it: the blank line above a definition, a decorator, or a line inside a multi-line literal. In one trialled file every anchor sat two lines above the def it named. The rule is now a single applicable test -- the line you cite must literally contain the thing you name, and a cited range must contain it on the first line. Quoted text is cited at the line the quoted characters are on, because a comment has its own line number and it is usually not the line of the code beside it. 3. "Never restate a count without the raw output in front of you" was ignored twice in that trial, so the rule flips from prohibition to requirement: any number stated must appear under Checks Run next to the command that produced it. Unwilling to show the command means describing the pattern rather than counting it. 4. Both Related Skills sections said the skill is one of seven and that the other five cover the remaining ground. Six, not five. Each section now names its sibling in this repository and links the remaining five out. Front matter is unchanged, so the generated README tables do not move.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two repo-agnostic review skills that answer questions the existing testing and documentation skills here don't.
What they do
test-gap-auditasks which behaviour is not covered, rather than how to write a test in a given framework. Given no scope it audits the whole repository: inventories the testable surfaces, then reports which routes, services, jobs and contracts have no tests, too few assertions, or only indirect coverage. Name a feature, PR or branch and it narrows to that.It bundles
coverage_map.py, which detects the test framework and naming convention, then matches every source file to tests three ways — by name, by mirrored path, and by what the test files actually import — and ranks the unmatched by risk keyword and size. The import scan is what makes "no tests found for X" worth reporting rather than merely plausible.docs-sync-auditcompares what the docs claim against what the code does. It bundlesdocs_drift.py, which checks documentednpm runscripts andmaketargets against the ones that exist, relative Markdown links against the filesystem, and environment variable names in both directions.It also reports a case a name-only comparison misses: a documented setting that is read by the code, but only inside a module nothing imports — configuration that reads as working and cannot take effect.
How they relate to what's already here
The existing testing and docs skills are framework-specific, which is where most of their value is. These are the repo-agnostic complement rather than an overlap:
pytest-coverageraises coverage to 100% inside a pytest project.test-gap-auditdecides where coverage is missing across a repository, in any language, and does not run tests.documentation-writerandcreate-readmewrite documentation.docs-sync-auditchecks whether existing documentation still matches the code.Neither restates something a frontier model already does well unprompted. The uplift is the discipline: a fixed report contract, and deterministic tooling for the parts a model does expensively and unreliably by hand — matching hundreds of source files to tests, or re-deriving which documented commands actually exist.
Conventions followed
P0–P3severity and apath:lineyou can open.npm run skill:validatepasses (418/418), andnpm starthas been run — the only generated change is the two new rows indocs/README.skills.md.One note on the build:
npm startalso rewrites a Dynatrace MCP URL indocs/README.agents.md, which comes from re-fetching external plugin data and is unrelated to this change. I reverted that file so this PR stays scoped to the skills.