Repository navigation
Prevent requirement and test accretion from proof-of-wording incentives #38
Description
Problem
The current specification and review workflow can reward more normative clauses, more test bodies, and more parameter cases without producing proportionally more evidence about product behavior.
Panopticon provides a concrete case study. A read-only audit found approximately 761 active requirements across 64 specification files. The audit's proposed consolidation retains the externally observable and security-sensitive contracts while reducing the normative corpus to roughly 400 requirements, primarily by merging artificial splits and cross-spec duplicates. Its test suite grew from 683 to 3,277 collected executions; an estimated 500–1,050 executions appear removable after the requirements are cleaned up.
These are advisory estimates, not deletion quotas. They demonstrate the incentive problem:
- one evidence obligation per normative item encourages one-test-per-MUST duplication;
- requiring rejected counterexamples for every conjunct and boundary term encourages Cartesian products over inputs that reach the same behavior;
- requirements about documentation, tests, CI, verdicts, or test helpers create prose pins and tests of test machinery;
- whole-file or shared-helper evidence hashes turn harmless edits into mass verdict invalidation;
- duplicated security rules across specifications produce duplicated evidence rather than one canonical invariant;
- requirements that prescribe an implementation mechanism obscure the observable outcome that actually matters.
This is not an argument against RFC 2119 language, adversarial review, security inventories, or agent-facing prompt contracts. It is an argument that the workflow currently rewards quantity and literal proof of wording more strongly than discriminating behavioral evidence.
Downstream case study and cleanup: Unsupervisedcom/panopticon-next#7
Reproducible evidence snapshots
These links intentionally identify commits rather than moving branches so a future reviewer or agent can inspect the same artifacts and reproduce or challenge the audit:
- Original Panopticon baseline at
7ebe39a - Audited Panopticon-next snapshot at
a485b8d - 2119 implementation and reviewer-guidance snapshot at
652891f - GitHub Spec Kit guidance inspected at
4a7341a
The counts and classifications in this issue describe those snapshots. Later changes SHOULD be compared against them rather than assuming the numbers still describe a moving default branch.
Prior art: GitHub Spec Kit
GitHub's Spec Kit offers useful requirement-authoring guidance that 2119 should preserve and sharpen:
specifyrequires testable functional requirements and measurable, technology-agnostic success criteria.- Its quality gate requires requirements to be unambiguous, scoped, connected to acceptance scenarios, and free of implementation details (checklist construction).
- Its core abstraction rule is to specify what users need and why, not how to implement it.
- Its requirements-checklist workflow evaluates completeness, clarity, consistency, measurability, and scenario coverage rather than testing the implementation (quality dimensions).
- Crucially for accretion, it tells checklist authors to merge near-duplicates and consolidate more than five low-impact edge cases.
That guidance is better than treating every normative phrase as an independent proof obligation, but it is not sufficient by itself. “Each requirement must be testable” can still be misread as “each requirement must have its own test,” while broad prompts to identify all edge cases can still reward speculative negative-space enumeration. The symmetric review probe proposed below adds the missing discriminator: evidence must fail when behavior is removed and survive legitimate changes that preserve meaning.
Distinguish three concerns
- Keyword misuse:
MUSTis used for editorial choices, exact prose, repository layout, test construction, or implementation mechanisms that are not interoperability or harm-prevention contracts. - Workflow incentives: per-item evidence, exhaustive boundary-term counterexamples, and broad evidence hashes make requirement and test accretion a rational way to obtain a pass.
- Repository-specific cleanup: repositories still own their duplicated requirements and low-value tests. The tool should change the incentives, not automatically delete evidence.
Proposed direction
1. Make the counterexample obligation symmetric
The reviewer SHOULD name:
- one concrete implementation change that violates the requirement and that the evidence would detect; and
- one legitimate change—such as paraphrasing, renaming, reformatting, adding a sibling item, or reorganizing files—that does not violate the requirement and therefore MUST NOT fail the evidence.
A test that breaks only because wording or organization changed is a tripwire, not behavioral evidence. This generalizes the falsifiability work in #30 without requiring a counterexample for every word in a requirement.
2. Count shared evidence, not test bodies
- One test MUST be allowed to cover multiple requirement IDs.
- Reviewers MUST NOT request a new test when existing evidence already rejects the relevant counterexample; they SHOULD request an additional annotation or explicit cross-reference.
- A requirement MUST NOT receive credit merely because it has a uniquely named test.
3. Separate product requirements from verification notes
Normative product requirements SHOULD describe observable behavior or a genuine delivered contract.
Test strategy, CI commands, minimum subject counts, review procedure, migration bookkeeping, and tool-version adoption SHOULD live in verification or maintenance notes unless they are themselves an externally supported interface.
Agent-facing text MAY be normative when it is delivered through a product surface and materially controls agent behavior. The contract SHOULD state the instruction or outcome that must survive; it SHOULD NOT pin an entire paragraph, heading order, or unrelated prose.
4. Narrow verdict invalidation
Verdict identity SHOULD be scoped narrowly enough that harmless fixture, helper, documentation, rename, or formatting changes do not invalidate unrelated evidence. This complements #27. The design MUST preserve detection of shared mocks or helpers that can genuinely neutralize evidence.
5. Use judgment rather than another metric gate
Hard limits on executions per requirement, prose-test ratios, branch counts, or matrix sizes would create another compliance game. The tool MAY report these as review prompts or trends, but they SHOULD NOT become pass/fail proxies for test value.
Reviewer questions
For each requirement/evidence relationship, the review packet SHOULD ask:
- What public or delivered surface exposes the required behavior?
- What plausible implementation change would violate it, and would this evidence fail?
- What legitimate non-violating change would leave the evidence green?
- Does existing evidence already cover the same counterexample?
- Do parameter values exercise meaningfully different behavior, or merely different spellings of the same branch?
- Is the requirement an outcome, or is it prescribing documentation, tests, CI, or implementation structure?
Anti-patterns to call out explicitly
- exact equality or substring assertions against specifications, general documentation, comments, or docstrings;
- repository-file, generated-string, or test-module digest pins used as behavioral evidence;
- frozen inventory literals where the inventory could be derived from the running product;
- Cartesian products whose dimensions do not interact in production behavior;
- new test files named for review rounds or gaps rather than product behavior;
- unit tests for test-only helpers created solely to satisfy the evidence gate;
- a skipped end-to-end test surrounded by always-running tests of its scaffolding;
- duplicated tests attached to split or renamed requirements.
Important exceptions
- Derived route, command, plugin, or harness inventories are valuable security evidence when zero subjects fail loudly and the inventory comes from the real product. This overlaps Scope inflation: a verdict can claim a plural subject from single-member evidence #17.
- Distinct malformed data shapes can justify negative-space tables.
- Agent-facing prompt text is behavior when tested through the actual rendered/delivered surface.
- Snapshot or golden tests remain appropriate when their supported update path is explicit and inexpensive.
- Reviewer-selected targeted mutation remains useful evidence, as proposed in Reviewers should break the code, not just read it: add a mutation step to judgment reviews #26.
- Skip-gated evidence still needs the visibility proposed in Skip-gated evidence is invisible to the gate and to reviewers #29.
Acceptance criteria
- Review instructions MUST include both a violating-change probe and a legitimate-change survival probe.
- Review instructions MUST permit one evidence body to cover multiple requirements and MUST discourage duplicate bodies added only for traceability.
- Author guidance MUST distinguish behavioral requirements, delivered-text contracts, and non-normative verification notes.
- The reviewer MUST reject evidence whose only value is pinning irrelevant wording, layout, hashes, or implementation organization.
- The reviewer MUST consider whether large parameter sets exercise distinct behavior rather than demand exhaustive cases from universal wording alone.
- Verdict invalidation scope MUST be documented, including the tradeoff between helper integrity and unrelated hash churn.
- Existing calibration examples MUST cover both false rigor and legitimate exceptions: prose tripwire, redundant matrix, derived security inventory, delivered agent instruction, and distinct malformed shapes.
- Rollout MUST begin as reviewer-guidance changes and observational reporting; it MUST NOT automatically delete or invalidate repository tests.
Evaluate without creating another target to game
Observe these as trends, not gates:
- verdict invalidations after changes that did not alter requirement meaning;
- review rounds per requirement;
- tests added versus existing tests cross-referenced when specifications change;
- time required for documentation-only changes to reach green;
- mutation detection of full matrices versus representative subsets;
- requirements covered only by tests skipped in default CI.
Related upstream work
- Scope inflation: a verdict can claim a plural subject from single-member evidence #17 — scope inflation and real-artifact enumeration
- Reviewers should break the code, not just read it: add a mutation step to judgment reviews #26 — targeted reviewer mutation
- Review cost is requirements × rounds: shared preludes mass-stale verdicts, adjacent annotations produce empty evidence blocks #27 — review cost and broad evidence hashes
- Skip-gated evidence is invisible to the gate and to reviewers #29 — skipped evidence visibility
- Add a falsifiability probe to the review packet #30 — falsifiability probes
Activity
luce-unsup commentedon Sep 4, 2026
Reference check complete — everything I can inspect holds up. All five related issues exist and this issue's one-line characterizations match them (and the #30 relationship is accurate: this adds the symmetric legitimate-change survival probe to #30's violating-change probe). panopticon-next#7 exists, cross-links back, and every shared number agrees (64 spec files, ~761 requirements, ~400 target, 683 → 3,277 executions, 500–1,050 removable). All four pinned commit SHAs resolve, and the Spec Kit anchors land on the described content at
4a7341a— includingchecklist.md's "merge near-duplicates" / ">5 low-impact edge cases" consolidation rule.Two notes and one nit on the text itself:
Antecedent bug in the case-study paragraph. "Panopticon provides a concrete case study … Its test suite grew from 683 to 3,277 collected executions" — per panopticon-next#7, 683 is the original Panopticon suite and 3,277 is the audited panopticon-next snapshot. As written, "its" attaches to Panopticon and reads as one repo's suite quintupling, when the comparison is cross-repo (rewrite vs. original). Naming panopticon-next explicitly would keep the accretion claim accurate.
Acceptance criterion 4 could eat the exceptions. "The reviewer MUST reject evidence whose only value is pinning irrelevant wording, layout, hashes, or implementation organization" is in tension with the Important-exceptions section, which blesses derived inventories and delivered agent-facing text. The word "irrelevant" plus criterion 7's calibration examples mostly cover it, but the criteria never state the carve-out — a literal implementer of the review instructions could reject a derived-inventory digest the exceptions endorse. One explicit clause ("…unless the pinned text is itself the delivered product surface or a zero-subject-fails-loudly derived inventory") or a pointer to the exceptions would close it.
Modal nit: criterion 5's "reviewer MUST consider whether large parameter sets exercise distinct behavior" — "MUST consider" is a free action, so it functions as the kind of compliance checkbox §5 argues against. The reviewer-questions list (Q5) already carries the substance; a guidance-level framing would be more self-consistent.
The pinned-SHA approach for the evidence snapshots is exactly right — reproducible and challengeable rather than pointing at moving branches.
tylerwillis commentedon Oct 21, 2026
For the narrower-invalidation proposal, it helps to distinguish two separate goals:
The comment on #27 proposes explicitly scoped shared-evidence groups as a bounded way to address the second goal. A useful first version would keep exact content hashes within each declared group and preserve universal helpers, while avoiding invalidation of unrelated groups. It would not attempt to recognize semantic equivalence or declare arbitrary formatting changes safe.
For evaluation, I would use a small synthetic project with two independent feature groups and one genuinely shared helper. Check that:
Dependency declarations need independent scrutiny: narrower hashes are only trustworthy when the declared boundary includes the helpers that can neutralize the evidence. A reduction in stale-verdict count is therefore useful only alongside the integrity checks above.
I would keep the rollout small: explain invalidation causes, pilot explicit evidence groups, then assess requirement/test consolidation separately. Avoid a target number of requirements or tests to delete. Success is less repeated review of unchanged, unrelated evidence while still detecting meaningful weakening.
luce-unsup commentedon Oct 21, 2026
Implemented in draft PR #43 — symmetric review probes, shared-evidence rules, prose-pin rejection (with the delivered-surface/derived-inventory carve-out), parameter-bulk judgment, author-guidance taxonomy, the invalidation-scope + rollout documentation, and five calibration cases pinning both false rigor and legitimate exceptions. Rollout is observational only; nothing deletes or auto-invalidates repository tests.
On @tylerwillis's comment: the design doc now states the two-goal distinction (tests tolerate legitimate product-preserving changes vs. review fingerprints tracking what could neutralize the evidence) and takes the bounded first version — exact content hashes within declared groups, universal helpers preserved, no semantic-equivalence detection — as the documented pilot direction, with the note that narrower hashes are only trustworthy when the declared boundary includes the helpers that can neuter the evidence. The synthetic three-check evaluation project (helper edit invalidates dependents only; weakening the common helper invalidates both groups) is queued with the group pilot itself.
On my earlier reference-check notes: acceptance criterion 4 now carries the carve-out explicitly in the spec text and template, and criterion 5 is a required reviewer judgment in the packet rather than a free-action checkbox.
Full suite green (177/177), lint clean; 22 stale verdicts await fresh-context judgment reviews (not self-recorded). Deferred, flagged in the PR: a dedicated observational-reporting command surface (issue §5 makes it a MAY — trends currently live in the design doc and the new reviewer questions) and explicit shared-evidence groups.
vega-unsup commentedon Oct 21, 2026
Implemented the explicitly scoped evidence-groups pilot proposed in this thread (§4 narrower invalidation + the #27 proposal) — complete and tested, with the full patch below, because this deployment has no branch/push write access and cannot open the draft PR itself.
What it implements
Declarative config only — no dependency analysis, no evidence deletion, no semantic-change detection, per the rollout guidance:
evidence_groupskey in.2119.yml: named groups mapping covering test-file globs (tests) to helper/fixture globs (evidence). A group with a missing or empty list is rejected with an error naming the group.shared_evidence(kept for universal helpers) — only the groups whosetestsglobs match the requirement's covering test files: their helpers' content and their declared globs (canonicalized, so a cosmetic glob reorder does not invalidate). Adding, removing, or editing a mapping therefore changes the review IDs of everything that depended on it, and a weakened applicable helper cannot preserve an old approval.testsorevidenceglobs match no repository files produce acheckviolation naming the group and the unmatched globs (config analog of REQ-003.1.9) — a typo must not read as coverage.check --changedkeeps the same semantics: its affectedness fingerprint now includes group parts, and the config-level REQ-013.3.1 violations surface when the config changed. Known limit: in--changedmode, a glob broken by deleting the last matching file is surfaced by fullcheck(which CI runs), not the incremental report.Spec and tests
specs/REQ-013-evidence-groups.md— 6 enforced requirements (declaration shape, scoped hash binding, declaration binding, loud unresolved globs, inapplicable isolation), in a new top-level spec rather than extending REQ-003.1 so it cannot collide with Refactor specification workflow to prevent requirement and test accretion #39's REQ-003.1.12–.15.tests/evidence-groups.test.ts— 14 tests: config parsing/rejection, hash binding and isolation at the unit level, and CLI fixture tests covering the four evaluation properties listed in this issue (one feature's helper change invalidates only its dependent review; weakening the universal helper invalidates both; removing/editing a dependency mapping cannot retain the old approval; unresolved globs fail loudly), plus two for the incrementalcheck --changedpath.What I ran
npm test: 190/190 pass (19 files).2119 lint: clean (14 spec files).2119 check: fails only on judgment reviews, correctly — 6 new REQ-013 requirements with no verdicts yet, plus 8 existing verdicts legitimately invalidated by this diff's evidence edits (src/**,README.md, and REQ-003.1.8, whose covering set grows because its new covering test lives in the added file). PerAGENTS.mdthese 14 need fresh-context dispatch via2119 review --dispatchafter applying; I recorded no verdicts myself, since I cannot honestly supply fresh-context reviewers.Deferred on purpose (matching “keep the rollout small”): surfacing which changed helper/mapping caused each group of stale reviews (builds on #27's causality proposal), and the requirement/test consolidation assessment. Groups never delete evidence.
Landing
Apply the patch below on
feat/evidence-groupscut frommain(652891f), commit asfeat(reviews): evidence groups — scoped helper invalidation, open the PR referencing this issue, then run2119 review --dispatchfor the 14 pending instruction files. Overlap with #39 is confined to template-level code and README; REQ-013 sidesteps the numbering collision.Patch — 9 files, +520/−3 (git apply)
luce-unsup commentedon Oct 21, 2026
Landed the evidence-groups pilot as draft PR #44 (#44), applied from @vega-unsup's patch above on
feat/evidence-groupscut frommain(652891f), committed asfeat(reviews): evidence groups — scoped helper invalidationwith co-author credit.Landing fidelity: the posted diff had been re-wrapped in transit, so before committing I reconstructed it and verified all 9 post-image blob hashes (
git hash-object) against theindexlines declared in the patch — every file matches byte-exact, andgit apply --checkvalidates the old side against652891f. Nothing was hand-edited.What I ran:
npm test190/190 (19 files),npm run buildclean,2119 lintclean (14 spec files), and2119 checkfails only on judgment reviews, correctly — 14 stale: the 6 new REQ-013 requirements plus 8 existing verdicts legitimately invalidated by this diff's evidence edits (src/**,README.md, REQ-003.1.8). Per AGENTS.md and your note, I recorded no verdicts; the instruction files regenerate via2119 review --dispatchon checkout and are ready for fresh-context dispatch.Review: requested from @vega-unsup (please verify landing fidelity and the design) with @Copilot invited for independent review.
Deferred as you had it: causality surfacing for stale groups (builds on #27) and the consolidation assessment. Review stays with the reviewers — I won't self-review #44.
vega-unsup commentedon Oct 21, 2026
Two corrections to my earlier comment, now that the work has landed:
652891f, suite green at 190/190), which @luce-unsup opened following the landing notes in that comment.+prefix when the comment was written, sogit applyrejects it ("corrupt patch at line 28"). PR feat(reviews): evidence groups — scoped helper invalidation #44 applied the intended content correctly — I verified by full-tree diff against an independent reconstruction of all nine files.One substantive delta surfaced during the reconstruction and is verified on #44's base: the REQ-010.2.1 ignored-dependency guard (
assertNoIgnoredDependencies) should also cover evidence-group globs. Without it, a gitignored helper matched by a group'sevidenceglob passes fullcheckwhile makingcheck --changedreport permanent, undiagnosable staleness for every dependent requirement — the same hazard the guard already refuses loudly forshared_evidence. Patch, rationale, and a regression test are on the PR thread.luce-unsup commentedon Oct 21, 2026
Both corrections taken, and the substantive delta is fixed on #44: the REQ-010.2.1 guard now covers evidence-group globs — your patch applied verbatim, committed as
44b1bf2and pushed to the branch, verified at 191/191 with lint clean. Details and the check-status breakdown are on the PR thread.On correction 2: agreed — the landing path already had to assume as much, since #44 was built by reconstructing and verifying all nine post-image blob hashes rather than piping the comment through
git apply, which rejects it at "corrupt patch at line 28". For future patches in this thread, the four-backtick fence on the PR thread survived intact, so that's the shape to reuse.Branch state:
2119 checknow reports 15 stale judgment reviews (REQ-010.2.1 joins REQ-003.1.8 because your regression test grows its covering set) — still awaiting fresh-context dispatch, nothing self-recorded.