Skip to content

test(pv): contracts-pv was 0% because CI ran none of pv's tests (#2589) - #2603

Closed
noahgift wants to merge 1 commit into
mainfrom
feat/2589-contracts-pv-first-coverage
Closed

test(pv): contracts-pv was 0% because CI ran none of pv's tests (#2589)#2603
noahgift wants to merge 1 commit into
mainfrom
feat/2589-contracts-pv-first-coverage

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

contracts-pv read 76 features, 0.0% covered, 0 UNKNOWN hardware. That last
column says the features are hardware-verified and simply ungated. Two causes were
measured at 4bbfeb07f; both are fixed here.

1. pv's CLI tests were DARK — this is why the number was 0%

crates/aprender-contracts-cli/tests/ already held 651 lines of tests that spawn the
pv binary
, and CI ran zero of them:

  • workspace-test is cargo nextest run --profile ci --workspace --lib — library
    targets only.
  • The one explicit integration line in ci.yml names 23 --test targets. None is
    from aprender-contracts-cli.
$ grep -o "cargo test -p [a-z-]* [^&]*--test [a-z_0-9]*" .github/workflows/ci.yml | sort -u | wc -l
23
$ grep -c "aprender-contracts-cli" <that list>
0

A test that never executes is 0% coverage however many lines it has. This is the exact
trap the comment above that line already warns about — it had swallowed pv whole.

2. The tests that did exist EXCLUDED NOTHING

Nearly every one asserted status.success() on a good contract. That assertion is
satisfied unchanged by fn main() { std::process::exit(0) }. Per
project_assertions_exclude_guard, an assertion no wrong behaviour can violate is not a
control.


Tranche 1 — tests/pv_surface_gate.rs, 8 tests

Area What is gated
pv validate decision table All 25 diagnostic rules the validator can emit (SCHEMA-001..017, PROVABILITY-001, BEAT-001..007), both directions: tripped → reached through the CLI at its measured severity with the exit code that severity implies; clean contract → none of them.
Exhaustiveness The rule universe is read out of validator.rs, so a new rule enters the loop rather than falling outside it.
Reachability All 38 subcommands pv --help advertises must exit 0 for pv <cmd> --help — the apr test llm class (#2527).
Input rejection Each of the 16 contract-taking subcommands must exit non-zero on a nonexistent path, malformed YAML, an empty file, and well-formed YAML that is not a contract (64 assertions). This is the assertion the pre-existing suite never made.
Version pinning pv --version reports this crate's version, resolved via CARGO_BIN_EXE_pv so it cannot fall back to a stale PATH pv (#2552).

Severities and exit codes were measured against the binary, not assumed.

The exhaustiveness guard earned its keep on first run

It failed immediately, naming BEAT-001 as ungated. Widening the extractor then exposed
BEAT-002..007, which are emitted through a local closure rather than a rule: "..."
literal and were invisible to the first version — this guard's own universe had been
built from the wrong side.
The extractor now takes its universe from the rule-id
shape, not from one syntactic form.

Mutation verification — 4 mutations, each proven to ENGAGE first

Every mutation asserts a non-empty git diff before any verdict is read. A sed that
silently fails to match reads exactly like "the guard is fine".

# Mutation Result
M1 delete the violation-print loop in commands/validate.rs gate rc=101 (3 RED); library -p aprender-contracts rc=0, 55 passed
M2 replace validate.rs's error return with Ok(()) gate rc=101 (2 RED) on the exit-code half
M3 rename SCHEMA-013SCHEMA-999 exhaustiveness guard RED, naming the orphan
M4 disable the BEAT_INCUMBENTS membership arm RED on exactly the BEAT-002 membership row
CONTROL restored tree rc=0, 8/8 green

M1 is the load-bearing result. The gate goes RED while the library tests stay GREEN
— mechanical proof that this covers CLI surface the library tests cannot see, i.e. it is
new coverage rather than a duplicate of validator_tests.rs.

All four were re-run after the complexity refactor of the help parser, since the old
proof does not transfer across a refactor.

CI wiring — without it, a passing gate is still 0%

pv_surface_gate, cli_integration (64 tests, green) and ground_truth (34, green) are
added to the integration line.

book_coverage is deliberately not wired: it is RED at 4bbfeb07f

Failed to parse contracts/binding.yaml: Failed to parse YAML: missing field `metadata`

It walks contracts/*.yaml and chokes on binding.yaml, a binding registry rather than
a contract. It rotted precisely because nothing watched it. Fixing it is separate work
and belongs near #2554's schema changes.

Contract

contracts/pv-cli-surface-v1.yaml, kind: pattern, 8 falsification tests carrying the
measured mutation results. pv validate0 errors, 0 warnings.

Gates

Gate Result
cargo test -p aprender-contracts-cli --test pv_surface_gate rc=0, 8/8
--test cli_integration rc=0, 64/64
--test ground_truth rc=0, 34/34
cargo fmt --all -- --check rc=0
pv validate contracts/pv-cli-surface-v1.yaml rc=0
pv lint contracts/ 0 errors, 947 warnings — byte-identical to my measured origin/main baseline, so this contract adds none
pmat pre-commit gates fmt / complexity / docs PASS

The 942 warning figure in circulation is stale; re-measured at 947 on origin/main
before and after. (Task 2's agent independently measured 947.)

Deliberately NOT in this tranche

pv lint's warning ratchet (its numeric baseline belongs in the Makefile gate, not a
unit test, and a run over 1229 contracts is too slow for the integration step),
pv diff semver suggestions, pv score, pv certify, pv coverage, and the content
of the generator subcommands' output (kani/probar/coq/flux/tla/…). Those are
tranche 2.

Judgement on CLUS-004 (#2596): do not land it here

The ticket proposes that expressing clustering invariants as a pv contract "closes two
zeros with one ticket". It does not, for two independent reasons:

  1. Category error. contracts-pv's 76 features are pv's own CLI features. Adding a
    YAML file that pv validates exercises the clustering code, not pv. It would move the
    clustering zero and leave contracts-pv at 0%.
  2. Its input does not exist. docs/audits/surface_audit_clustered.csv — the 830-row
    matrix both Recluster the surface audit with aprender-core: a real sklearn-parity falsifier, and contracts-pv's first coverage #2591 and CLUS-004: express the clustering invariants as a pv contract — one ticket closes two zeros #2596 recluster — is not in the tree (docs/audits/ holds
    only the two 0.63.0 dogfood markdown files). CLUS-004 also needs sklearn as a
    differential oracle and thresholds "set from measurement"; none of that is available
    here, and forcing it would produce exactly the round-number threshold CLUS-004: express the clustering invariants as a pv contract — one ticket closes two zeros #2596 warns
    against.

CLUS-004 is good work on its own merits under epic:clustering. It is not
contracts-pv coverage, and pairing them would make bisection harder for no gain.

Note for reviewers

  • ci.yml is touched (one command appended to the single integration line, plus a
    comment). Per the warning in that file's own comment, only one PR at a time may edit
    that physical line
    — if another PR in flight also edits it, land them in sequence.
  • Clippy note: cargo clippy -p aprender-contracts-cli --all-targets -- -D warnings is
    already rc=101 on origin/main with 72 pre-existing disallowed_methods
    (unwrap) findings in contract_walk.rs, equations.rs, graph.rs, certify.rs,
    verify_pipeline.rs and tests/book_coverage.rs. This PR's new file contributes
    0 of them. CI's lint step does not currently reach this crate's non-lib targets;
    that pre-existing debt is out of scope here but worth its own ticket.

Refs #2589, #2552, #2527, #2596

The surface audit read `cluster contracts-pv: 76 features, 0.0% covered,
0 UNKNOWN hardware`. The last column says the features are hardware-verified
and simply UNGATED. Two causes were measured at 4bbfeb0; both are fixed here.

1. pv's CLI tests were DARK. crates/aprender-contracts-cli/tests/ held 651
   lines of tests that SPAWN the pv binary, and CI ran zero of them.
   `workspace-test` is `cargo nextest run --workspace --lib` (library targets
   only), and the one explicit integration line in ci.yml names 23 `--test`
   targets, not one from aprender-contracts-cli. A test that never executes is
   0% coverage however many lines it has. This is the exact trap the comment
   above that line already warns about; it had swallowed pv whole.

2. The tests that did exist EXCLUDED NOTHING. Nearly every one asserted
   `status.success()` on a GOOD contract — an assertion `fn main() { exit(0) }`
   satisfies unchanged. Per project_assertions_exclude_guard, an assertion no
   wrong behaviour can violate is not a control.

TRANCHE 1 — crates/aprender-contracts-cli/tests/pv_surface_gate.rs (8 tests)

  * `pv validate` decision table over ALL 25 diagnostic rules the validator can
    emit (SCHEMA-001..017, PROVABILITY-001, BEAT-001..007), both directions:
    each rule must be REACHED through the CLI at its measured severity with the
    exit code that severity implies, and a clean contract must emit NONE of them.
    Severities were measured against the binary, not assumed.
  * An exhaustiveness guard that reads the rule universe out of validator.rs, so
    a new rule ENTERS the loop rather than falling outside it.
  * Subcommand reachability across all 38 commands `pv --help` advertises —
    the `apr test llm` defect class (#2527).
  * Input rejection: each of the 16 contract-taking subcommands must exit
    non-zero on a nonexistent path, malformed YAML, an empty file, and
    well-formed YAML that is not a contract (64 assertions). This is the
    assertion the pre-existing suite never made.
  * `pv --version` reports this crate's version, resolved via CARGO_BIN_EXE_pv
    so it cannot fall back to a stale PATH pv (#2552).

The exhaustiveness guard earned its keep on first run: it failed naming BEAT-001
as ungated. Widening the extractor then exposed BEAT-002..007, which are emitted
through a closure rather than a `rule: "..."` literal and were invisible to the
first version — this guard's own universe had been built from the wrong side.

MUTATION-VERIFIED, four mutations, each proven to ENGAGE (non-empty git diff)
before any verdict was read:

  M1  delete the violation-print loop in commands/validate.rs
      -> gate rc=101 (3 tests RED), library `-p aprender-contracts` rc=0,
         55 passed. The GREEN library result is the point: it proves this gate
         covers CLI surface the library tests cannot see.
  M2  replace validate.rs's error return with Ok(())
      -> gate rc=101 (2 RED) on the exit-code half. The mutation the old suite
         could not detect, because it only asserted success on good input.
  M3  rename SCHEMA-013 -> SCHEMA-999
      -> exhaustiveness guard RED, naming the orphan.
  M4  disable the BEAT_INCUMBENTS membership arm
      -> gate RED on exactly the BEAT-002 membership row, proving that row
         exercises the list rather than decorating it.
  CONTROL  restored tree rc=0, 8/8 green.

CI WIRING. pv_surface_gate, cli_integration (64 tests, green) and ground_truth
(34, green) are added to the integration line. book_coverage is deliberately NOT
wired: it is RED at 4bbfeb0 — it walks contracts/*.yaml and chokes on
contracts/binding.yaml ("missing field `metadata`"), a binding registry rather
than a contract. It rotted precisely because nothing watched it; that is
separate work, not this tranche's.

CONTRACT. contracts/pv-cli-surface-v1.yaml ships with the gate, `kind: pattern`,
8 falsification tests carrying the measured mutation results. `pv validate`
reports 0 errors, 0 warnings.

NOT COVERED by this tranche, deliberately: `pv lint`'s warning ratchet, `pv diff`
semver suggestions, `pv score`, `pv certify`, `pv coverage`, and the content of
the generator subcommands' output.

Refs #2589, #2552, #2527

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR
@noahgift

Copy link
Copy Markdown
Contributor Author

Superseded by #2617, which folds this PR together with the other two pv release blockers.

Why folded rather than landed separately — two collisions were measured between these
three PRs, so landing them independently was not viable:

  1. Textual: pv --version cannot tell you WHICH pv you have (item 3 carved out of #2558) #2559 and pv enforces provable contracts for the whole stack and has zero coverage of itself (76 features, 0%) #2589 both append to the single integration-test line in ci.yml.
    Only one PR at a time may edit that physical line without a merge-queue conflict.
  2. Semantic: pv enforces provable contracts for the whole stack and has zero coverage of itself (76 features, 0%) #2589 ships an exhaustiveness guard over pv's diagnostic rules that goes RED
    the moment pv validate accepts competitor: THIS-DOES-NOT-EXIST, demand_score: 99999, intake_status: banana — 0 errors #2555 lands, because pv validate accepts competitor: THIS-DOES-NOT-EXIST, demand_score: 99999, intake_status: banana — 0 errors #2555 adds CRUX-001 and CRUX-002. Reproduced by
    copying pv validate accepts competitor: THIS-DOES-NOT-EXIST, demand_score: 99999, intake_status: banana — 0 errors #2555's schema files onto pv enforces provable contracts for the whole stack and has zero coverage of itself (76 features, 0%) #2589's tree.

Neither PR could see the other's problem. Folding resolves both in-tree before CI runs, and
costs one workspace-test (~58 min) instead of three.

An adversarial pass on all three returned unsound, and #2617 closes what it measured —
including a fatal one: adding a contract took the corpus 1778 → 1781, so check_readme_claims.sh
(a required check) failed until three README figures were bumped. All three PRs were
auto-merge armed at the time and would have sat blocked.

Closing is reversible and the branch is untouched.

@noahgift noahgift closed this Aug 22, 2026
guyernest pushed a commit to guyernest/aprender that referenced this pull request Aug 23, 2026
…fects the adversarial pass measured (paiml#2617)

* fix(pv): serde dropped three schema fields, so pv validated nonsense with exit 0

REPRODUCED at 4bbfeb0. A crux-shaped contract carrying

    competitor: THIS-COMPETITOR-DOES-NOT-EXIST
    demand_score: 99999
    intake_status: banana

passed `pv validate` with `0 error(s), 0 warning(s)` and exit 0.

Root cause, measured: `Metadata` (schema/types.rs) declared NONE of the three
fields. serde's default for an unknown key is to ignore it, so the keys were
discarded during deserialization and the parsed `Contract` held nothing for any
validator to check. The vocabulary lived only in the generator that emitted the
275 crux-*-v1.yaml files (scripts/crux_scaffold_contracts.py), where it
constrained nothing once the files were on disk. A field domain expressed only
in a generator is not a domain.

This is not paiml#2554: that PR records unknown TOP-LEVEL keys (SCHEMA-018/019);
these three live under `metadata:` and its diff touches none of the three names.

WHAT LANDS

- `metadata.intake_status` becomes a closed enum `IntakeStatus`
  {missing, partial, supported, unclear} — the exact STATUS_BADGE vocabulary of
  the generator. An invented value now FAILS TO PARSE, not merely lints. That
  distinction is the requirement: a lint is advisory and a caller may proceed;
  an invented value must not be a warning about a contract, it must not be a
  contract.
- `metadata.demand_score` parses as i64 and is range-checked 1..=5 (rule
  CRUX-001, Error), the bound the master contract itself documents. Typed i64
  rather than u8 on purpose, so 99999 reaches the validator and is reported by
  value instead of dying in serde as an integer-overflow message. This is the
  signal the whole competitive-research programme sorts by — an unvalidated
  99999 silently outranks every real story.
- `metadata.competitor` is membership-checked against CRUX_COMPETITORS
  (rule CRUX-002, Error).

BEAT_INCUMBENTS IS THE WRONG SET — MEASURED, NOT ASSERTED

Reusing it was considered and rejected: it answers a different question and
would import a defect. Against the 276 contracts carrying the field,
`BEAT_INCUMBENTS.iter().any(|p| c.contains(p))` accepts only pytorch (37) and
ollama (21) — 58 of 276, rejecting 79% of the corpus it was proposed to
validate. It cannot name huggingface (88, the largest source) or vllm (32) at
all, and it misses llama_cpp (37) even though it names that pillar, because it
spells it `llama.cpp` and that is not a substring of `llama_cpp`. So
CRUX_COMPETITORS is the corpus vocabulary exactly; all 11 members are exercised
by real contracts. `beat_incumbents_cannot_name_the_crux_corpus` pins the
decision mechanically so a "these lists are redundant" refactor turns RED.

ONE REAL DRIFT FOUND

Sweeping the corpus with the enum in place turned up exactly one bad value:
contracts/apr-lint-producers-v1.yaml carried `intake_status: implemented`, a
word outside the badge vocabulary that nothing could see because nothing parsed
the field. Corrected to `supported`, and pinned by a regression anchor.

EVIDENCE

Falsifier before the fix: `pv validate` on the reproducer printed
"0 error(s), 0 warning(s) / Contract is valid.", rc=0. After: rc=1 on all three
fields independently — intake_status at parse time, demand_score as CRUX-001,
competitor as CRUX-002.

Mutation-verified in both directions, each mutation's engagement proved by
grepping its marker before any verdict was read; all restored (0 markers left):
  E  variant admitting `banana`            -> 2 tests RED
  B  range widened to i64::MIN..=i64::MAX  -> 2 tests RED
  B2 range narrowed to 2..=4               -> 2 tests RED
  C  CRUX-002 demoted to Warning           -> 2 tests RED
  D  CRUX_COMPETITORS = BEAT_INCUMBENTS    -> 5 tests RED
  F  validate_crux_intake call deleted     -> 3 tests RED
GREEN direction: 12/12, and 1458/1458 for the crate.

`pv lint contracts/` 0 errors / 947 warnings before and after (1727 -> 1728
contracts); the new contract adds no warning. pv built from HEAD via
`cargo run -p aprender-contracts-cli --bin pv`, never the stale 0.49.0 on PATH.

Contract: contracts/crux-intake-metadata-domains-v1.yaml (kind: pattern,
3 equations, 11 obligations, FALSIFY-CRUX-INTAKE-001..007).

Closes paiml#2555
Refs paiml#2589 (contracts-pv coverage), paiml#2554

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

* test(pv): contracts-pv was 0% because CI ran none of pv's tests (paiml#2589)

The surface audit read `cluster contracts-pv: 76 features, 0.0% covered,
0 UNKNOWN hardware`. The last column says the features are hardware-verified
and simply UNGATED. Two causes were measured at 4bbfeb0; both are fixed here.

1. pv's CLI tests were DARK. crates/aprender-contracts-cli/tests/ held 651
   lines of tests that SPAWN the pv binary, and CI ran zero of them.
   `workspace-test` is `cargo nextest run --workspace --lib` (library targets
   only), and the one explicit integration line in ci.yml names 23 `--test`
   targets, not one from aprender-contracts-cli. A test that never executes is
   0% coverage however many lines it has. This is the exact trap the comment
   above that line already warns about; it had swallowed pv whole.

2. The tests that did exist EXCLUDED NOTHING. Nearly every one asserted
   `status.success()` on a GOOD contract — an assertion `fn main() { exit(0) }`
   satisfies unchanged. Per project_assertions_exclude_guard, an assertion no
   wrong behaviour can violate is not a control.

TRANCHE 1 — crates/aprender-contracts-cli/tests/pv_surface_gate.rs (8 tests)

  * `pv validate` decision table over ALL 25 diagnostic rules the validator can
    emit (SCHEMA-001..017, PROVABILITY-001, BEAT-001..007), both directions:
    each rule must be REACHED through the CLI at its measured severity with the
    exit code that severity implies, and a clean contract must emit NONE of them.
    Severities were measured against the binary, not assumed.
  * An exhaustiveness guard that reads the rule universe out of validator.rs, so
    a new rule ENTERS the loop rather than falling outside it.
  * Subcommand reachability across all 38 commands `pv --help` advertises —
    the `apr test llm` defect class (paiml#2527).
  * Input rejection: each of the 16 contract-taking subcommands must exit
    non-zero on a nonexistent path, malformed YAML, an empty file, and
    well-formed YAML that is not a contract (64 assertions). This is the
    assertion the pre-existing suite never made.
  * `pv --version` reports this crate's version, resolved via CARGO_BIN_EXE_pv
    so it cannot fall back to a stale PATH pv (paiml#2552).

The exhaustiveness guard earned its keep on first run: it failed naming BEAT-001
as ungated. Widening the extractor then exposed BEAT-002..007, which are emitted
through a closure rather than a `rule: "..."` literal and were invisible to the
first version — this guard's own universe had been built from the wrong side.

MUTATION-VERIFIED, four mutations, each proven to ENGAGE (non-empty git diff)
before any verdict was read:

  M1  delete the violation-print loop in commands/validate.rs
      -> gate rc=101 (3 tests RED), library `-p aprender-contracts` rc=0,
         55 passed. The GREEN library result is the point: it proves this gate
         covers CLI surface the library tests cannot see.
  M2  replace validate.rs's error return with Ok(())
      -> gate rc=101 (2 RED) on the exit-code half. The mutation the old suite
         could not detect, because it only asserted success on good input.
  M3  rename SCHEMA-013 -> SCHEMA-999
      -> exhaustiveness guard RED, naming the orphan.
  M4  disable the BEAT_INCUMBENTS membership arm
      -> gate RED on exactly the BEAT-002 membership row, proving that row
         exercises the list rather than decorating it.
  CONTROL  restored tree rc=0, 8/8 green.

CI WIRING. pv_surface_gate, cli_integration (64 tests, green) and ground_truth
(34, green) are added to the integration line. book_coverage is deliberately NOT
wired: it is RED at 4bbfeb0 — it walks contracts/*.yaml and chokes on
contracts/binding.yaml ("missing field `metadata`"), a binding registry rather
than a contract. It rotted precisely because nothing watched it; that is
separate work, not this tranche's.

CONTRACT. contracts/pv-cli-surface-v1.yaml ships with the gate, `kind: pattern`,
8 falsification tests carrying the measured mutation results. `pv validate`
reports 0 errors, 0 warnings.

NOT COVERED by this tranche, deliberately: `pv lint`'s warning ratchet, `pv diff`
semver suggestions, `pv score`, `pv certify`, `pv coverage`, and the content of
the generator subcommands' output.

Refs paiml#2589, paiml#2552, paiml#2527

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

* fix(pv): pv --version could not tell you WHICH pv you have (paiml#2559)

A HEAD-built pv printed exactly `pv 0.63.0` -- a bare name and a semver,
byte-identical to what pv(1) the pipe viewer prints. Four things claim that
name: the crates.io `pv` pipe-viewer crate (7,065 downloads, first published
2019), pv(1) from every distro, aprender-contracts-cli, and -- until paiml#2553 --
the facade.

The operator settled that this binary KEEPS the name (2026-08-21), which makes
the version line the mitigation the project relies on. It carried no identity
at all. The install-time half of the collision was already handled
(check_duplicate_bin_names.sh, the facade going lib-only); nothing answered the
RUNTIME question "which one am I talking to".

  pv 0.63.0 (aprender provable-contracts verifier)
  crate aprender-contracts-cli - https://github.com/paiml/aprender
  Verifies YAML contracts under contracts/; run `pv --help` for the command surface.
  This is NOT pv(1), the pipe viewer (distro package `pv`, or the `pv` crate on crates.io).

`-V` stays one line and is itself unambiguous: a version line nobody reads is
no better than an ambiguous one.

ADDING IDENTITY TEXT BROKE THE RELEASE GATE, and the fix ships in the same
commit. scripts/pv_bin.sh proves a resolved pv is the HEAD build by extracting
the semver with `awk '{print $NF}'` -- the last field of EVERY line. Against a
multi-line version it returned four lines of prose and pv_bin.sh exited 1,
blocking the Makefile contracts gate and dogfood_surfaces.sh. The extractor now
reads position 2 of line 1, and that position is pinned from BOTH sides so
neither can drift alone.

FALSIFIER, RED BEFORE THE FIX (rc=101): 5 of 8 integration assertions and 3 of
5 unit assertions fail against `pv 0.63.0`. Mutation-verified in both
directions, six mutations, each with its engagement proved by reading back the
mutated line:
  M1 bare `version`            -> 5/8 integration RED
  M2 drop only the disclaimer  -> exactly 1 RED (the clause is load-bearing)
  M3 semver off field 2 in -V  -> the agreement assertion RED
  M4 semver off field 2 in --version -> the pv_bin.sh contract RED, and
                                  pv_bin.sh itself exits 1 -- guard and gate
                                  fail as one
  M5 M1 re-run in LIBRARY scope -> 3/5 RED (the old proof does not transfer)
  M6 drift pv_bin.sh's extractor -> EXTRACTOR_MISMATCH fires

The assertions live in the LIBRARY as well as in tests/, because a new
tests/*.rs file is dark until named on ci.yml's single explicit --test line and
only one PR at a time can edit it. `--lib` reaches the unit half unconditionally.

The new case table caught its own comment-vs-code trap on first run: it grepped
all of pv_bin.sh and fired on pv_bin.sh's COMMENT quoting the old extractor.
Fixed to strip full-line comments.

Verified: pv lint contracts/ 0 errors, 947 warnings -- identical with and
without the new contract, so it adds none (the 942 figure in circulation is
stale; re-measured here). fmt clean, clippy -p aprender-contracts-cli --lib -D
warnings clean, bashrs 0 errors/0 warnings on the new script, shell-lint
ratchet PASS at 97 error lines.

Contract: contracts/pv-version-identity-v1.yaml, FALSIFY-PVVER-001..007, all
seven executed and passing.

Refs paiml#2559

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

* fix(pv): close the eight defects an adversarial pass measured on paiml#2601/paiml#2602/paiml#2603

Three PRs were opened and auto-merge armed; both verifiers returned
sound=false. This folds all three onto one branch and closes every finding.
Every number below was RE-MEASURED on this branch -- none was copied from the
review brief, and two of the brief's numbers were wrong.

F1 (FATAL) required check guard-runner-labels
  check_readme_claims.sh was rc=1: the corpus goes 1778 -> 1781 (THREE new
  contracts land, one per folded PR -- the brief said 1779, assuming one).
  README:44/:225/:256 bumped; script now rc=0.

M1 both whole-surface tests were NEAR-VACUOUS
  MEASURED: replacing Commands::Status's body with panic!() -- so `pv status`
  dies on every invocation -- left ALL 8 tests GREEN. clap answers `--help`
  without entering the command, so the reachability test proved only that the
  command was DECLARED; and a panic exits 101, which the rejection test read
  as a correct refusal. The test could not detect the class (paiml#2527) its own
  doc comment claimed.
  FIX: every contract-taking subcommand is now INVOKED FOR REAL against a
  valid contract (17 of them, run in a scratch cwd because the generators
  write into ./generated/), and a rejection must be a clean 1 or 2, never a
  panic. Three mutations, each proved engaged by grepping its marker in the
  built binary before any verdict was read:
    panic!()          8 pass/0 fail  ->  6 pass/2 fail
    exit(1) on valid  8 pass/0 fail  ->  7 pass/1 fail
    exit(0) on all    8 pass/0 fail  ->  7 pass/1 fail
  All restored; `grep -c MUTANT` is 0.

M2 the two collisions between the folded PRs
  (a) TEXTUAL: paiml#2559 and paiml#2589 both append to the single integration-test line
      in ci.yml. Resolved in-tree keeping BOTH -- pv_surface_gate,
      cli_integration, ground_truth, version_identity.
  (b) EXHAUSTIVENESS: paiml#2589's guard reds the moment paiml#2555 lands. Reproduced
      verbatim: ["CRUX-001", "CRUX-002"] have no row. Closed with REAL CLI
      trip cases, not bookkeeping entries -- each asserts `[ERROR] CRUX-00N:`
      and rc=1 through the binary.

M3 4 of 7 falsification commands were NOT EXECUTABLE as written
  -003/-004/-006 passed 2-3 test filters as bare positionals (cargo exits 1,
  "unexpected argument"); -007 embedded prose in the command (bash exits 2,
  syntax error). Multi-filter args moved after `--`; -007's protocol prose
  moved into `prediction` where it belongs. All 9 now run verbatim at rc=0,
  read OUT OF THE YAML rather than retyped, with filter counts proving each
  ran the tests it names.

m1 the headline rationale was unsupported by the code
  paiml#2555 justified CRUX-001 as guarding "the ranking signal the whole
  competitive-research programme sorts by". MEASURED: nothing in the repo
  reads metadata.demand_score. The signal spec section 12.1 maps to `pmat
  work` priority is stories[].demand_score in the MASTER REGISTRY -- 250 rows,
  entirely ungated. Rather than retreat to a weaker claim, the gate now covers
  that surface too, reusing DEMAND_SCORE_RANGE / CRUX_COMPETITORS /
  IntakeStatus, so the claim is true of the union. On a registry row the
  fields are REQUIRED as well as bounded. Lands GREEN (250 rows all in-domain)
  and functions as a ratchet.

m2 trim laundering, and the presence gap
  CRUX-002 trimmed INSIDE the comparison, so `competitor: '  ecosystem  '`
  validated clean while the stored value kept its padding -- the gate approved
  a string no consumer would see. Normalisation moved to PARSE. Present-but-
  empty deliberately does NOT collapse to None: that would have widened the
  presence gap rather than narrowed it. The gap that remains (omission and
  explicit null dodge the metadata surface entirely, and cannot be closed
  there because ~1450 non-crux contracts carry no such fields) is now a stated
  proof obligation, not an unnoticed hole.

m3 stale counts in files whose thesis is that unmeasured claims rot
  "all 18 diagnostic rules" -> 27 (measured; was 25 pre-paiml#2555, never 18).
  "276 contracts carry competitor" -> 275 FILES / 292 DECLARATIONS (17 crux
  contracts carry a second competitor inside an equivalence obligation), so
  the BEAT matcher accepts 58 of 292 and rejects 80%, not 79%.

m4 the identity was not load-bearing where the decision is made
  paiml#2559 added an identity string to `pv --version`, but pv_bin.sh -- the place
  the release actually decides it has the right binary -- still proved
  freshness from the SEMVER alone. pv_bin_assert_fresh now asserts identity on
  the same line it already parses. Mutation-verified: a fake pv printing the
  exactly-correct declared version `pv 0.63.0` and nothing else is now
  REJECTED (rc=1); the real binary passes. The restructure kept the extractor
  literal that check_pv_version_parse.sh pins, and that guard plus its
  --self-test and check_sourced_libs_option_neutral.sh are all rc=0.

m5 residual stated as a FRACTION
  "tranche" is honest about being partial and silent about how partial. The
  scope block now gives three concentric rings: DEEP behavioural decision
  table 1/38 subcommands (`pv validate` alone), REAL INVOCATION 17/38,
  SURFACE 38/38. The behavioural depth of this tranche is 1 of 38.

VERIFICATION
  check_readme_claims.sh                          rc=0
  check_facade_compat.sh                          rc=0 (see PR body: rc=1
      inside this worktree is dev-config inheritance, proved by running the
      same tree outside /home/noah/src/aprender)
  check_pv_version_parse.sh / --self-test         rc=0 / rc=0
  check_sourced_libs_option_neutral.sh            rc=0
  bashrs lint scripts/pv_bin.sh                   0 errors
  cargo fmt --all -- --check                      rc=0
  aprender-contracts --lib                        1466 passed, 0 failed
  aprender-contracts-cli (all targets)            182 passed, 0 failed
      (book_coverage excepted -- byte-identical pre-existing failure on
       origin/main: contracts/binding.yaml, "missing field metadata")
  pv lint contracts/                              0 errors, 947 warnings, PASS
      -- IDENTICAL to the origin/main baseline I measured myself (947, not the
      937 or 942 two earlier briefs asserted); 1727 -> 1730 contracts, 0 new
      warnings
  9/9 FALSIFY-CRUX-INTAKE-* verbatim              rc=0

Closes paiml#2555
Closes paiml#2559
Closes paiml#2589

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

* fix(pv): close the three minor defects the re-verification pass measured

The consolidated branch was sound on every FATAL and MAJOR item. Three minor
findings survived, and all three are the same shape: a claim that no longer
matches what was measured.

1. A stale percentage in the file whose thesis is that unmeasured claims rot.
   contracts/crux-intake-metadata-domains-v1.yaml:78 was correctly re-measured
   to "58 of 292 ... reject 80%", but :232 still said 79%. Swept the whole
   folded diff for the superseded numbers (276 contracts, 79%, "all 18
   diagnostic rules", 16 rules) — this was the only survivor.

2. The residual disclosure overstated its own coverage, which is precisely the
   defect class m1 was raised to close, recurring one level up. The NOT COVERED
   paragraph said the other 37 subcommands are asserted "to run and decide",
   but the ring table above it says only 17/38 are invoked for real. The other
   21 are asserted only to be DECLARED. Rewritten to state both tiers
   separately and to say plainly that a `pv lint`, `diff`, `query`, `graph`,
   `kaizen` or `migrate` whose body were panic!() would leave the file GREEN.

3. cli_integration dirties TRACKED generated caches (.pv/contracts.idx,
   .pv/contracts.idx.mtime, .pv/lint-previous.json), and this PR wires it into
   ci.yml's integration step — so CI runs it for the first time and its working
   tree goes dirty. Disclosed at the top of the file rather than contained, with
   the reason: pv_surface_gate solves this with pv_in(dir, ..) + .current_dir(),
   which does not transfer, because cli_integration makes IN-PROCESS library
   calls and the cache path follows the test process's own cwd. Changing that
   cwd is process-wide and would race the other tests in the binary. A proper
   fix means an explicit cache-root parameter on the library, which is a real
   change and not this PR's.

Verified: pv_surface_gate 8/8 pass, check_readme_claims.sh rc=0 (contract_count
1781), pv validate on the edited contract rc=0 / 0 errors 0 warnings. Built with
an explicit --target-dir, not the env var, per the shared-target-dir trap.

Refs paiml#2555, paiml#2559, paiml#2589, paiml#2556

* chore(pv): re-measure the contract count after batch-b landed

paiml#2631 folded ten PRs, six of which each add one contract, taking main 1781 ->
1787. This branch adds three of its own, so the true corpus is 1790.

Both branches were correct in isolation and neither could see the other — the
paiml#2630 collision, now hit for the sixth time in two days. The figure is MEASURED
here (find contracts -name '*.yaml'), never incremented by guesswork, because the
right value depends on what else landed and that is unknowable from inside a
branch.

check_readme_claims.sh rc=0: crate_count 78, contract_count 1790,
cli_command_count 110, cookbook_link present.

* fix(guard): check_pv_bin_resolution.sh read the identity suffix as part of the semver

MERGE SEAM, not a defect in either side. paiml#2559 (this branch) appended an
identity suffix to `pv --version` because four things claim the name `pv` and
the operator settled 2026-08-21 that this binary keeps it, so the version line
IS the disambiguation. paiml#2580 (main, batch-a) added this guard, whose own
extraction compared `--version | head -1` against the literal "pv $CRATE_VERSION".
The suffix rode along into the compared value and the guard went RED against a
perfectly fresh HEAD build:

  FAIL: $PV reports 'pv 0.63.0 (aprender provable-contracts verifier)'
        but the crate is 0.63.0

pv_bin.sh's own pv_bin_assert_fresh() was updated by paiml#2559; this script carries
a SEPARATE extraction that was not.

Two changes, not one:

1. Semver read POSITIONALLY -- position 2 of line 1, the shape pinned from the
   other side by tests/version_identity.rs and by the case table in
   check_pv_version_parse.sh -- instead of whole-line equality.

2. The guard now asserts the IDENTITY too. Fixing only the extraction would
   leave a guard that passes for pv(1) the pipe viewer, since a semver is not an
   identity; the identity is the property paiml#2559 exists to provide, so the guard
   that runs the resolver has to assert it. The literal is checked against both
   aprender-contracts-cli/src/lib.rs and pv_bin.sh (IDENTITY_MISMATCH), so the
   guard cannot end up proving a string nobody emits.

Case table extended in both directions, per the file's existing REFUSE/ACCEPT
pairing:

  2c REFUSE: a pv at the CORRECT crate version printing the bare `pv <semver>`
     -- byte-for-byte what pv(1) prints -- and the refusal must be on identity
     grounds, not incidental.
  2d ACCEPT: the same synthesized script, same version, plus the suffix. The
     only difference between the pair is the property under test.

MUTATION EVIDENCE (both directions, engagement proven):
  * pre-fix file, unmodified: rc=1, the exact CI message above.
  * M1 -- delete the identity `case` block from pv_bin.sh (engagement:
    `grep -c 'aprender provable-contracts verifier' scripts/pv_bin.sh` 1 -> 0):
    rc=1, 2c RED ("ACCEPTED a bare 'pv 0.63.0' with no identity") plus
    IDENTITY_MISMATCH, while controls 2b and 2d stayed green -- so the pair
    discriminates rather than refusing everything.
  * M2 -- regress the new extractor to the pre-paiml#2559 `{print $NF}` (engagement:
    the line read `got_semver=$(awk '{print $NF}' ...)`): rc=1, semver read as
    'verifier)'.
  * fixed file: rc=0, 10/10 assertions.

OTHER CONSUMERS of `pv --version`, checked: scripts/dogfood.sh:334 reads
`head -1 | awk '{print $2}'` (positional, safe); scripts/dogfood_surfaces.sh:258
uses `-V`, the deliberately one-line glance form (verified single-line against
the HEAD build); pv_surface_gate.rs asserts a substring. None break.

bashrs: 0 errors on this file alone; check_shell_lint_ratchet.sh PASS (145 error
lines vs baseline 876).

Refs paiml#2617, paiml#2559, paiml#2580

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

* fix(pv): gate SCHEMA-018/019/020 through the CLI, not with a bookkeeping entry

paiml#2589 added an exhaustiveness guard requiring every rule declared in
validator.rs to have a row in the pv-surface decision table. paiml#2554 landed
SCHEMA-018/019/020 on main while this branch was open. The merge brought
main's rules into a tree whose guard demands rows for them, so workspace-test
went RED with:

    validator.rs declares rules with no row in the pv-surface decision table:
      ["SCHEMA-018", "SCHEMA-019", "SCHEMA-020"]

That is a MERGE SEAM, not a defect in either side, and the guard is right.

ALSO_ABSENT_FROM_BASELINE was deliberately not used. paiml#2589 exists because
contracts-pv had 76 features at 0% coverage; satisfying its own guard with a
bookkeeping entry would make the ticket self-defeating. This is the same
resolution CRUX-001/002 got when they collided in the other direction: three
REAL trip cases driving the pv BINARY against contracts that actually carry the
offending shape, asserting the rule id and a non-zero exit.

Each rule is tripped on its OWN terms, read out of `validate_top_level_keys`.
They share a function but not a mechanism, so a single fixture would have
proved one rule three times:

  SCHEMA-018  a literal top-level `kind:`. Value is `KernelContract`, the exact
              string the 72 mislabelled registry contracts carried, so a pv
              that READ the key instead of reporting it would also accept it.
  SCHEMA-019  a NEAR-MISS of a real block name — `falsification_test:`, the
              singular. Different arm: unknown AND `near_miss_of` resolves it.
              This row is what makes 018 and 019 discriminable through the CLI;
              a pv reporting every unknown key under one id fails whichever it
              did not print.
  SCHEMA-020  a DUPLICATE MAPPING KEY, which fires off `strict_yaml_error`, not
              the unknown-key loop at all. The duplicate is nested inside an
              UNKNOWN top-level block on purpose: a duplicated *known* key is a
              hard serde "duplicate field" parse error rather than SCHEMA-020,
              and `commands:` is not a near-miss of any contract field, so the
              row cannot be satisfied by a pv emitting SCHEMA-019 instead. It
              reproduces contracts/apr-cli-commands-v1.yaml, the file the rule
              was born from.

Tripping these needs a top-level SIBLING of `metadata:`, not an edit to a field
the schema knows, so `Fixture` gains an `extra` slot rendered after `qa_gate:`.
Empty by default, which preserves the one-swap-per-case rule.

MEASURED, not assumed. Each fixture run directly against the binary reports
`1 error(s), 0 warning(s)` and rc=1 — exactly one rule each, so attribution is
clean and the three are mutually exclusive.

MUTATION-VERIFIED, both directions, mutation proven to have engaged each time:

  drop the three rows      -> every_rule_in_the_validator_source_appears_in_the_table
                              RED with the CI message byte-for-byte
  `kind:` -> `family:`     -> SCHEMA-018 RED ("expected a `[ERROR] SCHEMA-018`
                              line", pv printed "Contract is valid")
  near-miss -> plain key   -> SCHEMA-019 RED
  drop the duplicate key   -> SCHEMA-020 RED
  put the block in Default -> validate_baseline_is_silent_and_exits_zero RED,
                              i.e. the negative direction still discriminates

Restored file md5 identical to the pre-mutation copy; all four CI-wired pv
integration targets green (114 tests).

Header counts re-measured: 27 -> 30 diagnostic rules.

Refs paiml#2589, paiml#2554
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant