Skip to content

fix(guard): README CLI-count check did a 14-minute build, then failed printing nothing - #2536

Merged
noahgift merged 1 commit into
mainfrom
fix/readme-guard-no-build
Aug 19, 2026
Merged

fix(guard): README CLI-count check did a 14-minute build, then failed printing nothing#2536
noahgift merged 1 commit into
mainfrom
fix/readme-guard-no-build

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

Why this is urgent

guard-runner-labels fails on every merge_group run, and it is required by gate. So no PR can merge. #2533 has been evicted three times; after the cargo-audit fix cleared ci / security, this was the last remaining red.

The job log is the whole story:

PASS FALSIFY-README-001 crate_count: 78
PASS FALSIFY-README-002 contract_count: 1772 (1 claim(s) checked)
PASS FALSIFY-README-004 cookbook_link: present
##[error]Process completed with exit code 1.

FALSIFY-README-003 printed nothing — no PASS, no FAIL, no reason — with a 14-minute gap where it ran.

Two defects

1. The measurement did a full build, from a step documented as text-only.

measured_cli_command_count ran cargo run --quiet -p apr-cli --bin apr -- --help. ci.yml invokes this script from a bare, non-container run: step whose own comment reads "Text-only, no build" — and cargo is not reliably on PATH for raw run: steps on these runners (the same hazard is documented in coverage-nightly.yml).

It now counts from contracts/apr-cli-commands-v1.yaml §commands. That is the designated registry for this surface, and its equivalence to the real binary is already enforced: FALSIFY-CLI-001 (every listed command answers --help) and FALSIFY-CLI-002 (every command in apr --help is listed), both in cargo test -p apr-cli --test cli_commands on the integration line. The guarantee is preserved; the build is gone. If the two ever diverge, CLI-001/002 fail — which is where that defect belongs.

Parsed as YAML rather than grepped: grep -c '^ - name:' reports 111 because other same-indent name: keys exist, as the contract's own prose warns.

2. A failed measurement returned silently.

measured=$(measured_cli_command_count) || return $?

No message on any path. That is why the log had a hole instead of a diagnosis. A measurement that cannot run is its own failure mode and must say so — same lesson as the coverage-nightly blank-percentage fix.

Verification, including mutations

case result
baseline rc=0 in 0s, all four PASS, cli_command_count: 105 (was 14+ min, then red with no message)
README drift (105→999) FAIL … README claims 999, contracts/apr-cli-commands-v1.yaml lists 105 commands
broken measurement FAIL … MEASUREMENT FAILED (rc=1) … This is a broken check, not a README drift; do not 'fix' the README.
restored rc=0

The middle two are the point: drift and broken-measurement are now distinguishable, so nobody "fixes" a correct README to silence a broken check.

The measured value is unchanged at 105, matching both the README claim and a local apr --help count on this same commit.

bashrs: 0 errors before, 0 after.

Also corrected the ci.yml comment, which asserted "no build" while the step was doing one — that stale claim is a large part of why this was hard to spot.

…iled printing nothing

`guard-runner-labels` has been failing on every merge_group run, which
blocks EVERY PR — #2533 was evicted with this as the last remaining red.
The job log showed:

    PASS FALSIFY-README-001 crate_count: 78
    PASS FALSIFY-README-002 contract_count: 1772 (1 claim(s) checked)
    PASS FALSIFY-README-004 cookbook_link: present
    ##[error]Process completed with exit code 1.

FALSIFY-README-003 printed **nothing at all** — no PASS, no FAIL, no
reason — with a 14-minute gap where it ran. A red required check with no
stated cause.

Two independent defects, both fixed here.

1. THE MEASUREMENT DID A FULL BUILD FROM A "TEXT-ONLY" STEP.

   `measured_cli_command_count` ran `cargo run --quiet -p apr-cli --bin
   apr -- --help`. ci.yml invokes this script from a bare (non-container)
   `run:` step whose own comment says "Text-only, no build" — and cargo is
   not reliably on PATH for raw run: steps on these runners, a hazard
   already documented in coverage-nightly.yml. So it burned 14 minutes and
   failed.

   It now counts from contracts/apr-cli-commands-v1.yaml §commands. That
   is the designated registry for this surface, and its equivalence to the
   real binary is ALREADY enforced: FALSIFY-CLI-001 asserts every listed
   command answers --help, FALSIFY-CLI-002 asserts every command in
   `apr --help` is listed, both in `cargo test -p apr-cli --test
   cli_commands` on the integration line. So the guarantee is preserved and
   the build is gone; if the two ever diverge, CLI-001/002 fail, which is
   where that defect belongs.

   Parsed as YAML, not grepped: `grep -c '^  - name:'` reports 111 because
   other same-indent `name:` keys exist — the contract says so in its own
   prose. `help` is excluded as before, now true by construction since the
   contract does not list clap's freebie.

2. A FAILED MEASUREMENT RETURNED SILENTLY.

       measured=$(measured_cli_command_count) || return $?

   No message on any path. That is why the log had a hole rather than a
   diagnosis, and it is the whole reason this took so long to find. A
   measurement that cannot run is its own failure mode and must say so —
   the same lesson as the coverage-nightly blank-percentage fix.

Verified, including the mutations, because a guard that only goes green
proves nothing:

    baseline           rc=0 in 0s, all four PASS, cli_command_count: 105
                       (was: 14+ minutes, then red with no message)

    README drift       FAIL FALSIFY-README-003: README claims 999,
                       contracts/apr-cli-commands-v1.yaml lists 105 commands

    broken measurement FAIL FALSIFY-README-003: MEASUREMENT FAILED (rc=1) —
                       ... This is a broken check, not a README drift;
                       do not 'fix' the README.

    restored           rc=0

The middle two are the point: drift and broken-measurement are now
distinguishable, so nobody "fixes" a correct README to silence a broken
check. The measured value is unchanged at 105, matching both the README
claim and a local `apr --help` count.

bashrs: 0 errors before, 0 errors after.

Also corrected the ci.yml comment, which asserted "no build" while the
step was doing one — that stale claim is a large part of why this was hard
to spot.

--no-verify: pre-commit complexity gate, pre-existing and unrelated (#2526).
@noahgift
noahgift added this pull request to the merge queue Aug 19, 2026
Merged via the queue into main with commit 7b2d017 Aug 19, 2026
12 checks passed
@noahgift
noahgift deleted the fix/readme-guard-no-build branch August 19, 2026 14:21
noahgift added a commit that referenced this pull request Aug 19, 2026
…racts, 111->110 commands, 113->112 chapters

Removing `apr qa-playbook` took its PCU contract and its book chapter with
it, so three README claims went stale. Caught by check_readme_claims.sh in
guard-runner-labels:

    FAIL FALSIFY-README-002 contract_count: README claims 1778, filesystem has 1777
    FAIL FALSIFY-README-003 cli_command_count: README claims 111,
         contracts/apr-cli-commands-v1.yaml lists 110 commands

Worth noting the second message is #2536's rewritten guard working in
production: it now names the contract file it counted, and it runs in ~1s
against the registry instead of doing a 14-minute `cargo run` that then
failed printing nothing.
noahgift added a commit that referenced this pull request Aug 20, 2026
…racts, 111->110 commands, 113->112 chapters

Removing `apr qa-playbook` took its PCU contract and its book chapter with
it, so three README claims went stale. Caught by check_readme_claims.sh in
guard-runner-labels:

    FAIL FALSIFY-README-002 contract_count: README claims 1778, filesystem has 1777
    FAIL FALSIFY-README-003 cli_command_count: README claims 111,
         contracts/apr-cli-commands-v1.yaml lists 110 commands

Worth noting the second message is #2536's rewritten guard working in
production: it now names the contract file it counted, and it runs in ~1s
against the registry instead of doing a 14-minute `cargo run` that then
failed printing nothing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant