fix(dogfood): the deepest probe in the sweep had never run once

[noahgift]

Harvested from fix/guard-batch — the one commit of that branch (f031b7c) whose content is not already in main. The branch's other 11 commits landed via #2527 and relatives, so both gaps below are live on main today.

scripts/dogfood_surfaces.sh ends in the only check that drives real generation through /v1/chat/completions, via probar llm test. Two independent gaps meant it never executed a single time — and neither was visible from its output, which printed a tidy skip or a confident FAIL either way.

1. The probe binary was built without --features llm

crates/aprender-test-cli/src/commands.rs:117 declares Llm(LlmArgs) with no #[cfg], so clap advertises llm and renders its --help in every build. Only the handler is gated (main.rs:74-79). A featureless binary therefore parses llm test and then returns:

Error: LLM features not enabled. Rebuild with --features llm

…which the sweep reported as FAIL probar llm test FAILED — blaming the server for a gap in the harness, the very thing the comment three lines below warns against for the config case.

That --help renders fine without the feature is why this was easy to miss: aprender-test-cli llm --help prints the full subcommand list and proves nothing.

2. It skipped unless an operator exported a config that was already committed

It skipped whenever DOGFOOD_PROBAR_CONFIG was unset, on the stated grounds that no committed config existed. One does — tests/fixtures/probar-llm-endpoint.yaml, 3416 bytes, in main. Now defaults to it when present; an explicit env var still wins; a genuinely absent file is still a skip-with-reason, never a FAIL.

Falsifier

scripts/check_dogfood_llm_probe_armed.sh, wired into ci.yml beside the other guards, with a --self-test case table.

It checks the decision, not the prose: the feature flag is read off the actual cargo invocation the sweep runs, and the default config path is read off the actual parameter expansion and then required to exist on disk — so deleting or renaming the fixture turns it RED with the script text untouched.

Mutation verification — both directions, mutation proven to engage

probe	result
against main's dogfood_surfaces.sh, unmodified	rc=1, both gaps named
against this tree	rc=0
--self-test case table	4/4

self-test A (unmutated tree is GREEN): PASS
self-test B (missing --features llm turns RED): PASS
self-test C (absent fixture turns RED): PASS
self-test D (no committed default turns RED): PASS

check_guards_are_wired.sh was RED before the ci.yml wiring (unwired guards grew 1 -> 2, NEW: check_dogfood_llm_probe_armed.sh) and PASS after — the wiring is proven, not assumed.

bashrs lint on the new guard alone: 0 errors. check_shell_lint_ratchet.sh PASS (97 error lines vs baseline 876; the new script adds none). The baseline is deliberately not re-recorded here — that improvement is pre-existing drift and does not belong in this PR.

Scope

3 files: the sweep (+17/-5), the new guard, the ci.yml wiring. No Rust touched. The CI edit is purely additive ("add a gate").

🤖 Generated with Claude Code

https://claude.ai/code/session_01CbMTnT8Upx6Ym18i5H11iR

[noahgift]

Duplicate-work note, and a retraction of the concern I raised on #2579 (now closed in favour of this PR).

I independently harvested the same commit (f031b7c on fix/guard-batch) and reached the same two-gap diagnosis. I verified it behaviourally rather than statically, and the A/B is worth recording here as a second, independent confirmation — same tree, same path, same minute, worktree at 4bbfeb0:

cargo build -p aprender-test-cli --bin aprender-test-cli
./aprender-test-cli llm test --config tests/fixtures/probar-llm-endpoint.yaml --url http://127.0.0.1:1
-> Error: LLM features not enabled. Rebuild with --features llm

cargo build -p aprender-test-cli --bin aprender-test-cli --features llm
./aprender-test-cli llm test --config tests/fixtures/probar-llm-endpoint.yaml --url http://127.0.0.1:1
-> chat_completions_answers_a_wide_margin_arithmetic_question ... ERROR: HTTP error
   system_prompt_is_honoured ... ERROR: HTTP error
   distinct_prompt_gets_a_distinct_answer_not_a_canned_one ... ERROR: HTTP error
   Results: 0/3 passed, 3 failed

With the feature the handler runs, parses the committed fixture, and executes all three cases, failing only because nothing listens on port 1. That is a probe. Without it there is nothing to fail. It also confirms the fixture is well-formed and yields exactly 3 cases.

Retraction. On #2579 I flagged that a text-scanning feature check can be blind — my own first version grepped the file it lived in, so the grep's own pattern string matched and deleting --features from the real build line left it GREEN. I have now read this guard's patch: it scans a different file ($sweep), and self-test case B copies the sweep into a temp tree, deletes the line with sed, and requires RED. The trap does not apply here. Disregard that concern.

One difference worth keeping in mind for a follow-up: your guard requires the sweep to spell --features llm on the cargo build line, which couples it to that line staying a single greppable string. If the invocation is ever moved into an argv array or a variable, the guard goes RED for the wrong reason. Not a change worth making now — just the failure mode to expect if someone refactors that call site.

[noahgift]

Superseded by #2613, the 0.64.0 integration batch.

This PR's commits are merged into batch/release-0-64-0 verbatim (--no-ff, never rebased),
and #2613's body carries the full provenance table — PR number, branch, merged head SHA, and
the issues each closes — so the detail survives the squash.

Why batched rather than landed individually: one workspace-test run is ~58 minutes on a
shared box. Thirteen PRs cost thirteen runs whether they go serially or in parallel; one
integration branch costs one. The same approach landed 24 branches previously.

Batching also found four defects that were invisible to every individual PR — most
notably the README contract count: #2548, #2549 and #2587 each add exactly one contract, each
is individually correct at 1779, and three +1s collide on one literal (correct value 1781).
That is the exact class that killed the previous batch.

Closing now, deliberately: an open PR that merges first moves #2613's base and forces
another full run. This is reversible and the branch is untouched — reopen if #2613 is
abandoned.