Context
When a snapshot file is missing, assert_match_snapshot writes it and counts a
snapshot assertion (src/assert_snapshot.sh:24-27,
bashunit::snapshot::initialize). The run exits 0. There is no mode in which
a first-time recording is treated as a failure.
Evidence
$ cat snap_test.sh
function test_new_snapshot_is_silently_recorded() {
assert_match_snapshot "hello world"
}
$ rm -rf snapshots && ./bashunit snap_test.sh >/dev/null 2>&1; echo $?
0
docs/ai-agents.md:67 already notes the underlying problem: exit code alone does
not distinguish a run that only recorded new snapshots. But nothing lets a caller
make that state fail.
Proposal
Add a way to require that every snapshot already exists — a --snapshot-ci /
--no-snapshot-create flag, or fold it into the existing --strict mode, so a
missing snapshot fails the test instead of being created:
✗ Failed: New snapshot is silently recorded
Snapshot does not exist: snapshots/snap_test_sh.test_....snapshot
Run without --snapshot-ci to record it.
Folding it into --strict is the smaller surface, but --strict currently means
set -euo pipefail for test bodies, which is a different axis — a separate flag
is probably cleaner. Worth deciding in discussion.
Pairs naturally with #900: one flag records deliberately, the other forbids
recording by accident.
Why this matters for agentic coding
The failure mode is a green run that asserted nothing. If a snapshot is
gitignored, never committed, or lost, CI records it on the fly and passes — and an
agent reading that green run reports the behaviour as verified. Agents are
especially exposed here because they add snapshot tests in bulk and trust the exit
code as their signal that work is done; a mode that fails on a missing snapshot is
what makes that signal trustworthy.
Acceptance criteria
Context
When a snapshot file is missing,
assert_match_snapshotwrites it and counts asnapshot assertion (
src/assert_snapshot.sh:24-27,bashunit::snapshot::initialize). The run exits 0. There is no mode in whicha first-time recording is treated as a failure.
Evidence
docs/ai-agents.md:67already notes the underlying problem: exit code alone doesnot distinguish a run that only recorded new snapshots. But nothing lets a caller
make that state fail.
Proposal
Add a way to require that every snapshot already exists — a
--snapshot-ci/--no-snapshot-createflag, or fold it into the existing--strictmode, so amissing snapshot fails the test instead of being created:
Folding it into
--strictis the smaller surface, but--strictcurrently meansset -euo pipefailfor test bodies, which is a different axis — a separate flagis probably cleaner. Worth deciding in discussion.
Pairs naturally with #900: one flag records deliberately, the other forbids
recording by accident.
Why this matters for agentic coding
The failure mode is a green run that asserted nothing. If a snapshot is
gitignored, never committed, or lost, CI records it on the fly and passes — and an
agent reading that green run reports the behaviour as verified. Agents are
especially exposed here because they add snapshot tests in bulk and trust the exit
code as their signal that work is done; a mode that fails on a missing snapshot is
what makes that signal trustworthy.
Acceptance criteria
in the message
assert_match_snapshotandassert_match_snapshot_ignore_colorsdocs/configuration.mddocs/snapshots.mdas the CI settingtests/acceptance/make sa+make lintgreen