Skip to content

feat(snapshot): add a mode where a missing snapshot fails instead of being recorded #901

Description

@Chemaclass

Context

When a snapshot file is missing, assert_match_snapshot writes it and counts a
snapshot assertion (src/assert_snapshot.sh:24-27,
bashunit::snapshot::initialize). The run exits 0. There is no mode in which
a first-time recording is treated as a failure.

Evidence

$ cat snap_test.sh
function test_new_snapshot_is_silently_recorded() {
  assert_match_snapshot "hello world"
}

$ rm -rf snapshots && ./bashunit snap_test.sh >/dev/null 2>&1; echo $?
0

docs/ai-agents.md:67 already notes the underlying problem: exit code alone does
not distinguish a run that only recorded new snapshots. But nothing lets a caller
make that state fail.

Proposal

Add a way to require that every snapshot already exists — a --snapshot-ci /
--no-snapshot-create flag, or fold it into the existing --strict mode, so a
missing snapshot fails the test instead of being created:

✗ Failed: New snapshot is silently recorded
    Snapshot does not exist: snapshots/snap_test_sh.test_....snapshot
    Run without --snapshot-ci to record it.

Folding it into --strict is the smaller surface, but --strict currently means
set -euo pipefail for test bodies, which is a different axis — a separate flag
is probably cleaner. Worth deciding in discussion.

Pairs naturally with #900: one flag records deliberately, the other forbids
recording by accident.

Why this matters for agentic coding

The failure mode is a green run that asserted nothing. If a snapshot is
gitignored, never committed, or lost, CI records it on the fly and passes — and an
agent reading that green run reports the behaviour as verified. Agents are
especially exposed here because they add snapshot tests in bulk and trust the exit
code as their signal that work is done; a mode that fails on a missing snapshot is
what makes that signal trustworthy.

Acceptance criteria

  • A missing snapshot fails the test under the new mode, with the resolved path
    in the message
  • Default behaviour (record and pass) unchanged
  • Applies to both assert_match_snapshot and
    assert_match_snapshot_ignore_colors
  • Env-var equivalent, documented in docs/configuration.md
  • Recommended in docs/snapshots.md as the CI setting
  • Acceptance tests in tests/acceptance/
  • Bash 3.0+ compatible, make sa + make lint green

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions