[cerebrate]

Summary

Phase 2: restore testing that runs on a workstation with Docker, never on the cluster.

scripts/restore-test.sh /mnt/db-backups              # the latest set of every target
scripts/restore-test.sh --set 20261009T021500Z /mnt/db-backups mariadb-main

For each target, the script:

• starts a throwaway scratch server matching the recorded server version (postgres:<major>, mariadb:<major.minor>, or mongo:<major.minor> as a single-node replica set, so --oplogReplay behaves as in production). Each one gets its own compose project and network, with no host ports.
• restores the set with the normal restore path, then checks the result against the set's recorded inventory.
• removes everything, then prints a summary with a pass/fail and the restore time for each target. It exits non-zero if any target fails.

The host needs nothing but Docker. Backups are mounted read-only and read as the backup root's owner, because pg_dump's directory archives are mode 0700.

Options: --set, --image, --build (build from the checkout), --tmpfs (scratch data in RAM), --keep (leave the scratch server up). The SCRATCH_IMAGE_* variables override the scratch images, e.g. to rehearse an upgrade.

New subcommands (no config file needed)

• dbbackup inspect PATH [--format text|env|json] describes a set. The script uses the env format to choose the scratch image.
• dbbackup restore-test PATH --host … | --uri-env … verifies the set, restores it, and compares it with the inventory:
  • Errors: a missing database, table/view/materialized view/collection, or user/role, or an object restored as a different kind.
  • Warnings: row-count differences.

Row counts

The first runs showed InnoDB's TABLE_ROWS estimates on freshly loaded tables as 41 and 0 for tables holding 3,000 and 500 rows. Warnings built on those would be misleading, so:

• The restored side is always counted exactly, since the scratch server is disposable.
• InnoDB estimates are no longer recorded (they're stored as unknown). Aria/MyISAM counts, pg_class.reltuples and Mongo's metadata counts are still recorded, and compared within a tolerance (25%, and at least 100 rows).
• New option exact_counts (in defaults or per target, off by default) records COUNT(*) for every table at backup time, compared exactly. It scans every table nightly, hence off by default.
• 0.1.x MariaDB sets, which recorded InnoDB estimates, have their counts skipped, with a note saying why.

Also

• The scratch-server start is retried up to three times. MariaDB's first-boot initialization can overrun its entrypoint's 30-second limit on busy hosts, which showed up here on WSL.
• make restore-test-check backs up the compose stack and runs the script on the result. CI now runs it after the integration suite.
• Version 0.2.0: the kustomize newTag and the docs' image references are bumped.

Test plan

• make lint; make test: 93 unit tests, including inventory comparison, exact vs estimated vs legacy counts, open_set, inspect env quoting, and the exact_counts config
• make integration: 17 end-to-end tests, now also asserting exact MariaDB counts (3000 / 500) and the counts mode in manifests
• make restore-test-check: all four targets PASSED, with no warnings
• A corrupted set fails on the checksum check, and in a mixed run the script prints the summary and exits 1
• An explicitly named target with no sets reports ERROR (no usable set) and exits 1
• A simulated 0.1.x MariaDB manifest with a wrong InnoDB estimate passes, with counts skipped and the note
• No scratch containers, networks or volumes are left behind
• CI
• Run against the real backup share's 0.1.1 sets

🤖 Generated with Claude Code