store cache: a corrupt cache entry is treated as repository corruption, check --repair deletes the healthy chunk

[ThomasWaldmann]

Summary

With BORG_STORE_CACHE enabled, a damaged file in the local pack cache is handled as if the repository was damaged:

• borg extract replaces the affected chunk by zeros and says it is "missing or corrupted in the repository",
• borg check --verify-data reports an integrity error for a repository that is fine,
• borg check --repair --verify-data deletes the healthy chunk from the repository - real data loss caused by a cache defect.

The bad cache entry is never invalidated or re-fetched from the primary storage, so the state persists over any number of runs.

Tested with master ec4fe8a + borgstore 0.6.3 (main 258fbd8), file:// repository, macOS.

How to reproduce

export BORG_REPO=/path/repo BORG_PASSPHRASE=test BORG_PACK_MAX_SIZE=8000000
borg repo-create -e aes256-ocb
borg create arch1 src                      # ~350 MB of random data -> 42 packs
export BORG_STORE_CACHE=/path/cache        # no size limit, so everything gets cached
mkdir out && (cd out && borg extract arch1)    # fills the cache
# flip one bit in one cached pack - the repository is not touched:
f=$(find /path/cache/packs -type f -size +7000k | sort | head -1)
python3 -c 'import sys; f=open(sys.argv[1],"r+b"); f.seek(3_000_000); b=f.read(1); f.seek(3_000_000); f.write(bytes([b[0]^1]))' $f

Observed:

$ borg extract arch1                       # rc 113
repository object b7c29d8a... corrupted (Data integrity error: Chunk b7c29d8a...: Could not decrypt
  [Authentication / EVP_DecryptFinal_ex failed]), returning 2107267 zero bytes.
src/big5/f017.bin: 1 chunk(s) missing or corrupted in the repository, replaced by all-zero data
Store cache hits: 46    Store cache misses: 0          <- everything came from the cache

$ borg check --verify-data                 # rc 1
chunk b7c29d8a..., integrity error: Data integrity error: Chunk b7c29d8a...: Could not decrypt [...]
Finished cryptographic data integrity verification, verified 3176 chunks with 1 integrity errors.

$ BORG_STORE_CACHE= borg check --verify-data    # rc 0 - the repository is fine

The cached pack still differs from the repository pack after these runs (cmp), nothing heals it.

Then, on a copy of the repository (same cache state):

$ borg check --repair --verify-data        # rc 0
chunk b7c29d8a..., integrity error: ...
Finished cryptographic data integrity verification, verified 3176 chunks with 1 integrity errors.
The following chunks are missing in the repository: ...

# now WITHOUT the cache:
$ borg check --verify-data                 # rc 1: "The following chunks are missing in the repository"
$ borg extract arch1                       # rc 113
repository object b7c29d8a... missing, returning 2107267 zero bytes.

packs/ of the repository copy shrank from 347.9 MB to 345.8 MB: the chunk that was perfectly fine in the repository is gone, 2.1 MB of f017.bin are lost.

A truncated cache file behaves alike ("object size inconsistent: expected 1605502 bytes, got 1000000", "object too small: expected at least 49 header bytes, got 0", zeros in the extracted files). borgstore's posixfs backend writes via temp file + rename, so truncation is less likely than bit rot, but the cache usually lives on less trustworthy storage than one would accept for a repository and nobody ever checks it.

Why

borgstore's writethrough load() returns whatever the cache backend has (also for ranged reads), without verification and without contacting the primary backend. borg can not tell where the bytes came from, so an AEAD authentication failure (or a bad object header) is attributed to the repository, and check's "re-read the defect chunk" retry reads the same cached bytes again.

The inverse case follows from the same mechanism: as long as a pack is served from the cache, borg check --verify-data verifies the cached copy, not the pack in the repository - corruption in the repository stays unnoticed for cached packs. (Not measured with check, but borg extract succeeds with a warm cache after moving the repository's packs/ directory away, i.e. the primary storage is not consulted at all on a cache hit.)

Ideas

1. borg check should not use the store cache at all (at the very least not with --repair). Its job is to verify the primary storage; with the cache it verifies the wrong thing in both directions (false positives → --repair deletes good data; false negatives → repository damage masked by good cached copies). Small borg-side change: do not pass cache_url to the Store for check.
2. Invalidate and retry for all readers: when an object read via the cache fails (header check / size check / decryption), call store.cache_invalidate("packs/<pack id>"), read again from the primary storage and only then treat it as repository corruption. Would make extract / mount / transfer / recreate self-healing against cache defects. Needs a place to hook in, because Repository.get() / get_many() return the raw bytes and the authentication failure only shows up later in RepoObj.parse().
3. Verification in borgstore: packs are named by the store hash of their content, so a cache entry can be verified against its name - cheap relative to the transfer for whole-item loads (that is what get_many does), not possible per ranged read without hashing the whole item. Could be an optional cache policy ("name is <algorithm> hash of the content": verify on full loads, or once per session on first use of an entry).
4. Error messages should not say "in the repository" when the data came from the cache (needs borgstore to tell, or follows from 2.).

1 closes the data-loss path and is tiny; 2 would fix the rest of the user-visible behaviour.

Related: #10396 (shared store cache dir vs. repo-create / repo-delete).