Skip to content

name content-addressed objects by blake3 instead of sha256 - #10363

Open
ThomasWaldmann wants to merge 3 commits into
borgbackup:masterfrom
ThomasWaldmann:blake3-pack-id
Open

name content-addressed objects by blake3 instead of sha256#10363
ThomasWaldmann wants to merge 3 commits into
borgbackup:masterfrom
ThomasWaldmann:blake3-pack-id

Conversation

@ThomasWaldmann

@ThomasWaldmann ThomasWaldmann commented Sep 12, 2026

Copy link
Copy Markdown
Member

Every object whose name is a digest of its content is now named by the 256 bit blake3 hash instead of sha256:

  • packs/, index/ fragments, keys/ (repokey borg keys) and locks/ in the repository,
  • automatically placed keyfiles (the key id borg shows is the same hash),
  • the files cache suffix (files.<hash of the series name>).

The checksums that are appended to content rather than used as a name (the checked-packs tracker and the per-archive reference caches) are blake3 as well, so sha256 is not used for any content addressing or integrity checksum of store objects any more.

Two helpers in crypto.key, blake3_256() (bytes) and blake3_256_hex() (the name form), compute the hash for all of them and hash big inputs (packs, index fragments) multi-threaded from the existing blake3 threshold on; Blake3ChecksumKey.id_hash reuses the former. Repository.NAME_HASH names the algorithm for borgstore's hash() and defrag(), which already support blake3 (borg already depends on borgstore[blake3]).

Not changed: the archive ids, which are the key mode's id hash.

Not backwards compatible with repositories and files caches written by earlier borg2 betas (as agreed, no compatibility with older betas is needed). A files cache from an older beta is not found any more, so the first backup per series re-reads its files once; a checked-packs tracker or reference cache from an older beta fails its checksum and is rebuilt.

Note for sftp repositories: the sftp check-file extension has no blake3, so borg check downloads packs to hash them client-side (borgstore falls back to that on its own). The REST server hashes with blake3 server-side as before.

Docs (data-structures.rst, packs.rst, security.rst, faq.rst, usage/key.rst), the check command help text and the tests are updated accordingly.

🤖 Generated with Claude Code

Every object whose name is a digest of its content is now named by the
256 bit blake3 hash instead of sha256: packs/, index/ fragments, keys/
(repokey borg keys) and locks/ in the repository, automatically placed
keyfiles (the key id is the same hash) and the files cache suffix.

One helper, crypto.key.blake3_256(), computes the hash for all of them
and hashes big inputs (packs, index fragments) multi-threaded from the
existing blake3 threshold on; Blake3ChecksumKey.id_hash reuses it.
Repository.NAME_HASH names the algorithm for borgstore's hash() and
defrag(), which already support blake3.

Not changed: checksums that are appended to content rather than used as
a name (checked-packs tracker, archive reference caches) and the archive
ids, which are the key mode's id hash.

Not backwards compatible with repositories and files caches written by
earlier borg2 betas.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.05882% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 88.05%. Comparing base (11d7d10) to head (8b6e1f8).
⚠️ Report is 9 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/borg/cache.py 83.33% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #10363      +/-   ##
==========================================
+ Coverage   87.93%   88.05%   +0.12%     
==========================================
  Files         103      103              
  Lines       18893    18919      +26     
  Branches     2917     2919       +2     
==========================================
+ Hits        16613    16660      +47     
+ Misses       1583     1566      -17     
+ Partials      697      693       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

ThomasWaldmann and others added 2 commits September 12, 2026 20:39
Object names are hex, so give the naming code a hex helper instead of
converting the bytes helper's result at every call site: the keyfile,
repokey, lock, index fragment and files cache naming now use
blake3_256_hex(). The tests use blake3(...).hexdigest() directly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The checked-packs tracker (cache/checked-packs) and the per-archive
reference caches (cache/referenced-by-archive.*) carry a checksum of
their content appended to it. Use blake3_256() for it, like for the
object names, so sha256 is not used for any store object checksum any
more. An object with an old checksum fails verification and is rebuilt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant