name content-addressed objects by blake3 instead of sha256 - #10363
Open
ThomasWaldmann wants to merge 3 commits into
Open
name content-addressed objects by blake3 instead of sha256#10363ThomasWaldmann wants to merge 3 commits into
ThomasWaldmann wants to merge 3 commits into
Conversation
Every object whose name is a digest of its content is now named by the 256 bit blake3 hash instead of sha256: packs/, index/ fragments, keys/ (repokey borg keys) and locks/ in the repository, automatically placed keyfiles (the key id is the same hash) and the files cache suffix. One helper, crypto.key.blake3_256(), computes the hash for all of them and hashes big inputs (packs, index fragments) multi-threaded from the existing blake3 threshold on; Blake3ChecksumKey.id_hash reuses it. Repository.NAME_HASH names the algorithm for borgstore's hash() and defrag(), which already support blake3. Not changed: checksums that are appended to content rather than used as a name (checked-packs tracker, archive reference caches) and the archive ids, which are the key mode's id hash. Not backwards compatible with repositories and files caches written by earlier borg2 betas. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #10363 +/- ##
==========================================
+ Coverage 87.93% 88.05% +0.12%
==========================================
Files 103 103
Lines 18893 18919 +26
Branches 2917 2919 +2
==========================================
+ Hits 16613 16660 +47
+ Misses 1583 1566 -17
+ Partials 697 693 -4 ☔ View full report in Codecov by Harness. |
Object names are hex, so give the naming code a hex helper instead of converting the bytes helper's result at every call site: the keyfile, repokey, lock, index fragment and files cache naming now use blake3_256_hex(). The tests use blake3(...).hexdigest() directly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The checked-packs tracker (cache/checked-packs) and the per-archive reference caches (cache/referenced-by-archive.*) carry a checksum of their content appended to it. Use blake3_256() for it, like for the object names, so sha256 is not used for any store object checksum any more. An object with an old checksum fails verification and is rebuilt. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every object whose name is a digest of its content is now named by the 256 bit blake3 hash instead of sha256:
packs/,index/fragments,keys/(repokey borg keys) andlocks/in the repository,files.<hash of the series name>).The checksums that are appended to content rather than used as a name (the checked-packs tracker and the per-archive reference caches) are blake3 as well, so sha256 is not used for any content addressing or integrity checksum of store objects any more.
Two helpers in
crypto.key,blake3_256()(bytes) andblake3_256_hex()(the name form), compute the hash for all of them and hash big inputs (packs, index fragments) multi-threaded from the existing blake3 threshold on;Blake3ChecksumKey.id_hashreuses the former.Repository.NAME_HASHnames the algorithm for borgstore'shash()anddefrag(), which already support blake3 (borg already depends onborgstore[blake3]).Not changed: the archive ids, which are the key mode's id hash.
Not backwards compatible with repositories and files caches written by earlier borg2 betas (as agreed, no compatibility with older betas is needed). A files cache from an older beta is not found any more, so the first backup per series re-reads its files once; a checked-packs tracker or reference cache from an older beta fails its checksum and is rebuilt.
Note for sftp repositories: the sftp check-file extension has no blake3, so
borg checkdownloads packs to hash them client-side (borgstore falls back to that on its own). The REST server hashes with blake3 server-side as before.Docs (
data-structures.rst,packs.rst,security.rst,faq.rst,usage/key.rst), the check command help text and the tests are updated accordingly.🤖 Generated with Claude Code