extract: do not abort on corrupted chunks, replace them by all-zero data with a warning, fixes #840 - #10357
Open
ThomasWaldmann wants to merge 1 commit into
Open
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10357 +/- ##
==========================================
+ Coverage 87.93% 87.95% +0.02%
==========================================
Files 103 103
Lines 18893 18926 +33
Branches 2917 2921 +4
==========================================
+ Hits 16613 16647 +34
+ Misses 1583 1581 -2
- Partials 697 698 +1 ☔ View full report in Codecov by Harness. |
ThomasWaldmann
commented
Sep 11, 2026
…ata with a warning, fixes borgbackup#840 Missing chunks were already replaced by all-zero data of the correct size during extraction (with an error log line, but no warning exit code and no file name), while a corrupted chunk (one that does not authenticate, decrypt or decompress) aborted the whole extraction with an IntegrityError, leaving the file at 0 bytes and all following files unextracted. Now DownloadPipeline.fetch_many treats a corrupted chunk like a missing one if the new replace_corrupted flag is set: it logs an error naming the chunk and yields an all-zero replacement of the correct size. extract (incl. --dry-run and --stdout) and mount -o allow_damaged_files set that flag, so a damaged file is still extracted completely (correct size and metadata, all-zero content where the damaged chunks were), the extraction continues with the next file, and each damaged file is reported with a warning (new BackupDamagedChunksError, rc 113). This also fixes the missing-chunk case, which now gets the same per-file warning and exit code instead of silently exiting with rc 0. recreate and transfer keep raising IntegrityError for corrupted chunks, as they would otherwise store all-zero data as if it were the content (see the *_wrong_chunk_content tests). The FUSE read handlers map a corrupted chunk to EIO (with an error log line) instead of an unhandled exception. Supersedes the borg 1.x-era PR borgbackup#7481 (extract --skip-errors). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ThomasWaldmann
force-pushed
the
extract-corrupt-chunks-840
branch
from
September 11, 2026 16:10
666da6c to
57a1605
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (#840)
borg extractaborted with anIntegrityError(rc 90) on the first chunk that does not authenticate / decrypt / decompress, leaving that file at 0 bytes and all following files unextracted.--continuefailed the same way. The only way out wasborg check --repair --verify-data, which modifies the repository and needs a full verification pass.Missing chunks, on the other hand, were already replaced by all-zero data of the correct size, but only with an error log line: no warning exit code and no file name (see the old TODO in
test_extract_file_with_missing_chunk).Change
DownloadPipeline.fetch_manygets areplace_corruptedflag: with it, a corrupted chunk is handled like a missing one, i.e. an error naming the chunk is logged and an all-zero replacement of the correct size is yielded. Adamagedlist parameter lets the caller learn which chunks were replaced.extract(incl.--dry-runand--stdout) sets the flag. A damaged file is extracted completely (correct size and metadata, all-zero content where the damaged chunks were), the extraction continues with the next file, and each damaged file is reported with a per-file warning (newBackupDamagedChunksError, rc 113). The missing-chunk case gets the same warning and exit code now instead of silently exiting with rc 0.--dry-runthus can be used to find unreadable files without writing anything.mount -o allow_damaged_filescovers corrupted chunks too. Without the option, the FUSE read handlers now map a corrupted chunk toEIO(with an error log line) instead of an unhandled exception.recreateandtransferkeep raisingIntegrityErrorfor corrupted chunks: they would otherwise store all-zero data as if it were the content (the*_wrong_chunk_contenttests still assert that).frontends.rstrc list.Note: after such an extraction,
extract --continueconsiders the damaged file done (size and mtime match), same as before for missing chunks.Tests
test_extract_file_with_damaged_chunk/test_extract_dry_run_with_damaged_chunk, parametrized over missing and corrupted (byte flipped in the pack viacorrupt_chunk_on_disk), check the rc, the warning with the file name, the correct size, that only the damaged chunk's region is zero and that the other files are intact.test_fuse_allow_damaged_filesparametrized the same way.test_verify_data_wrong_chunk_contentadapted: withBORG_ASSERT_ID=read, extract now reports the file (rc 113) instead of aborting.Verified manually as well: 4 MB random file, 4 bytes flipped in the data pack,
extract --dry-runandextractboth report the file with rc 113, the file has the right size, the small file after it is extracted.🤖 Generated with Claude Code