fix(backup): alert when a rotation leaves the repo unopenable (audit M14) - #444
Draft
passcod wants to merge 1 commit into
Draft
fix(backup): alert when a rotation leaves the repo unopenable (audit M14)#444passcod wants to merge 1 commit into
passcod wants to merge 1 commit into
Conversation
`Recovery::Broken` — neither the committed passphrase nor the in-flight candidate opens the repo — is documented as an alerting condition, but its only sink was `bail!`, i.e. an `error!` line. Every comparable backup failure files an issue. That state is as bad as it gets for a group: backups and restores are both dead, and Canopy can't recover on its own because it holds the only copies of both passphrases and neither works. Meanwhile the dashboard stayed green, every device backup silently failed against an unopenable repo, and the first operator-visible signal was backup-staleness noticing hours later. It now files a group-scoped `backup-rotation-broken` check, escalating — restorability is already gone, so this must not wait out incident grace — with documentation pointing at object-lock format-blob recovery. A later reconcile that does open the repo (abandon or promote) clears it. Filing is best-effort around the existing `bail!`: the caller has already reached its verdict and a failure to record must not mask it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes M14 (medium) from the audit in #370.
The bug
Recovery::Broken— neither the committed passphrase nor the in-flight candidate opens the repo — is documented as an alerting condition ("corrupt/locked; alert"), but its only sink wasbail!, which surfaces as anerror!line. Every comparable backup failure files an issue.That state is as bad as it gets for a group: backups and restores are both dead, and Canopy can't recover on its own, because it holds the only copies of both passphrases and neither works. Meanwhile the dashboard stayed green, every device backup silently failed against an unopenable repo, and the first operator-visible signal was
backup-stalenessnoticing hours later. The condition also recurs every rotation period, so it re-logs and re-drops indefinitely.The fix
A new group-scoped check,
backup-rotation-broken, filed throughfile_checklike every other backup alert:default_escalates: true) — restorability is already gone, so it must not wait out incident grace;ROTATION_BROKEN_DOC) with the actual remedy: recover the format blob from object-lock history, don't run maintenance in the meantime;Filing is best-effort around the existing
bail!— the caller has already reached its verdict, and failing to record it must not mask it.The two alert helpers take a connection rather than the
Workerso they're directly testable; a smallwith_db_best_effortwrapper keeps the pool-checkout-and-log behaviour at the call site.Tests
broken_repo_files_an_escalating_group_alert_and_clears(jobs, against aTestDb) — asserts the alert exists, is active, isfailed, and escalates; that re-filing coalesces into one issue rather than piling up each period; and that clearing deactivates it while keeping the row as history.The kopia-driven path into
Brokenisn't covered — it needs a real kopia binary — but the decision that reaches it is already unit-tested byreconcile_decision_covers_all_states.Generated by Claude Code