Skip to content

fix(backup): alert when a rotation leaves the repo unopenable (audit M14) - #444

Draft
passcod wants to merge 1 commit into
mainfrom
claude/pr-370-fix-m14-rotation-broken
Draft

fix(backup): alert when a rotation leaves the repo unopenable (audit M14)#444
passcod wants to merge 1 commit into
mainfrom
claude/pr-370-fix-m14-rotation-broken

Conversation

@passcod

@passcod passcod commented Aug 1, 2026

Copy link
Copy Markdown
Member

Fixes M14 (medium) from the audit in #370.

The bug

Recovery::Broken — neither the committed passphrase nor the in-flight candidate opens the repo — is documented as an alerting condition ("corrupt/locked; alert"), but its only sink was bail!, which surfaces as an error! line. Every comparable backup failure files an issue.

That state is as bad as it gets for a group: backups and restores are both dead, and Canopy can't recover on its own, because it holds the only copies of both passphrases and neither works. Meanwhile the dashboard stayed green, every device backup silently failed against an unopenable repo, and the first operator-visible signal was backup-staleness noticing hours later. The condition also recurs every rotation period, so it re-logs and re-drops indefinitely.

The fix

A new group-scoped check, backup-rotation-broken, filed through file_check like every other backup alert:

  • escalating (default_escalates: true) — restorability is already gone, so it must not wait out incident grace;
  • documented (ROTATION_BROKEN_DOC) with the actual remedy: recover the format blob from object-lock history, don't run maintenance in the meantime;
  • cleared when a later reconcile does open the repo (either the abandon or the promote path).

Filing is best-effort around the existing bail! — the caller has already reached its verdict, and failing to record it must not mask it.

The two alert helpers take a connection rather than the Worker so they're directly testable; a small with_db_best_effort wrapper keeps the pool-checkout-and-log behaviour at the call site.

Tests

broken_repo_files_an_escalating_group_alert_and_clears (jobs, against a TestDb) — asserts the alert exists, is active, is failed, and escalates; that re-filing coalesces into one issue rather than piling up each period; and that clearing deactivates it while keeping the row as history.

The kopia-driven path into Broken isn't covered — it needs a real kopia binary — but the decision that reaches it is already unit-tested by reconcile_decision_covers_all_states.


Generated by Claude Code

`Recovery::Broken` — neither the committed passphrase nor the in-flight
candidate opens the repo — is documented as an alerting condition, but its
only sink was `bail!`, i.e. an `error!` line. Every comparable backup failure
files an issue.

That state is as bad as it gets for a group: backups and restores are both
dead, and Canopy can't recover on its own because it holds the only copies of
both passphrases and neither works. Meanwhile the dashboard stayed green,
every device backup silently failed against an unopenable repo, and the first
operator-visible signal was backup-staleness noticing hours later.

It now files a group-scoped `backup-rotation-broken` check, escalating —
restorability is already gone, so this must not wait out incident grace — with
documentation pointing at object-lock format-blob recovery. A later reconcile
that does open the repo (abandon or promote) clears it.

Filing is best-effort around the existing `bail!`: the caller has already
reached its verdict and a failure to record must not mask it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants