Skip to content

fix(backup): stop passphrase rotation being starved by maintenance (audit H5) - #429

Draft
passcod wants to merge 1 commit into
mainfrom
claude/pr-370-fix-h5-rotation-slot
Draft

fix(backup): stop passphrase rotation being starved by maintenance (audit H5)#429
passcod wants to merge 1 commit into
mainfrom
claude/pr-370-fix-h5-rotation-slot

Conversation

@passcod

@passcod passcod commented Aug 1, 2026

Copy link
Copy Markdown
Member

Fixes H5 (high) from the audit in #370.

The bug

jitter_slot(group, window) is a pure function of the group id and the window, so two jobs sharing a window land on the identical second for every group. The default rotation period is 7 days — exactly full maintenance's window — so rotation's slot was always maintenance's slot, for every group in the fleet. (86400 divides 604800, so it collided with the daily quick-maintenance time-of-day too.)

They also share the per-group in-flight lock. Maintenance uses slot_deadline_due (stays due until it runs, so it always wins) and holds the group for minutes; rotation used slot_is_due, which fires only if a tick lands inside a single 60-second window and never catches up. So on the collision minute maintenance claimed the group, rotation's only eligible tick found it in-flight and skipped, and the window passed.

Net effect: the passphrase never rotated for any group whose full maintenance takes longer than a minute — effectively all of them — and the only trace was a debug-level rotation tick ok.

The fix

Two changes, both needed. Domain separation alone still leaves rotation with one no-catch-up tick per week; catch-up alone still puts it behind maintenance every period.

1. Domain-separated slots. jitter_slot_in(domain, group, window) draws the slot from a per-job space (FNV-1a over the domain name, XORed into the group hash — FNV rather than DefaultHasher because the latter explicitly doesn't promise stability across processes or releases, and a slot that moves between deploys defeats the point). Rotation uses domain "rotation". Unseeded jitter_slot is unchanged by construction, so no group's maintenance slot moves.

2. Deadline scheduling with catch-up. Rotation moves to slot_deadline_due_in, the domain-seeded form of the rule maintenance and inspection already use, so losing its moment to another op costs a tick rather than a period.

That needs a persisted anchor, which rotation didn't have — hence the migration adding server_group_backup_config.repo_password_rotated_at, stamped on each successful rotation (best-effort: the passphrase has rotated, so failing the op over the bookkeeping write would misreport it). Existing rows are NULL, i.e. due at their next target, which is correct — their passphrase is already older than a period.

Tests

  • domain_seeded_slots_never_coincide_with_the_unseeded_one — checks 500 groups over the weekly window, not just an average.
  • domain_seeded_slots_are_stable_bounded_and_domain_distinct.
  • seeded_deadline_still_catches_up_after_a_missed_slot — the catch-up property survives the seeding, and a rotation that happens is done for the window.

Spec

Added one sentence to BKJ's rotation section: losing the per-group interlock defers rotation within the period, not to the next one. The existing "rotates on a cadence" requirement was already being violated; this makes the contention case explicit.


Generated by Claude Code

`jitter_slot(group, window)` is a pure function of the group and the window,
so two jobs sharing a window land on the identical second for every group.
The default rotation period is a week — exactly full maintenance's window —
so rotation's slot was always maintenance's slot. They also share the
per-group in-flight lock, and maintenance holds it for minutes, so on the
collision minute maintenance claimed the group first and rotation's one
eligible tick found it busy and skipped.

Rotation used `slot_is_due`, which fires only on the tick inside a single 60s
slot and never catches up, so that skip deferred the rotation a full period —
every period. The passphrase simply never rotated for any group whose full
maintenance takes over a minute, i.e. all of them, and the only trace was a
debug-level "rotation tick ok".

Two changes, both needed:

- `jitter_slot_in(domain, ..)` draws the slot from a per-job space, so
  rotation no longer schedules itself onto maintenance's second. A test pins
  this for 500 groups rather than on average.
- Rotation moves to the deadline-with-catch-up rule the other periodic jobs
  already use, so losing its moment to another op costs a tick rather than a
  period. That needs a persisted anchor, hence the new
  `repo_password_rotated_at` column, stamped on each successful rotation.

Existing unseeded slots are unchanged, so no group's maintenance moves.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants