fix(backup): stop passphrase rotation being starved by maintenance (audit H5) - #429
Draft
passcod wants to merge 1 commit into
Draft
fix(backup): stop passphrase rotation being starved by maintenance (audit H5)#429passcod wants to merge 1 commit into
passcod wants to merge 1 commit into
Conversation
`jitter_slot(group, window)` is a pure function of the group and the window, so two jobs sharing a window land on the identical second for every group. The default rotation period is a week — exactly full maintenance's window — so rotation's slot was always maintenance's slot. They also share the per-group in-flight lock, and maintenance holds it for minutes, so on the collision minute maintenance claimed the group first and rotation's one eligible tick found it busy and skipped. Rotation used `slot_is_due`, which fires only on the tick inside a single 60s slot and never catches up, so that skip deferred the rotation a full period — every period. The passphrase simply never rotated for any group whose full maintenance takes over a minute, i.e. all of them, and the only trace was a debug-level "rotation tick ok". Two changes, both needed: - `jitter_slot_in(domain, ..)` draws the slot from a per-job space, so rotation no longer schedules itself onto maintenance's second. A test pins this for 500 groups rather than on average. - Rotation moves to the deadline-with-catch-up rule the other periodic jobs already use, so losing its moment to another op costs a tick rather than a period. That needs a persisted anchor, hence the new `repo_password_rotated_at` column, stamped on each successful rotation. Existing unseeded slots are unchanged, so no group's maintenance moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGfH1cdFKPnKpM7ytRThft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes H5 (high) from the audit in #370.
The bug
jitter_slot(group, window)is a pure function of the group id and the window, so two jobs sharing a window land on the identical second for every group. The default rotation period is 7 days — exactly full maintenance's window — so rotation's slot was always maintenance's slot, for every group in the fleet. (86400 divides 604800, so it collided with the daily quick-maintenance time-of-day too.)They also share the per-group in-flight lock. Maintenance uses
slot_deadline_due(stays due until it runs, so it always wins) and holds the group for minutes; rotation usedslot_is_due, which fires only if a tick lands inside a single 60-second window and never catches up. So on the collision minute maintenance claimed the group, rotation's only eligible tick found it in-flight and skipped, and the window passed.Net effect: the passphrase never rotated for any group whose full maintenance takes longer than a minute — effectively all of them — and the only trace was a debug-level
rotation tick ok.The fix
Two changes, both needed. Domain separation alone still leaves rotation with one no-catch-up tick per week; catch-up alone still puts it behind maintenance every period.
1. Domain-separated slots.
jitter_slot_in(domain, group, window)draws the slot from a per-job space (FNV-1a over the domain name, XORed into the group hash — FNV rather thanDefaultHasherbecause the latter explicitly doesn't promise stability across processes or releases, and a slot that moves between deploys defeats the point). Rotation uses domain"rotation". Unseededjitter_slotis unchanged by construction, so no group's maintenance slot moves.2. Deadline scheduling with catch-up. Rotation moves to
slot_deadline_due_in, the domain-seeded form of the rule maintenance and inspection already use, so losing its moment to another op costs a tick rather than a period.That needs a persisted anchor, which rotation didn't have — hence the migration adding
server_group_backup_config.repo_password_rotated_at, stamped on each successful rotation (best-effort: the passphrase has rotated, so failing the op over the bookkeeping write would misreport it). Existing rows are NULL, i.e. due at their next target, which is correct — their passphrase is already older than a period.Tests
domain_seeded_slots_never_coincide_with_the_unseeded_one— checks 500 groups over the weekly window, not just an average.domain_seeded_slots_are_stable_bounded_and_domain_distinct.seeded_deadline_still_catches_up_after_a_missed_slot— the catch-up property survives the seeding, and a rotation that happens is done for the window.Spec
Added one sentence to BKJ's rotation section: losing the per-group interlock defers rotation within the period, not to the next one. The existing "rotates on a cadence" requirement was already being violated; this makes the contention case explicit.
Generated by Claude Code