feat(db)!: retry a failed run, and report the stack's verdict

[aaaaahaaaaa]

What

Phase 2 of the retry system: a failed run queues its own next attempt, backfills and hooks report the stack's outcome rather than any one attempt's.

Implements the retry spec sections 5 to 8, following the phase 2 plan. Builds on phase 1 (#364), which added the policy and operation-level retry.

The shape

A run is one attempt; a stack is the attempts of one unit of work, keyed by a new root_run_id. scheduled_for holds the backoff.

The successor is created in RunStore.complete, the single terminal path every failure takes: executor verdict, executor exception, queue launch failure, reaper timeout. Doing it there rather than in a sweeping controller is the load-bearing choice, because it makes "has a successor" true in the same transaction that marks the run failed. That is what lets the hook evaluator gate on the successor's existence without knowing anything about budgets or backoff, with no window in which a doomed attempt looks final.

Everything else follows from the stack:

• Dispatch: the claim skips a run whose scheduled_for has not passed, so a retry waits without a status of its own and without holding the head of the queue.
• Backfills: the verdict reads each stack's latest attempt, so an attempt a later one healed no longer condemns the batch, while a queued successor still counts as in flight and keeps it open. This is what the earlier manual-retry work punted, and it cannot be punted: a cron lookback window is a backfill.
• The executions view: ranks by attempt before severity, so a healed operation reads as the success it ended as instead of being outranked forever by its own first attempt. Gains an attempts column.
• Hooks: observe a verdict, never an attempt. The context gains the stack's position, so a message can say the work succeeded on the second attempt or failed after three.

Nothing retries until a job declares a policy. There is no instance-wide default, by design.

For the reviewer

Three constraints found during implementation, each of which would have shipped a bug:

The migration must be idempotent. create_all builds tables from the models and then runs the chain, so on a fresh database the columns already exist by the time 004 runs. I confirmed the non-idempotent form fails (column "root_run_id" of relation "runs" already exists) before fixing it. Without IF NOT EXISTS this breaks make dev-reset and every new deployment.

attempts is the view's last column. CREATE OR REPLACE VIEW may only append; inserting a column mid-list reads to Postgres as renaming the one that was there, and it refuses. Only a live server catches this, which is why there is now an integration test for the view rather than relying on the SQLite stand-in that exercises the mapping, not the query.

root_run_id is stamped by a before_insert listener, not at each creation site. It cannot be a column default because it references the row's own id, and a table=True model skips pydantic validation so a validator never fires. The plan called for assigning it at every creation site; there are 37 Run(...) constructions, and every future one would have been a trap. All 37 are untouched.

One existing test changed legitimately: the hook metadata assertion, which now carries attempt and attempts.

Migration

Revision 004: two columns, a recursive backfill folding existing retry_of chains into stacks, two indexes, and the view replacement. A run whose predecessor was deleted becomes its own root, which is correct: its lineage is gone.

Verified against a populated database as well as a fresh one. On a populated one all existing runs came out correctly self-rooted with no nulls.

Verification

uv run ruff check, uv run ty check and uv run pytest: 2905 passed, 9 skipped. The 9 skips are the Postgres integration tests, which pass against a live server: INTERLOPER_TEST_POSTGRES_DSN=... uv run pytest packages/interloper-db/tests/migrations -m integration gives 9 passed.

Next

Phase 3 (surfaces) makes the runs listing stack-native and teaches the app to read a stack as one piece of work.

By Digitl

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!