Skip to content

feat(checks): durable verdict delivery — bucketed positions, quota reserve, reconcile sweep - #237

Merged
debuggingfuture merged 3 commits into
mainfrom
feat/verdict-delivery
Oct 22, 2026
Merged

debuggingfuture merged 3 commits into
mainfrom
feat/verdict-delivery

Conversation

@debuggingfuture

@debuggingfuture debuggingfuture commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

What

Durable verdict delivery from ADR-0009: a computed verdict reaches the pull request even when the installation's GitHub quota runs short.

  • Bucketed queue positions. A waiter writes its queue position only when it crosses a bucket boundary (1, 2-5, 6-20, over 20) and at most once a minute (admission-summary.ts, admission loop in workflow.ts). The check-run shows the bucket, not the exact count.
  • Retry-After and reset honoured. Each retry in the checks client (packages/runtime-cf/src/checks-github.ts) waits at least as long as GitHub asked. A wait longer than 30 s degrades the call and marks the installation exhausted until the wait lapses, so later calls do not spend requests on a wait GitHub already named.
  • Verdict reserve. Every check-run response's x-ratelimit-remaining / -limit / -reset feeds a per-installation quota cache (github-quota.ts). At or under 10% of the window, progress stops; create, update and conclude proceed. The cache holds GitHub's last answer per isolate. The quota itself lives on GitHub and each response refreshes it, so the worst case is one extra call per isolate past the reserve.
  • Create at conclusion. ChecksService.conclude completes the check-run, or creates it already completed when the opener degraded to the noop sentinel id. It is total and returns delivered (with the check-run id), undelivered (worth sending again: rate limit, outage, missing credentials) or refused (404 or 422: GitHub rejects that write for good). The workflow runs it in a durable deliver-verdict step, so a replay never creates a second check-run.
  • executions.delivered_at. Migration 0012 adds delivered_at, delivery_refused_at and installation_id. The workflow stores the installation with the check-run id and stamps delivered_at when the conclusion lands.
  • Reconcile sweep. */10 * * * * (armed in wrangler.jsonc, routed by routes/scheduled.ts) re-sends the conclusion of every finished execution (success / failure / skipped) that has an installation and neither delivered_at nor delivery_refused_at. It works oldest first, skips rows finished less than 10 minutes ago and rows older than 7 days, and handles at most 50 rows a tick. The first undelivered row of an installation stops the sweep for that installation until the next tick. A refused row stamps delivery_refused_at and leaves the queue, so it never holds up the newer rows of its installation. Cron parity exempts the sweep cron and asserts it is armed.

Why

Without a write budget, a burst of queued runs spent an installation's hourly quota on position updates. Check-runs then failed to open or never concluded, and verdicts computed in D1 never reached GitHub, which looks the same as a hang.

Notes

Spec: specs/adr/0009-fast-trustworthy-verdicts.md § Durable verdict delivery (proposed in #232)

Testing: pnpm lint, pnpm typecheck and pnpm test (207 files, 2854 passed) are green. New tests cover the bucket boundaries and the once-a-minute bound (admission-summary.test.ts); quota headers and the observe hook (http.test.ts, check-runs.test.ts); reserve, exhaustion and Retry-After waits (github-quota.test.ts, checks-github.test.ts); create-at-conclusion, undelivered and refused outcomes (checks-github.test.ts); sweep order, bounds, per-installation stop, refused rows leaving the queue and noop-id recording (delivery-sweep.test.ts); and sweep cron routing (scheduled.test.ts, cron-parity.test.ts).

…serve, create-at-conclusion, reconcile sweep

Queue-position updates write only on a bucket change (1, 2-5, 6-20, over 20)
and at most once a minute. The checks client honours Retry-After and the
rate-limit reset, keeps 10% of each installation's quota for create and
conclude, and creates the check-run with the verdict when the opener
degraded. executions.delivered_at records a landed conclusion; a cron sweep
re-sends the rest, oldest first.

@flaredispatch-fractalboxdev flaredispatch-fractalboxdev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI code review — 💬 Comment

Risk tier: full · 0 critical · 2 warnings · 1 suggestions

Reviewers: security 0 · performance 1 · code-quality 0 · documentation 2 · release-management 0 · compliance 0 · agents-md 0

1. ⚠️ Warning — Sweep processes all 50 deliveries serially

📍 apps/dispatcher/src/delivery-sweep.ts:121-131

Each row waits for 'conclude' and then 'markDelivered' before the next row starts. With 50 rows, GitHub latency and retry/backoff delays can make one tick run for many minutes or overlap the next 10-minute cron tick, causing duplicate reconciliation work and poor throughput. Use bounded concurrency (ideally grouped/limited per installation) or otherwise enforce a sweep deadline and resume pagination on the next tick.

2. 💡 Suggestion — Migration comment points to the wrong source path

📍 infra/migrations/0012_verdict_delivery.sql:16-17

The comment references 'apps/dispatcher/delivery-sweep.ts', but the implementation is under 'apps/dispatcher/src/delivery-sweep.ts'. Update the path so operators can locate the sweep directly.

3. ⚠️ Warning — Sweep comment incorrectly says no verdict is owed without credentials

📍 apps/dispatcher/src/routes/scheduled.ts:159-160

A deployment without App credentials can still contain executions with an installation_id and a missing delivered_at from an earlier deployment. The sweep is skipped because delivery cannot currently be attempted, not because no verdict is owed. Reword this comment to avoid implying those rows are not owed delivery.

📋 View full logs & reviewed diff ↗

A conclude GitHub rejects with 404 or 422 now returns refused. The sweep
stamps executions.delivery_refused_at and moves on to the installation's
next row, so a dead row no longer holds that installation's newer verdicts
behind it for the seven-day window.
# Conflicts:
#	packages/runtime-cf/package.json
@debuggingfuture
debuggingfuture merged commit bd2364f into main Oct 22, 2026
4 of 5 checks passed
@debuggingfuture
debuggingfuture deleted the feat/verdict-delivery branch October 22, 2026 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant