Repository navigation
feat(checks): durable verdict delivery — bucketed positions, quota reserve, reconcile sweep - #237
Conversation
…serve, create-at-conclusion, reconcile sweep Queue-position updates write only on a bucket change (1, 2-5, 6-20, over 20) and at most once a minute. The checks client honours Retry-After and the rate-limit reset, keeps 10% of each installation's quota for create and conclude, and creates the check-run with the verdict when the opener degraded. executions.delivered_at records a landed conclusion; a cron sweep re-sends the rest, oldest first.
There was a problem hiding this comment.
AI code review — 💬 Comment
Risk tier: full · 0 critical · 2 warnings · 1 suggestions
Reviewers: security 0 · performance 1 · code-quality 0 · documentation 2 · release-management 0 · compliance 0 · agents-md 0
1. ⚠️ Warning — Sweep processes all 50 deliveries serially
📍 apps/dispatcher/src/delivery-sweep.ts:121-131
Each row waits for 'conclude' and then 'markDelivered' before the next row starts. With 50 rows, GitHub latency and retry/backoff delays can make one tick run for many minutes or overlap the next 10-minute cron tick, causing duplicate reconciliation work and poor throughput. Use bounded concurrency (ideally grouped/limited per installation) or otherwise enforce a sweep deadline and resume pagination on the next tick.
2. 💡 Suggestion — Migration comment points to the wrong source path
📍 infra/migrations/0012_verdict_delivery.sql:16-17
The comment references 'apps/dispatcher/delivery-sweep.ts', but the implementation is under 'apps/dispatcher/src/delivery-sweep.ts'. Update the path so operators can locate the sweep directly.
3. ⚠️ Warning — Sweep comment incorrectly says no verdict is owed without credentials
📍 apps/dispatcher/src/routes/scheduled.ts:159-160
A deployment without App credentials can still contain executions with an installation_id and a missing delivered_at from an earlier deployment. The sweep is skipped because delivery cannot currently be attempted, not because no verdict is owed. Reword this comment to avoid implying those rows are not owed delivery.
A conclude GitHub rejects with 404 or 422 now returns refused. The sweep stamps executions.delivery_refused_at and moves on to the installation's next row, so a dead row no longer holds that installation's newer verdicts behind it for the seven-day window.
# Conflicts: # packages/runtime-cf/package.json
What
Durable verdict delivery from ADR-0009: a computed verdict reaches the pull request even when the installation's GitHub quota runs short.
admission-summary.ts, admission loop inworkflow.ts). The check-run shows the bucket, not the exact count.packages/runtime-cf/src/checks-github.ts) waits at least as long as GitHub asked. A wait longer than 30 s degrades the call and marks the installation exhausted until the wait lapses, so later calls do not spend requests on a wait GitHub already named.x-ratelimit-remaining/-limit/-resetfeeds a per-installation quota cache (github-quota.ts). At or under 10% of the window,progressstops;create,updateandconcludeproceed. The cache holds GitHub's last answer per isolate. The quota itself lives on GitHub and each response refreshes it, so the worst case is one extra call per isolate past the reserve.ChecksService.concludecompletes the check-run, or creates it already completed when the opener degraded to thenoopsentinel id. It is total and returnsdelivered(with the check-run id),undelivered(worth sending again: rate limit, outage, missing credentials) orrefused(404 or 422: GitHub rejects that write for good). The workflow runs it in a durabledeliver-verdictstep, so a replay never creates a second check-run.executions.delivered_at. Migration0012addsdelivered_at,delivery_refused_atandinstallation_id. The workflow stores the installation with the check-run id and stampsdelivered_atwhen the conclusion lands.*/10 * * * *(armed inwrangler.jsonc, routed byroutes/scheduled.ts) re-sends the conclusion of every finished execution (success/failure/skipped) that has an installation and neitherdelivered_atnordelivery_refused_at. It works oldest first, skips rows finished less than 10 minutes ago and rows older than 7 days, and handles at most 50 rows a tick. The firstundeliveredrow of an installation stops the sweep for that installation until the next tick. Arefusedrow stampsdelivery_refused_atand leaves the queue, so it never holds up the newer rows of its installation. Cron parity exempts the sweep cron and asserts it is armed.Why
Without a write budget, a burst of queued runs spent an installation's hourly quota on position updates. Check-runs then failed to open or never concluded, and verdicts computed in D1 never reached GitHub, which looks the same as a hang.
Notes
0012follows0011_spend_guard.sqlfrom feat(spend): spend guard and admission classes — budget, burn ceiling, scaled polls, interactive class #236. Merge feat(spend): spend guard and admission classes — budget, burn ceiling, scaled polls, interactive class #236 first, or renumber if it lands later.installation_idis a column beyond the ADR'sdelivered_at. It marks a row as owing a verdict on GitHub: rows from a deploy or dispatch with no liveChecksLayer, and every row written before this migration, carry none, so the sweep leaves them alone.undelivered(an App-wide credential fault the sweep keeps retrying), 404 and 422 asrefused. The workflow logs either; the sweep sends its own short summary once more and stampsdelivery_refused_atif GitHub refuses that too.Spec:
specs/adr/0009-fast-trustworthy-verdicts.md§ Durable verdict delivery (proposed in #232)Testing:
pnpm lint,pnpm typecheckandpnpm test(207 files, 2854 passed) are green. New tests cover the bucket boundaries and the once-a-minute bound (admission-summary.test.ts); quota headers and theobservehook (http.test.ts,check-runs.test.ts); reserve, exhaustion and Retry-After waits (github-quota.test.ts,checks-github.test.ts); create-at-conclusion, undelivered and refused outcomes (checks-github.test.ts); sweep order, bounds, per-installation stop, refused rows leaving the queue and noop-id recording (delivery-sweep.test.ts); and sweep cron routing (scheduled.test.ts,cron-parity.test.ts).