Repository navigation
feat(spend): spend guard and admission classes — budget, burn ceiling, scaled polls, interactive class - #236
Conversation
…end (ADR-0006) Admission reads month-to-date spend before it enqueues and before each claim; at or over the monthly budget the run ends RunSkipped (neutral check, skipped row). The claim admits only while admitted hourly rates across every pool plus the claimant's fit under the burn ceiling. Fan-out parents read spend before spawning. The meter prices admission to completion on the provisioned list rate, from one sandbox-to-instance table checked against wrangler.jsonc.
There was a problem hiding this comment.
AI code review — 💬 Comment
Risk tier: full · 0 critical · 4 warnings · 1 suggestions
Reviewers: security
1. ⚠️ Warning — Spend status is recomputed on every admission poll
📍 apps/dispatcher/src/workflow.ts:913-918
'spendCheck' runs 'readSpendStatus' inside a durable step before each claim attempt, and the admission loop invokes it for every poll at lines 965-966. This adds an aggregate D1 spend query per queued run per polling interval, potentially multiplying database load and workflow latency for runs waiting near the queue timeout. Cache the status for the admission interval or make the claim query return the spend decision atomically, while preserving replay determinism.
2. 💡 Suggestion — Dashboard requests now perform an additional spend read
📍 apps/dispatcher/src/http-app.ts:369-380
Every dashboard HTML request performs 'readSpendNow', adding a CONFIG_KV lookup and spend-status D1 aggregation beside the existing executions query. On a frequently refreshed dashboard this creates repeated database work unrelated to the requested execution data. Consider a short-lived cache or including the spend status in the existing dashboard data read.
3. ⚠️ Warning — Update the referenced cost and instance documentation
📍 packages/core/src/cost.ts:39-84
The rate card now adds provisioned disk sizes and disk GB-second pricing, but this change does not update the cited 'specs/05-byoc.md' and 'specs/06-cost.md' documentation. Those documents should define the new 'diskGb' values, disk pricing, and the revised per-execution cost examples so the implementation and documentation remain consistent.
4. ⚠️ Warning — Document the changed metering window
📍 packages/core/src/cost.ts:238-251
The implementation changes container costing from execution/boot-style timing to the admission-to-completion window, while continuing to cite 'specs/06-cost.md'. Update that specification with the admitted-active meter, including how queued time and unfinished executions are handled, rather than leaving the behavior documented only in source comments.
5. ⚠️ Warning — Avoid exposing internal spend-read errors to API callers
📍 apps/dispatcher/src/routes/spend.ts:85-87
The Access-gated '/api/spend' endpoint serializes 'error.message' directly in its 503 response. D1, CONFIG_KV, or runtime errors can contain internal schema, query, binding, or infrastructure details, unnecessarily disclosing them to viewers and potentially downstream clients. Return only the stable 'spend_unreadable' error publicly and log the detailed error server-side.
…ls settle, then skips contextful-release caught no RunSkipped from fanOut, so a budget closing at a later spawn ended the parent while running cells still wrote to the draft release, and the typed failure thrown inside the step was retried by the platform before it surfaced. The spawn steps now checkpoint the skip as an answer: no further cell starts, the running ones settle, and the release ends RunSkipped naming the unstarted cells. A closed budget at the formula spawn skips the same way. The child-runs fake's spendClosed also takes a function, so a test closes the budget mid-run. The cost.ts header names the SANDBOX_CONTAINER table.
…p and start jitter (ADR-0008, ADR-0009) - A waiter sleeps min(60 s, 20 s x ceil(position / cap)) between claims, from the position its claim step observed; ADMISSION_WAITER_TTL_MS is three of the longest sleeps. - limits.admissionClass (default batch); oxlint, pr-review, offload-test and worker-deploy are interactive. The claim and the queue position rank interactive ahead of batch, FIFO within a class; migration 0011 adds run_admissions.admission_class. - ADMISSION_CAP_BROWSER bounds the browser pool; routing tests assert every pool's cap fits its container's max_instances. - An admitted run waits 0-2 s, derived from its execution id, before its container starts.
…pper bounds with stop headroom (ADR-0006, ADR-0009)
There was a problem hiding this comment.
AI code review — 🛑 Request changes
Risk tier: full · 1 critical · 2 warnings · 0 suggestions
Reviewers: security
1. ⚠️ Warning — Document new spend settings in the operator documentation
📍 apps/dispatcher/src/env.ts:467-500
The change adds four spend-guard variables plus browser admission-cap behavior, including parsing rules, defaults, unlimited semantics, and CONFIG_KV precedence, but only documents them in TypeScript comments. Add the operator-facing configuration/reference documentation (and deployment examples) so operators can discover and correctly configure these settings without reading source.
2. ⚠️ Warning — Synchronize the referenced pricing specifications
📍 packages/core/src/cost.ts:60-105
The code now introduces provisioned disk sizes and disk-second billing, while explicitly treating specs/05-byoc.md and specs/06-cost.md as the authoritative sources. No corresponding specification/documentation changes are present in the diff. Update those references with the disk allocations, rate, hourly examples, and revised per-execution calculation so the canonical cost documentation does not contradict the implementation.
3. 🛑 Critical — Monthly spend budget is not enforced against concurrent admitted runs
📍 apps/dispatcher/src/workflow.ts:901-936
The admission gate only reads completed month-to-date spend before enqueue and before each claim; it does not reserve the projected cost of a run or atomically include in-flight admissions in the budget decision. Multiple concurrent runs can therefore all observe spend below the budget, be admitted, and later finish with aggregate cost far above the configured monthly limit, contradicting the documented 'at or over it, admission skips every run' guard. Add an atomic budget reservation (including in-flight/projected admitted cost) or otherwise serialize/admission-account for concurrent runs before allowing claims.
Admission caps what a deploy spends and decides who waits. Every run passes admission before it starts a container, so the limits are enforced there on every dispatch path: webhook, Action and cron.
Spend guard (ADR-0006)
executions.admitted_at), at the list rate of its container class on provisioned vCPU, memory and disk. Queue wait costs nothing. One table (SANDBOX_CONTAINER, runtime-cfexecution-cost.ts) maps each sandbox image to its class and instance type, andexecution-cost.test.tschecks it against the committedwrangler.jsonc. The lean image is now priced atstandard-4, the size its class runs. Rows priced before this change keep their value and readcost_meter = 'modeled-wall'.SPEND_OVERHEAD_FACTOR. At or over budget, the run endsRunSkipped('spend budget reached: $X of $Y this month'), which concludes the check-runneutraland records the executionskipped.fanOutchecks the budget before it spawns children.contextful-gatethen posts its unrun parts as cancelled and skips.contextful-releasestarts no further cell, lets the running cells settle, and skips naming the unstarted cells; both catch the skip inside the spawn step, so it checkpoints instead of being retried.RunSkipped('spend ceiling: …'). A timeout under the pool cap alone staysAdmissionTimedOut.SPEND_BUDGET_MONTHLY_USD(50),SPEND_BURN_MAX_USD_PER_HOUR(1.50),SPEND_OVERHEAD_FACTOR(1.10) andSPEND_WARN_RATIO(0.8). The defaults live in code, andunlimiteddisables a limit. TheCONFIG_KVkeysspend.budgetMonthlyUsdandspend.burnMaxUsdPerHouroverride the vars. A value that does not parse falls back to the default limit, never to no limit.spend.warn, sends one notice (useCase: spend-warn), and writes thespend.warned:<YYYY-MM>marker.GET /api/spendis Access-gated like the analytics read. It returns month-to-date spend, the budget, the ceiling, the admitted burn and the reset time. The dashboard shows the same line.0011_spend_guard.sqladdsexecutions.admitted_at,executions.cost_meter,run_admissions.hourly_rate_micro_usd,run_admissions.admission_classand theexecutions(admitted_at)index that the spend query scans.Admission (ADR-0008, ADR-0009)
min(60 s, 20 s × ceil(position / cap))between claims, from the position its claim step observed, so a replay sleeps the same length. The head keeps the 20-second poll. A sleep never runs past the queue-age limit by more than one 20-second poll.ADMISSION_WAITER_TTL_MSis 180 s, three of the longest sleeps. The attempt count is still derived from the 20-second sleep, so it covers the queue age; the age read inside each claim step ends a slower loop sooner. Each attempt is three steps, so the 6-hour override bounds a waiter at 3,240 steps.limits.admissionClassisinteractiveorbatch(default).oxlint,pr-review,offload-testandworker-deployare interactive. The claim's no-barge check and the queue position rank interactive ahead of batch, FIFO within a class; a row with no class ranks as batch. The burn ceiling never reorders the line: a batch claimant whose lower rate would fit still waits behind an interactive waiter the ceiling holds.ADMISSION_CAP_BROWSER(default 16) bounds the browser pool likeADMISSION_CAP_AGENTandADMISSION_CAP_RELEASE. Each cap is an upper bound; the burn ceiling sets operating concurrency. The routing tests assert every pool's cap ≤ its container'smax_instancesin the committedwrangler.jsonc, under both the committed vars and the code defaults.admission-jitter, a durable step) before its container starts. The length is an FNV-1a hash of the execution id, so a replay neither redraws nor re-sleeps it.Every spend read, limit read, claim and warning runs inside
step.do, so a replay reads the checkpointed answer.apps/substrateis unchanged. The docs benchmark now imports the core per-second rate, so its figures include disk.Spec:
specs/adr/0006-spend-guard.md,specs/adr/0008-cheaper-waiting-and-transport.md(position-scaled admission polls),specs/adr/0009-fast-trustworthy-verdicts.md(interactive admission class, pools sized under the spend guard) (#232).Testing:
pnpm install --frozen-lockfile,pnpm lint,pnpm typecheckandpnpm test(210 files, 2883 passed, 1 skipped) pass. New tests cover:the spend math (
core/spend.test.ts)the rate card with disk (
core/cost.test.ts)the claim SQL's ceiling clause,
admitted_atstamping and the spend read and warning against real D1/KV (*.workers.test.ts)the instance table against
wrangler.jsoncthe
fanOutspend gatethe gate's spend stop
the release's spend stop, at the first spawn and mid-window (
runs/contextful-release.test.ts)the neutral-skip conclusion and
/api/spend(routes/spend.test.ts)the scaled sleep, the attempt bound and the jitter range (
core/run-admission.test.ts)class ordering, FIFO within a class, pre-0011 rows, dead interactive waiters and the ceiling-held interactive waiter against real D1 (
run-admission-d1.workers.test.ts)the browser cap and cap ≤
max_instancesfor every pool (sandbox-routing.test.ts)the interactive run set (
runs/admission-class.test.ts)Pools. The lean pool cap defaults to 48 and every container class keeps 8 instances of headroom above its pool cap (ADR-0009). The burn ceiling, not the cap, sets concurrency.