Skip to content

feat(spend): spend guard and admission classes — budget, burn ceiling, scaled polls, interactive class - #236

Merged
debuggingfuture merged 4 commits into
mainfrom
feat/spend-guard
Oct 22, 2026
Merged

debuggingfuture merged 4 commits into
mainfrom
feat/spend-guard

Conversation

@debuggingfuture

@debuggingfuture debuggingfuture commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Admission caps what a deploy spends and decides who waits. Every run passes admission before it starts a container, so the limits are enforced there on every dispatch path: webhook, Action and cron.

Spend guard (ADR-0006)

  • Meter. An execution is priced from admission to completion (executions.admitted_at), at the list rate of its container class on provisioned vCPU, memory and disk. Queue wait costs nothing. One table (SANDBOX_CONTAINER, runtime-cf execution-cost.ts) maps each sandbox image to its class and instance type, and execution-cost.test.ts checks it against the committed wrangler.jsonc. The lean image is now priced at standard-4, the size its class runs. Rows priced before this change keep their value and read cost_meter = 'modeled-wall'.
  • Monthly budget. Admission reads month-to-date spend before it enqueues a run and again before each claim: finished cost plus in-flight rate × time, times SPEND_OVERHEAD_FACTOR. At or over budget, the run ends RunSkipped('spend budget reached: $X of $Y this month'), which concludes the check-run neutral and records the execution skipped. fanOut checks the budget before it spawns children. contextful-gate then posts its unrun parts as cancelled and skips. contextful-release starts no further cell, lets the running cells settle, and skips naming the unstarted cells; both catch the skip inside the spawn step, so it checkpoints instead of being retried.
  • Burn ceiling. The claim SQL admits only when the summed hourly rates of live admitted rows across all pools, plus the claimant's own rate, fit under the ceiling. A claimant whose own rate exceeds the ceiling is admitted only when the admitted set is empty. A queue timeout while the ceiling held the run ends RunSkipped('spend ceiling: …'). A timeout under the pool cap alone stays AdmissionTimedOut.
  • Limits. SPEND_BUDGET_MONTHLY_USD (50), SPEND_BURN_MAX_USD_PER_HOUR (1.50), SPEND_OVERHEAD_FACTOR (1.10) and SPEND_WARN_RATIO (0.8). The defaults live in code, and unlimited disables a limit. The CONFIG_KV keys spend.budgetMonthlyUsd and spend.burnMaxUsdPerHour override the vars. A value that does not parse falls back to the default limit, never to no limit.
  • Warning and visibility. The first admission after spend crosses the warn ratio logs spend.warn, sends one notice (useCase: spend-warn), and writes the spend.warned:<YYYY-MM> marker. GET /api/spend is Access-gated like the analytics read. It returns month-to-date spend, the budget, the ceiling, the admitted burn and the reset time. The dashboard shows the same line.
  • Storage. Migration 0011_spend_guard.sql adds executions.admitted_at, executions.cost_meter, run_admissions.hourly_rate_micro_usd, run_admissions.admission_class and the executions(admitted_at) index that the spend query scans.

Admission (ADR-0008, ADR-0009)

  • Position-scaled polls. A waiter sleeps min(60 s, 20 s × ceil(position / cap)) between claims, from the position its claim step observed, so a replay sleeps the same length. The head keeps the 20-second poll. A sleep never runs past the queue-age limit by more than one 20-second poll. ADMISSION_WAITER_TTL_MS is 180 s, three of the longest sleeps. The attempt count is still derived from the 20-second sleep, so it covers the queue age; the age read inside each claim step ends a slower loop sooner. Each attempt is three steps, so the 6-hour override bounds a waiter at 3,240 steps.
  • Interactive class. limits.admissionClass is interactive or batch (default). oxlint, pr-review, offload-test and worker-deploy are interactive. The claim's no-barge check and the queue position rank interactive ahead of batch, FIFO within a class; a row with no class ranks as batch. The burn ceiling never reorders the line: a batch claimant whose lower rate would fit still waits behind an interactive waiter the ceiling holds.
  • Pools under the spend guard. ADMISSION_CAP_BROWSER (default 16) bounds the browser pool like ADMISSION_CAP_AGENT and ADMISSION_CAP_RELEASE. Each cap is an upper bound; the burn ceiling sets operating concurrency. The routing tests assert every pool's cap ≤ its container's max_instances in the committed wrangler.jsonc, under both the committed vars and the code defaults.
  • Start jitter. An admitted run sleeps 0–2 s (admission-jitter, a durable step) before its container starts. The length is an FNV-1a hash of the execution id, so a replay neither redraws nor re-sleeps it.

Every spend read, limit read, claim and warning runs inside step.do, so a replay reads the checkpointed answer. apps/substrate is unchanged. The docs benchmark now imports the core per-second rate, so its figures include disk.

Spec: specs/adr/0006-spend-guard.md, specs/adr/0008-cheaper-waiting-and-transport.md (position-scaled admission polls), specs/adr/0009-fast-trustworthy-verdicts.md (interactive admission class, pools sized under the spend guard) (#232).

Testing: pnpm install --frozen-lockfile, pnpm lint, pnpm typecheck and pnpm test (210 files, 2883 passed, 1 skipped) pass. New tests cover:

  • the spend math (core/spend.test.ts)

  • the rate card with disk (core/cost.test.ts)

  • the claim SQL's ceiling clause, admitted_at stamping and the spend read and warning against real D1/KV (*.workers.test.ts)

  • the instance table against wrangler.jsonc

  • the fanOut spend gate

  • the gate's spend stop

  • the release's spend stop, at the first spawn and mid-window (runs/contextful-release.test.ts)

  • the neutral-skip conclusion and /api/spend (routes/spend.test.ts)

  • the scaled sleep, the attempt bound and the jitter range (core/run-admission.test.ts)

  • class ordering, FIFO within a class, pre-0011 rows, dead interactive waiters and the ceiling-held interactive waiter against real D1 (run-admission-d1.workers.test.ts)

  • the browser cap and cap ≤ max_instances for every pool (sandbox-routing.test.ts)

  • the interactive run set (runs/admission-class.test.ts)

  • Pools. The lean pool cap defaults to 48 and every container class keeps 8 instances of headroom above its pool cap (ADR-0009). The burn ceiling, not the cap, sets concurrency.

…end (ADR-0006)

Admission reads month-to-date spend before it enqueues and before each claim; at or over the monthly budget the run ends RunSkipped (neutral check, skipped row). The claim admits only while admitted hourly rates across every pool plus the claimant's fit under the burn ceiling. Fan-out parents read spend before spawning. The meter prices admission to completion on the provisioned list rate, from one sandbox-to-instance table checked against wrangler.jsonc.

@flaredispatch-fractalboxdev flaredispatch-fractalboxdev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI code review — 💬 Comment

Risk tier: full · 0 critical · 4 warnings · 1 suggestions

Reviewers: security ⚠️ · performance 2 · code-quality ⚠️ · documentation 2 · release-management ⚠️ · compliance 1 · agents-md ⚠️

1. ⚠️ Warning — Spend status is recomputed on every admission poll

📍 apps/dispatcher/src/workflow.ts:913-918

'spendCheck' runs 'readSpendStatus' inside a durable step before each claim attempt, and the admission loop invokes it for every poll at lines 965-966. This adds an aggregate D1 spend query per queued run per polling interval, potentially multiplying database load and workflow latency for runs waiting near the queue timeout. Cache the status for the admission interval or make the claim query return the spend decision atomically, while preserving replay determinism.

2. 💡 Suggestion — Dashboard requests now perform an additional spend read

📍 apps/dispatcher/src/http-app.ts:369-380

Every dashboard HTML request performs 'readSpendNow', adding a CONFIG_KV lookup and spend-status D1 aggregation beside the existing executions query. On a frequently refreshed dashboard this creates repeated database work unrelated to the requested execution data. Consider a short-lived cache or including the spend status in the existing dashboard data read.

3. ⚠️ Warning — Update the referenced cost and instance documentation

📍 packages/core/src/cost.ts:39-84

The rate card now adds provisioned disk sizes and disk GB-second pricing, but this change does not update the cited 'specs/05-byoc.md' and 'specs/06-cost.md' documentation. Those documents should define the new 'diskGb' values, disk pricing, and the revised per-execution cost examples so the implementation and documentation remain consistent.

4. ⚠️ Warning — Document the changed metering window

📍 packages/core/src/cost.ts:238-251

The implementation changes container costing from execution/boot-style timing to the admission-to-completion window, while continuing to cite 'specs/06-cost.md'. Update that specification with the admitted-active meter, including how queued time and unfinished executions are handled, rather than leaving the behavior documented only in source comments.

5. ⚠️ Warning — Avoid exposing internal spend-read errors to API callers

📍 apps/dispatcher/src/routes/spend.ts:85-87

The Access-gated '/api/spend' endpoint serializes 'error.message' directly in its 503 response. D1, CONFIG_KV, or runtime errors can contain internal schema, query, binding, or infrastructure details, unnecessarily disclosing them to viewers and potentially downstream clients. Return only the stable 'spend_unreadable' error publicly and log the detailed error server-side.

📋 View full logs & reviewed diff ↗

…ls settle, then skips

contextful-release caught no RunSkipped from fanOut, so a budget closing
at a later spawn ended the parent while running cells still wrote to the
draft release, and the typed failure thrown inside the step was retried by
the platform before it surfaced. The spawn steps now checkpoint the skip
as an answer: no further cell starts, the running ones settle, and the
release ends RunSkipped naming the unstarted cells. A closed budget at the
formula spawn skips the same way.

The child-runs fake's spendClosed also takes a function, so a test closes
the budget mid-run. The cost.ts header names the SANDBOX_CONTAINER table.
…p and start jitter (ADR-0008, ADR-0009)

- A waiter sleeps min(60 s, 20 s x ceil(position / cap)) between claims,
  from the position its claim step observed; ADMISSION_WAITER_TTL_MS is
  three of the longest sleeps.
- limits.admissionClass (default batch); oxlint, pr-review, offload-test and
  worker-deploy are interactive. The claim and the queue position rank
  interactive ahead of batch, FIFO within a class; migration 0011 adds
  run_admissions.admission_class.
- ADMISSION_CAP_BROWSER bounds the browser pool; routing tests assert every
  pool's cap fits its container's max_instances.
- An admitted run waits 0-2 s, derived from its execution id, before its
  container starts.
@debuggingfuture debuggingfuture changed the title feat(spend): spend guard — meter, monthly budget, burn ceiling, /api/spend feat(spend): spend guard and admission classes — budget, burn ceiling, scaled polls, interactive class Oct 22, 2026
…pper bounds with stop headroom (ADR-0006, ADR-0009)

@flaredispatch-fractalboxdev flaredispatch-fractalboxdev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI code review — 🛑 Request changes

Risk tier: full · 1 critical · 2 warnings · 0 suggestions

Reviewers: security ⚠️ · performance ⚠️ · code-quality ⚠️ · documentation 2 · release-management 0 · compliance 1 · agents-md ⚠️

1. ⚠️ Warning — Document new spend settings in the operator documentation

📍 apps/dispatcher/src/env.ts:467-500

The change adds four spend-guard variables plus browser admission-cap behavior, including parsing rules, defaults, unlimited semantics, and CONFIG_KV precedence, but only documents them in TypeScript comments. Add the operator-facing configuration/reference documentation (and deployment examples) so operators can discover and correctly configure these settings without reading source.

2. ⚠️ Warning — Synchronize the referenced pricing specifications

📍 packages/core/src/cost.ts:60-105

The code now introduces provisioned disk sizes and disk-second billing, while explicitly treating specs/05-byoc.md and specs/06-cost.md as the authoritative sources. No corresponding specification/documentation changes are present in the diff. Update those references with the disk allocations, rate, hourly examples, and revised per-execution calculation so the canonical cost documentation does not contradict the implementation.

3. 🛑 Critical — Monthly spend budget is not enforced against concurrent admitted runs

📍 apps/dispatcher/src/workflow.ts:901-936

The admission gate only reads completed month-to-date spend before enqueue and before each claim; it does not reserve the projected cost of a run or atomically include in-flight admissions in the budget decision. Multiple concurrent runs can therefore all observe spend below the budget, be admitted, and later finish with aggregate cost far above the configured monthly limit, contradicting the documented 'at or over it, admission skips every run' guard. Add an atomic budget reservation (including in-flight/projected admitted cost) or otherwise serialize/admission-account for concurrent runs before allowing claims.

📋 View full logs & reviewed diff ↗

@debuggingfuture
debuggingfuture merged commit 40cf162 into main Oct 22, 2026
6 checks passed
@debuggingfuture
debuggingfuture deleted the feat/spend-guard branch October 22, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant