From 6acca82cabecd03fa451e7c11a8724a844cbd8b1 Mon Sep 17 00:00:00 2001 From: debuggingfuture Date: Thu, 22 Oct 2026 16:28:11 +0800 Subject: [PATCH 1/3] docs(adr): spend guard and cost-efficient execution (ADR-0006, ADR-0007) --- specs/adr/0006-spend-guard.md | 126 +++++++++++++++++++++ specs/adr/0007-cost-efficient-execution.md | 100 ++++++++++++++++ specs/adr/README.md | 4 +- 3 files changed, 229 insertions(+), 1 deletion(-) create mode 100644 specs/adr/0006-spend-guard.md create mode 100644 specs/adr/0007-cost-efficient-execution.md diff --git a/specs/adr/0006-spend-guard.md b/specs/adr/0006-spend-guard.md new file mode 100644 index 0000000..5ac151a --- /dev/null +++ b/specs/adr/0006-spend-guard.md @@ -0,0 +1,126 @@ +# ADR-0006 — Spend guard: a monthly budget closes admission, a burn ceiling bounds concurrency + +**Status:** proposed +**Related:** `apps/dispatcher/src/workflow.ts` · `packages/runtime-cf/src/run-admission-d1.ts` · `packages/runtime-cf/src/execution-cost.ts` · `packages/core/src/cost.ts` · `infra/migrations/0005_execution_cost.sql` · ADR-0007 + +## Context + +Cloudflare offers no spend cap for Workers, Workflows or Containers. A budget +notification sends one email when month-to-date spend first crosses its +threshold and stops nothing. Container compute is almost all of a deploy's +bill, and it scales with pushes times fan-out width: a 30-part gate on a busy +pull request spends in an afternoon what a month of standard checks spends. + +Every run passes the admission semaphore before it starts a container, and a +run refused there costs nothing. That makes admission the one place a hard +limit holds for every dispatch path: webhook, Action and cron. + +The per-execution cost rollup does not support a limit today. It prices the +execution's wall time, queue wait included, and maps the lean image to +`standard-2` while the lean container class runs `standard-4`. + +## Decision + +### Meter + +- An execution's container-active time runs from admission to completion. + Admission writes `executions.admitted_at`; queue wait costs nothing. +- Its compute cost is the list rate of its container class's instance type, + priced on provisioned vCPU, memory and disk, times active seconds, plus its + metered model tokens. A provisioned-size list rate is an upper bound on the + billed rate, so the guard stops early rather than late. +- The instance type per sandbox image comes from one table that a test + checks against the container classes in `wrangler.jsonc`. +- The finish path writes `cost_micro_usd` from this rule. Rows written before + the rule keep their value and are labelled `modeled-wall`. + +### Spend read + +Month-to-date spend is the sum over the current UTC calendar month of: + +- `cost_micro_usd` of every finished execution admitted this month; +- for every admitted, unfinished execution, its hourly rate times the time + since `admitted_at`; + +times an overhead factor for Workflow steps and Durable Objects +(`SPEND_OVERHEAD_FACTOR`, default `1.10`). One indexed D1 query answers it. + +### Monthly budget + +- `SPEND_BUDGET_MONTHLY_USD` sets the budget. Unset, the budget is **90 USD**. + `unlimited` disables the monthly check. A `CONFIG_KV` value at + `spend.budgetMonthlyUsd` overrides the variable, so an operator raises the + budget without a deploy. +- Admission reads spend before it enqueues a run and again before each claim. + At or over budget, the run ends `RunSkipped` with the reason + `spend budget reached: $ of $ this month`, which concludes + its check-run `neutral` and records the execution `skipped`. No run is + exempt. +- A fan-out parent reads spend before it spawns children. Over budget, it + skips and spawns none. A child spawned before exhaustion skips at its own + admission. +- Runs already admitted finish. The overshoot is bounded by the burn ceiling + times the longest run body still in flight. + +### Burn ceiling + +- `SPEND_BURN_MAX_USD_PER_HOUR` caps the summed hourly rate of all admitted + executions across every pool. Unset, the ceiling is **1.50 USD per hour**, + which admits three `standard-4` containers, or eleven `standard-2`, at once. + `unlimited` disables it. `CONFIG_KV` `spend.burnMaxUsdPerHour` overrides it. +- `run_admissions` records each admitted row's hourly rate. A claim admits + only when the pool has a free slot and the admitted rates plus the + claimant's own rate fit under the ceiling. A run whose own rate exceeds the + ceiling is admitted only into an empty set, so it never waits forever. +- A run that times out in the queue while the ceiling, not the pool cap, held + it back ends `RunSkipped` with the reason `spend ceiling`. A queue timeout + under the pool cap alone keeps its current verdict. +- `max_instances` per container class stays the platform's hard limit. The + ceiling makes the configured `ADMISSION_CAP` an upper bound, not the + operating point. + +### Warning and visibility + +- The first admission after spend crosses `SPEND_WARN_RATIO` of the budget + (default `0.8`) in a month logs `spend.warn` and sends one notification on + the dispatcher's notify path. `CONFIG_KV` `spend.warned:` marks + the month as warned. +- `GET /api/spend` returns month-to-date spend, the budget, the burn ceiling, + the admitted hourly rate and the month's reset time. The dashboard shows + the same line. + +## Rationale + +The budget has to bind on every path, and only admission sees every run +before it costs anything. A neutral skip tells the developer the check did +not run without reading as a failure. + +The defaults cap a deploy that sets nothing. The operator overlay replaces the +committed `wrangler.jsonc`, so a default in committed config never reaches a +deploy whose overlay predates it. The defaults therefore live in code. + +A monthly budget alone lets one burst spend the month in a day and leaves the +rest of the month with no checks. The burn ceiling spreads the budget over +time and bounds the overshoot when the budget closes. A dollar rate, rather +than a container count, holds across pools whose instance types differ. + +An upper-bound meter can stop runs while real spend sits below the budget. +That is the safe direction for a cap. The monthly reconcile against the +Billable Usage API shows how far apart the two are. + +## Consequences + +- At the defaults, checks skip once a month's estimated spend reaches 90 USD. + Wide fan-outs queue behind the burn ceiling and run slower. +- Raising either limit is a single `CONFIG_KV` write. +- The meter estimates spend; it is not the invoice. The Billable Usage API + stays the source of truth for the month's bill. +- Workers requests, R2, KV, D1 and AI Gateway are inside the overhead factor + rather than metered per execution. + +## Revisit triggers + +- Measured monthly billed spend differs from the meter's month-to-date by more + than 25%. +- Cloudflare ships an account-level spend cap for Containers. +- A run class needs to run regardless of budget, such as a release. diff --git a/specs/adr/0007-cost-efficient-execution.md b/specs/adr/0007-cost-efficient-execution.md new file mode 100644 index 0000000..b85d53b --- /dev/null +++ b/specs/adr/0007-cost-efficient-execution.md @@ -0,0 +1,100 @@ +# ADR-0007 — Cost-efficient execution: dispatch only work, upload each cache key once, build heavy native dependencies once + +**Status:** proposed +**Related:** `apps/dispatcher/src/routes/webhook.ts` · `runs/check.ts` · `runs/offload-test.ts` · `runs/worker-deploy.ts` · `packages/runtime-cf/src/cache-r2.ts` · `packages/runtime-cf/src/cache-restore-or.ts` · `runs/contextful-gate.ts` · `infra/Dockerfile.sandbox-demo` · ADR-0006 + +## Context + +A deploy pays for container time, and three patterns spend it on nothing: + +- A webhook run on a repo that never opted in creates a Workflow, opens a + check-run, waits for an admission slot, starts a container, and only then + reads `CONFIG_KV` and ends green with nothing done. Each one also holds a + pool slot that real work waits for. +- A cache restore that finds its archive but cannot apply it falls through to + the miss path: the run installs from scratch, then packs and uploads the + same content-addressed archive again. On the http sandbox transport every + stream restore fails this way, so every run of a fan-out re-uploads the + archive that already sits in R2. +- Every part of a fan-out gate clones and compiles the same dependency graph + in a fresh container. A bundled C++ dependency such as DuckDB rebuilds its + engine from source in each part and dominates that compile. + +ADR-0006 caps what a deploy spends. This record lowers what each verdict costs, +so a budget buys more checks. + +## Decision + +### Dispatch only work + +- A run may declare `optIn: (ctx) => readonly string[]`, the `CONFIG_KV` keys + whose presence means the repo opted in. `check`, `offload-test` and + `worker-deploy` declare the command keys their bodies read today. +- The webhook receiver reads those keys before it creates a Workflow. With + none present it dispatches nothing, opens no check-run, and lists the run in + the response's `skipped` array with reason `not-configured`. +- A dispatch that names its command explicitly, from an Action or a fan-out + parent, skips the opt-in read. +- The run bodies keep their own opt-in check, so a key deleted between + dispatch and run still ends in a no-op. + +### Upload each cache key once + +- A restore reports one of three outcomes: `hit`, `miss` (no archive under + the key), or `unapplied` (the archive exists, applying it failed). +- `unapplied` runs the miss path and skips the save: the key is + content-addressed, so the archive in R2 already holds this content. +- A save checks `bucket.head` for its key before it packs. An existing + archive skips the `tar czf` and the upload. Parallel misses on one key then + upload once instead of once per part. +- A restore on a transport that cannot stream into the container reports + `unapplied` without reading the archive body, and logs the transport once + per execution. + +### Fan-out parts install nothing their stage repeats + +- A gate part whose stage command resolves its own dependencies, as `cargo` + does on build, spawns with `install: false` and restores only what the + stage cannot rebuild cheaply, such as a prebuilt tool binary. + +### Prebuilt heavy native dependencies in the image + +- The lean sandbox image ships sha256-pinned prebuilt shared libraries and + headers for native dependencies that dominate a cold compile, DuckDB first, + and exports the paths the dependency's build script reads in linked mode + (`DUCKDB_LIB_DIR`, `DUCKDB_INCLUDE_DIR`). +- The variables change nothing for a consumer that builds the dependency + bundled: the build script reads them only in linked mode. A consumer opts + in by building the dependency linked in CI. The consumer owns that choice, + and it asserts the loaded library's version at startup, because a linked + library of the wrong version compiles cleanly and misbehaves at runtime. +- The image's library version and the consumer's pinned crate version move + together. + +## Rationale + +The receiver is the last point where skipping costs nothing: past it, a skip +still pays for a Workflow, a check-run and a slot. Keeping the body's check +makes the receiver read an optimisation rather than the only line of defence. + +A content-addressed key makes a second upload pure waste, so the head check is +always safe. Separating `unapplied` from `miss` keeps a broken restore visible +in logs without paying for it in container time. + +Building a C++ engine from source in every part buys nothing a prebuilt +library does not, and only the consumer can decide whether CI may link it +dynamically while release builds stay static. + +## Consequences + +- Webhook responses on unconfigured repos list `skipped` runs. Those repos no + longer receive green no-op check-runs. +- A broken restore still costs a cold install per part until the transport is + fixed, but no longer a repeated pack and upload. +- The lean image grows by the prebuilt libraries' size. + +## Revisit triggers + +- A restore path reaches the container without passing through the Worker, + such as a signed R2 URL, at a throughput that makes a shared build cache pay. +- A consumer's opt-in needs more than the presence of a `CONFIG_KV` key. diff --git a/specs/adr/README.md b/specs/adr/README.md index 3bdbdd8..cb9a0ab 100644 --- a/specs/adr/README.md +++ b/specs/adr/README.md @@ -15,4 +15,6 @@ interfaces or assume they exist. | [0002](./0002-memory-capability.md) | proposed | Org context is an optional, total `memory` capability with MCP-speaking adapters; runs consume, never produce | | [0003](./0003-no-context-relay.md) | proposed | Context backends ingest GitHub directly; FlareDispatch builds no relay, export, or pull endpoint for them | | [0004](./0004-serialized-run-groups.md) | proposed | A run may declare a serialization group: one execution in flight, newest waiter wins, never cancel the running one; state in D1 | -| [0005](./0005-superseded-run-groups.md) | proposed | A run may declare a supersession group: the highest-ranked member cancels every lower live one with its children, slots and checks | +| [0005](./0005-superseded-run-groups.md) | proposed | A run may declare a supersession group: the highest-ranked member cancels every lower live one with its children, slots and checks | +| [0006](./0006-spend-guard.md) | proposed | A monthly budget closes admission and a burn ceiling caps the summed hourly rate of admitted runs; both default low in code | +| [0007](./0007-cost-efficient-execution.md) | proposed | Dispatch only opted-in work, upload each content-addressed cache key once, build heavy native dependencies once in the image | From 15a19fbf45b508fecc3450a1ad5a21e665e6ca1d Mon Sep 17 00:00:00 2001 From: debuggingfuture Date: Thu, 22 Oct 2026 16:58:23 +0800 Subject: [PATCH 2/3] docs(adr): cheaper waiting (ADR-0008) and fast, trustworthy verdicts (ADR-0009) --- specs/adr/0007-cost-efficient-execution.md | 2 +- .../adr/0008-cheaper-waiting-and-transport.md | 70 ++++++++++++++ specs/adr/0009-fast-trustworthy-verdicts.md | 92 +++++++++++++++++++ specs/adr/README.md | 20 ++-- 4 files changed, 174 insertions(+), 10 deletions(-) create mode 100644 specs/adr/0008-cheaper-waiting-and-transport.md create mode 100644 specs/adr/0009-fast-trustworthy-verdicts.md diff --git a/specs/adr/0007-cost-efficient-execution.md b/specs/adr/0007-cost-efficient-execution.md index b85d53b..431f102 100644 --- a/specs/adr/0007-cost-efficient-execution.md +++ b/specs/adr/0007-cost-efficient-execution.md @@ -59,7 +59,7 @@ so a budget buys more checks. ### Prebuilt heavy native dependencies in the image -- The lean sandbox image ships sha256-pinned prebuilt shared libraries and +- The Rust variant of the lean sandbox image ships sha256-pinned prebuilt shared libraries and headers for native dependencies that dominate a cold compile, DuckDB first, and exports the paths the dependency's build script reads in linked mode (`DUCKDB_LIB_DIR`, `DUCKDB_INCLUDE_DIR`). diff --git a/specs/adr/0008-cheaper-waiting-and-transport.md b/specs/adr/0008-cheaper-waiting-and-transport.md new file mode 100644 index 0000000..e5b8edf --- /dev/null +++ b/specs/adr/0008-cheaper-waiting-and-transport.md @@ -0,0 +1,70 @@ +# ADR-0008 — Cheaper waiting: rpc transport by default, position-scaled admission polls, counted supersessions + +**Status:** proposed +**Related:** `packages/runtime-cf/src/runtime.ts` · `packages/runtime-cf/src/sandbox-cf.ts` · `apps/dispatcher/src/workflow.ts` · `packages/runtime-cf/src/run-admission-d1.ts` · `apps/dispatcher/src/supersede.ts` · ADR-0005 · ADR-0007 + +## Context + +- The sandbox transport defaults to http, and only a per-repo `CONFIG_KV` pin + selects rpc. Streaming a file into a container needs rpc, so on the default + transport every cache restore fails and every run installs from scratch. +- A queued run polls admission every 20 seconds whatever its position. Each + poll is a claim step plus a sleep step, and Workflows bills each step. A run + fifty places back pays for hundreds of polls that cannot admit it. +- ADR-0005 cancels a superseded gate's children, but nothing counts how often + that happens or how much work it stops, so nobody can tell whether it is + working. + +## Decision + +### rpc transport by default + +- `SANDBOX_TRANSPORT` defaults to `rpc` in code when unset. +- `CONFIG_KV` `sandbox.transport:` still overrides the default per repo, + in both directions, so an operator pins a repo back to `http` without a + deploy. +- A container keeps the transport it started with for its whole life, as + `sandbox-cf.ts` pins it today. + +### Position-scaled admission polls + +- A waiter at FIFO position `p` in a pool with cap `c` sleeps + `min(60 s, 20 s × ceil(p / c))` between claims. The head of the queue keeps + the 20-second poll, and a waiter many slots back polls up to three times + less often. +- The waiter liveness window (`ADMISSION_WAITER_TTL_MS`) is at least three + times the longest sleep, so a slow-polling waiter never reads as dead. +- The sleep length derives from the position observed inside the claim step, + so a replay sleeps the same length. + +### Counted supersessions + +- Each supersession logs one `supersede.cancelled` line naming the group, the + cancelled root, and the number of children, admitted slots and open + check-runs it ended. +- The dashboard's execution list labels a cancelled execution with the + execution that superseded it. + +## Rationale + +rpc is the only transport on which every capability in `@cloudflare/sandbox` +works, file streaming included, so an http default makes the cache a cost +rather than a saving. Keeping the per-repo pin gives a deploy a rollback +without a release. + +Polling far from the head buys nothing: a waiter cannot move more than one +position per released slot. Scaling the sleep with position cuts the step bill +of a long queue while the head still admits within one short poll. + +A log line per supersession makes the saving measurable without a new table. + +## Consequences + +- Deploys that never set the variable switch to rpc on their next deploy. +- Waiters deep in a long queue admit up to 40 seconds later after a burst of + releases. + +## Revisit triggers + +- rpc fails a capability that http serves. +- Admission moves from polling to an event that wakes the head waiter. diff --git a/specs/adr/0009-fast-trustworthy-verdicts.md b/specs/adr/0009-fast-trustworthy-verdicts.md new file mode 100644 index 0000000..3288031 --- /dev/null +++ b/specs/adr/0009-fast-trustworthy-verdicts.md @@ -0,0 +1,92 @@ +# ADR-0009 — Fast, trustworthy verdicts: an interactive admission class, durable delivery, a two-tier gate, pools sized under the spend guard + +**Status:** proposed +**Related:** `apps/dispatcher/src/workflow.ts` · `packages/runtime-cf/src/run-admission-d1.ts` · `packages/core/src/services/checks.ts` · `runs/contextful-gate.ts` · `runs/check.ts` · ADR-0005 · ADR-0006 + +## Context + +- Every run shares one FIFO per pool. A fan-out gate enqueues tens of parts + at once, and a one-minute lint or review queued behind them waits for the + whole fan-out, then times out red. +- Check-run updates have no budget. The queue-position update is a GitHub + write per position change per waiter, and a burst exhausts the + installation's hourly quota. After that, check-runs fail to open or never + conclude, and computed verdicts never reach the pull request. +- A wide gate's floor is its slowest part, tens of minutes on any platform. + Running every heavy part on every pull-request push buys a verdict nobody + waits for, at the highest cost per push. +- A larger pool shortens queues, but under ADR-0006 it is the burn ceiling, + not the pool cap, that sets concurrency. + +## Decision + +### Interactive admission class + +- A run may declare `limits.admissionClass: "interactive" | "batch"`. + Unset, the class is `batch`. `oxlint`, `pr-review`, `offload-test` and + `worker-deploy` declare `interactive`. +- `run_admissions` records the class. A claim admits the oldest interactive + waiter before any batch waiter. The no-barge rule and the queue position + apply within a class, interactive ahead of batch. +- An interactive run never waits behind a batch run that has not yet been + admitted. + +### Durable verdict delivery + +- Queue-position updates are bucketed: a waiter writes its position only when + it crosses a bucket boundary (1, 2-5, 6-20, over 20) and at most once a + minute. +- Every GitHub call honours `Retry-After` and the rate-limit reset header. The + checks client keeps a per-installation reserve of 10% of the hourly quota + for create and conclude calls; position updates stop first. +- A check-run that failed to open is created at conclusion, with the + verdict, instead of left absent. +- `executions.delivered_at` records when the concluding check-run write + succeeded. A cron reconcile sweep re-sends the conclusion of every finished + execution without `delivered_at`, oldest first, inside the reserve. + +### Two-tier gate + +- A gate's plan marks each part `light` or `heavy`. `CONFIG_KV` + `gate.lightLabels:` lists the light labels. Unset, every part is + heavy, which keeps today's behaviour. +- On a pull-request event the gate runs light parts only, as interactive + runs. On a push to the default branch it runs the heavy parts once per head + under a serialization group (ADR-0004), so a newer head supersedes an older + one. +- Branch protection on the light tier is the repository owner's setting, not + the dispatcher's. + +### Pools sized under the spend guard + +- `ADMISSION_CAP` per pool is an upper bound. ADR-0006's burn ceiling sets the + operating concurrency. +- The browser pool has its own `ADMISSION_CAP_BROWSER`, like the agent and + release pools. +- `max_instances` per container class is at least the pool's cap plus + headroom for containers still stopping. A test asserts cap ≤ `max_instances` + for every pool. +- A claim that admits adds a jitter of up to 2 seconds before the container + starts, so a burst of admissions does not hit the container start-rate + limit at once. + +## Rationale + +A class costs nothing when the pool is free and removes the one wait a +developer feels most. Delivery is the product: a computed verdict that never +reaches GitHub costs the same as one that does and reads as a hang. A two-tier +gate puts the minutes-scale check where the merge decision is made and runs +the slow matrix where it can still catch regressions. + +## Consequences + +- Batch work waits longer when interactive work is queued. +- GitHub shows coarser queue positions. +- Heavy failures surface on the default branch after merge, not on the pull + request. + +## Revisit triggers + +- A batch run starves behind sustained interactive load. +- GitHub raises installation limits, or the checks API gains a cheaper + progress channel. diff --git a/specs/adr/README.md b/specs/adr/README.md index cb9a0ab..828d01f 100644 --- a/specs/adr/README.md +++ b/specs/adr/README.md @@ -9,12 +9,14 @@ A record is only binding once its Status reads `accepted`. Treat `proposed` rows under discussion — in particular, do not implement against a proposed record's interfaces or assume they exist. -| ADR | Status | Decision | -| -------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [0001](./0001-cloudflare-workflows-scope.md) | proposed | How runs use Cloudflare Workflows: one instance per dispatch, hibernation reserved for bounded human decisions, entity lifecycles keep state in the system of record, container disk is never durable state | -| [0002](./0002-memory-capability.md) | proposed | Org context is an optional, total `memory` capability with MCP-speaking adapters; runs consume, never produce | -| [0003](./0003-no-context-relay.md) | proposed | Context backends ingest GitHub directly; FlareDispatch builds no relay, export, or pull endpoint for them | -| [0004](./0004-serialized-run-groups.md) | proposed | A run may declare a serialization group: one execution in flight, newest waiter wins, never cancel the running one; state in D1 | -| [0005](./0005-superseded-run-groups.md) | proposed | A run may declare a supersession group: the highest-ranked member cancels every lower live one with its children, slots and checks | -| [0006](./0006-spend-guard.md) | proposed | A monthly budget closes admission and a burn ceiling caps the summed hourly rate of admitted runs; both default low in code | -| [0007](./0007-cost-efficient-execution.md) | proposed | Dispatch only opted-in work, upload each content-addressed cache key once, build heavy native dependencies once in the image | +| ADR | Status | Decision | +| ----------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| [0001](./0001-cloudflare-workflows-scope.md) | proposed | How runs use Cloudflare Workflows: one instance per dispatch, hibernation reserved for bounded human decisions, entity lifecycles keep state in the system of record, container disk is never durable state | +| [0002](./0002-memory-capability.md) | proposed | Org context is an optional, total `memory` capability with MCP-speaking adapters; runs consume, never produce | +| [0003](./0003-no-context-relay.md) | proposed | Context backends ingest GitHub directly; FlareDispatch builds no relay, export, or pull endpoint for them | +| [0004](./0004-serialized-run-groups.md) | proposed | A run may declare a serialization group: one execution in flight, newest waiter wins, never cancel the running one; state in D1 | +| [0005](./0005-superseded-run-groups.md) | proposed | A run may declare a supersession group: the highest-ranked member cancels every lower live one with its children, slots and checks | +| [0006](./0006-spend-guard.md) | proposed | A monthly budget closes admission and a burn ceiling caps the summed hourly rate of admitted runs; both default low in code | +| [0007](./0007-cost-efficient-execution.md) | proposed | Dispatch only opted-in work, upload each content-addressed cache key once, build heavy native dependencies once in the image | +| [0008](./0008-cheaper-waiting-and-transport.md) | proposed | rpc sandbox transport by default, admission polls scaled by queue position, one log line per supersession | +| [0009](./0009-fast-trustworthy-verdicts.md) | proposed | Interactive admission class ahead of batch, budgeted and reconciled check-run delivery, a light pull-request tier with the heavy gate on the default branch, pool caps as upper bounds under the spend guard | From 3dbc1bd3cc109464a2bef18c63433e7dc8ee2ca5 Mon Sep 17 00:00:00 2001 From: debuggingfuture Date: Thu, 22 Oct 2026 17:48:25 +0800 Subject: [PATCH 3/3] docs(adr): 50 USD default budget; pool sizes and their burn-ceiling reconciliation --- specs/adr/0006-spend-guard.md | 4 +-- specs/adr/0009-fast-trustworthy-verdicts.md | 28 ++++++++++++++++----- 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/specs/adr/0006-spend-guard.md b/specs/adr/0006-spend-guard.md index 5ac151a..53277b6 100644 --- a/specs/adr/0006-spend-guard.md +++ b/specs/adr/0006-spend-guard.md @@ -47,7 +47,7 @@ times an overhead factor for Workflow steps and Durable Objects ### Monthly budget -- `SPEND_BUDGET_MONTHLY_USD` sets the budget. Unset, the budget is **90 USD**. +- `SPEND_BUDGET_MONTHLY_USD` sets the budget. Unset, the budget is **50 USD**. `unlimited` disables the monthly check. A `CONFIG_KV` value at `spend.budgetMonthlyUsd` overrides the variable, so an operator raises the budget without a deploy. @@ -110,7 +110,7 @@ Billable Usage API shows how far apart the two are. ## Consequences -- At the defaults, checks skip once a month's estimated spend reaches 90 USD. +- At the defaults, checks skip once a month's estimated spend reaches 50 USD. Wide fan-outs queue behind the burn ceiling and run slower. - Raising either limit is a single `CONFIG_KV` write. - The meter estimates spend; it is not the invoice. The Billable Usage API diff --git a/specs/adr/0009-fast-trustworthy-verdicts.md b/specs/adr/0009-fast-trustworthy-verdicts.md index 3288031..743577e 100644 --- a/specs/adr/0009-fast-trustworthy-verdicts.md +++ b/specs/adr/0009-fast-trustworthy-verdicts.md @@ -60,12 +60,28 @@ ### Pools sized under the spend guard - `ADMISSION_CAP` per pool is an upper bound. ADR-0006's burn ceiling sets the - operating concurrency. -- The browser pool has its own `ADMISSION_CAP_BROWSER`, like the agent and - release pools. -- `max_instances` per container class is at least the pool's cap plus - headroom for containers still stopping. A test asserts cap ≤ `max_instances` - for every pool. + operating concurrency: a pool admits at most + `min(cap, ceiling / instance hourly rate)` at once, summed across pools. +- The lean pool's cap is 48 by default. It moves to 64 once a week at 48 shows + no container start-rate errors and no placement retries. The browser pool + has its own `ADMISSION_CAP_BROWSER`, like the agent and release pools. +- The caps take effect only as the operator raises the burn ceiling. At the + default 1.50 USD an hour the guard admits three `standard-4` containers, + whatever the cap. Full use of the cap needs the ceiling below: + + | lean concurrency | burn ceiling (`standard-4` at list rate) | + | ---------------- | ---------------------------------------- | + | 3 | 1.50 USD/h (default) | + | 16 | 6.50 USD/h | + | 48 | 19.50 USD/h | + | 64 | 26.00 USD/h | + + An operator raises the ceiling together with the monthly budget, so a + larger pool shortens queues without lifting the cap on the month. + +- `max_instances` per container class is the pool's cap plus 8, headroom for + containers still stopping. A test asserts cap < `max_instances` for every + pool, on the committed vars and on the code defaults. - A claim that admits adds a jitter of up to 2 seconds before the container starts, so a burst of admissions does not hit the container start-rate limit at once.