docs(adr): spend guard, cost-efficient execution, cheaper waiting, fast verdicts

specs/adr/0006-spend-guard.md

@@ -0,0 +1,126 @@
+# ADR-0006 — Spend guard: a monthly budget closes admission, a burn ceiling bounds concurrency
+
+**Status:** proposed
+**Related:** `apps/dispatcher/src/workflow.ts` · `packages/runtime-cf/src/run-admission-d1.ts` · `packages/runtime-cf/src/execution-cost.ts` · `packages/core/src/cost.ts` · `infra/migrations/0005_execution_cost.sql` · ADR-0007
+
+## Context
+
+Cloudflare offers no spend cap for Workers, Workflows or Containers. A budget
+notification sends one email when month-to-date spend first crosses its
+threshold and stops nothing. Container compute is almost all of a deploy's
+bill, and it scales with pushes times fan-out width: a 30-part gate on a busy
+pull request spends in an afternoon what a month of standard checks spends.
+
+Every run passes the admission semaphore before it starts a container, and a
+run refused there costs nothing. That makes admission the one place a hard
+limit holds for every dispatch path: webhook, Action and cron.
+
+The per-execution cost rollup does not support a limit today. It prices the
+execution's wall time, queue wait included, and maps the lean image to
+`standard-2` while the lean container class runs `standard-4`.
+
+## Decision
+
+### Meter
+
+- An execution's container-active time runs from admission to completion.
+  Admission writes `executions.admitted_at`; queue wait costs nothing.
+- Its compute cost is the list rate of its container class's instance type,
+  priced on provisioned vCPU, memory and disk, times active seconds, plus its
+  metered model tokens. A provisioned-size list rate is an upper bound on the
+  billed rate, so the guard stops early rather than late.
+- The instance type per sandbox image comes from one table that a test
+  checks against the container classes in `wrangler.jsonc`.
+- The finish path writes `cost_micro_usd` from this rule. Rows written before
+  the rule keep their value and are labelled `modeled-wall`.
+
+### Spend read
+
+Month-to-date spend is the sum over the current UTC calendar month of:
+
+- `cost_micro_usd` of every finished execution admitted this month;
+- for every admitted, unfinished execution, its hourly rate times the time
+  since `admitted_at`;
+
+times an overhead factor for Workflow steps and Durable Objects
+(`SPEND_OVERHEAD_FACTOR`, default `1.10`). One indexed D1 query answers it.
+
+### Monthly budget
+
+- `SPEND_BUDGET_MONTHLY_USD` sets the budget. Unset, the budget is **50 USD**.
+  `unlimited` disables the monthly check. A `CONFIG_KV` value at
+  `spend.budgetMonthlyUsd` overrides the variable, so an operator raises the
+  budget without a deploy.
+- Admission reads spend before it enqueues a run and again before each claim.
+  At or over budget, the run ends `RunSkipped` with the reason
+  `spend budget reached: $<spend> of $<budget> this month`, which concludes
+  its check-run `neutral` and records the execution `skipped`. No run is
+  exempt.
+- A fan-out parent reads spend before it spawns children. Over budget, it
+  skips and spawns none. A child spawned before exhaustion skips at its own
+  admission.
+- Runs already admitted finish. The overshoot is bounded by the burn ceiling
+  times the longest run body still in flight.
+
+### Burn ceiling
+
+- `SPEND_BURN_MAX_USD_PER_HOUR` caps the summed hourly rate of all admitted
+  executions across every pool. Unset, the ceiling is **1.50 USD per hour**,
+  which admits three `standard-4` containers, or eleven `standard-2`, at once.
+  `unlimited` disables it. `CONFIG_KV` `spend.burnMaxUsdPerHour` overrides it.
+- `run_admissions` records each admitted row's hourly rate. A claim admits
+  only when the pool has a free slot and the admitted rates plus the
+  claimant's own rate fit under the ceiling. A run whose own rate exceeds the
+  ceiling is admitted only into an empty set, so it never waits forever.
+- A run that times out in the queue while the ceiling, not the pool cap, held
+  it back ends `RunSkipped` with the reason `spend ceiling`. A queue timeout
+  under the pool cap alone keeps its current verdict.
+- `max_instances` per container class stays the platform's hard limit. The
+  ceiling makes the configured `ADMISSION_CAP` an upper bound, not the
+  operating point.
+
+### Warning and visibility
+
+- The first admission after spend crosses `SPEND_WARN_RATIO` of the budget
+  (default `0.8`) in a month logs `spend.warn` and sends one notification on
+  the dispatcher's notify path. `CONFIG_KV` `spend.warned:<YYYY-MM>` marks
+  the month as warned.
+- `GET /api/spend` returns month-to-date spend, the budget, the burn ceiling,
+  the admitted hourly rate and the month's reset time. The dashboard shows
+  the same line.
+
+## Rationale
+
+The budget has to bind on every path, and only admission sees every run
+before it costs anything. A neutral skip tells the developer the check did
+not run without reading as a failure.
+
+The defaults cap a deploy that sets nothing. The operator overlay replaces the
+committed `wrangler.jsonc`, so a default in committed config never reaches a
+deploy whose overlay predates it. The defaults therefore live in code.
+
+A monthly budget alone lets one burst spend the month in a day and leaves the
+rest of the month with no checks. The burn ceiling spreads the budget over
+time and bounds the overshoot when the budget closes. A dollar rate, rather
+than a container count, holds across pools whose instance types differ.
+
+An upper-bound meter can stop runs while real spend sits below the budget.
+That is the safe direction for a cap. The monthly reconcile against the
+Billable Usage API shows how far apart the two are.
+
+## Consequences
+
+- At the defaults, checks skip once a month's estimated spend reaches 50 USD.
+  Wide fan-outs queue behind the burn ceiling and run slower.
+- Raising either limit is a single `CONFIG_KV` write.
+- The meter estimates spend; it is not the invoice. The Billable Usage API
+  stays the source of truth for the month's bill.
+- Workers requests, R2, KV, D1 and AI Gateway are inside the overhead factor
+  rather than metered per execution.
+
+## Revisit triggers
+
+- Measured monthly billed spend differs from the meter's month-to-date by more
+  than 25%.
+- Cloudflare ships an account-level spend cap for Containers.
+- A run class needs to run regardless of budget, such as a release.

specs/adr/0007-cost-efficient-execution.md

@@ -0,0 +1,100 @@
+# ADR-0007 — Cost-efficient execution: dispatch only work, upload each cache key once, build heavy native dependencies once
+
+**Status:** proposed
+**Related:** `apps/dispatcher/src/routes/webhook.ts` · `runs/check.ts` · `runs/offload-test.ts` · `runs/worker-deploy.ts` · `packages/runtime-cf/src/cache-r2.ts` · `packages/runtime-cf/src/cache-restore-or.ts` · `runs/contextful-gate.ts` · `infra/Dockerfile.sandbox-demo` · ADR-0006
+
+## Context
+
+A deploy pays for container time, and three patterns spend it on nothing:
+
+- A webhook run on a repo that never opted in creates a Workflow, opens a
+  check-run, waits for an admission slot, starts a container, and only then
+  reads `CONFIG_KV` and ends green with nothing done. Each one also holds a
+  pool slot that real work waits for.
+- A cache restore that finds its archive but cannot apply it falls through to
+  the miss path: the run installs from scratch, then packs and uploads the
+  same content-addressed archive again. On the http sandbox transport every
+  stream restore fails this way, so every run of a fan-out re-uploads the
+  archive that already sits in R2.
+- Every part of a fan-out gate clones and compiles the same dependency graph
+  in a fresh container. A bundled C++ dependency such as DuckDB rebuilds its
+  engine from source in each part and dominates that compile.
+
+ADR-0006 caps what a deploy spends. This record lowers what each verdict costs,
+so a budget buys more checks.
+
+## Decision
+
+### Dispatch only work
+
+- A run may declare `optIn: (ctx) => readonly string[]`, the `CONFIG_KV` keys
+  whose presence means the repo opted in. `check`, `offload-test` and
+  `worker-deploy` declare the command keys their bodies read today.
+- The webhook receiver reads those keys before it creates a Workflow. With
+  none present it dispatches nothing, opens no check-run, and lists the run in
+  the response's `skipped` array with reason `not-configured`.
+- A dispatch that names its command explicitly, from an Action or a fan-out
+  parent, skips the opt-in read.
+- The run bodies keep their own opt-in check, so a key deleted between
+  dispatch and run still ends in a no-op.
+
+### Upload each cache key once
+
+- A restore reports one of three outcomes: `hit`, `miss` (no archive under
+  the key), or `unapplied` (the archive exists, applying it failed).
+- `unapplied` runs the miss path and skips the save: the key is
+  content-addressed, so the archive in R2 already holds this content.
+- A save checks `bucket.head` for its key before it packs. An existing
+  archive skips the `tar czf` and the upload. Parallel misses on one key then
+  upload once instead of once per part.
+- A restore on a transport that cannot stream into the container reports
+  `unapplied` without reading the archive body, and logs the transport once
+  per execution.
+
+### Fan-out parts install nothing their stage repeats
+
+- A gate part whose stage command resolves its own dependencies, as `cargo`
+  does on build, spawns with `install: false` and restores only what the
+  stage cannot rebuild cheaply, such as a prebuilt tool binary.
+
+### Prebuilt heavy native dependencies in the image
+
+- The Rust variant of the lean sandbox image ships sha256-pinned prebuilt shared libraries and
+  headers for native dependencies that dominate a cold compile, DuckDB first,
+  and exports the paths the dependency's build script reads in linked mode
+  (`DUCKDB_LIB_DIR`, `DUCKDB_INCLUDE_DIR`).
+- The variables change nothing for a consumer that builds the dependency
+  bundled: the build script reads them only in linked mode. A consumer opts
+  in by building the dependency linked in CI. The consumer owns that choice,
+  and it asserts the loaded library's version at startup, because a linked
+  library of the wrong version compiles cleanly and misbehaves at runtime.
+- The image's library version and the consumer's pinned crate version move
+  together.
+
+## Rationale
+
+The receiver is the last point where skipping costs nothing: past it, a skip
+still pays for a Workflow, a check-run and a slot. Keeping the body's check
+makes the receiver read an optimisation rather than the only line of defence.
+
+A content-addressed key makes a second upload pure waste, so the head check is
+always safe. Separating `unapplied` from `miss` keeps a broken restore visible
+in logs without paying for it in container time.
+
+Building a C++ engine from source in every part buys nothing a prebuilt
+library does not, and only the consumer can decide whether CI may link it
+dynamically while release builds stay static.
+
+## Consequences
+
+- Webhook responses on unconfigured repos list `skipped` runs. Those repos no
+  longer receive green no-op check-runs.
+- A broken restore still costs a cold install per part until the transport is
+  fixed, but no longer a repeated pack and upload.
+- The lean image grows by the prebuilt libraries' size.
+
+## Revisit triggers
+
+- A restore path reaches the container without passing through the Worker,
+  such as a signed R2 URL, at a throughput that makes a shared build cache pay.
+- A consumer's opt-in needs more than the presence of a `CONFIG_KV` key.

specs/adr/0008-cheaper-waiting-and-transport.md

@@ -0,0 +1,70 @@
+# ADR-0008 — Cheaper waiting: rpc transport by default, position-scaled admission polls, counted supersessions
+
+**Status:** proposed
+**Related:** `packages/runtime-cf/src/runtime.ts` · `packages/runtime-cf/src/sandbox-cf.ts` · `apps/dispatcher/src/workflow.ts` · `packages/runtime-cf/src/run-admission-d1.ts` · `apps/dispatcher/src/supersede.ts` · ADR-0005 · ADR-0007
+
+## Context
+
+- The sandbox transport defaults to http, and only a per-repo `CONFIG_KV` pin
+  selects rpc. Streaming a file into a container needs rpc, so on the default
+  transport every cache restore fails and every run installs from scratch.
+- A queued run polls admission every 20 seconds whatever its position. Each
+  poll is a claim step plus a sleep step, and Workflows bills each step. A run
+  fifty places back pays for hundreds of polls that cannot admit it.
+- ADR-0005 cancels a superseded gate's children, but nothing counts how often
+  that happens or how much work it stops, so nobody can tell whether it is
+  working.
+
+## Decision
+
+### rpc transport by default
+
+- `SANDBOX_TRANSPORT` defaults to `rpc` in code when unset.
+- `CONFIG_KV` `sandbox.transport:<repo>` still overrides the default per repo,
+  in both directions, so an operator pins a repo back to `http` without a
+  deploy.
+- A container keeps the transport it started with for its whole life, as
+  `sandbox-cf.ts` pins it today.
+
+### Position-scaled admission polls
+
+- A waiter at FIFO position `p` in a pool with cap `c` sleeps
+  `min(60 s, 20 s × ceil(p / c))` between claims. The head of the queue keeps
+  the 20-second poll, and a waiter many slots back polls up to three times
+  less often.
+- The waiter liveness window (`ADMISSION_WAITER_TTL_MS`) is at least three
+  times the longest sleep, so a slow-polling waiter never reads as dead.
+- The sleep length derives from the position observed inside the claim step,
+  so a replay sleeps the same length.
+
+### Counted supersessions
+
+- Each supersession logs one `supersede.cancelled` line naming the group, the
+  cancelled root, and the number of children, admitted slots and open
+  check-runs it ended.
+- The dashboard's execution list labels a cancelled execution with the
+  execution that superseded it.
+
+## Rationale
+
+rpc is the only transport on which every capability in `@cloudflare/sandbox`
+works, file streaming included, so an http default makes the cache a cost
+rather than a saving. Keeping the per-repo pin gives a deploy a rollback
+without a release.
+
+Polling far from the head buys nothing: a waiter cannot move more than one
+position per released slot. Scaling the sleep with position cuts the step bill
+of a long queue while the head still admits within one short poll.
+
+A log line per supersession makes the saving measurable without a new table.
+
+## Consequences
+
+- Deploys that never set the variable switch to rpc on their next deploy.
+- Waiters deep in a long queue admit up to 40 seconds later after a burst of
+  releases.
+
+## Revisit triggers
+
+- rpc fails a capability that http serves.
+- Admission moves from polling to an event that wakes the head waiter.