fix(bootstrap): first-boot FK race + single-flight

[mmcintosh]

fix(bootstrap): document-type seeding must precede its FK consumers on a fresh DB

Fixes #1017

TL;DR

On a fresh database's first boot, bootstrapMiddleware seeds system data with a
flat Promise.all that races a foreign-key dependency: two steps create
document_types rows while two other steps insert documents that FK-reference
those rows. When a consumer insert wins the race, D1 throws FOREIGN KEY constraint failed; the per-step .catch() swallows it, so RBAC roles/verbs and every core
plugin are silently left unseeded. A second facet compounds it: there is no
single-flight guard, so two concurrent requests on a cold isolate both enter bootstrap
and seed in parallel — re-opening the same FK window across the two runs. The bootstrap
then writes an unconditional KV skip-marker (24h TTL), which latches the broken
state: every subsequent request/isolate takes the KV fast-path, skips D1 entirely,
and the site runs with no roles and no plugins until the marker expires or is cleared.

This PR (1) makes the two document-type producers run and finish before the consumer
inserts, (2) adds a single-flight in-flight promise so concurrent cold-isolate requests
await one bootstrap instead of racing parallel ones, and (3) makes the "bootstrap
complete" latch + KV marker conditional on success.

Update — re-grounded onto current main

This branch sat for about a month; main moved independently in that window (a KV
freshness guard on the fast-path, a Better-Auth session skip-path, a new
bootstrapDefaultContent seed step). Re-grounded onto current main with a hand-merge
through the one resulting conflict — the fix's Phase A/B restructuring, applied around
all of main's independent additions. bootstrapDefaultContent folds into the same
per-step failure tracking as the rest of Phase B (a failure there no longer withholds
the completion latch either — a decorative demo-post seed shouldn't share blast radius
with RBAC/plugin seeding).

The original regression test proved the DB/FK premise directly but never exercised
bootstrapMiddleware itself — confirmed by reverting the Phase A/B split and rerunning
it: still green. Added a second test that mounts the real middleware in a Hono app and
fires a request through it against an FK-enforced SQLite DB, asserting the RBAC
documents actually land. Verified that test isn't a placebo either, the same way: with
the fix temporarily reverted it fails exactly as expected (0 RBAC rows written, and the
new "completed WITH ERRORS" log line fires), then reverted back clean.

The ~25 pre-existing unit failures this branch used to show relative to a month-old
main are gone — that was always a separate, unrelated test-infra issue (per-DB
caches + an executionCtx guard), fixed independently since. Full suite is green now:
1734 passed, 0 failed.

---

How we hit it (real, not theoretical)

We were spinning up a clean local greenfield stack (fresh D1, empty KV) on
beta.25 to validate an incoming feature branch. The very first request logged:

[PluginBootstrap] Installing plugin: Authentication System
✘ [ERROR] [PluginBootstrap] Error ensuring plugin Authentication System:
    Error: D1_ERROR: FOREIGN KEY constraint failed: SQLITE_CONSTRAINT_FOREIGNKEY
  [cause]: Error: FOREIGN KEY constraint failed: SQLITE_CONSTRAINT_FOREIGNKEY
    at PluginService.installPlugin (services/plugin-service.ts:103)
    at PluginBootstrapService.ensurePluginInstalled (services/plugin-bootstrap.ts:131)
    at PluginBootstrapService.bootstrapCorePlugins (services/plugin-bootstrap.ts:80)
    at async Promise.all (index 4)
✘ [ERROR] [Bootstrap] Error seeding RBAC documents:
    Error: D1_ERROR: FOREIGN KEY constraint failed: SQLITE_CONSTRAINT_FOREIGNKEY

Two things stood out:

1. Every failing step is inside the await Promise.all([...]) in
   bootstrap.ts (at async Promise.all (index 4)), and the failures are exactly
   the steps that write documents rows (RBAC seed at index 3, plugin bootstrap at
   index 4). The steps that create document_types are indices 0–1 of the same
   array.
2. After that first boot, the errors never appeared again — but the site was broken:
   admin routes returned 403 Insufficient permissions (no RBAC roles), and core
   plugins were inactive. Restarting the worker didn't help. Only clearing the KV
   bootstrap marker (rm -rf .wrangler/state/v3/kv locally) and re-booting fixed it —
   because by the second boot the document_types rows happened to already exist, so
   the race didn't reproduce.

That second-boot-heals-it behavior is the tell: the bug is a write-ordering race on
a cold DB, and the KV marker turns a transient race into a sticky outage.

Root cause

documents.type_id is NOT NULL REFERENCES document_types(id)
(migrations/0002_documents.sql:30). The seed phase in bootstrap.ts runs five
steps in one Promise.all:

Step	Function	Role
0	bootstrapDocumentTypes	producer — inserts document_types rows
1	autoRegisterCollectionDocumentTypes	producer — inserts document_types rows
2	repairMissingCredentialAccounts	independent (auth account table)
3	rbacService.ensureSystemRbacSeed	consumer — INSERT INTO documents (rbac_role/verb/grant), FK → document_types
4	PluginBootstrapService.bootstrapCorePlugins	consumer — installPlugin → INSERT INTO documents, FK → document_types

Because all five are launched concurrently, on a cold DB a consumer's documents
insert can execute before the producer has committed the matching document_types
row → FOREIGN KEY constraint failed. D1 enforces this FK; local SQLite in the test
harness disables it (which is why unit tests never caught it — see Testing).

Second facet: no single-flight guard

bootstrapComplete (module-level) only flips to true at the end of a successful
run. On a cold isolate, two requests that arrive close together both read
bootstrapComplete === false, both miss the KV marker (not written yet), and both run
the full seed concurrently. Even with the ordering fix applied per-run, two
concurrent runs can interleave (run A creating types while run B inserts documents) and
duplicate every seed write. There is no promise that a second caller can await.

Third defect: the marker latches failures

After the batch:

bootstrapComplete = true;                        // in-memory latch — unconditional
await cacheKv.put(BOOTSTRAP_KV_KEY(), '1', { expirationTtl: 86400 })  // KV marker — unconditional

Both are written even when steps in the batch failed. The KV marker is the cold-start
fast-path (if (kvDone === '1') { … return next() } near the top of the middleware),
so once it's set, no isolate re-runs the D1 seed for 24h — the partial bootstrap
is cached as if it were complete.

Who this affects

Any environment whose first-ever request lands on a fresh, unseeded database:

• New Cloudflare deploys — the first request after a fresh D1 is provisioned.
• Preview/branch deploys with their own D1.
• Local dev on a clean .wrangler state.
• CI that boots against a freshly-migrated DB.

Warm/existing databases are unaffected (the document_types already exist, so the
race can't lose). That's precisely why it's easy to miss in day-to-day dev against an
already-bootstrapped DB, and why it bites new installs specifically.

The fix

packages/core/src/middleware/bootstrap.ts — the seed phase becomes two phases:

• Phase A (await, sequential): the two producers —
  bootstrapDocumentTypes then autoRegisterCollectionDocumentTypes — so all
  document_types rows are committed before any documents insert references them.
  These are two fast, idempotent registrations; awaiting them adds negligible latency.
• Phase B (parallel, as before): the two consumers (RBAC seed, plugin bootstrap)
  plus the independent credential-account repair, kept in a Promise.all so
  cold-start latency is preserved now that the FK precondition holds. main's newer
  bootstrapDefaultContent step runs after Phase B, logged on failure but not gating
  the completion latch (see "Update" above).

Single-flight guard. A module-level bootstrapInFlight: Promise<void> | null holds
the running bootstrap. A request that arrives mid-bootstrap awaits it and proceeds
rather than starting a second seed. The check-and-set is synchronous (no await
between the if (bootstrapInFlight) test and the assignment), so exactly one request
wins the latch; it is released in a finally. This closes the concurrency facet and
also removes duplicate cold-start work.

A small runStep(label, fn) wrapper records whether any step threw. The
bootstrapComplete latch and the KV skip-marker are now written only when every
step succeeded. If something still fails transiently (a D1 hiccup, say), nothing is
cached, and the next request retries the idempotent bootstrap and converges — instead
of locking a broken state for 24h.

Net behavior on the happy path is unchanged: types get registered, consumers seed,
the marker is written. The only difference is ordering (so the FK holds) and that a
failed boot no longer poisons the cache.

Why not just widen the KV TTL / bump the version key

c99a5673 ("bump version to beta.25 — invalidates KV bootstrap key") already showed
the marker can cache a bad state — but a version bump only busts the key once per
release; the next fresh DB re-enters the same race and re-latches. The self-host path
633023e4 ("entrypoint-based seed prevents concurrent SQLite corruption") applies the
right idea — seed sequentially before serving — but only to the Docker/SQLite runtime.
This PR brings that ordering guarantee to the Workers/D1 cold-start path, which is
where new cloud installs boot.

Testing

packages/core/src/__tests__/middleware/bootstrap-fk-ordering.test.ts:

• DB/FK-premise tests (original, real SQLite, foreign_keys = ON): the shared
  d1-sqlite harness disables FK enforcement by default (D1 doesn't reliably enforce
  it and the services delete derived rows explicitly) — here the FK is the subject, so
  the test re-enables it. Proves a documents insert for an unregistered type rejects
  with FOREIGN KEY constraint failed and writes nothing, and that registering the
  type first lets it land.
• End-to-end middleware test (new): mounts the real bootstrapMiddleware in a
  minimal Hono app (plugin bootstrap disabled via config.plugins.disableAll to keep
  the FK dynamic isolated to RBAC seeding, which shares the identical Phase A/B
  wiring), fires one request against the FK-enforced harness, and asserts the RBAC
  document_types and documents rows both actually exist afterward. Verified this
  test regresses: reverting Phase A back into Phase B's Promise.all makes it fail
  (0 RBAC rows, "completed WITH ERRORS" logged); reverted the break, confirmed clean.

Reality-tested end-to-end on fresh greenfield boots (clean D1 + empty KV) against the
beta.25 runtime:

• Single first request: clean boot, zero FK errors, DB fully seeded (21
  document types, 4 RBAC roles, 6 verbs, 4 plugins) — no KV-clear-and-reboot needed.
• Four concurrent cold-isolate requests: the bootstrap body runs exactly once
  (Starting system initialization logged once), completes cleanly, zero FK errors
  — proving the single-flight guard.

Before the fix, the same first boot always FK-failed on Installing plugin: Authentication System + RBAC seed and required a KV-clear + second boot to recover.

Gates: tsc --noEmit — 0 errors. Full unit suite — 1734 passed, 0 failed
(re-grounded onto current main; the pre-existing test-infra failures this branch used
to inherit are gone independently of this PR).

Scope / not in scope

• No change to the KV fast-path read, the infra-path skips, or plugin gating.
• The self-host entrypoint seed path (633023e4) is unchanged.

[mmcintosh]

Note on the red CI. The failing check here is the unit-test step (about 25 tests), and those failures are pre-existing on main — this branch is cut straight from current main, and none of them touch bootstrap.ts. They are the known beta.25 test-harness issue (a per-database cache leaks generated columns across in-memory test DBs, producing "no such column: q_*" errors) and are fixed by the test-infra PR (#1018, Fixes #1016). Type-check passes here, and this PR's purpose is to fix the E2E against preview failure: the loginAsAdmin timeout caused by the first-boot FK race that leaves the RBAC roles and core plugins unseeded (#1017). The two are a pair: #1018 fixes this PR's unit suite, and this PR fixes #1018's E2E preview. Recommend landing #1018 first, then this one — after both, CI is green across the stack.