The governor saved the machine when it was one laptop and one build. It is now the bottleneck and the hazard on the big boxes: a 32-core EPYC serialises to ONE build per user, and when the 30-minute lock wait expires every queued waiter proceeds UNSERIALISED at full width at the same moment -- N x (cores-2) compile threads, which is exactly the OOM from the 2026-09-01 overnight incident. With the bacon fleet landing (a watcher per VS Code window, commonly ~10), lock traffic multiplies and the single mutex also turns the realtime diagnostics stream into minutes of queue latency.
What the current shim does (read at /usr/local/bin/hyperi-rust-govern): SERIALIZE=1 flock per user, LOCK_WAIT=1800 then per-waiter fallthrough to unserialised full-width builds, CARGO_BUILD_JOBS = cores - RESERVE regardless of concurrency, rust-build.slice + CPUQuota as the outer budget, and an admitted flock hole -- a daemon spawned by a build inherits fd 9 and holds the lock for its own lifetime.
The redesign, sized for cores/RAM instead of assuming one build:
- SLOTS, not a mutex: K concurrent governed builds, K and jobs-per-slot from rust-governor.conf with defaults derived from nproc and MemTotal (roughly one slot per 8 cores / 8 GB, floor 1, cap 6). Total width never exceeds cores - reserve.
- CLASSES via HYPERI_RUST_GOVERN_CLASS: interactive (default -- a human or agent invoked it) outranks background (bacon watchers, cache warmers). Background builds get a lower job cap (cores/4 capped) and never queue long: a short wait, then run DEGRADED (jobs=2, nice 19) rather than blocking the watcher or stampeding.
- Kill the stampede: on wait expiry a waiter proceeds degraded (jobs=2 inside the slice), individually, with 0-10s jitter -- never all-at-once at full width. That is the incident fix.
- Stale holders: probe the holder pid (fuser already in the script) and break a dead or daemon-inherited lock instead of waiting 30 minutes; per-slot flock files keep crash-release on Linux, the mkdir+pid pattern (already in the macOS path) covers Darwin. Stays bash 3.2 / POSIX, no new deps.
- The slice stays the outer budget; per-scope CPUQuota follows the actual per-slot jobs.
Done-when: 10 background bacons plus 2 interactive builds on a 32-core box -- interactive never waits past ~30s, background never blocks interactive, nothing OOMs, and the unserialised-stampede log line cannot occur.
Companion (hyperi-ai side): bacon_up will export the background class and cap bacon's cargo jobs -- tracked there. Boxes this must fit: the Macs, dragonfly (12c/30G), desktop-derek (32-core EPYC, 64G and likely doubling).
The governor saved the machine when it was one laptop and one build. It is now the bottleneck and the hazard on the big boxes: a 32-core EPYC serialises to ONE build per user, and when the 30-minute lock wait expires every queued waiter proceeds UNSERIALISED at full width at the same moment -- N x (cores-2) compile threads, which is exactly the OOM from the 2026-09-01 overnight incident. With the bacon fleet landing (a watcher per VS Code window, commonly ~10), lock traffic multiplies and the single mutex also turns the realtime diagnostics stream into minutes of queue latency.
What the current shim does (read at /usr/local/bin/hyperi-rust-govern): SERIALIZE=1 flock per user, LOCK_WAIT=1800 then per-waiter fallthrough to unserialised full-width builds, CARGO_BUILD_JOBS = cores - RESERVE regardless of concurrency, rust-build.slice + CPUQuota as the outer budget, and an admitted flock hole -- a daemon spawned by a build inherits fd 9 and holds the lock for its own lifetime.
The redesign, sized for cores/RAM instead of assuming one build:
Done-when: 10 background bacons plus 2 interactive builds on a 32-core box -- interactive never waits past ~30s, background never blocks interactive, nothing OOMs, and the unserialised-stampede log line cannot occur.
Companion (hyperi-ai side): bacon_up will export the background class and cap bacon's cargo jobs -- tracked there. Boxes this must fit: the Macs, dragonfly (12c/30G), desktop-derek (32-core EPYC, 64G and likely doubling).