fix: gate managed zccache sidecar install and surface blocking status to clients

[zackees]

Context

The managed zccache sidecar is installed lazily from the build path:

• crates/fbuild-build/src/zccache.rs resolves zccache through find_zccache().
• find_zccache() uses a process-local OnceLock<Option<PathBuf>>, so within a single daemon process only one thread runs managed_zccache::ensure() and other same-process callers block on that initialization.
• crates/fbuild-build/src/managed_zccache.rs::ensure() checks for the final binary, then downloads SHA256SUMS plus the release archive, verifies it, extracts into a unique staging dir, and finally renames into ~/.fbuild/<mode>/bin/zccache-<version>/.
• The final rename handles a lost race: if another installer already placed the final binary, the loser deletes its staging dir and succeeds.

That final atomic rename prevents a half-installed zccache from being observed, but it does not prevent duplicate work before the rename. Multiple fbuild daemons/processes that hit first-run zccache setup at the same time can each download and extract the same archive. The daemon also does not expose a distinct "managed zccache download/install in progress" state; blocked clients just see a generic build/install wait.

Current daemon behavior:

• POST /api/build and POST /api/install-deps serialize per project through ctx.project_lock(project_dir).
• Streaming builds return the NDJSON response immediately and can wait up to the CLI's 1800s request timeout.
• Non-streaming operations also use the generic 1800s daemon POST timeout.
• /ws/status and daemon status snapshots expose operation_in_progress and current_operation, but not a structured dependency-install phase or wait reason.
• OperationGuard updates in-memory daemon state but does not currently broadcast a status update when operation text changes.

The desired behavior is that the daemon owns first-run sidecar install coordination: one installer performs the download, other clients block intentionally, and blocked clients receive a clear status/progress signal instead of appearing hung or timing out.

Proposal

Add a daemon/process-shared managed dependency install gate for the zccache sidecar.

Suggested shape:

1. Add a cross-process lock around managed zccache installation before network fetch begins, not only at final rename.
   • Lock scope should be the final managed version dir or a sibling lock file under ~/.fbuild/<mode>/bin/.
   • Within the lock, re-check managed_zccache_exe().is_file() before downloading.
   • Other processes should wait on the lock instead of starting duplicate downloads.
2. Add daemon-visible install state for managed sidecar setup.
   • At minimum: dependency_install_in_progress, dependency_name = "zccache", dependency_version, phase (waiting_for_lock, downloading, verifying, extracting, installed), and a human message.
   • This should update /health//api/daemon/info or a dedicated status endpoint, and /ws/status should broadcast phase changes.
3. Ensure clients do not time out while intentionally blocked behind a sidecar install.
   • Streaming clients should receive periodic NDJSON/status events while waiting.
   • Non-streaming clients should either use the existing 1800s timeout safely or get a clear long-running operation response contract if the wait exceeds a threshold.
   • The status should distinguish "waiting for another fbuild process to install zccache" from "compiling".
4. Keep failure behavior explicit.
   • If the installer fails, waiters should observe the failure and either retry once or fall back to ambient zccache discovery according to the existing find_zccache() policy.
   • Do not leave stale lock state that blocks future clients forever.

Acceptance criteria

• Two concurrent fbuild clients starting with no managed zccache installed result in exactly one network download/extract attempt for the managed zccache version.
• Other clients block on the daemon/process-shared install gate and do not start duplicate downloads.
• While blocked, clients can observe a clear status such as "waiting for managed zccache 1.12.7 install" rather than only "building".
• Streaming build clients receive periodic status/log events during long sidecar install waits so the connection stays alive and looks intentional.
• Non-streaming build/install-deps clients do not hit a request timeout solely because another client is installing zccache.
• A failed or interrupted installer does not leave a permanent stale lock; a later fbuild invocation can recover.
• Tests cover same-process contention, cross-process/file-lock contention, and stale/interrupted install recovery.

Open questions

• Should this gate live in fbuild-build::managed_zccache as a generic sidecar-install lock, or in fbuild-daemon so it can update daemon status directly?
• Should zccache install status be represented as a new daemon state, or as structured fields alongside the existing Building state?
• Should the same mechanism become a generic managed-dependency install gate for future sidecars, not just zccache?

Related issues

• chore: remove zccache crate linkage and use the managed sidecar for cache APIs #600 removes compile-time zccache crate linkage. This issue is separate: it covers runtime sidecar install coordination and client-visible blocking behavior.

[zackees]

Additional investigation: broker/cache strategy risk

There is a broader broker-design issue here beyond the zccache install lock.

fbuild has very large, shared artifacts: toolchains, framework/platform packages, downloaded archives, managed sidecars, and compile/cache data. Those need to behave like a per-user/global repository (~/.fbuild/<mode>/cache, or an explicit FBUILD_CACHE_DIR) with stable package identity. We should not let broker adoption accidentally turn these into private stores scoped to whichever fbuild daemon/backend version the broker negotiated.

Current broker-facing code records fbuild cache roots through CacheManifest in crates/fbuild-daemon/src/broker/service.rs:

• CacheData = get_cache_root()
• CacheIndex = <cache>/index
• CacheTmp = <fbuild_root>/tmp
• CacheLogs / CacheLocks = daemon dir
• CacheRuntime = daemon binary dir
• CacheConfig = get_fbuild_root()

That is directionally good because it points at fbuild's real cache roots, not a broker-created private backend cache. The risk is policy: if the broker starts resolving multiple fbuild backend versions as separate runnable backends, clients could observe different daemon versions while sharing or publishing cache manifests inconsistently. For fbuild, the artifact repository has to be authoritative and shared, while daemon version negotiation should not imply per-version artifact ownership.

This points to a strategy decision before deeper broker migration:

1. Single-fbuild policy: enforce one active fbuild daemon per user/cache root, and make the broker resolve to that daemon. If a newer fbuild is requested, replace/upgrade the daemon rather than running parallel versioned daemons against separate manifests.
2. Broker resolver policy: if multiple fbuild backend versions exist, the broker needs a deterministic strategy based on cache-root identity and compatibility, not just service version. The selected backend must use the same global artifact repository and must not fork package/cache state by backend version.
3. Explicit cache-owner contract: fbuild owns packages/toolchains/framework archives and sidecars. The broker can discover/report roots, but should not create private versioned stores for these artifacts unless fbuild explicitly opts into a separate trust/cache domain such as CI EXPLICIT_INSTANCE.

Suggested follow-up acceptance criteria for the broker side:

• Document that local fbuild uses one shared per-user artifact repository, independent of daemon backend version.
• Add a guard/test that broker-negotiated fbuild daemons publish the same global cache root for the same FBUILD_CACHE_DIR/dev-mode inputs.
• Decide whether local development should enforce exactly one active fbuild daemon per cache root.
• If parallel backend versions are allowed, specify the resolver rule and cache compatibility rule before routing build/deploy traffic through the broker.
• Keep CI explicit-instance isolation as the exception, not the local default.

[zackees]

Broker/global cache investigation split

Breaking the concern in the broker/cache strategy comment into sub-issues for focused investigation:

1. Sub-issue A: daemon identity and broker resolution

   • Should local fbuild enforce exactly one active daemon per user/cache root?
   • Does the current running-process service definition allow parallel backend versions that could route clients inconsistently?

2. Sub-issue B: global artifact repository ownership

   • Are package/toolchain/framework archives and managed sidecars already rooted in stable per-user paths independent of daemon version?
   • Where could broker adoption accidentally create private/versioned stores?

3. Sub-issue C: large download/install concurrency and client blocking

   • Which existing locks prevent duplicate package/toolchain/sidecar downloads across concurrent fbuild uses?
   • What status should blocked clients see so they wait intentionally rather than timing out?

4. Sub-issue D: compatibility, migration, and recovery policy

   • If old and new fbuild daemons coexist, what compatibility rule protects the shared repository?
   • What stale-lock or interrupted-install recovery behavior is required?

I?m going to append separate repo-backed findings for each sub-issue.

[zackees]

Sub-issue B findings: global artifact repository ownership

Findings from the current tree:

• The canonical per-user roots are centralized in crates/fbuild-paths/src/lib.rs: get_fbuild_root() is ~/.fbuild/{dev|prod} (FBUILD_DEV_MODE split) and get_cache_root() is FBUILD_CACHE_DIR or <fbuild_root>/cache (crates/fbuild-paths/src/lib.rs:19, crates/fbuild-paths/src/lib.rs:61). This is independent of the fbuild daemon/backend version.
• Package/toolchain/platform/framework-ish installs are rooted under that cache root. The legacy Cache facade builds packages/, toolchains/, platforms/, libraries/, and core/ below cache_root (crates/fbuild-packages/src/cache.rs:51, :55, :59, :63, :67), and PackageBase::new() uses Cache::new(project_dir) plus DiskCache::open() (crates/fbuild-packages/src/lib.rs:211, :212). PackageBase::staged_install() commits into the resolved cache path and writes .install_complete (crates/fbuild-packages/src/lib.rs:314, :369, :376). No daemon-version segment is part of these paths.
• The newer two-phase DiskCache also opens at fbuild_paths::get_cache_root() (crates/fbuild-packages/src/disk_cache/mod.rs:41) and defines archives/, installed/, and index.sqlite under the same root (crates/fbuild-packages/src/disk_cache/paths.rs:118, :123, :128). Kind covers packages, toolchains, platforms, libraries, frameworks, and lnk blobs (crates/fbuild-packages/src/disk_cache/paths.rs:18). This is the right ownership model for large archives and cache data, but there is still a mixed layout: the legacy package installer records installs into the DiskCache index only if the install path is under dc.cache_root() (crates/fbuild-packages/src/lib.rs:289, :291), while the physical install path can be the legacy <cache>/toolchains/... / <cache>/platforms/... shape rather than <cache>/installed/....
• Managed zccache is a sidecar, but it is not in get_cache_root(): fbuild installs the pinned binary under ~/.fbuild/<mode>/bin/zccache-<MANAGED_ZCCACHE_VERSION>/ via managed_dir() (crates/fbuild-build/src/managed_zccache.rs:23, :47, :48). find_zccache() intentionally prefers FBUILD_ZCCACHE_BIN and otherwise this managed binary (crates/fbuild-build/src/zccache.rs:31, :34, :55, :70). This is stable per user/mode and versioned by zccache, not by fbuild daemon version, but it will not follow FBUILD_CACHE_DIR.
• Broker CacheManifest is directionally correct for artifacts: CacheData is get_cache_root(), CacheIndex is <cache>/index, and config/temp/log/lock come from get_fbuild_root()/daemon dirs (crates/fbuild-daemon/src/broker/service.rs:44, :53, :59; crates/fbuild-paths/src/running_process.rs:117, :134, :138, :139, :140, :143, :144). The inventory doc says the same and explicitly treats local dev as SHARED_BROKER, with CI using EXPLICIT_INSTANCE ci-trusted (docs/running-process/inventory.md:19, :20).

Where broker adoption could accidentally create private/versioned stores:

1. If broker-spawned backends derive package/cache roots from service-definition location, CacheRuntime, relocated daemon binary dirs, broker instance id, or backend version instead of fbuild_paths::get_cache_root(), they could fork packages/toolchains/frameworks into private per-version stores.
2. CacheRuntime is explicitly the daemon binary directory (crates/fbuild-daemon/src/broker/service.rs:49, :58). That is useful runtime metadata, but it should not become the owner for artifact sidecars. The managed zccache sidecar already lives under get_fbuild_root()/bin; if broker relocation adds another per-version bin root, zccache installs could split by backend runtime path.
3. CI docs correctly say to cache FBUILD_CACHE_DIR and not ~/.fbuild/*/daemon/ (docs/CI_CACHE.md:15, :19; docs/CI_CACHING.md:92, :93, :94, :97, :99). Broker adoption should keep CI EXPLICIT_INSTANCE isolation as a trust-domain exception, not as the local default.

Actionable recommendations:

• Add a broker adoption invariant/test: for the same environment (FBUILD_DEV_MODE, FBUILD_CACHE_DIR, HOME/USERPROFILE), two fbuild backend versions must publish identical CacheData/artifact roots and must not include service version, broker instance, or daemon runtime path in package/toolchain/framework/cache roots.
• Treat CacheRuntime as executable provenance only. Do not use it to root package archives, extracted installs, zccache sidecars, lnk blobs, or cache indexes.
• Decide and document whether managed zccache belongs under get_fbuild_root()/bin permanently or should move/report as a sidecar root. Either way, make the contract explicit because it is stable per user/mode but outside FBUILD_CACHE_DIR.
• Reconcile docs/tests around the mixed cache layout: docs describe <cache>/archives, <cache>/installed, and index.sqlite, while current package installs can still land in legacy <cache>/toolchains/<cache>/platforms. That is not a broker-versioning bug, but it is the main place future cache-root changes could accidentally fork ownership.

[zackees]

Sub-issue C findings: download/install concurrency and client blocking

Findings from the current Rust daemon/package paths:

• crates/fbuild-daemon/src/context.rs keeps per-project locks as DashMap<PathBuf, Arc<tokio::sync::Mutex<()>>> and project_lock() keys them by the exact PathBuf. crates/fbuild-daemon/src/handlers/operations/build.rs takes that lock for both streaming and non-streaming builds, and crates/fbuild-daemon/src/handlers/operations/install_deps.rs takes the same lock before install_platform_deps(). This serializes concurrent requests for the same project inside one daemon, but it does not protect shared package/toolchain/framework installs across different projects or across separate daemon processes.

• The generic package installer in crates/fbuild-packages/src/lib.rs is staged but not package-locked. PackageBase::staged_install() checks install_path.exists(), uses a deterministic sibling {final}_staging, deletes any existing staging dir, downloads with downloader::download_file(), extracts, validates, then renames staging to final and writes .install_complete. Two fbuild clients installing the same uncached package can delete or reuse each other's staging dir; if they are different projects/daemons, the project lock will not prevent this. Most toolchains and framework packages call this shared path via ensure_installed() (crates/fbuild-packages/src/toolchain/*.rs, crates/fbuild-packages/src/library/*.rs).

• crates/fbuild-build/src/managed_zccache.rs already has the safer pattern: unique per-process staging dirs under ~/.fbuild/<mode>/bin, atomic rename to final, and an explicit "lost race with concurrent installer" success path if the final binary exists. This should be the model for package/framework/toolchain installs.

• Download progress plumbing exists but is not wired into package installs. crates/fbuild-packages/src/downloader.rs has DownloadProgress and download_file_with_progress(), but PackageBase::staged_install() and library_downloader.rs call buffered download_file(), which loads the whole response into memory and emits no client-visible progress. That is especially rough for ESP32/toolchain/framework downloads.

• Blocked streaming clients currently get an open NDJSON response but no queued/progress event while waiting for the project lock. build.rs logs waiting for project lock and warns after 10s, then waits indefinitely, but it does not send a stream event until build log lines start. Non-streaming clients just hold the HTTP request. crates/fbuild-cli/src/daemon_client.rs has 1800s request timeouts for build/post paths, 100ms connect timeout, and returns stream ended without a result event only if the stream closes without a terminal result.

• Status surfaces are too coarse for this case. /api/locks/status in crates/fbuild-daemon/src/handlers/locks.rs reports project locks as { project_dir, is_held } only, with no holder request/client, waiters, wait duration, phase, package URL/name, or package install state. /ws/status in crates/fbuild-daemon/src/handlers/websockets.rs reports state, current_operation, operation_in_progress, and progress_percent: null; I did not find build/package paths broadcasting status updates through BroadcastHub.

Actionable recommendations:

1. Add a package-install coordination layer keyed by (kind, cache_key, version) before PackageBase::staged_install(): at minimum unique staging dirs plus race-aware final commit like managed_zccache; preferably also a cross-process lock/lease so only one daemon downloads/extracts a given package while others wait and re-check final/sentinel state.
2. Treat install_path.exists() alone as insufficient completion. Gate cache hits on the .install_complete sentinel or validation, and handle stale _staging/partial directories without deleting another active installer's work.
3. Wire download_file_with_progress() into toolchain/framework/library installs and propagate structured events: queued_for_project_lock, waiting_for_package_install, download_started, download_progress, extracting, install_complete. Streaming /api/build can emit these as NDJSON before compile logs; /ws/status and /api/locks/status should expose the same phase for blocked clients.
4. Enrich project lock status with holder request id/caller pid/cwd/current operation, waiters, and held duration. For package locks, expose package name/version/cache key and current owner/progress so a second client says "waiting for ESP32 toolchain download (42%)" instead of appearing hung.
5. Keep the existing 1800s client request timeout, but send heartbeat/progress events while queued or downloading. That prevents idle-looking clients and makes real timeout failures distinguishable from normal first-run installs.

[zackees]

Sub-issue A findings: daemon identity and broker resolution

Findings from the current tree:

• The intended local model is one persistent per-user daemon, split only by dev/prod mode. docs/running-process/inventory.md:12 says local fbuild is a single persistent per-user daemon, with FBUILD_DEV_MODE=1 using ~/.fbuild/dev/ + port 8865 and prod using ~/.fbuild/prod/ + port 8765. docs/running-process/inventory.md:20 sets the target local broker isolation to SHARED_BROKER, with CI as the EXPLICIT_INSTANCE "ci-trusted" exception.
• The running-process service definition matches that intent only at the service-name/isolation level: crates/fbuild-daemon/src/broker/service.rs:63 builds the default SHARED_BROKER service definition for service fbuild, and crates/fbuild-daemon/src/broker/service.rs:95 uses ServiceDefinitionBuilder::shared_broker(SERVICE_NAME, daemon_binary). The manifest hard-codes broker instance "shared" at crates/fbuild-daemon/src/broker/service.rs:148, so it does not distinguish cache roots or dev/prod roots in the broker instance id.
• The actual direct-path daemon identity is still the daemon dir + port file under ~/.fbuild/{dev|prod}/daemon, not the artifact cache root. crates/fbuild-paths/src/lib.rs:19 defines get_fbuild_root() as ~/.fbuild/{dev|prod}; crates/fbuild-paths/src/lib.rs:31, :36, and :41 derive daemon dir/PID/port files from that root. FBUILD_CACHE_DIR affects only get_cache_root() at crates/fbuild-paths/src/lib.rs:61, so two invocations with different FBUILD_CACHE_DIR values but the same dev/prod mode still share the same daemon PID/port identity.
• Port routing has additional escape hatches/fallbacks that are independent of broker negotiation. get_daemon_port() first honors FBUILD_DAEMON_PORT, then current-mode daemon.port, then the other mode's daemon.port, then the mode default (crates/fbuild-paths/src/lib.rs:171). That cross-mode fallback is useful for legacy discovery, but it means a prod client can route to a dev daemon if its own current-mode port file is absent.
• CLI acquisition currently negotiates a broker backend and then validates the legacy HTTP endpoint separately. crates/fbuild-cli/src/daemon_client.rs:897 calls try_acquire_broker_daemon() before direct fallback; on success, crates/fbuild-cli/src/daemon_client.rs:927 records the negotiated broker endpoint/version, but crates/fbuild-cli/src/daemon_client.rs:936 immediately creates a normal DaemonClient and waits on client.health() from get_daemon_url(). That means the broker-selected backend endpoint and the HTTP port-file endpoint can diverge.
• The daemon writes a single PID/port file after binding (crates/fbuild-daemon/src/main.rs:155, :161, :164) and installs the running-process service definition afterward (crates/fbuild-daemon/src/main.rs:168, :180). I did not find a startup call that publishes the CacheManifest; the helper exists in crates/fbuild-daemon/src/broker/service.rs:127, but current startup appears to install only the service definition. If the broker later relies on manifests for cache-root identity, this needs to be made explicit.
• Broker-framed traffic is still diagnostic-only in this slice. docs/running-process/README.md:11 says broker-spawned daemons answer broker health/daemon-info diagnostics, while CLI/PyO3 continue using HTTP for operations. crates/fbuild-daemon/src/broker/backend.rs:117 serves only Health and DaemonInfo; other ops return an error telling callers to use HTTP fallback. So inconsistent routing is currently most likely at acquisition/health/diagnostic time, but it will become a build/deploy correctness issue once operations move onto broker frames.

Conclusion: local fbuild should enforce exactly one active daemon per effective local daemon domain, but the domain needs to be defined deliberately. Today the implementation is effectively one daemon per user + dev/prod root + HTTP port, not one daemon per FBUILD_CACHE_DIR. If the desired policy is truly one daemon per user/cache root, the current path model does not implement it. If the desired policy is one daemon per user/mode with optional shared FBUILD_CACHE_DIR, then broker resolution should key and document that explicitly so cache-root changes cannot silently route to the wrong long-lived daemon.

Actionable recommendations:

1. Define the local daemon identity tuple in docs and tests. Suggested tuple: {user, mode/dev-prod root, effective artifact cache root, broker isolation/trust domain}. If FBUILD_CACHE_DIR is intentionally not part of identity, document that and reject/diagnose cache-root changes against an already-running daemon.
2. Make broker negotiation and HTTP routing converge. After broker adopt, use the negotiated backend's reported HTTP port/daemon identity, or verify that client.health_full() returns the same PID/version/cache_dir/daemon_dir as the negotiated backend before returning success.
3. Add a guard for parallel backend versions. For local SHARED_BROKER, either replace/upgrade the active daemon when a newer binary is requested, or require the broker resolver to select by daemon identity/cache-root compatibility, not only service name/version. CI EXPLICIT_INSTANCE "ci-trusted" should remain the exception.
4. Remove or narrow cross-mode HTTP fallback once broker acquisition is authoritative, or at least surface it in diagnostics (daemon running-process/daemon status) so users can see when prod is talking to a dev daemon via the other mode's port file.
5. Publish and validate the CacheManifest during daemon startup if broker cache-root identity is part of routing policy. Add a test that two same-domain daemons cannot advertise different CacheData/CacheLocks roots under the same SHARED_BROKER instance.

[zackees]

Sub-issue D findings: compatibility, migration, and recovery policy

Scope checked: broker docs/session/service, cache roots/manifest, disk-cache schema + GC, package staging, managed_zccache, daemon PID/status/lock handling, CLI/Python acquisition paths.

Findings:

• Old/new daemon coexistence is still a mixed model. crates/fbuild-paths/src/running_process.rs:42-88 still reports broker mode as broker-requested-direct-fallback, while CLI/Python both try broker adoption and then continue using the HTTP health/endpoint path (crates/fbuild-cli/src/daemon_client.rs:894-947, crates/fbuild-python/src/daemon.rs:84-124). This is a reasonable migration bridge, but the policy should be explicit: direct HTTP remains the compatibility fallback until broker request routing is complete, and typed broker VersionUnsupported / VersionBlocked refusals should stay fatal rather than spawning a second incompatible daemon (crates/fbuild-cli/src/daemon_client.rs:955-963, crates/fbuild-python/src/daemon.rs:160-168).
• Broker instance/cache metadata is present and mostly guarded. Local service definitions are SHARED_BROKER; CI has an explicit EXPLICIT_INSTANCE "ci-trusted" builder (crates/fbuild-daemon/src/broker/service.rs:63-117) and the diagnostic exposes the distinction (crates/fbuild-cli/src/cli/daemon_cmd.rs:147-188). One gap: manifest_builder() always advertises broker instance "shared" (crates/fbuild-daemon/src/broker/service.rs:144-155), so a future CI manifest path needs either a CI-specific manifest builder or a test proving cache manifests are intentionally shared even when service isolation is explicit.
• Cache schema migration is append-only and backward-compatible today. schema_migrations is authoritative, cache_meta.schema_version is mirrored only for legacy tooling (crates/fbuild-packages/src/disk_cache/index/mod.rs:6-17, :127-176), and m002_add_leases_refcount fixes old leases rows (crates/fbuild-packages/src/disk_cache/index/migrations.rs:18-36, :86-103). Existing regression coverage seeds a pre-migration DB and verifies lease acquisition (crates/fbuild-packages/tests/disk_cache_schema_migration.rs:21-102). Add a guard that fails if existing migration IDs are reordered/renamed and a compatibility test opening a DB with all known old objects plus unknown future cache_meta keys.
• Interrupted installs are recoverable in the disk-cache path. Disk cache paths define .partial staging and .install_complete sentinels (crates/fbuild-packages/src/disk_cache/paths.rs:147-183); reconcile() removes missing rows, incomplete installs, and .partial directories (crates/fbuild-packages/src/disk_cache/gc.rs:137-197, :236-249); daemon startup runs reconciliation in the background (crates/fbuild-daemon/src/main.rs:302-320). PackageBase still uses a legacy <version>_staging sibling and final-path existence check (crates/fbuild-packages/src/lib.rs:307-382), so add tests for crash points before rename, after rename before sentinel, and stale _staging cleanup; also consider aligning PackageBase staging names with disk_cache .partial so one recovery rule covers both.
• Stale locks are mostly in-memory, not durable files. Project locks live in DaemonContext.project_locks and are cleared when unlocked by /api/locks/clear or periodic cleanup (crates/fbuild-daemon/src/context.rs:151-156, crates/fbuild-daemon/src/handlers/locks.rs:46-74, crates/fbuild-daemon/src/main.rs:277-297). Disk-cache leases use PID+nonce refcounts and reap dead PIDs during GC (crates/fbuild-packages/src/disk_cache/lease.rs:1-15, crates/fbuild-packages/src/disk_cache/index/queries.rs:131-174, :264-290). Add an explicit test that a dead-PID lease is reaped and unpins entries, plus a daemon lock-status test documenting that stale_locks is currently always empty (crates/fbuild-daemon/src/handlers/locks.rs:38-43) unless that field is made meaningful.
• Daemon recovery has PID/port/status protections but needs coexistence tests. Startup removes dead PID files before binding (crates/fbuild-daemon/src/main.rs:136-151, :406-423), binds with platform-specific reuse/exclusive-address policy and retry (crates/fbuild-daemon/src/main.rs:465-490), writes PID/port files (crates/fbuild-daemon/src/main.rs:155-166), and writes daemon_status.json atomically with a Windows fallback (crates/fbuild-daemon/src/status_manager.rs:194-225). Add integration tests around stale PID file cleanup, live PID refusal/no cleanup, and status temp-file interruption leaving either old-valid or new-valid JSON.
• Managed zccache install is concurrency-tolerant but orphan cleanup is not covered. It extracts to a unique sibling temp dir then renames, treating an existing final binary as a won race (crates/fbuild-build/src/managed_zccache.rs:59-85, :109-157). Add tests for concurrent ensure() races and a startup/ensure cleanup policy for stale .zccache-*.tmp dirs from interrupted downloads.

Recommended policy text for this sub-issue:

1. Compatibility: direct HTTP remains the rollback path; broker version-block/refusal errors are fatal; non-refusal broker unavailability may fall back to direct until full broker routing lands.
2. Cache schema: migrations are append-only, IDs immutable, old DBs are upgraded in place, and cache_meta.schema_version is compatibility-only.
3. Recovery: incomplete installs are discardable, completed installs require validation/sentinel or revalidation, dead-PID leases are reaped by GC, and PID/port/status files are advisory state that must never block a healthy restart.
4. CI: EXPLICIT_INSTANCE "ci-trusted" should be tested as a deliberate exception, including whether its cache manifest instance should be ci-trusted or intentionally shared.

[zackees]

Broker/global cache investigation synthesis

The four sub-issue passes are posted:

• Sub-issue A: daemon identity and broker resolution ? fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
• Sub-issue B: global artifact repository ownership ? fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
• Sub-issue C: download/install concurrency and client blocking ? fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
• Sub-issue D: compatibility, migration, and recovery policy ? fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)

Main conclusion: the safe local policy should be one active fbuild daemon identity per shared artifact repository, not one private repository per broker-selected daemon/backend version. Today fbuild mostly behaves like one daemon per dev/prod root and legacy port file, but that is not the same thing as one daemon per FBUILD_CACHE_DIR, and the broker path can introduce a second routing identity unless we define and enforce the tuple explicitly.

Recommended follow-up shape:

1. Define the daemon identity tuple, likely (user, dev/prod mode, canonical FBUILD_CACHE_DIR or default cache root, trust domain), and document that backend version is not a cache-owner dimension.
2. Make local SHARED_BROKER resolution converge on that identity. If a different fbuild version is requested, either reuse/upgrade/replace the daemon for that cache root, or refuse with a clear compatibility message; do not silently create a private versioned artifact store.
3. Treat the broker CacheRuntime root as daemon provenance only. CacheData/package/toolchain/framework archives remain owned by fbuild's global repository.
4. Add package/toolchain/framework/sidecar install gates below the project lock level. Same-project serialization is not enough because different projects and different daemon instances can hit the same global package roots concurrently.
5. Surface lock wait/install state to clients: project lock waits, package install waits, zccache sidecar waits, download/verify/extract phases, and whether the caller is the active installer or a waiter.
6. Add compatibility/recovery tests: same FBUILD_CACHE_DIR across broker-negotiated versions publishes the same CacheData; CI EXPLICIT_INSTANCE remains isolated; stale _staging/partial sidecar installs recover; migration IDs remain immutable; dead-PID leases do not block forever.

Open policy question to settle before implementation: should local fbuild strictly refuse parallel daemon versions for the same cache root, or allow them only when a declared cache schema compatibility check passes?

[zackees]

Created parent/meta burndown issue for this broker/global-artifact work:

• meta: broker-safe global artifact repository feature burndown #603

That tracker consolidates the four sub-issue findings from this thread into implementation workstreams and acceptance criteria. This issue can remain focused on the original managed zccache install gate and detailed investigation history.