Daemon startup: wait for a live daemon instead of a fixed 15 s deadline on slow or overloaded hosts #3782

Purpose

On a slow or overloaded host, the CLI kills a healthy daemon that is still starting. The result is daemon_startup_failed, and the run fails before any work starts. This happens on any CI that can clog: shared macOS runners, small Linux containers, Windows runners, and nested virtualization. It also happens on a developer machine under load. A slow host must make startup slower, not fail it.

#3342 records this failure class (more than 12 instances on unrelated diffs since 2026-10-08; re-runs at the same head pass). This issue is the fix contract for that class.

Current behavior (origin/main)

startLocalDaemon in src/daemon-client/daemon-client-lifecycle.ts:

• It sets one wall-clock deadline: DAEMON_STARTUP_TIMEOUT_MS = 15_000 (line 83, used at line 480).
• waitForDaemonStartup (line 818) polls until the daemon publishes a reachable daemon.json, or until the process exits, or until the deadline.
• At the deadline, the client retires the daemon, even when the process is alive. A timeout result never retries (retry is true only for early_exit), so DAEMON_STARTUP_ATTEMPTS = 2 does not help. Every report shows startupAttempts: 1.
• No setting can change the 15 s budget.

Evidence from run 39019020372 (PR #3760, 2026-10-17):

• The step pnpm clean:daemon took 70 s on that runner, which shows that the host was overloaded.
• The CLI spawned daemon pid 6849 at 09:30:03. The client killed it 15 s later (cleanupResults: retired / forced / exited).
• daemon.log was empty (0 bytes), so the daemon did not reach its first log line. The process was alive but did not get enough CPU.
• The same signature occurred on refactor/host-environment-root (run 39008970059) and fix/provider-not-registered (run 38993655494), which do not contain that PR's commits.

Required behavior

1. Wait for a live daemon. While the spawned daemon process is alive and has not exited, the client continues to wait for registration. A slow start is not a failure.

2. Fail fast on a real failure. If the daemon process exits before it registers, the client reports the failure immediately, as it does now (early_exit, unproven).

3. Keep a hard ceiling. A daemon that stays alive but never registers must not hang the client. The client stops at a ceiling. The default ceiling must be large enough for an overloaded CI host (proposal: 120 s). Measure the slowest successful startup in CI logs before you choose the number.

4. Show progress. The daemon writes startup phase markers (for example spawned, modules-loaded, lock-acquired, listening, registered) with timestamps, to the place the client already reads (daemon.log tail or the startup metadata). The client can then tell "slow but progressing" from "stuck". If a marker shows progress, the client keeps waiting up to the ceiling. If no marker appears for a long time, the client can stop earlier than the ceiling. Put the stall window next to the ceiling, as one constant.

5. Make the budget configurable. An environment variable sets the ceiling, for example AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS. It applies to the CLI, MCP, and the Node client, because all three use startLocalDaemon (via ensureDaemon(), feat(client): daemonLifecycle attach mode and ensureDaemon() #3765 and feat(cli): ensure the daemon explicitly before attaching in the CLI and MCP #3766). Document it with the other daemon environment variables. Do not add CI auto-detection. Rule 1 makes the default safe on every host.

6. Use typed failure reasons. details.kind stays daemon_startup_failed. A new details.reason tells the caller what happened:

   • startup-exited: the process exited before it registered.
   • startup-stalled: the process is alive, but no progress marker appeared within the stall window.
   • startup-ceiling: the process made progress, but did not register before the ceiling.

   Each reason has its own hint. For startup-stalled and startup-ceiling, the hint says that the host can be overloaded, and names the environment variable. Callers branch on details.reason, never on the message.

7. Record timings. The failure details include the time of each phase marker that the client saw, and startupTimeoutMs shows the effective ceiling.

Example failure payload:

{
  "code": "COMMAND_FAILED",
  "message": "Failed to start daemon",
  "details": {
    "kind": "daemon_startup_failed",
    "reason": "startup-stalled",
    "startupTimeoutMs": 120000,
    "phases": { "spawned": 0, "modules-loaded": 41250 },
    "hint": "The daemon is alive but made no startup progress for 60 s. The host can be overloaded. Retry, or raise AGENT_DEVICE_DAEMON_STARTUP_TIMEOUT_MS."
  }
}

Completion conditions

• A test starts a daemon that is alive but slow (for example a test-only startup delay of 20 s, longer than the old 15 s budget). Startup succeeds, and the client does not kill the daemon.
• A test starts a daemon that exits during startup. The client reports startup-exited within 1 s after the exit.
• A test starts a daemon that stays alive and writes no phase marker. The client reports startup-stalled at the stall window, not at the ceiling.
• A test sets the environment variable to a small value and sees startup-ceiling with the new startupTimeoutMs.
• The existing lifecycle and startup-race tests pass without change in meaning.
• The daemon-client help or user docs name the environment variable and the three reasons.
• After the change merges, iOS smoke lane: preflight 'prepare ios-runner' fails job with daemon_startup_failed on PR lanes (untracked third class) #3342 gets a Current state: comment. Then the iOS Smoke preflight step is observed for one week. The issue closes if no daemon_startup_failed with reason startup-stalled or startup-ceiling appears below the new ceiling.

Out of scope

• Re-running CI lanes to hide the failure.
• The slow pnpm clean:daemon step in .github/workflows/ios.yml (70 s on an overloaded runner). That is a separate CI cost. Report it in iOS smoke lane: preflight 'prepare ios-runner' fails job with daemon_startup_failed on PR lanes (untracked third class) #3342 if it repeats.

Dependencies

• iOS smoke lane: preflight 'prepare ios-runner' fails job with daemon_startup_failed on PR lanes (untracked third class) #3342: failure-class log and evidence.
• feat(client): daemonLifecycle attach mode and ensureDaemon() #3765 and feat(cli): ensure the daemon explicitly before attaching in the CLI and MCP #3766 (merged): ensureDaemon() and attach mode. They call startLocalDaemon, so the change applies to all clients.