diff --git a/CHANGELOG.md b/CHANGELOG.md index 89500c54b4..725a0793e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## Unreleased +- Add `ensureDaemon({ stateDir })` and the client config `daemonLifecycle: 'attach' | 'managed'`. An `attach` client connects to the daemon registered in the state directory and never starts, replaces or kills one, including after a request times out. It fails with `details.reason: 'daemon-not-running'` (the hint points at `ensureDaemon()`) or `'daemon-incompatible'` (both identities in `details`, the hint names `agent-device daemon stop`). `ensureDaemon()` starts a missing daemon and replaces an incompatible one that answers only through an authenticated quiesce request the daemon answers itself (a resumable upload still streaming and a human-control hold that names no lease are not counted, so a replacement can end them): it refuses with `daemon-busy` while any request is in flight or any session, repair session or live lease exists, and a daemon from a release before this one is never replaced. A registered daemon that answers nothing is retired, as is one that times out on the quiesce request and then fails repeated liveness probes; neither can be asked. A remote `daemonBaseUrl` connection never starts or retires a local daemon in either mode. `managed` stays the default and behaves as before until 0.22. - Add `uninstall ` and `client.apps.uninstall({ app })` to remove an installed app on adb-backed Android targets, iOS simulators, and CoreDevice iOS devices. Both report `wasInstalled`. A package name or bundle id that is not installed succeeds with `wasInstalled: false`. Other targets, Appium and Limrun sessions included, refuse it before anything runs. - Add `settings mobile-data on|off` for adb-backed Android targets. It switches mobile data through `svc data` and reports the `mobileData` state stored in the global `mobile_data` row; multi-SIM read-back is not covered yet. Every other target refuses it before anything runs, and `--app` is refused because the switch is device-wide. - On iOS, a `fill` that runs under a degraded XCTest snapshot channel now aims at a point, including ref, selector and replay fills. When the runner identifies no text input under that point, it refuses with `NO_TEXT_INPUT_AT_POINT` (`error.details.reason: no_text_input_at_point`, `dispatched: no`) before dispatching its focus tap, and no field changes; when the input lookup itself cannot answer, it refuses with `TEXT_INPUT_PROBE_UNAVAILABLE` (`error.details.reason: text_input_probe_unavailable`). Previously such a fill cleared and typed through the app-level first responder, so it could rewrite a field the caller never named, or report a clear failure after emptying it. A ref or selector fill whose rect centre misses the real field (an outer container, a non-input node) now needs a point inside the input's own rect. diff --git a/packages/contracts/src/client-connection.ts b/packages/contracts/src/client-connection.ts index 07cbf69ad4..bcb2239c03 100644 --- a/packages/contracts/src/client-connection.ts +++ b/packages/contracts/src/client-connection.ts @@ -2,6 +2,8 @@ import type { DaemonLockPolicy, + DaemonServerMode, + DaemonTransportPreference, DaemonRequest, DaemonResponse, LeaseBackend, @@ -15,8 +17,58 @@ import type { RemoteConnectionProfileFields, } from './remote-config-fields.ts'; +/** + * Who owns the daemon on a client's state directory. + * + * - `managed`: the client starts a missing daemon and replaces an incompatible one itself, and may + * retire a hung one after a timeout. This is the default until 0.22. + * - `attach`: the client only connects to the daemon registered in the state directory. It never + * starts, replaces or kills one. A missing daemon fails with `details.reason: 'daemon-not-running'` + * and an incompatible one with `'daemon-incompatible'`; `ensureDaemon()` is the explicit way to + * start or replace one. + * + * Independent of `daemonTransport`, `daemonServerMode` and `daemonBaseUrl`: a remote connection + * never starts or retires a local daemon in either mode. + */ +export type DaemonLifecycle = 'attach' | 'managed'; + +/** What a client can learn about a daemon without talking to it: how it identifies itself on disk. */ +export type DaemonIdentity = { + pid: number; + version?: string; + /** `installed` for a published package, `checkout` for a source tree. */ + codeOrigin?: 'installed' | 'checkout'; + codeSignature?: string; + processStartTime?: string; +}; + +export type EnsureDaemonOptions = { + /** The state directory the daemon owns; defaults to `AGENT_DEVICE_STATE_DIR`, then the user default. */ + stateDir?: string; + daemonTransport?: DaemonTransportPreference; + daemonServerMode?: DaemonServerMode; + /** + * Called when this call started a daemon, before `ensureDaemon` returns. `missing` means no + * usable daemon was registered; `incompatible` means a daemon this client cannot use was + * replaced after it agreed to leave. `previous` is the daemon that was replaced, when there was + * one: a `missing` start with a `previous` replaced a registered daemon that answered nothing, + * either unreachable or hung and force-retired. + */ + onStart?: (reason: 'missing' | 'incompatible', previous?: DaemonIdentity) => void; +}; + +/** A reachable local daemon. It carries no credentials. */ +export type DaemonConnection = { + stateDir: string; + daemon: DaemonIdentity; + /** True when this call started the daemon. */ + started: boolean; +}; + export type AgentDeviceDaemonTransportContext = { authToken?: string; + /** How this request treats the local daemon; `managed` when omitted. */ + daemonLifecycle?: DaemonLifecycle; /** * Cancels this one in-flight request. A built-in transport that sees an abort destroys the * request's connection, which makes the daemon mark the request canceled; the promise rejects @@ -33,6 +85,8 @@ export type AgentDeviceDaemonTransport = ( export type AgentDeviceClientConfig = RemoteConnectionProfileFields & CloudProviderProfileFields & { + /** See `DaemonLifecycle`. */ + daemonLifecycle?: DaemonLifecycle; session?: string; lockPolicy?: DaemonLockPolicy; lockPlatform?: PlatformSelector; diff --git a/packages/contracts/src/facades/client.ts b/packages/contracts/src/facades/client.ts index 78499ed2da..5af90eef2f 100644 --- a/packages/contracts/src/facades/client.ts +++ b/packages/contracts/src/facades/client.ts @@ -34,7 +34,11 @@ export type { AgentDeviceIdentifiers, AgentDeviceRequestOverrides, AgentDeviceSelectionOptions, + DaemonConnection, + DaemonIdentity, + DaemonLifecycle, DeviceCommandBaseOptions, + EnsureDaemonOptions, } from '../client-connection.ts'; export type { AgentDeviceCapabilitiesResult, diff --git a/packages/daemon-contracts/src/daemon-resolution-quiesce.test.ts b/packages/daemon-contracts/src/daemon-resolution-quiesce.test.ts new file mode 100644 index 0000000000..67897417b5 --- /dev/null +++ b/packages/daemon-contracts/src/daemon-resolution-quiesce.test.ts @@ -0,0 +1,39 @@ +import assert from 'node:assert/strict'; +import { test } from 'vitest'; +import { readDaemonQuiesceCensus, readDaemonQuiesceFence } from './daemon-resolution.ts'; + +test('a census is read only when every count is a non-negative integer', () => { + const census = { + startingUp: false, + inFlightRequests: 1, + sessions: 0, + repairSessions: 2, + leases: 3, + }; + assert.deepEqual(readDaemonQuiesceCensus({ reason: 'daemon-busy', ...census }), census); + + for (const malformed of [-1, 1.5, '1', Number.NaN, Number.POSITIVE_INFINITY, null]) { + assert.equal(readDaemonQuiesceCensus({ ...census, leases: malformed }), undefined); + } + assert.equal(readDaemonQuiesceCensus({ sessions: 1 }), undefined); + // Only a daemon that is starting up says so; an older one omits the field. + assert.equal(readDaemonQuiesceCensus({ ...census, startingUp: true })?.startingUp, true); + const { startingUp: _omitted, ...withoutStartingUp } = census; + assert.equal(readDaemonQuiesceCensus(withoutStartingUp)?.startingUp, false); + assert.equal(readDaemonQuiesceCensus({ ...census, startingUp: 'yes' }), undefined); + assert.equal(readDaemonQuiesceCensus(undefined), undefined); +}); + +test('a fence names an integer pid and a start time or null, nothing else', () => { + assert.deepEqual(readDaemonQuiesceFence({ pid: 7, processStartTime: 'Fri' }), { + pid: 7, + processStartTime: 'Fri', + }); + assert.deepEqual(readDaemonQuiesceFence({ pid: 7, processStartTime: null }), { + pid: 7, + processStartTime: null, + }); + assert.equal(readDaemonQuiesceFence({ pid: 7 }), undefined); + assert.equal(readDaemonQuiesceFence({ pid: 7.5, processStartTime: null }), undefined); + assert.equal(readDaemonQuiesceFence('7'), undefined); +}); diff --git a/packages/daemon-contracts/src/daemon-resolution.ts b/packages/daemon-contracts/src/daemon-resolution.ts index 8a5b1d1ff3..f0ae06c273 100644 --- a/packages/daemon-contracts/src/daemon-resolution.ts +++ b/packages/daemon-contracts/src/daemon-resolution.ts @@ -5,6 +5,7 @@ import { expandUserHomePath, resolveUserPath } from '@agent-device/host-kit/file import { findProjectRoot, isSourceCheckoutProjectRoot } from '@agent-device/host-kit/version'; import { type EnvMap } from '@agent-device/kernel/source-value'; +import type { DaemonWireRequest } from '@agent-device/contracts/command'; import type { DaemonServerMode, DaemonTransportPreference } from '@agent-device/kernel/contracts'; export type { DaemonServerMode, DaemonTransportPreference }; @@ -134,6 +135,22 @@ export type DaemonQuiesceCensus = Readonly<{ leases: number; }>; +export function buildDaemonQuiesceRequest(params: { + token: string; + fence: DaemonQuiesceFence; + requestId: string; +}): DaemonWireRequest { + const { token, fence, requestId } = params; + return { + token, + session: 'default', + command: DAEMON_QUIESCE_COMMAND, + positionals: [], + input: { pid: fence.pid, processStartTime: fence.processStartTime }, + meta: { requestId }, + }; +} + export function readDaemonQuiesceFence(input: unknown): DaemonQuiesceFence | undefined { if (typeof input !== 'object' || input === null) return undefined; const { pid, processStartTime } = input as Record; @@ -146,3 +163,23 @@ export function readDaemonCapabilities(value: unknown): readonly DaemonCapabilit if (!Array.isArray(value)) return []; return DAEMON_CAPABILITIES.filter((capability) => value.includes(capability)); } + +export function readDaemonQuiesceCensus(details: unknown): DaemonQuiesceCensus | undefined { + if (typeof details !== 'object' || details === null) return undefined; + const { startingUp, inFlightRequests, sessions, repairSessions, leases } = details as Record< + string, + unknown + >; + if (startingUp !== undefined && typeof startingUp !== 'boolean') return undefined; + const counts = [inFlightRequests, sessions, repairSessions, leases]; + if (!counts.every((count) => Number.isSafeInteger(count) && (count as number) >= 0)) { + return undefined; + } + return { + startingUp: startingUp === true, + inFlightRequests: inFlightRequests as number, + sessions: sessions as number, + repairSessions: repairSessions as number, + leases: leases as number, + }; +} diff --git a/packages/daemon/src/server/http/daemon-runtime-quiesce.test.ts b/packages/daemon/src/server/http/daemon-runtime-quiesce.test.ts index f14afa29b1..d097a68b2b 100644 --- a/packages/daemon/src/server/http/daemon-runtime-quiesce.test.ts +++ b/packages/daemon/src/server/http/daemon-runtime-quiesce.test.ts @@ -3,28 +3,12 @@ import net from 'node:net'; import { afterEach, expect, test, vi } from 'vitest'; import { mkdtempForTestSync } from '@agent-device/test-kit'; import { - DAEMON_QUIESCE_COMMAND, + buildDaemonQuiesceRequest, resolveDaemonPaths, - type DaemonQuiesceFence, } from '@agent-device/daemon-contracts/daemon-resolution'; import { readCurrentOwnerIdentity } from '@agent-device/host-kit/process'; import type { DaemonResponse } from '@agent-device/kernel/contracts'; -function buildDaemonQuiesceRequest(params: { - token: string; - fence: DaemonQuiesceFence; - requestId: string; -}) { - return { - token: params.token, - session: 'default', - command: DAEMON_QUIESCE_COMMAND, - positionals: [], - input: { pid: params.fence.pid, processStartTime: params.fence.processStartTime }, - meta: { requestId: params.requestId }, - }; -} - // A quiesce is the daemon's own answer to "may I leave?", so these tests boot the real runtime and // ask over its real socket: the census, the closing of admission and the drain are one flow. const probe = vi.hoisted(() => ({ diff --git a/src/__tests__/agent-device-client-daemon-lifecycle.test.ts b/src/__tests__/agent-device-client-daemon-lifecycle.test.ts new file mode 100644 index 0000000000..c74faff254 --- /dev/null +++ b/src/__tests__/agent-device-client-daemon-lifecycle.test.ts @@ -0,0 +1,47 @@ +import { test } from 'vitest'; +import assert from 'node:assert/strict'; +import type { + AgentDeviceDaemonTransport, + AgentDeviceDaemonTransportContext, + DaemonLifecycle, +} from '@agent-device/contracts/client'; +import { createAgentDeviceClient } from '../agent-device-client.ts'; + +// `daemonLifecycle` is a client choice that only the transport can act on, so the client's job is to +// hand it over exactly and to refuse a value it does not know. + +function recordingTransport() { + const contexts: Array = []; + const transport: AgentDeviceDaemonTransport = async (_request, context) => { + contexts.push(context); + return { ok: true, data: { devices: [] } }; + }; + return { contexts, transport }; +} + +test('the configured daemon lifecycle reaches the transport', async () => { + const { contexts, transport } = recordingTransport(); + const attached = createAgentDeviceClient({ daemonLifecycle: 'attach' }, { transport }); + const plain = createAgentDeviceClient({}, { transport }); + + const managed = createAgentDeviceClient({ daemonLifecycle: 'managed' }, { transport }); + + await attached.devices.list(); + await managed.devices.list(); + await plain.devices.list(); + + assert.deepEqual( + contexts.map((context) => context?.daemonLifecycle), + ['attach', 'managed', undefined], + ); +}); + +test('an unknown daemon lifecycle is refused instead of becoming managed', () => { + const { transport } = recordingTransport(); + const mistyped = 'Attach' as DaemonLifecycle; + + assert.throws( + () => createAgentDeviceClient({ daemonLifecycle: mistyped }, { transport }), + (error: { code?: string }) => error.code === 'INVALID_ARGS', + ); +}); diff --git a/src/__tests__/daemon-registration-owner.test.ts b/src/__tests__/daemon-registration-owner.test.ts index da8e9e7227..da720dd78c 100644 --- a/src/__tests__/daemon-registration-owner.test.ts +++ b/src/__tests__/daemon-registration-owner.test.ts @@ -27,7 +27,10 @@ import { spawnRegisteredDaemonFixture, finishRegisteredDaemonFixture, } from './test-utils/registered-daemon-fixture.ts'; -import { ensureDaemon, resolveClientSettings } from '../daemon-client/daemon-client-lifecycle.ts'; +import { + ensureManagedDaemon, + resolveClientSettings, +} from '../daemon-client/daemon-client-lifecycle.ts'; import { inspectProcessLock } from '@agent-device/host-kit/file'; import { AppError } from '@agent-device/kernel/errors'; import { sleep } from '@agent-device/host-kit/retry'; @@ -791,7 +794,7 @@ test('cutover refuses an already-running legacy daemon before signaling or chang assert.ok(disposition, 'a new daemon must refuse an occupied legacy file'); assert.equal(disposition.exitCode, DAEMON_STARTUP_EXIT_CODES.unproven); await assert.rejects( - ensureDaemon( + ensureManagedDaemon( resolveClientSettings({ session: 'default', command: 'devices', diff --git a/src/agent-device-client.ts b/src/agent-device-client.ts index 421b6ddc9f..4bc838f0a0 100644 --- a/src/agent-device-client.ts +++ b/src/agent-device-client.ts @@ -16,6 +16,7 @@ import type { CaptureScreenshotResult, CaptureSnapshotOptions, CaptureSnapshotResult, + DaemonLifecycle, DragOptions, FlingOptions, MaterializationReleaseOptions, @@ -84,6 +85,8 @@ export function createAgentDeviceClient( deps: { transport?: AgentDeviceDaemonTransport } = {}, ): AgentDeviceClient { const transport = deps.transport ?? sendToDaemon; + // A typo in a JSON or JS config must not fall through to the default and manage a daemon. + readDaemonLifecycle(config.daemonLifecycle); // A non-default responseLevel (digest/full) makes the daemon return a leveled // shape; the per-command client normalizers assume the default shape, so the @@ -120,7 +123,11 @@ export function createAgentDeviceClient( // mark the request canceled. const response = await cancellation.guard( async () => - await transport(request, { authToken: merged.daemonAuthToken, signal: merged.signal }), + await transport(request, { + authToken: merged.daemonAuthToken, + signal: merged.signal, + daemonLifecycle: merged.daemonLifecycle, + }), ); if (!response.ok) { throwDaemonError(response.error); @@ -585,6 +592,13 @@ function optionalSnapshotResponseFields( }; } +function readDaemonLifecycle(value: unknown): DaemonLifecycle | undefined { + if (value === undefined || value === 'attach' || value === 'managed') return value; + throw new AppError('INVALID_ARGS', "daemonLifecycle must be 'attach' or 'managed'.", { + daemonLifecycle: value, + }); +} + function mergeClientOptions( config: AgentDeviceClientConfig, options: InternalRequestOptions, diff --git a/src/cli/commands/__tests__/proxy.test.ts b/src/cli/commands/__tests__/proxy.test.ts index ef33cd29e5..8c206fc623 100644 --- a/src/cli/commands/__tests__/proxy.test.ts +++ b/src/cli/commands/__tests__/proxy.test.ts @@ -1,6 +1,6 @@ // #3420: the proxy command's startup path — daemon-backed upstream resolution, client-token // sourcing, and real loopback binding — is pinned through the command handler itself. The -// process-spawn seam (`ensureDaemon`) and the output sink are mocked at the module boundary the +// process-spawn seam (`ensureManagedDaemon`) and the output sink are mocked at the module boundary the // command consumes; the proxy server and its listen are the real ones. import assert from 'node:assert/strict'; import http from 'node:http'; @@ -14,7 +14,7 @@ import { import { resolveDaemonPaths } from '@agent-device/daemon-contracts/daemon-resolution'; const mocks = vi.hoisted(() => ({ - ensureDaemon: vi.fn(), + ensureManagedDaemon: vi.fn(), writeCommandOutput: vi.fn(async (..._args: unknown[]) => undefined), createdServers: [] as http.Server[], listenArgs: [] as { port: number; host: string }[], @@ -22,7 +22,7 @@ const mocks = vi.hoisted(() => ({ vi.mock('../../../daemon-client/daemon-client-lifecycle.ts', async (importOriginal) => ({ ...(await importOriginal()), - ensureDaemon: mocks.ensureDaemon, + ensureManagedDaemon: mocks.ensureManagedDaemon, })); vi.mock('@agent-device/proxy', async (importOriginal) => { const actual = await importOriginal(); @@ -96,11 +96,11 @@ test('proxy rejects positional arguments', async () => { }), (error: { code?: string }) => error.code === 'INVALID_ARGS', ); - assert.equal(mocks.ensureDaemon.mock.calls.length, 0); + assert.equal(mocks.ensureManagedDaemon.mock.calls.length, 0); }); test('proxy refuses to start when the local daemon has no HTTP endpoint', async () => { - mocks.ensureDaemon.mockResolvedValue({ + mocks.ensureManagedDaemon.mockResolvedValue({ info: { token: 'daemon-secret', pid: 4242 }, startedByClient: false, }); @@ -120,7 +120,7 @@ test('proxy refuses to start when the local daemon has no HTTP endpoint', async test('proxy binds a loopback listener and reports the supplied client token', async (t) => { if (await skipWhenLoopbackUnavailable(t)) return; const stateDir = mkdtempForTestSync('agent-device-proxy-start-'); - mocks.ensureDaemon.mockResolvedValue({ + mocks.ensureManagedDaemon.mockResolvedValue({ info: { httpPort: 54321, token: 'daemon-secret', pid: 4242 }, startedByClient: false, }); @@ -148,14 +148,14 @@ test('proxy binds a loopback listener and reports the supplied client token', as assert.match(startupCard, /Proxy listening at/); assert.ok(startupCard.includes(proxyBaseUrl)); // The daemon the proxy fronts is the local one of the resolved state dir, never a remote URL. - const settings = mocks.ensureDaemon.mock.calls[0]![0]; + const settings = mocks.ensureManagedDaemon.mock.calls[0]![0]; assert.equal(settings.paths.baseDir, resolveDaemonPaths(stateDir).baseDir); assert.equal(settings.remoteBaseUrl, undefined); }); test('proxy honors proxy host and port flags and mints a token when none is supplied', async (t) => { if (await skipWhenLoopbackUnavailable(t)) return; - mocks.ensureDaemon.mockResolvedValue({ + mocks.ensureManagedDaemon.mockResolvedValue({ info: { httpPort: 54322, token: 'daemon-secret', pid: 4242 }, startedByClient: false, }); diff --git a/src/cli/commands/proxy.ts b/src/cli/commands/proxy.ts index 0d3036ab71..f277c1d886 100644 --- a/src/cli/commands/proxy.ts +++ b/src/cli/commands/proxy.ts @@ -2,7 +2,7 @@ import { randomBytes } from 'node:crypto'; import { createDaemonProxyServer } from '@agent-device/proxy'; import { buildDaemonHttpBaseUrl } from '@agent-device/contracts/daemon-http'; import { - ensureDaemon, + ensureManagedDaemon, resolveClientSettings, } from '../../daemon-client/daemon-client-lifecycle.ts'; import { AppError } from '@agent-device/kernel/errors'; @@ -41,7 +41,7 @@ async function startProxy(flags: CliFlags): Promise { daemonServerMode: 'http', }, }); - const daemon = await ensureDaemon(settings); + const daemon = await ensureManagedDaemon(settings); const upstreamBaseUrl = resolveLocalDaemonBaseUrl(daemon.info.httpPort); const token = resolveProxyClientToken(flags); const server = createDaemonProxyServer({ diff --git a/src/daemon-client/__tests__/daemon-client-explicit-lifecycle.test.ts b/src/daemon-client/__tests__/daemon-client-explicit-lifecycle.test.ts new file mode 100644 index 0000000000..72b31163b0 --- /dev/null +++ b/src/daemon-client/__tests__/daemon-client-explicit-lifecycle.test.ts @@ -0,0 +1,670 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import path from 'node:path'; +import { afterEach, test, vi } from 'vitest'; +import { closeLoopbackServer, supportsLoopbackBind } from '@agent-device/test-kit'; +import { AppError } from '@agent-device/kernel/errors'; +import { DAEMON_QUIESCE_COMMAND } from '@agent-device/daemon-contracts/daemon-resolution'; +import { DAEMON_STARTUP_EXIT_CODES } from '@agent-device/daemon-contracts/daemon-registration-owner'; +import { isProcessAlive } from '@agent-device/host-kit/process'; +import { readVersion } from '@agent-device/host-kit/version'; +import { + finishRegisteredDaemonFixtures, + spawnRegisteredDaemonFixture, + waitForRegisteredDaemonFixture, +} from '../../__tests__/test-utils/registered-daemon-fixture.ts'; +import { currentDaemonCodeSignature } from '../../__tests__/test-utils/daemon-http-fixture.ts'; +import { + busy, + draining, + launchesFreshDaemonAt as launchSeamAnswersAt, + okResult, + registerDaemon, + startScriptedDaemon, + startScriptedSocketDaemon, + stateDirAndPaths, +} from './explicit-lifecycle-fixture.ts'; + +vi.mock('@agent-device/host-kit/command', async (importOriginal) => ({ + ...(await importOriginal()), + runCmdDetachedMonitored: vi.fn(), +})); +vi.mock('@agent-device/host-kit/retry', async (importOriginal) => ({ + ...(await importOriginal()), + sleep: vi.fn(async () => {}), +})); +import { runCmdDetachedMonitored } from '@agent-device/host-kit/command'; +import { sleep } from '@agent-device/host-kit/retry'; +import * as transport from '../daemon-client-transport.ts'; +import { sendToDaemon } from '../daemon-client.ts'; +import { ensureDaemon } from '../daemon-client-ensure.ts'; + +// The explicit lifecycle: what an attach client may never do, and what `ensureDaemon` may do only +// through the daemon's own answer. Daemons here are loopback stand-ins registered by real children, +// so a kill or a spawn is observable as a dead pid or a call to the launch seam. + +const spawn = vi.mocked(runCmdDetachedMonitored); +const actualCommand = await vi.importActual( + '@agent-device/host-kit/command', +); +const actualRetry = await vi.importActual( + '@agent-device/host-kit/retry', +); + +afterEach(async () => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + spawn.mockReset(); + vi.mocked(sleep).mockReset(); + vi.mocked(sleep).mockImplementation(async () => {}); + await finishRegisteredDaemonFixtures(); +}); + +/** Real waits, so a start can poll the child it launched. */ +function launchesFreshDaemonAt( + paths: ReturnType['paths'], + httpPort: number, +): void { + vi.mocked(sleep).mockImplementation(actualRetry.sleep); + launchSeamAnswersAt(spawn, paths, httpPort); +} + +function request(stateDir: string, extra: Record = {}) { + return { + session: 'default', + command: 'devices', + positionals: [], + flags: { stateDir, daemonTransport: 'http' as const, ...extra }, + meta: { requestId: 'req-explicit-lifecycle' }, + }; +} + +function identityIn(error: AppError, key: 'daemon' | 'client'): { pid?: number; version?: string } { + return (error.details?.[key] ?? {}) as { pid?: number; version?: string }; +} + +async function failureOf(run: Promise): Promise { + try { + await run; + } catch (error) { + assert.ok(error instanceof AppError, 'expected a typed AppError'); + return error; + } + return assert.fail('expected the call to reject'); +} + +async function withLoopback(t: { skip: (note: string) => void }): Promise { + if (await supportsLoopbackBind()) return true; + t.skip('loopback listeners are not permitted in this environment'); + return false; +} + +test('attach reports a missing daemon without starting one', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + + const error = await failureOf(sendToDaemon(request(stateDir), { daemonLifecycle: 'attach' })); + + assert.equal(error.details?.reason, 'daemon-not-running'); + assert.equal(spawn.mock.calls.length, 0); + assert.equal(fs.existsSync(paths.infoPath), false); +}); + +test('attach reports an incompatible daemon with both identities and leaves it running', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const stale = await startScriptedDaemon(okResult({ via: 'stale' })); + const { child } = await registerDaemon(paths, { + httpPort: stale.port, + version: '0.0.0-stale', + capabilities: ['quiesce'], + }); + + try { + const error = await failureOf(sendToDaemon(request(stateDir), { daemonLifecycle: 'attach' })); + + assert.equal(error.details?.reason, 'daemon-incompatible'); + assert.equal(identityIn(error, 'daemon').version, '0.0.0-stale'); + assert.equal(identityIn(error, 'client').version, readVersion()); + assert.match(String(error.details?.hint), /agent-device daemon stop/); + assert.equal(isProcessAlive(child.pid), true, 'attach never kills the daemon'); + assert.equal(fs.existsSync(paths.infoPath), true); + assert.equal(spawn.mock.calls.length, 0); + assert.deepEqual(stale.rpcCommands, [], 'attach never asks the daemon to leave either'); + } finally { + await closeLoopbackServer(stale.server); + } +}); + +test('attach reports a registered daemon that answers nothing as not running and leaves it', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const closed = await startScriptedDaemon(okResult({})); + await closeLoopbackServer(closed.server); + const { child } = await registerDaemon(paths, { httpPort: closed.port }); + + const error = await failureOf(sendToDaemon(request(stateDir), { daemonLifecycle: 'attach' })); + + assert.equal(error.details?.reason, 'daemon-not-running'); + assert.equal(identityIn(error, 'daemon').pid, child.pid); + assert.equal(isProcessAlive(child.pid), true); + assert.equal(spawn.mock.calls.length, 0); +}); + +test('attach uses a compatible daemon and never lets a timeout retire it', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const live = await startScriptedDaemon(okResult({ via: 'live' })); + await registerDaemon(paths, { httpPort: live.port }); + const send = vi.spyOn(transport, 'sendRequest'); + + try { + const response = await sendToDaemon(request(stateDir), { daemonLifecycle: 'attach' }); + assert.deepEqual(response, { ok: true, data: { via: 'live' } }); + assert.equal(send.mock.calls.at(-1)?.[5]?.timeoutRecovery, 'none'); + + await sendToDaemon(request(stateDir)); + assert.equal(send.mock.calls.at(-1)?.[5]?.timeoutRecovery, 'retire-unresponsive'); + assert.equal(spawn.mock.calls.length, 0); + } finally { + await closeLoopbackServer(live.server); + } +}); + +test('a remote connection never starts a local daemon in either lifecycle', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir } = stateDirAndPaths(); + const unreachable = await startScriptedDaemon(okResult({})); + await closeLoopbackServer(unreachable.server); + const remote = { daemonBaseUrl: `http://127.0.0.1:${unreachable.port}` }; + + for (const daemonLifecycle of ['attach', 'managed'] as const) { + const error = await failureOf(sendToDaemon(request(stateDir, remote), { daemonLifecycle })); + assert.match(error.message, /Remote daemon is unavailable/); + } + assert.equal(spawn.mock.calls.length, 0); +}); + +test('ensureDaemon starts a missing daemon and reports why', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const fresh = await startScriptedDaemon(okResult({})); + launchesFreshDaemonAt(paths, fresh.port); + const started: Array<[string, unknown]> = []; + + try { + const connection = await ensureDaemon({ + stateDir, + daemonTransport: 'http', + onStart: (reason, previous) => started.push([reason, previous]), + }); + + assert.equal(connection.started, true); + assert.equal(connection.stateDir, paths.baseDir); + assert.equal(spawn.mock.calls.length, 1); + assert.deepEqual(started, [['missing', undefined]]); + assert.equal('token' in connection.daemon, false, 'the connection carries no credentials'); + + const again = await ensureDaemon({ stateDir, daemonTransport: 'http' }); + assert.equal(again.started, false); + assert.equal(again.daemon.pid, connection.daemon.pid); + assert.equal(spawn.mock.calls.length, 1, 'a usable daemon is reused'); + } finally { + await closeLoopbackServer(fresh.server); + } +}); + +test('ensureDaemon ignores a remote base URL in the environment', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const fresh = await startScriptedDaemon(okResult({})); + launchesFreshDaemonAt(paths, fresh.port); + vi.stubEnv('AGENT_DEVICE_DAEMON_BASE_URL', 'http://127.0.0.1:1'); + + try { + const connection = await ensureDaemon({ stateDir, daemonTransport: 'http' }); + assert.equal(connection.started, true); + } finally { + await closeLoopbackServer(fresh.server); + } +}); + +test('ensureDaemon replaces an incompatible idle daemon through a fenced quiesce request', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const stale = await startScriptedDaemon(draining); + const fresh = await startScriptedDaemon(okResult({})); + const { child, info } = await registerDaemon(paths, { + httpPort: stale.port, + version: '0.0.0-stale', + capabilities: ['quiesce'], + }); + launchesFreshDaemonAt(paths, fresh.port); + const started: Array<[string, { pid: number; version?: string } | undefined]> = []; + + try { + const connection = await ensureDaemon({ + stateDir, + daemonTransport: 'http', + onStart: (reason, previous) => started.push([reason, previous]), + }); + + assert.equal(connection.started, true); + assert.deepEqual(stale.rpcCommands, [DAEMON_QUIESCE_COMMAND]); + assert.equal(stale.rpcRequests[0]?.params.token, 'local-secret'); + assert.deepEqual(stale.rpcRequests[0]?.params.input, { + pid: info.pid, + processStartTime: info.processStartTime ?? null, + }); + assert.deepEqual(started, [ + [ + 'incompatible', + { + pid: child.pid, + version: '0.0.0-stale', + codeOrigin: 'checkout', + codeSignature: currentDaemonCodeSignature(), + processStartTime: info.processStartTime, + }, + ], + ]); + assert.equal( + isProcessAlive(child.pid), + false, + 'the drained daemon is gone before its successor', + ); + assert.equal(spawn.mock.calls.length, 1); + } finally { + await closeLoopbackServer(stale.server); + await closeLoopbackServer(fresh.server); + } +}); + +test.for([ + { + name: 'a request in flight', + census: { inFlightRequests: 1, sessions: 0, repairSessions: 0, leases: 0 }, + }, + { + name: 'a live lease', + census: { inFlightRequests: 0, sessions: 0, repairSessions: 0, leases: 1 }, + }, +])('ensureDaemon refuses to replace a daemon holding $name', async ({ census }, t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const stale = await startScriptedDaemon(busy(census)); + const { child } = await registerDaemon(paths, { + httpPort: stale.port, + version: '0.0.0-stale', + capabilities: ['quiesce'], + }); + + try { + const error = await failureOf(ensureDaemon({ stateDir, daemonTransport: 'http' })); + + assert.equal(error.details?.reason, 'daemon-busy'); + assert.equal(error.details?.inFlightRequests, census.inFlightRequests); + assert.equal(error.details?.leases, census.leases); + assert.equal(isProcessAlive(child.pid), true); + assert.equal(spawn.mock.calls.length, 0); + } finally { + await closeLoopbackServer(stale.server); + } +}); + +test('ensureDaemon takes a refusal from its typed reason, not from its wording', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const stale = await startScriptedDaemon({ + ok: false, + error: { + code: 'COMMAND_FAILED', + message: 'Daemon is busy and was not asked to leave.', + details: { reason: 'something-else' }, + }, + }); + await registerDaemon(paths, { + httpPort: stale.port, + version: '0.0.0-stale', + capabilities: ['quiesce'], + }); + + try { + const error = await failureOf(ensureDaemon({ stateDir, daemonTransport: 'http' })); + // The refusal's own typed reason is what surfaces; its wording decided nothing. + assert.equal(error.details?.reason, 'something-else'); + assert.equal(spawn.mock.calls.length, 0); + } finally { + await closeLoopbackServer(stale.server); + } +}); + +test('ensureDaemon never replaces a daemon that predates the quiesce capability', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const old = await startScriptedDaemon(draining); + const { child } = await registerDaemon(paths, { httpPort: old.port, version: '0.0.0-old' }); + + try { + const error = await failureOf(ensureDaemon({ stateDir, daemonTransport: 'http' })); + + assert.equal(error.details?.reason, 'daemon-incompatible'); + assert.equal(error.details?.mismatch, 'no-quiesce'); + assert.deepEqual(old.rpcCommands, [], 'a daemon that cannot answer is not asked'); + assert.equal(isProcessAlive(child.pid), true); + assert.equal(spawn.mock.calls.length, 0); + } finally { + await closeLoopbackServer(old.server); + } +}); + +test('ensureDaemon retires a daemon that timed out on quiesce and answers no liveness probe', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + // Bound together before the dead one is closed, so the fresh daemon cannot be handed its port. + const closed = await startScriptedDaemon(okResult({})); + const fresh = await startScriptedDaemon(okResult({})); + await closeLoopbackServer(closed.server); + const { child } = await registerDaemon(paths, { + httpPort: closed.port, + version: '0.0.0-stale', + capabilities: ['quiesce'], + }); + launchesFreshDaemonAt(paths, fresh.port); + vi.spyOn(transport, 'sendRequest').mockRejectedValueOnce( + new AppError('COMMAND_FAILED', 'Daemon request timed out', { + reason: 'daemon_transport_timeout', + }), + ); + + try { + const started: Array<[string, number | undefined]> = []; + const connection = await ensureDaemon({ + stateDir, + daemonTransport: 'http', + onStart: (reason, previous) => started.push([reason, previous?.pid]), + }); + + assert.equal(connection.started, true); + assert.equal(isProcessAlive(child.pid), false); + // Never shown to be idle, so it is reported like an unreachable daemon, not an incompatible one. + assert.deepEqual(started, [['missing', child.pid]]); + } finally { + await closeLoopbackServer(fresh.server); + } +}); + +test('ensureDaemon keeps a daemon that timed out on quiesce but still answers a probe', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const slow = await startScriptedDaemon(okResult({})); + const { child } = await registerDaemon(paths, { + httpPort: slow.port, + version: '0.0.0-stale', + capabilities: ['quiesce'], + }); + vi.spyOn(transport, 'sendRequest').mockRejectedValueOnce( + new AppError('COMMAND_FAILED', 'Daemon request timed out', { + reason: 'daemon_transport_timeout', + }), + ); + + try { + const error = await failureOf(ensureDaemon({ stateDir, daemonTransport: 'http' })); + + assert.equal(error.details?.reason, 'daemon_transport_timeout'); + assert.equal(isProcessAlive(child.pid), true); + assert.equal(spawn.mock.calls.length, 0); + } finally { + await closeLoopbackServer(slow.server); + } +}); + +const instanceMismatch = { + ok: false, + error: { + code: 'COMMAND_FAILED', + message: 'Quiesce request names a different daemon instance.', + details: { reason: 'daemon-instance-mismatch' }, + }, +}; + +test('a live daemon on another transport is quiesced, never killed as unreachable', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const live = await startScriptedSocketDaemon({ ok: true, data: { outcome: 'draining' } }); + const fresh = await startScriptedDaemon(okResult({})); + const { child } = await registerDaemon(paths, { + socketPort: live.port, + capabilities: ['quiesce'], + }); + launchesFreshDaemonAt(paths, fresh.port); + const started: string[] = []; + + try { + // Same version and code, served over the socket only; this client asks for HTTP. + await ensureDaemon({ + stateDir, + daemonTransport: 'http', + onStart: (reason) => started.push(reason), + }); + + assert.deepEqual(live.commands, [DAEMON_QUIESCE_COMMAND]); + assert.deepEqual(started, ['incompatible']); + assert.equal(isProcessAlive(child.pid), false, 'it left because it agreed to'); + } finally { + await closeLoopbackServer(live.server); + await closeLoopbackServer(fresh.server); + } +}); + +test('a live daemon on another transport is left alone when it cannot be asked to leave', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const live = await startScriptedSocketDaemon({ ok: true, data: { outcome: 'draining' } }); + const { child } = await registerDaemon(paths, { socketPort: live.port }); + + try { + const ensured = await failureOf(ensureDaemon({ stateDir, daemonTransport: 'http' })); + assert.equal(ensured.details?.reason, 'daemon-incompatible'); + assert.equal(ensured.details?.mismatch, 'no-quiesce'); + + const attached = await failureOf( + sendToDaemon(request(stateDir), { daemonLifecycle: 'attach' }), + ); + assert.equal(attached.details?.reason, 'daemon-incompatible'); + assert.equal(attached.details?.mismatch, 'transport'); + + assert.equal(isProcessAlive(child.pid), true); + assert.equal(spawn.mock.calls.length, 0); + assert.deepEqual(live.commands, []); + } finally { + await closeLoopbackServer(live.server); + } +}); + +test('ensureDaemon judges again after the daemon it addressed turned out to be gone', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const stale = await startScriptedDaemon(draining); + const fresh = await startScriptedDaemon(okResult({})); + await registerDaemon(paths, { + httpPort: stale.port, + version: '0.0.0-stale', + capabilities: ['quiesce'], + }); + launchesFreshDaemonAt(paths, fresh.port); + const actualSend = transport.sendRequest; + vi.spyOn(transport, 'sendRequest').mockResolvedValueOnce(instanceMismatch); + + try { + const connection = await ensureDaemon({ stateDir, daemonTransport: 'http' }); + + assert.equal(connection.started, true); + assert.equal(typeof actualSend, 'function'); + assert.deepEqual( + stale.rpcCommands, + [DAEMON_QUIESCE_COMMAND], + 'the second judgement asked again', + ); + } finally { + await closeLoopbackServer(stale.server); + await closeLoopbackServer(fresh.server); + } +}); + +test('ensureDaemon reports contention when the daemon keeps changing under it', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const stale = await startScriptedDaemon(instanceMismatch); + const { child } = await registerDaemon(paths, { + httpPort: stale.port, + version: '0.0.0-stale', + capabilities: ['quiesce'], + }); + + try { + const error = await failureOf(ensureDaemon({ stateDir, daemonTransport: 'http' })); + + assert.equal(error.details?.reason, 'daemon-lifecycle-contended'); + assert.equal(isProcessAlive(child.pid), true); + assert.equal(spawn.mock.calls.length, 0); + } finally { + await closeLoopbackServer(stale.server); + } +}); + +test('ensureDaemon does not start a daemon over one it could not confirm retired', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const closed = await startScriptedDaemon(okResult({})); + await closeLoopbackServer(closed.server); + const { child } = await registerDaemon(paths, { + httpPort: closed.port, + version: '0.0.0-stale', + capabilities: ['quiesce'], + }); + vi.spyOn(transport, 'sendRequest').mockRejectedValueOnce( + new AppError('COMMAND_FAILED', 'Daemon request timed out', { + reason: 'daemon_transport_timeout', + }), + ); + const actualKill = process.kill.bind(process); + vi.spyOn(process, 'kill').mockImplementation((pid, signal) => { + if (pid === child.pid && signal !== 0) { + throw Object.assign(new Error('refused'), { code: 'EPERM' }); + } + return actualKill(pid, signal); + }); + + const error = await failureOf(ensureDaemon({ stateDir, daemonTransport: 'http' })); + + assert.equal(error.details?.reason, 'daemon_retirement_unconfirmed'); + assert.equal(spawn.mock.calls.length, 0); +}); + +test('ensureDaemon quiesces an incompatible daemon that wins a startup race instead of retiring it', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const stale = await startScriptedDaemon(draining); + const fresh = await startScriptedDaemon(okResult({})); + const barrier = path.join(paths.baseDir, 'contender-barrier'); + fs.mkdirSync(paths.baseDir, { recursive: true }); + vi.mocked(sleep).mockImplementation(actualRetry.sleep); + let winner: ReturnType | undefined; + spawn.mockImplementationOnce((_command, _args, options) => { + // An incompatible daemon registers just ahead of the one this call launches, which then exits busy. + winner = spawnRegisteredDaemonFixture( + paths, + { + httpPort: stale.port, + token: 'local-secret', + version: '0.0.0-stale', + codeOrigin: 'checkout', + codeSignature: currentDaemonCodeSignature(), + capabilities: ['quiesce'], + }, + undefined, + ); + // Released whether or not the winner published, so the launched child never waits forever. + const release = () => fs.writeFileSync(barrier, 'go'); + void waitForRegisteredDaemonFixture(paths, winner).then(release, release); + // Our launch loses the race once the winner is registered: it exits with the busy code. + return actualCommand.runCmdDetachedMonitored( + process.execPath, + [ + '-e', + `const fs=require("fs");setInterval(()=>{if(fs.existsSync(process.argv[1]))process.exit(${DAEMON_STARTUP_EXIT_CODES.busy})},10)`, + barrier, + ], + options, + ); + }); + launchesFreshDaemonAt(paths, fresh.port); + + try { + const connection = await ensureDaemon({ stateDir, daemonTransport: 'http' }); + + assert.equal(connection.started, true); + assert.deepEqual(stale.rpcCommands, [DAEMON_QUIESCE_COMMAND], 'the racer was asked to leave'); + assert.ok(winner); + assert.equal(isProcessAlive(winner.pid), false); + assert.equal(spawn.mock.calls.length, 2, 'the second start follows the quiesce'); + } finally { + await closeLoopbackServer(stale.server); + await closeLoopbackServer(fresh.server); + } +}); + +test('a registration whose process is gone is not running, whatever version it names', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const stale = await startScriptedDaemon(draining); + const fresh = await startScriptedDaemon(okResult({})); + fs.mkdirSync(paths.baseDir, { recursive: true }); + fs.writeFileSync( + paths.infoPath, + JSON.stringify({ + httpPort: stale.port, + token: 'local-secret', + pid: 999_999_999, + version: '0.0.0-stale', + processStartTime: 'gone', + capabilities: ['quiesce'], + }), + ); + + try { + const attached = await failureOf( + sendToDaemon(request(stateDir), { daemonLifecycle: 'attach' }), + ); + assert.equal(attached.details?.reason, 'daemon-not-running'); + + // ensureDaemon replaces the dead registration without asking anyone to leave. + launchesFreshDaemonAt(paths, fresh.port); + const connection = await ensureDaemon({ stateDir, daemonTransport: 'http' }); + assert.equal(connection.started, true); + assert.deepEqual(stale.rpcCommands, []); + } finally { + await closeLoopbackServer(stale.server); + await closeLoopbackServer(fresh.server); + } +}); + +test('a quiesce the daemon never hears is judged again, not reported as a transport failure', async (t) => { + if (!(await withLoopback(t))) return; + const { stateDir, paths } = stateDirAndPaths(); + const closed = await startScriptedDaemon(draining); + await closeLoopbackServer(closed.server); + const { child } = await registerDaemon(paths, { + httpPort: closed.port, + version: '0.0.0-stale', + capabilities: ['quiesce'], + }); + + const error = await failureOf(ensureDaemon({ stateDir, daemonTransport: 'http' })); + + assert.equal(error.details?.reason, 'daemon-lifecycle-contended'); + assert.equal(isProcessAlive(child.pid), true); + assert.equal(spawn.mock.calls.length, 0); +}); diff --git a/src/daemon-client/__tests__/daemon-client-lease-beat.test.ts b/src/daemon-client/__tests__/daemon-client-lease-beat.test.ts index 6c2881b2b1..8db255bfca 100644 --- a/src/daemon-client/__tests__/daemon-client-lease-beat.test.ts +++ b/src/daemon-client/__tests__/daemon-client-lease-beat.test.ts @@ -580,6 +580,8 @@ describe('buildUploadLeaseHeartbeat', () => { const settings = { paths: resolveDaemonPaths('/tmp/agent-device-upload-lease'), transportPreference: 'socket' as const, + daemonLifecycle: 'managed' as const, + retireIncompatibleContender: true, serverMode: 'socket' as const, }; diff --git a/src/daemon-client/__tests__/daemon-client-startup-race.test.ts b/src/daemon-client/__tests__/daemon-client-startup-race.test.ts index 6687da63d3..726c45031a 100644 --- a/src/daemon-client/__tests__/daemon-client-startup-race.test.ts +++ b/src/daemon-client/__tests__/daemon-client-startup-race.test.ts @@ -594,7 +594,7 @@ test.for(['default', 'private replay'] as const)( if (!(await supportsLoopbackBind())) return t.skip('loopback unavailable'); const http = await startHttpDaemonFixture({ devices: [], sessionActive: false }); const defaultPaths = resolveDaemonPaths(mkdtempForTestSync('daemon-missed-birth-')); - const ensure = vi.spyOn(lifecycle, 'ensureDaemon'); + const ensure = vi.spyOn(lifecycle, 'ensureManagedDaemon'); let paths: DaemonPaths | undefined; let child: ReturnType | undefined; let now = Date.now(); diff --git a/src/daemon-client/__tests__/daemon-client-timeout-route.test.ts b/src/daemon-client/__tests__/daemon-client-timeout-route.test.ts index 619a2e2411..a266293a47 100644 --- a/src/daemon-client/__tests__/daemon-client-timeout-route.test.ts +++ b/src/daemon-client/__tests__/daemon-client-timeout-route.test.ts @@ -37,6 +37,7 @@ vi.mock('@agent-device/host-kit/command', async () => { }); import { AppError } from '@agent-device/kernel/errors'; +import { isProcessAlive } from '@agent-device/host-kit/process'; import { PUBLIC_COMMANDS } from '@agent-device/command-registry/catalog'; import { sendRequest } from '../daemon-client-transport.ts'; import type { DaemonInfo } from '../daemon-client-metadata.ts'; @@ -454,6 +455,38 @@ test('a refused timeout fallback preserves the timeout without an unhandled reje } }); +test('request-timeout route: an attach client leaves a daemon that answers no probe running', async (t) => { + if (await skipWhenLoopbackUnavailable(t)) return; + // The same wedged daemon the reset row retires. An attach client does not own it, so the probe + // still tells the caller what it found, and nothing is signaled or deleted. + const daemon = await startStandIn('socket', 'refuse'); + const paths = dummyStatePaths(); + const info = await registeredOwner(paths, daemon.port, 'socket'); + try { + await expectRouteError( + sendRequest( + info, + buildRequest(RESET_POLICY_COMMAND, undefined), + 'socket', + paths, + TIMEOUT_MS, + { timeoutRecovery: 'none' }, + ), + /did not answer the liveness probe.*does not retire a daemon it did not start.*agent-device daemon stop/s, + ); + assert.equal( + daemon.connections(), + 2, + 'the probe still asks, so the hint can say what it found', + ); + assert.equal(isProcessAlive(info.pid), true); + assert.ok(fs.existsSync(paths.infoPath)); + assert.ok(fs.existsSync(paths.lockPath)); + } finally { + await closeLoopbackServer(daemon.server); + } +}); + test('request-timeout route: the reset acts on the resolved state paths, not the request flags', async (t) => { if (await skipWhenLoopbackUnavailable(t)) return; // Path resolution happens upstream of `sendRequest`, which receives the state paths the caller diff --git a/src/daemon-client/__tests__/daemon-launch-spec.test.ts b/src/daemon-client/__tests__/daemon-launch-spec.test.ts index d69d0aa727..19a0581e25 100644 --- a/src/daemon-client/__tests__/daemon-launch-spec.test.ts +++ b/src/daemon-client/__tests__/daemon-launch-spec.test.ts @@ -118,6 +118,7 @@ test('an unreachable newer daemon is replaced like any version mismatch', async { kind: 'replace', reason: `version mismatch (client v${readVersion()})`, + cause: 'version', }, ); }); @@ -151,7 +152,7 @@ test('a same-version daemon the client transport cannot reach is replaced', asyn runningDaemon({ codeOrigin: 'installed' }), onlyOnAnotherTransport(), ), - { kind: 'replace', reason: 'unreachable' }, + { kind: 'replace', reason: 'unreachable', cause: 'unreachable' }, ); }); @@ -159,6 +160,7 @@ test('a reachable daemon older than the client is replaced', async () => { assert.deepEqual(await resolveDaemonTakeover(runningDaemon({ version: '0.0.1' }), reachable()), { kind: 'replace', reason: `version mismatch (client v${readVersion()})`, + cause: 'version', }); }); @@ -236,7 +238,11 @@ test('an installed client replaces a daemon that reports a source checkout (#245 runningDaemon({ codeOrigin: 'checkout', codeSignature: 'edited-checkout' }), reachable(), ), - { kind: 'replace', reason: 'code origin mismatch (daemon checkout, client installed)' }, + { + kind: 'replace', + reason: 'code origin mismatch (daemon checkout, client installed)', + cause: 'code', + }, ); }); @@ -248,6 +254,7 @@ test('an installed client replaces a daemon that predates the code origin field { kind: 'replace', reason: 'code origin mismatch (daemon unreported, client installed)', + cause: 'code', }, ); }); @@ -279,6 +286,7 @@ test('a source checkout replaces a daemon whose code signature differs', async ( { kind: 'replace', reason: 'code-signature mismatch', + cause: 'code', }, ); }); @@ -296,6 +304,7 @@ test('a source checkout replaces a daemon that reports an installed package', as { kind: 'replace', reason: 'code origin mismatch (daemon installed, client checkout)', + cause: 'code', }, ); }); @@ -308,11 +317,16 @@ test('a source checkout judges a daemon that predates the code origin field by i assert.deepEqual(await resolveDaemonTakeover(runningDaemon({}), reachable()), { kind: 'replace', reason: 'code-signature mismatch', + cause: 'code', }); }); test('a mismatched version replaces the daemon whichever tree the client runs from', async () => { - const expected = { kind: 'replace', reason: `version mismatch (client v${readVersion()})` }; + const expected = { + kind: 'replace', + reason: `version mismatch (client v${readVersion()})`, + cause: 'version', + }; for (const sourceCheckout of [false, true]) { useClientTree(sourceCheckout); @@ -340,6 +354,7 @@ test('a reachable daemon of a matching identity survives, an unreachable one doe { kind: 'replace', reason: 'unreachable', + cause: 'unreachable', }, ); }); diff --git a/src/daemon-client/__tests__/explicit-lifecycle-fixture.ts b/src/daemon-client/__tests__/explicit-lifecycle-fixture.ts new file mode 100644 index 0000000000..299e3afa00 --- /dev/null +++ b/src/daemon-client/__tests__/explicit-lifecycle-fixture.ts @@ -0,0 +1,135 @@ +import http from 'node:http'; +import net from 'node:net'; +import type { Mock } from 'vitest'; +import { listenOnLoopback, mkdtempForTestSync } from '@agent-device/test-kit'; +import { + resolveDaemonPaths, + type DaemonPaths, +} from '@agent-device/daemon-contracts/daemon-resolution'; +import type { runCmdDetachedMonitored } from '@agent-device/host-kit/command'; +import { readVersion } from '@agent-device/host-kit/version'; +import { + spawnRegisteredDaemonFixture, + waitForRegisteredDaemonFixture, +} from '../../__tests__/test-utils/registered-daemon-fixture.ts'; +import { currentDaemonCodeSignature } from '../../__tests__/test-utils/daemon-http-fixture.ts'; + +// Stand-ins for the daemons the explicit lifecycle judges: a loopback server that answers every +// `/rpc` with one scripted result, and a real child process that owns its registration. + +export type ScriptedDaemon = { + server: http.Server; + port: number; + rpcCommands: string[]; + rpcRequests: Record[]; +}; + +/** Answers `/health` and answers every `/rpc` with the one JSON-RPC `result` it was given. */ +export async function startScriptedDaemon( + result: Record, +): Promise { + const rpcCommands: string[] = []; + const rpcRequests: Record[] = []; + const server = http.createServer((req, res) => { + if (req.method === 'GET' && req.url === '/health') { + res.writeHead(200); + res.end('ok'); + return; + } + const chunks: Buffer[] = []; + req.on('data', (chunk) => chunks.push(Buffer.from(chunk))); + req.on('end', () => { + const rpc = JSON.parse(Buffer.concat(chunks).toString('utf8')) as Record; + rpcRequests.push(rpc); + rpcCommands.push(String(rpc.params?.command)); + res.writeHead(200, { 'content-type': 'application/json' }); + res.end(JSON.stringify({ jsonrpc: '2.0', id: rpc.id, result })); + }); + }); + return { server, port: await listenOnLoopback(server), rpcCommands, rpcRequests }; +} + +/** A socket-only stand-in: one JSON line in, the scripted response out. */ +export async function startScriptedSocketDaemon( + response: Record, +): Promise<{ server: net.Server; port: number; commands: string[] }> { + const commands: string[] = []; + const server = net.createServer((socket) => { + socket.setEncoding('utf8'); + socket.on('error', () => {}); + let buffered = ''; + socket.on('data', (chunk) => { + buffered += chunk; + if (!buffered.includes('\n')) return; + commands.push(String((JSON.parse(buffered) as { command?: unknown }).command)); + socket.write(`${JSON.stringify(response)}\n`); + }); + }); + return { server, port: await listenOnLoopback(server), commands }; +} + +export const okResult = (data: Record) => ({ ok: true, data }); +export const draining = okResult({ outcome: 'draining' }); +export const busy = (census: Record) => ({ + ok: false, + error: { + code: 'COMMAND_FAILED', + message: 'Daemon is busy and was not asked to leave.', + details: { reason: 'daemon-busy', ...census }, + }, +}); + +export function stateDirAndPaths(): { stateDir: string; paths: DaemonPaths } { + const stateDir = mkdtempForTestSync('agent-device-explicit-lifecycle-'); + return { stateDir, paths: resolveDaemonPaths(stateDir) }; +} + +type Registration = { + httpPort?: number; + socketPort?: number; + version?: string; + capabilities?: readonly string[]; +}; + +/** A live child owning the registration, advertising `fields` at the stand-in's port. */ +export async function registerDaemon(paths: DaemonPaths, fields: Registration) { + const child = spawnRegisteredDaemonFixture( + paths, + { + ...(fields.httpPort !== undefined ? { httpPort: fields.httpPort } : {}), + ...(fields.socketPort !== undefined ? { socketPort: fields.socketPort } : {}), + token: 'local-secret', + version: fields.version ?? readVersion(), + codeOrigin: 'checkout', + codeSignature: currentDaemonCodeSignature(), + ...(fields.capabilities ? { capabilities: fields.capabilities as ['quiesce'] } : {}), + }, + undefined, + ); + return { child, info: await waitForRegisteredDaemonFixture(paths, child) }; +} + +/** + * Answers the daemon launch seam with a child that registers and advertises the stand-in at + * `httpPort`, so a start the client performs yields a daemon it can reach. + */ +export function launchesFreshDaemonAt( + spawn: Mock, + paths: DaemonPaths, + httpPort: number, +): void { + spawn.mockImplementation((_command, _args, options) => + spawnRegisteredDaemonFixture( + paths, + { + httpPort, + token: 'local-secret', + version: readVersion(), + codeOrigin: 'checkout', + codeSignature: currentDaemonCodeSignature(), + capabilities: ['quiesce'], + }, + options, + ), + ); +} diff --git a/src/daemon-client/daemon-client-attach.ts b/src/daemon-client/daemon-client-attach.ts new file mode 100644 index 0000000000..bf0d35a7f1 --- /dev/null +++ b/src/daemon-client/daemon-client-attach.ts @@ -0,0 +1,41 @@ +import { + assertDaemonPolicyMatches, + ensureManagedDaemon, + observeLocalDaemon, + type DaemonClientSettings, + type EnsuredDaemon, +} from './daemon-client-lifecycle.ts'; +import { daemonIncompatibleError, daemonNotRunningError } from './daemon-lifecycle-errors.ts'; + +/** + * Connects to the local daemon registered in the state directory and nothing else: it never starts, + * replaces or kills one. What the registration says is judged by the same ladder a managed client + * uses, so a daemon an attach client accepts is one a managed client would also reuse. + */ +async function attachLocalDaemon(settings: DaemonClientSettings): Promise { + const stateDir = settings.paths.baseDir; + const observed = await observeLocalDaemon(settings, { refineUnreachable: true }); + if (!observed) throw daemonNotRunningError(stateDir); + const { info, decision } = observed; + switch (decision.kind) { + case 'reuse': + return { info, startedByClient: false }; + case 'refuseNewer': + throw await daemonIncompatibleError({ info, mismatch: 'newer', stateDir }); + case 'replace': + throw decision.cause === 'unreachable' + ? daemonNotRunningError(stateDir, info) + : await daemonIncompatibleError({ info, mismatch: decision.cause, stateDir }); + } +} + +/** The daemon a request goes to, by the lifecycle its client chose. A remote one is never ours. */ +export async function connectToDaemon(settings: DaemonClientSettings): Promise { + if (settings.remoteBaseUrl || settings.daemonLifecycle !== 'attach') { + return await ensureManagedDaemon(settings); + } + const attached = await attachLocalDaemon(settings); + // Checked on this branch too: attaching must not hand a policy-bound caller a daemon without it. + await assertDaemonPolicyMatches(attached.info, settings.paths.baseDir); + return attached; +} diff --git a/src/daemon-client/daemon-client-ensure.ts b/src/daemon-client/daemon-client-ensure.ts new file mode 100644 index 0000000000..034d6fdb84 --- /dev/null +++ b/src/daemon-client/daemon-client-ensure.ts @@ -0,0 +1,172 @@ +import type { + DaemonConnection, + DaemonIdentity, + EnsureDaemonOptions, +} from '@agent-device/contracts/client'; +import { sleep } from '@agent-device/host-kit/retry'; +import { AppError } from '@agent-device/kernel/errors'; +import { + assertDaemonPolicyMatches, + observeLocalDaemon, + resolveLocalDaemonSettings, + retireDaemonForTakeover, + startLocalDaemon, + type DaemonClientSettings, + type EnsuredDaemon, +} from './daemon-client-lifecycle.ts'; +import { requestDaemonQuiesce } from './daemon-client-quiesce.ts'; +import { probeDaemonResponsive } from './daemon-client-liveness-probe.ts'; +import type { DaemonInfo } from './daemon-client-metadata.ts'; +import { + DAEMON_CONTENDED_REASON, + daemonIdentityOf, + daemonIncompatibleError, +} from './daemon-lifecycle-errors.ts'; + +// A racing `ensureDaemon` in another process can retire or replace the daemon this call judged, or +// a daemon this call quiesced can still be leaving. A judgement that went stale is made again after +// a short wait, for as long as a drain can take. +const ENSURE_ATTEMPTS = 40; +const JUDGE_AGAIN_DELAY_MS = 250; +const UNRESPONSIVE_PROBES = 3; +const UNRESPONSIVE_PROBE_GAP_MS = 200; + +type EnsureStep = + | { kind: 'connected'; daemon: EnsuredDaemon; started: boolean } + | { kind: 'judge-again' }; + +/** + * Makes the local daemon on a state directory usable by this client, and says what it did. + * + * A missing daemon is started. An incompatible one is replaced only if it agrees to leave: the + * daemon itself refuses while it holds a request, a session, a repair session or a live lease, so + * this never destroys another client's work, and a daemon that cannot be asked is never replaced. + * A daemon that answers nothing is retired after repeated failed liveness probes. A remote + * connection has no local daemon to ensure, so this takes no remote option and reads no remote + * environment. + */ +export async function ensureDaemon(options: EnsureDaemonOptions = {}): Promise { + const settings = resolveLocalDaemonSettings(options); + for (let attempt = 1; attempt <= ENSURE_ATTEMPTS; attempt += 1) { + const step = await ensureOnce(settings, options.onStart); + if (step.kind === 'judge-again') { + await sleep(JUDGE_AGAIN_DELAY_MS); + continue; + } + await assertDaemonPolicyMatches(step.daemon.info, settings.paths.baseDir); + return { + stateDir: settings.paths.baseDir, + daemon: daemonIdentityOf(step.daemon.info), + started: step.started, + }; + } + throw new AppError('COMMAND_FAILED', 'The daemon kept changing while it was being ensured.', { + reason: 'daemon-lifecycle-contended', + stateDir: settings.paths.baseDir, + hint: 'Another process is starting or replacing the daemon on this state directory. Retry.', + }); +} + +async function ensureOnce( + settings: DaemonClientSettings, + onStart: EnsureDaemonOptions['onStart'], +): Promise { + const observed = await observeLocalDaemon(settings, { refineUnreachable: true }); + if (!observed) return await start(settings, onStart, 'missing'); + const { info, decision } = observed; + switch (decision.kind) { + case 'reuse': + return { kind: 'connected', daemon: { info, startedByClient: false }, started: false }; + case 'refuseNewer': + throw await daemonIncompatibleError({ + info, + mismatch: 'newer', + stateDir: settings.paths.baseDir, + }); + case 'replace': + return decision.cause === 'unreachable' + ? await replaceUnreachable(settings, info, onStart) + : await replaceIncompatible(settings, info, onStart); + } +} + +async function replaceUnreachable( + settings: DaemonClientSettings, + info: DaemonInfo, + onStart: EnsureDaemonOptions['onStart'], +): Promise { + // `observeLocalDaemon` probed it repeatedly before calling it unreachable. + await retireDaemonForTakeover(info, settings.paths); + return await start(settings, onStart, 'missing', daemonIdentityOf(info)); +} + +async function replaceIncompatible( + settings: DaemonClientSettings, + info: DaemonInfo, + onStart: EnsureDaemonOptions['onStart'], +): Promise { + // A daemon that predates the capability cannot say whether it is idle, so it is never replaced. + if (!info.capabilities?.includes('quiesce')) { + throw await daemonIncompatibleError({ + info, + mismatch: 'no-quiesce', + stateDir: settings.paths.baseDir, + }); + } + const outcome = await quiesceOrRetireUnresponsive(info, settings); + if (outcome === 'gone') return { kind: 'judge-again' }; + if (outcome === 'draining') await retireDaemonForTakeover(info, settings.paths); + // A daemon force-retired for answering nothing is reported like an unreachable one: it was never + // shown to be idle, so "incompatible" would overstate what happened to it. + const reason = outcome === 'retired' ? 'missing' : 'incompatible'; + return await start(settings, onStart, reason, daemonIdentityOf(info)); +} + +/** + * The one place a request timeout can lead to killing a daemon: a quiesce that timed out, against + * a daemon that then fails its liveness probe every time, is a hung process nobody can ask to leave. + */ +async function quiesceOrRetireUnresponsive( + info: DaemonInfo, + settings: DaemonClientSettings, +): Promise<'draining' | 'gone' | 'retired'> { + try { + return await requestDaemonQuiesce(info, settings); + } catch (error) { + if (!(error instanceof AppError) || error.details?.reason !== 'daemon_transport_timeout') + throw error; + if (await answersLivenessProbe(info)) throw error; + await retireDaemonForTakeover(info, settings.paths, 'force'); + return 'retired'; + } +} + +async function answersLivenessProbe(info: DaemonInfo): Promise { + for (let probe = 1; probe <= UNRESPONSIVE_PROBES; probe += 1) { + if (await probeDaemonResponsive(info)) return true; + if (probe < UNRESPONSIVE_PROBES) await sleep(UNRESPONSIVE_PROBE_GAP_MS); + } + return false; +} + +async function start( + settings: DaemonClientSettings, + onStart: EnsureDaemonOptions['onStart'], + reason: 'missing' | 'incompatible', + previous?: DaemonIdentity, +): Promise { + let daemon: EnsuredDaemon; + try { + daemon = await startLocalDaemon(settings); + } catch (error) { + // The winner of a startup race is judged again from the top, so an incompatible one is quiesced + // rather than retired. + if (error instanceof AppError && error.details?.reason === DAEMON_CONTENDED_REASON) { + return { kind: 'judge-again' }; + } + throw error; + } + // A start can resolve to a daemon another caller raced in, which this call did not start. + if (daemon.startedByClient) onStart?.(reason, previous); + return { kind: 'connected', daemon, started: daemon.startedByClient }; +} diff --git a/src/daemon-client/daemon-client-lifecycle.ts b/src/daemon-client/daemon-client-lifecycle.ts index 205f4784b3..28d5803e89 100644 --- a/src/daemon-client/daemon-client-lifecycle.ts +++ b/src/daemon-client/daemon-client-lifecycle.ts @@ -1,5 +1,7 @@ import type { DaemonWireRequest } from '@agent-device/contracts/command'; +import type { DaemonLifecycle, EnsureDaemonOptions } from '@agent-device/contracts/client'; import type { DaemonResponse } from '@agent-device/kernel/contracts'; +import { daemonContendedError, daemonStopCommand } from './daemon-lifecycle-errors.ts'; import { attachActiveSessionAddressHint } from './daemon-client-address-hints.ts'; import fs from 'node:fs'; import { processLockHoldsDaemonIdentity } from '@agent-device/daemon-contracts/daemon-registration'; @@ -7,7 +9,6 @@ import net from 'node:net'; import { AppError, normalizeError, type NormalizedError } from '@agent-device/kernel/errors'; import { readReplayDivergenceResume } from '@agent-device/ad-replay/divergence'; import { type ExecDetachedExit } from '@agent-device/host-kit/command'; -import { shellQuoteIfNeeded } from '@agent-device/kernel/device-shell'; import { emitDiagnostic } from '@agent-device/host-kit/diagnostics'; import { isProcessAlive } from '@agent-device/host-kit/process'; import { sleep } from '@agent-device/host-kit/retry'; @@ -54,6 +55,13 @@ import { export type DaemonClientSettings = { paths: DaemonPaths; + daemonLifecycle: DaemonLifecycle; + /** + * Whether a daemon that wins a startup race may be retired when this client cannot use it. A + * managed client may; `ensureDaemon` may not, because it replaces a daemon only through a quiesce + * and judges the winner again instead. + */ + retireIncompatibleContender: boolean; transportPreference: DaemonTransportPreference; serverMode: DaemonServerMode; ownedStateDir?: OwnedReplayStateDir; @@ -86,15 +94,21 @@ LOOPBACK_BLOCK_LIST.addSubnet('::ffff:127.0.0.0', 104, 'ipv6'); export function resolveClientSettings( req: Omit, suppliedAuthToken?: string, + daemonLifecycle: DaemonLifecycle = 'managed', ): DaemonClientSettings { const explicitStateDir = resolveExplicitStateDir(req); const remote = resolveRemoteClientSettings(req, suppliedAuthToken); - const transport = resolveTransportClientSettings(req, remote.remoteBaseUrl); - const ownedStateDir = shouldUseOwnedReplayStateDir(req, explicitStateDir, remote.rawBaseUrl) - ? createOwnedReplayStateDir() - : undefined; + const transport = resolveTransportClientSettings(req.flags, remote.remoteBaseUrl); + // An attach client owns no daemon, and a private state directory would hold one nobody started. + const ownedStateDir = + daemonLifecycle !== 'attach' && + shouldUseOwnedReplayStateDir(req, explicitStateDir, remote.rawBaseUrl) + ? createOwnedReplayStateDir() + : undefined; return { paths: ownedStateDir?.paths ?? resolveDaemonPaths(explicitStateDir), + daemonLifecycle, + retireIncompatibleContender: true, transportPreference: transport.preference, serverMode: transport.serverMode, ownedStateDir, @@ -103,6 +117,21 @@ export function resolveClientSettings( }; } +/** + * The settings of a local daemon with no request behind them, for the caller that ensures one. It + * reads no remote base URL: a remote connection has no local daemon to ensure. + */ +export function resolveLocalDaemonSettings(options: EnsureDaemonOptions): DaemonClientSettings { + const transport = resolveTransportClientSettings(options, undefined); + return { + paths: resolveDaemonPaths(options.stateDir ?? process.env.AGENT_DEVICE_STATE_DIR), + daemonLifecycle: 'managed', + retireIncompatibleContender: false, + transportPreference: transport.preference, + serverMode: transport.serverMode, + }; +} + function resolveExplicitStateDir(req: Omit): string | undefined { return req.flags?.stateDir ?? process.env.AGENT_DEVICE_STATE_DIR; } @@ -123,10 +152,10 @@ function resolveRemoteClientSettings( } function resolveTransportClientSettings( - req: Omit, + flags: Partial> | undefined, remoteBaseUrl: string | undefined, ): { preference: DaemonTransportPreference; serverMode: DaemonServerMode } { - const rawTransport = req.flags?.daemonTransport ?? process.env.AGENT_DEVICE_DAEMON_TRANSPORT; + const rawTransport = flags?.daemonTransport ?? process.env.AGENT_DEVICE_DAEMON_TRANSPORT; const preference = resolveDaemonTransportPreference(rawTransport); if (remoteBaseUrl && preference === 'socket') { throw new AppError( @@ -136,7 +165,7 @@ function resolveTransportClientSettings( ); } const rawServerMode = - req.flags?.daemonServerMode ?? + flags?.daemonServerMode ?? process.env.AGENT_DEVICE_DAEMON_SERVER_MODE ?? (rawTransport === 'dual' ? 'dual' : undefined); return { @@ -153,7 +182,7 @@ function shouldUseOwnedReplayStateDir( return isOneShotReplayCommand(req.command) && !explicitStateDir && !rawRemoteBaseUrl; } -export async function ensureDaemon(settings: DaemonClientSettings): Promise { +export async function ensureManagedDaemon(settings: DaemonClientSettings): Promise { if (settings.remoteBaseUrl) { return await ensureRemoteDaemon(settings); } @@ -191,10 +220,26 @@ async function ensureRemoteDaemon(settings: DaemonClientSettings): Promise { + options: { + deadline?: number; + waitForLiveStartup?: boolean; + /** Tell a live daemon on another transport from one that answers nowhere. */ + refineUnreachable?: boolean; + } = {}, +): Promise { const { deadline } = options; const inspection = inspectProcessLock(settings.paths.lockPath); if (inspection.state === 'unproven') throw daemonRegistrationUnprovenError(settings, inspection); @@ -209,11 +254,44 @@ async function readReusableLocalDaemon( canReachReusableDaemon(existing, settings.transportPreference, deadline), onAnyAdvertisedTransport: () => canReachReusableDaemon(existing, 'auto', deadline), }); + return { info: existing, decision: await refineReplacement(existing, decision, options) }; +} + +/** + * Tells apart what the ladder lumps together, for callers that must not kill a daemon on a guess: a + * registration whose process is gone is unreachable whatever its version, and one that answers on + * another transport is alive. + */ +async function refineReplacement( + info: DaemonInfo, + decision: DaemonTakeoverDecision, + options: { deadline?: number; refineUnreachable?: boolean }, +): Promise { + if (!options.refineUnreachable || decision.kind !== 'replace') return decision; + if (!isProcessAlive(info.pid)) return { ...decision, cause: 'unreachable' }; + if ( + decision.cause === 'unreachable' && + (await canReachReusableDaemon(info, 'auto', options.deadline)) + ) { + return { ...decision, cause: 'transport' }; + } + return decision; +} + +async function readReusableLocalDaemon( + settings: DaemonClientSettings, + options: { deadline?: number; waitForLiveStartup?: boolean } = {}, +): Promise { + const { deadline } = options; + const observed = await observeLocalDaemon(settings, options); + if (!observed) return null; + const { info: existing, decision } = observed; if (decision.kind === 'reuse') return existing; if (decision.kind === 'refuseNewer') { throw newerDaemonRefusedError(existing, decision, settings.paths.baseDir); } + if (!settings.retireIncompatibleContender) throw daemonContendedError(); if (remainingStartupBudget(deadline) < MINIMUM_DAEMON_TAKEOVER_BUDGET_MS) return null; emitDaemonTakeoverNotice(existing, decision.reason, settings.paths.baseDir); await retireDaemonForTakeover(existing, settings.paths); @@ -254,11 +332,15 @@ function daemonRegistrationUnprovenError( }); } -async function retireDaemonForTakeover(existing: DaemonInfo, paths: DaemonPaths): Promise { +export async function retireDaemonForTakeover( + existing: DaemonInfo, + paths: DaemonPaths, + mode: 'graceful' | 'force' = 'graceful', +): Promise { const retirement = await stopAndRetireDaemon({ paths: paths, observed: { pid: existing.pid, startTime: existing.processStartTime ?? null }, - mode: 'graceful', + mode, }); if (retirement.status === 'retained') { throw new AppError('COMMAND_FAILED', 'Daemon replacement could not be confirmed.', { @@ -304,7 +386,10 @@ async function canReachReusableDaemon( * ADR 0029: a caller that names a daemon policy must not silently use a daemon that enforces a * different one (or none). A caller that names no policy uses whatever the daemon enforces. */ -async function assertDaemonPolicyMatches(existing: DaemonInfo, stateDir: string): Promise { +export async function assertDaemonPolicyMatches( + existing: DaemonInfo, + stateDir: string, +): Promise { if (!process.env.AGENT_DEVICE_DAEMON_POLICY?.trim()) return; const { loadDaemonPolicy } = await import('@agent-device/daemon-contracts/daemon-policy-file'); const expected = loadDaemonPolicy(process.env)?.digest; @@ -316,7 +401,7 @@ async function assertDaemonPolicyMatches(existing: DaemonInfo, stateDir: string) reason: 'DAEMON_POLICY_MISMATCH', expectedPolicyDigest: expected, daemonPolicyDigest: existing.policyDigest ?? null, - hint: `Stop the running daemon (agent-device daemon stop --state-dir ${shellQuoteIfNeeded(stateDir)}), then retry so a daemon starts with this policy.`, + hint: `Stop the running daemon (${daemonStopCommand(stateDir)}), then retry so a daemon starts with this policy.`, }, ); } @@ -347,7 +432,7 @@ function newerDaemonRefusedError( daemonPid: info.pid, daemonVersion, clientVersion, - hint: `Use the agent-device v${daemonVersion} CLI that started it, or stop it deliberately: agent-device daemon stop --state-dir ${shellQuoteIfNeeded(stateDir)}`, + hint: `Use the agent-device v${daemonVersion} CLI that started it, or stop it deliberately: ${daemonStopCommand(stateDir)}`, }, ); } @@ -369,7 +454,7 @@ type FailedDaemonStartup = { retry: boolean; }; -async function startLocalDaemon(settings: DaemonClientSettings): Promise { +export async function startLocalDaemon(settings: DaemonClientSettings): Promise { const deadline = Date.now() + DAEMON_STARTUP_TIMEOUT_MS; const cleanupResults: DaemonRetirementResult[] = []; let failure: FailedDaemonStartup | undefined; diff --git a/src/daemon-client/daemon-client-quiesce.ts b/src/daemon-client/daemon-client-quiesce.ts new file mode 100644 index 0000000000..7f35e3eb33 --- /dev/null +++ b/src/daemon-client/daemon-client-quiesce.ts @@ -0,0 +1,74 @@ +import { + DAEMON_QUIESCE_REASONS, + DAEMON_SHUTTING_DOWN_REASON, + buildDaemonQuiesceRequest, + readDaemonQuiesceCensus, +} from '@agent-device/daemon-contracts/daemon-resolution'; +import { createRequestId } from '@agent-device/host-kit/diagnostics'; +import { AppError, throwDaemonError } from '@agent-device/kernel/errors'; +import type { DaemonInfo } from './daemon-client-metadata.ts'; +import type { DaemonClientSettings } from './daemon-client-lifecycle.ts'; +import { isDaemonTransportUnavailableError, sendRequest } from './daemon-client-transport.ts'; +import { daemonBusyError } from './daemon-lifecycle-errors.ts'; + +const QUIESCE_TIMEOUT_MS = 10_000; + +// A refused or reset connection: the daemon left, is leaving, or was never there. Nothing was +// accepted, so judging again is safe. +function isConnectionFailure(error: unknown): boolean { + return error instanceof AppError && error.details?.reason === 'daemon_transport_failure'; +} + +/** + * `draining`: the daemon accepted and is leaving. `gone`: the daemon the request was addressed to + * is no longer there to answer, because it left, is leaving already, or was replaced. + */ +export type QuiesceOutcome = 'draining' | 'gone'; + +/** + * Asks the daemon to leave, fenced to the instance `info` names. A daemon that holds work refuses + * with `daemon-busy`, and that refusal is the answer: nothing here signals it. A timeout is not + * recovered here either; the caller decides what an unresponsive daemon deserves. + */ +export async function requestDaemonQuiesce( + info: DaemonInfo, + settings: DaemonClientSettings, +): Promise { + const request = buildDaemonQuiesceRequest({ + token: info.token, + fence: { pid: info.pid, processStartTime: info.processStartTime ?? null }, + requestId: createRequestId(), + }); + let response; + try { + response = await sendRequest( + info, + request, + // Whichever transport the daemon serves: a client that cannot reach it on its own preference + // is exactly the one that has to ask it to leave. + 'auto', + settings.paths, + QUIESCE_TIMEOUT_MS, + { timeoutRecovery: 'none' }, + ); + } catch (error) { + if (isDaemonTransportUnavailableError(error) || isConnectionFailure(error)) return 'gone'; + throw error; + } + if (response.ok) return 'draining'; + const reason = response.error.details?.reason; + if (reason === DAEMON_QUIESCE_REASONS.busy) { + throw daemonBusyError( + info, + settings.paths.baseDir, + readDaemonQuiesceCensus(response.error.details), + ); + } + if ( + reason === DAEMON_QUIESCE_REASONS.instanceMismatch || + reason === DAEMON_SHUTTING_DOWN_REASON + ) { + return 'gone'; + } + return throwDaemonError(response.error); +} diff --git a/src/daemon-client/daemon-client-timeout.ts b/src/daemon-client/daemon-client-timeout.ts index ffbf596c34..4b33a04b40 100644 --- a/src/daemon-client/daemon-client-timeout.ts +++ b/src/daemon-client/daemon-client-timeout.ts @@ -23,6 +23,11 @@ export async function handleRequestTimeout( info: DaemonInfo; statePaths: DaemonPaths; remote: boolean; + /** + * Whether this caller may retire a daemon that answers nothing. An attach client never may: the + * daemon is not its to kill, and `ensureDaemon` is where an unresponsive one is replaced. + */ + retireUnresponsive: boolean; timeoutMs: number; requestId: string | undefined; command: string | undefined; @@ -32,8 +37,18 @@ export async function handleRequestTimeout( action?: string; }>, ): Promise { - const { info, statePaths, remote, timeoutMs, requestId, command, platform, session, action } = - params; + const { + info, + statePaths, + remote, + retireUnresponsive, + timeoutMs, + requestId, + command, + platform, + session, + action, + } = params; // The command's declared policy (`timeoutPolicy.onTimeout`, ADR 0008) decides whether a local // timed-out request is RESET-ELIGIBLE; the liveness probe decides whether the eligibility runs. // A remote client cannot reach the host's process table or the daemon's state dir, so a remote @@ -51,16 +66,14 @@ export async function handleRequestTimeout( }) : undefined; const unresponsive = probeAnswered === false; - let retirement: DaemonRetirementResult | undefined; - if (unresponsive) { - const { stopAndRetireDaemon } = - await import('@agent-device/daemon-contracts/daemon-registration-owner'); - retirement = await stopAndRetireDaemon({ - paths: statePaths, - observed: { pid: info.pid, startTime: info.processStartTime ?? null }, - mode: 'force', - }); - } + const mayRetire = unresponsive && retireUnresponsive; + // Loaded only here: a timeout is the one place that needs it, and the CLI's eager closure must + // not pay for it. + const stopCommand = + unresponsive && !retireUnresponsive + ? (await import('./daemon-lifecycle-errors.ts')).daemonStopCommand(statePaths.baseDir) + : undefined; + const retirement = mayRetire ? await forceRetireDaemon(info, statePaths) : undefined; const resetDaemon = retirement?.status === 'retired'; const preserved = retirement?.status === 'retained' && retirement.termination?.status !== 'exited'; @@ -74,7 +87,7 @@ export async function handleRequestTimeout( daemonPidReset: resetDaemon ? info.pid : undefined, daemonPidForceKilled: resetDaemon ? daemonWasForceKilled(retirement) : undefined, daemonRetirement: retirement, - daemonPreservedAfterTimeout: preserved || (!remote && !unresponsive), + daemonPreservedAfterTimeout: preserved || (!remote && !mayRetire), daemonLivenessProbeAnswered: probeAnswered, daemonBaseUrl: info.baseUrl, }, @@ -90,6 +103,7 @@ export async function handleRequestTimeout( : resolveRequestTimeoutHint({ remote, resetDaemon: resetDaemon, + stopCommand, command, applePlatformDeclared: isAffirmativelyApplePlatform(platform), session, @@ -98,6 +112,19 @@ export async function handleRequestTimeout( }); } +async function forceRetireDaemon( + info: DaemonInfo, + statePaths: DaemonPaths, +): Promise { + const { stopAndRetireDaemon } = + await import('@agent-device/daemon-contracts/daemon-registration-owner'); + return await stopAndRetireDaemon({ + paths: statePaths, + observed: { pid: info.pid, startTime: info.processStartTime ?? null }, + mode: 'force', + }); +} + // Whether a timed-out request is eligible to tear down the local daemon is declared on the // command's descriptor (ADR 0008, `timeoutPolicy.onTimeout`): read-only capture/polling commands // preserve the daemon so sessions survive and evidence commands still work; everything else is @@ -132,6 +159,8 @@ function isAffirmativelyApplePlatform(platform: PlatformSelector | undefined): b export function resolveRequestTimeoutHint(params: { remote: boolean; resetDaemon: boolean; + /** Set when the daemon answered nothing and this caller is not allowed to retire it. */ + stopCommand?: string; command: string | undefined; /** Only ever consulted on the kept-alive branch, where an Apple hint has evidence to stand on. */ applePlatformDeclared: boolean; @@ -139,7 +168,8 @@ export function resolveRequestTimeoutHint(params: { action?: string; session?: string; }): string { - const { remote, resetDaemon, command, applePlatformDeclared, session, action } = params; + const { remote, resetDaemon, stopCommand, command, applePlatformDeclared, session, action } = + params; // A daemon that survives this client window may still be exporting a `record stop` that ran out // of time — a stop still queued for the device lock is dropped before any export starts, hence // "may" — and a finished file stays retrievable by asking again. This holds whether the daemon @@ -158,6 +188,9 @@ export function resolveRequestTimeoutHint(params: { // are not proven stopped from here, so the hint claims nothing about them. return 'Retry with --debug and check daemon diagnostics logs. The daemon did not answer the liveness probe and was reset after the timeout.'; } + if (stopCommand) { + return `Retry with --debug and check daemon diagnostics logs. The daemon did not answer the liveness probe, and this request does not retire a daemon it did not start. Stop it with ${stopCommand}, then retry. SDK clients call ensureDaemon() to start a fresh one.`; + } const iosPrepareHint = applePlatformDeclared && command === PUBLIC_COMMANDS.snapshot ? ' If this was the first Apple-platform snapshot on the device, run agent-device prepare ios-runner with the same --platform before snapshot/test so runner startup is handled explicitly.' diff --git a/src/daemon-client/daemon-client-transport.ts b/src/daemon-client/daemon-client-transport.ts index ff146929b5..a9a879287f 100644 --- a/src/daemon-client/daemon-client-transport.ts +++ b/src/daemon-client/daemon-client-transport.ts @@ -37,6 +37,12 @@ type SendRequestOptions = { * canceled-request error. An abort is never a timeout: it never reaches `handleRequestTimeout`. */ signal?: AbortSignal; + /** + * What a timed-out request may do to the daemon. `retire-unresponsive`, the default, lets the + * liveness probe retire a daemon that answers nothing; `none` leaves every daemon alone, which is + * how an attach client and the quiesce request itself time out. + */ + timeoutRecovery?: 'retire-unresponsive' | 'none'; }; /** @@ -384,6 +390,7 @@ export async function sendRequest( throw await handleRequestTimeout({ info, statePaths, + retireUnresponsive: options.timeoutRecovery !== 'none', ...timeoutRequestContext(req, isRemoteDaemon(info), expiredBudget), }); } @@ -687,7 +694,10 @@ function requestTimeoutError(timeoutMs: number, requestId: string | undefined): // The fields a timed-out request is described by, read once so a socket and an HTTP timeout cannot // describe the same request differently. -type TimeoutRequestFields = Omit[0], 'info' | 'statePaths'>; +type TimeoutRequestFields = Omit< + Parameters[0], + 'info' | 'statePaths' | 'retireUnresponsive' +>; function timeoutRequestContext( req: DaemonWireRequest, diff --git a/src/daemon-client/daemon-client.ts b/src/daemon-client/daemon-client.ts index b8e103c0b6..d72826b882 100644 --- a/src/daemon-client/daemon-client.ts +++ b/src/daemon-client/daemon-client.ts @@ -30,8 +30,9 @@ export async function sendToDaemon( req: Omit, options: DaemonTransportOptions = {}, ): Promise { - const { resolveClientSettings, ensureDaemon, attachSessionAddressHints } = + const { resolveClientSettings, attachSessionAddressHints } = await import('./daemon-client-lifecycle.ts'); + const { connectToDaemon } = await import('./daemon-client-attach.ts'); const requestId = req.meta?.requestId ?? createRequestId(); const debug = Boolean(req.meta?.debug || req.flags?.verbose); // A few internal callers build DaemonRequest directly instead of using the @@ -45,6 +46,7 @@ export async function sendToDaemon( const settings = resolveClientSettings( requestWithoutAuthFlag, options.authToken ?? flagAuthToken, + options.daemonLifecycle, ); const requestTimeoutMs = resolveCommandRequestTimeoutMs( resolveCommandTimeoutPolicy(requestWithoutAuthFlag.command), @@ -55,10 +57,14 @@ export async function sendToDaemon( const cancellation = createRequestGuard({ signal: options.signal, requestId }); cancellation.refuseIfAborted(); const daemon = await cancellation.guard(async () => { - return await withDiagnosticTimer('daemon_startup', async () => await ensureDaemon(settings), { - requestId, - session: req.session, - }); + return await withDiagnosticTimer( + 'daemon_startup', + async () => await connectToDaemon(settings), + { + requestId, + session: req.session, + }, + ); }); const info = daemon.info; const preparedRemoteRequest = await cancellation.guard( @@ -103,7 +109,12 @@ export async function sendToDaemon( settings.transportPreference, settings.paths, requestTimeoutMs, - { onProgress: options.onProgress, signal: options.signal }, + { + onProgress: options.onProgress, + signal: options.signal, + timeoutRecovery: + settings.daemonLifecycle === 'attach' ? 'none' : 'retire-unresponsive', + }, ), { requestId, command: req.command }, ); diff --git a/src/daemon-client/daemon-launch-spec.ts b/src/daemon-client/daemon-launch-spec.ts index 22c30f5315..2ef78c3adc 100644 --- a/src/daemon-client/daemon-launch-spec.ts +++ b/src/daemon-client/daemon-launch-spec.ts @@ -107,10 +107,17 @@ export async function resolveLocalDaemonCodeIdentity(): Promise> { + const code = await resolveLocalDaemonCodeIdentity(); + return { + version: readVersion(), + codeOrigin: code.origin, + ...(code.origin === 'checkout' ? { codeSignature: code.codeSignature } : {}), + }; +} + +/** The command that ends a daemon deliberately; the one recovery every surface can name. */ +export function daemonStopCommand(stateDir: string): string { + return `agent-device daemon stop --state-dir ${shellQuoteIfNeeded(stateDir)}`; +} + +/** A daemon won a startup race that this caller may not settle by retiring it. */ +export const DAEMON_CONTENDED_REASON = 'daemon-contended'; + +export function daemonContendedError(): AppError { + return new AppError( + 'COMMAND_FAILED', + 'Another process started a daemon while this one was starting.', + { + reason: DAEMON_CONTENDED_REASON, + }, + ); +} + +export function daemonNotRunningError(stateDir: string, registered?: DaemonInfo): AppError { + return new AppError('COMMAND_FAILED', 'No daemon is running for this state directory.', { + reason: DAEMON_LIFECYCLE_REASONS.notRunning, + stateDir, + ...(registered ? { daemon: daemonIdentityOf(registered) } : {}), + hint: `Start one with a local agent-device command that uses the daemon (devices, for example), or with ensureDaemon({ stateDir: ${JSON.stringify(stateDir)} }) from the SDK, before attaching.${registered ? ` The registered daemon (pid ${registered.pid}) does not answer; ${daemonStopCommand(stateDir)} clears it.` : ''}`, + }); +} + +export async function daemonIncompatibleError(params: { + info: DaemonInfo; + mismatch: DaemonTakeoverCause | 'newer' | 'no-quiesce'; + stateDir: string; +}): Promise { + const { info, mismatch, stateDir } = params; + return new AppError('COMMAND_FAILED', 'The running daemon cannot serve this client.', { + reason: DAEMON_LIFECYCLE_REASONS.incompatible, + mismatch, + stateDir, + daemon: daemonIdentityOf(info), + client: await readClientIdentity(), + hint: `Stop it with ${daemonStopCommand(stateDir)}, then retry. SDK clients call ensureDaemon() to start a compatible one.`, + }); +} + +export function daemonBusyError( + info: DaemonInfo, + stateDir: string, + census: DaemonQuiesceCensus | undefined, +): AppError { + return new AppError('COMMAND_FAILED', 'The running daemon is busy and was not replaced.', { + reason: DAEMON_LIFECYCLE_REASONS.busy, + stateDir, + daemon: daemonIdentityOf(info), + ...census, + hint: census?.startingUp + ? `The daemon is still starting up. Retry in a moment; if it never finishes starting, stop it with ${daemonStopCommand(stateDir)}.` + : `Wait for its requests, sessions and leases to end and retry, or stop it deliberately with ${daemonStopCommand(stateDir)}.`, + }); +} diff --git a/src/sdk/ensure-daemon.ts b/src/sdk/ensure-daemon.ts new file mode 100644 index 0000000000..d4328459b4 --- /dev/null +++ b/src/sdk/ensure-daemon.ts @@ -0,0 +1,13 @@ +import type { DaemonConnection, EnsureDaemonOptions } from '@agent-device/contracts/client'; + +/** + * Starts the local daemon on a state directory if none is usable, and replaces an incompatible one + * only when it holds no request, session, repair session or live lease. See `DaemonLifecycle`. + * + * Loaded on first call: a client that only attaches never pays for the startup and replacement + * machinery. + */ +export async function ensureDaemon(options?: EnsureDaemonOptions): Promise { + const { ensureDaemon: ensure } = await import('../daemon-client/daemon-client-ensure.ts'); + return await ensure(options); +} diff --git a/src/sdk/index.ts b/src/sdk/index.ts index d0f0d738a9..2f6d65414a 100644 --- a/src/sdk/index.ts +++ b/src/sdk/index.ts @@ -1,4 +1,5 @@ export { createAgentDeviceClient } from '../agent-device-client.ts'; +export { ensureDaemon } from './ensure-daemon.ts'; export { createLocalArtifactAdapter } from '../io.ts'; export { AppError, @@ -14,6 +15,12 @@ export { isUnreadableCaptureContentError } from '@agent-device/contracts/android // `ReturnType` and `Awaited>`, and typed the // nodes a snapshot returns by hand. export type * from '../client/client-types.ts'; +export type { + DaemonConnection, + DaemonIdentity, + DaemonLifecycle, + EnsureDaemonOptions, +} from '@agent-device/contracts/client'; // Two results `client-types.ts` imports for its signatures without re-exporting them. export type { AgentArtifactsResult, diff --git a/test/integration/installed-package-metro.test.ts b/test/integration/installed-package-metro.test.ts index 57e330d08f..d6e917f45a 100644 --- a/test/integration/installed-package-metro.test.ts +++ b/test/integration/installed-package-metro.test.ts @@ -382,6 +382,7 @@ test('installed package exposes Node APIs and packaged companion tunnel entrypoi 'centerOfRect', 'createAgentDeviceClient', 'createLocalArtifactAdapter', + 'ensureDaemon', 'isAgentDeviceError', 'isUnreadableCaptureContentError', 'normalizeAgentDeviceError', diff --git a/website/docs/docs/client-api.md b/website/docs/docs/client-api.md index d9d506a74a..c2b06719f5 100644 --- a/website/docs/docs/client-api.md +++ b/website/docs/docs/client-api.md @@ -31,6 +31,7 @@ Entry points for driving devices from Node.js: - `agent-device` - `createAgentDeviceClient(options?)` + - `ensureDaemon(options?)` - `createLocalArtifactAdapter(options?)` - `AppError`, `isAgentDeviceError(error)`, `normalizeAgentDeviceError(error)` - `isUnreadableCaptureContentError(error)`: true when an Android helper capture found no readable @@ -39,6 +40,7 @@ Entry points for driving devices from Node.js: platform, backend, and mechanism failure. - `centerOfRect(rect)` - types: `AgentDeviceClient`, `AgentDeviceClientConfig`, `AgentDeviceDevice` + - types: `DaemonLifecycle`, `EnsureDaemonOptions`, `DaemonConnection`, `DaemonIdentity` - types: every option and result type of the client's methods, among them `AppOpenOptions`, `AppOpenResult`, `CaptureSnapshotOptions`, `CaptureSnapshotResult`, `CaptureScreenshotResult`, `PressOptions` @@ -84,6 +86,41 @@ if ('cloudArtifacts' in result) { } ``` +## Daemon lifecycle + +A client talks to one local daemon per state directory. `daemonLifecycle` says who owns that daemon. + +| Mode | What the client does with the daemon | +| --- | --- | +| `'managed'` (default) | Starts a missing daemon, replaces one that does not match the client, and retires an unresponsive one after a timeout. | +| `'attach'` | Connects to the running daemon. Never starts, replaces or stops one, even after a timeout. | + +`'managed'` stays the default until 0.22. Use `'attach'` when your code, or another process, owns the daemon, and call `ensureDaemon()` to start or replace it deliberately: + +```ts +import { createAgentDeviceClient, ensureDaemon } from 'agent-device'; + +const stateDir = './.my-app/agent-device-state'; + +await ensureDaemon({ + stateDir, + onStart: (reason, previous) => console.log(`started a daemon: ${reason}`, previous), +}); +const client = createAgentDeviceClient({ stateDir, daemonLifecycle: 'attach' }); +``` + +`ensureDaemon()` starts a daemon when none runs. It replaces a daemon that does not match the client only if that daemon holds no work: no request in flight, no session, no repair session and no live lease. A resumable upload still streaming and a host human-control hold that names no lease are not counted, so a replacement can end them. Otherwise it fails and leaves the daemon alone. Two cases retire a daemon without that proof, because it cannot be asked: a registered daemon that answers nothing on any transport, and one that times out on the quiesce request and then fails repeated liveness probes. It returns `{ stateDir, daemon, started }`; `daemon` holds the pid, version and code identity, and no credentials. `onStart` runs when the call started a daemon: `'incompatible'` when it replaced a daemon that agreed to leave, with that daemon as `previous`, and `'missing'` otherwise. A `'missing'` start that also has a `previous` replaced a registered daemon that answered nothing. + +These failures carry a typed `error.details.reason`: + +| Reason | Meaning | +| --- | --- | +| `daemon-not-running` | An `'attach'` client found no daemon to connect to. Call `ensureDaemon()`. | +| `daemon-incompatible` | The running daemon does not match the client. `details.daemon` and `details.client` hold both identities. Stop it with `agent-device daemon stop --state-dir `, then call `ensureDaemon()`. A daemon from a release that predates quiescing cannot be asked to leave, so `ensureDaemon()` reports it the same way. | +| `daemon-busy` | `ensureDaemon()` found an incompatible daemon that still holds work. `details` counts `inFlightRequests`, `sessions`, `repairSessions` and `leases`. Wait for them to end and retry. | + +`ensureDaemon()` only ever manages the local daemon on `stateDir`. A client with `daemonBaseUrl` never starts or stops a local daemon in either mode, and `daemonLifecycle` does not change `daemonTransport` or `daemonServerMode`. + ## Device cloud sessions You drive Limrun, BrowserStack, AWS Device Farm, and TestMu AI devices with the same typed client methods. Use the CLI `connect` flow to persist connection state locally. Pass provider settings in the client config when your Node.js integration already owns credentials and provider selectors.