Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 62 additions & 17 deletions test/integration/command-coverage/declarations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,10 @@ const C = PUBLIC_COMMANDS;
* load time (`projectCoverage`), so adding a public command is one edit here rather than six
* parallel edits in six manifests.
*
* A gap row is exactly one of two kinds: a `known-gap` row is owed work and its builder pins the
* platform's open tracking issue; an `accepted-gap` row states a `reason` for why no evidence lane
* will cover it and has no tracking issue. `assertCoverageGapRowsAreOwned` enforces both.
*
* `satisfies Record<PublicCommand, CommandCoverageDeclaration>` is the exhaustiveness check:
* a command added to the catalog without a row here fails type-checking, and each platform's
* coverage smoke test fails the same omission at runtime.
Expand All @@ -68,8 +72,9 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'artifacts lists a daemon-tracked artifact produced during a web session',
'daemon artifact listing round-trips a tracked artifact for a web-backed session',
),
linux: linux.gap(
'No Linux live command creates a downloadable daemon artifact for inventory yet',
linux: linux.accepted(
'No Linux live command creates a downloadable daemon artifact for inventory',
'The Linux lanes run the local CLI, where command outputs are caller-owned local paths; request finalization inventories a response artifact only when the request carries a client artifact path (remote transport) or it is the sparse-snapshot fallback screenshot, which no Linux lane observes, so a Linux lane cannot produce a downloadable inventory entry without a local-transport product decision.',
),
},
[C.devices]: {
Expand Down Expand Up @@ -135,7 +140,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'smoke:inventory-install',
'doctor discovers the installed fixture app',
),
macos: macos.gap('No command-specific macOS doctor evidence exists yet'),
macos: macos.accepted(
Comment thread
thymikee marked this conversation as resolved.
'No command-specific macOS doctor evidence exists',
'The doctor checks are daemon-owned host diagnostics — version, state dir, inventory probes, Metro — pinned platform-neutrally by the daemon doctor tests, and no macOS lane invokes the command: neither the System Settings replay nor the desktop and recording provider scenarios carry a doctor step.',
),
tvos: tvos.gap('No tvOS-specific doctor command evidence exists yet'),
web: web.contract(
'packages/platform-web/src/__tests__/doctor.test.ts',
Expand Down Expand Up @@ -204,14 +212,20 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'full:device-lifecycle',
'shutdown succeeds and inventory reports the selected simulator stopped',
),
macos: macos.gap('No command-specific macOS shutdown evidence exists yet'),
macos: macos.accepted(
'No command-specific macOS shutdown evidence exists',
'The Apple owner declares shutdownTarget unavailable on the macOS leaf (unsupported-platform-leaf): agent-device never powers off the host Mac it runs on, so the command refuses at admission and no lane will observe a successful macOS shutdown.',
),
tvos: tvos.gap('No tvOS-specific shutdown command evidence exists yet'),
web: web.contract(
'packages/platform-web/src/runtime.test.ts',
'boot and shutdown report the runtime-owned unavailable readiness fact',
'the web runtime fact rejects device shutdown',
),
linux: linux.gap('No Linux-specific shutdown command evidence exists yet'),
linux: linux.accepted(
'No Linux-specific shutdown command evidence exists',
'The Linux owner declares shutdownTarget unavailable (unsupported-platform-leaf): the Linux desktop is the host machine itself and agent-device never powers off the machine it runs on, so the command refuses at admission and no lane will observe a successful shutdown.',
),
},
[C.appState]: {
androidEmulator: androidEmulator.live(
Expand Down Expand Up @@ -280,7 +294,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'logs reports the runtime-owned unavailable app-log facts',
'the web runtime fact rejects native app-log commands',
),
linux: linux.gap('No Linux-specific app-log command evidence exists yet'),
linux: linux.accepted(
'No Linux-specific app-log command evidence exists',
'The Linux runtime declares the whole app-log family unavailable (unsupported-platform-leaf) by owner decision: a Linux desktop session has no runtime-owned app log stream, so the command refuses at admission and no lane will observe a Linux app-log stream.',
),
},
[C.events]: {
androidEmulator: androidEmulator.live(
Expand All @@ -291,7 +308,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'full:observability-artifacts',
'timeline contains commands from this session',
),
macos: macos.gap('No command-specific macOS event timeline evidence exists yet'),
macos: macos.accepted(
'No command-specific macOS event timeline evidence exists',
'The events timeline is daemon-owned session bookkeeping, pinned by request-router-events.test.ts for a daemon-backed web session, and no macOS lane invokes the command: neither the System Settings replay nor the desktop and recording provider scenarios carry an events step.',
),
tvos: tvos.gap('No tvOS-specific session-event command evidence exists yet'),
web: web.contract(
'packages/daemon/src/server/__tests__/request-router-events.test.ts',
Expand Down Expand Up @@ -388,8 +408,9 @@ const COMMAND_COVERAGE_DECLARATIONS = {
),
macos: macos.gap('No command-specific macOS test-suite evidence exists yet'),
tvos: tvos.gap('No tvOS-specific test-suite command evidence exists yet'),
web: web.gap(
"Web test-suite execution has no executable web evidence: ReplayTestPlatform = Exclude<PlatformSelector, 'web'> structurally excludes web from the declared-platform filter, so `test --platform web` can never select a script (proven by a regression test in session-command-replay.test.ts) — that is evidence of what the command cannot do, not that it works on web",
web: web.accepted(
'No command-specific web test-suite evidence exists',
"ReplayTestPlatform = Exclude<PlatformSelector, 'web'> structurally excludes web from the declared-platform filter, so `test --platform web` can never select a script (proven by a regression test in session-command-replay.test.ts); there is no successful web test-suite execution for any lane to observe, and adding web replay-test support is a product decision, not scheduled work.",
),
linux: linux.commandEvidenceLive(
'the command-evidence lane runs a dedicated Linux script as a test suite',
Expand Down Expand Up @@ -474,7 +495,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'install and reinstall share the runtime-owned unavailable deploy fact',
'the web runtime fact rejects native app installation',
),
linux: linux.gap('No Linux-specific application installation command evidence exists yet'),
linux: linux.accepted(
'No Linux-specific application installation command evidence exists',
'The Linux owner declares the app-deployment family unavailable (unsupported-platform-leaf): a Linux desktop app is launched by name through the desktop interactor and there is no installable package model, so the command refuses at admission and no lane will observe a Linux install.',
),
},
[C.reinstall]: {
androidEmulator: androidEmulator.contract(
Expand All @@ -500,7 +524,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'install and reinstall share the runtime-owned unavailable deploy fact',
'the web runtime fact rejects native app reinstallation',
),
linux: linux.gap('No Linux-specific application reinstallation command evidence exists yet'),
linux: linux.accepted(
'No Linux-specific application reinstallation command evidence exists',
'Reinstall reuses the app-deployment family the Linux runtime declares unavailable (unsupported-platform-leaf): there is no installable package model for the desktop interactor to reinstall, so the command refuses at admission and no lane will observe a Linux reinstall.',
),
},
[C.push]: {
androidEmulator: androidEmulator.live(
Expand All @@ -527,7 +554,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'push reports the runtime-owned unavailable readiness and push facts',
'the web runtime fact rejects native push notification delivery',
),
linux: linux.gap('No Linux-specific push delivery command evidence exists yet'),
linux: linux.accepted(
'No Linux-specific push delivery command evidence exists',
'The Linux runtime declares push delivery unavailable (unsupported-platform-leaf): the desktop interactor has no OS push channel to deliver through, so the command refuses at admission and no lane will observe a Linux push delivery.',
),
},
[C.triggerAppEvent]: {
androidEmulator: androidEmulator.contract(
Expand Down Expand Up @@ -763,7 +793,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'full:observability-artifacts',
'Release fixture returns typed detected=false and dismissed=false overlay state',
),
macos: macos.gap('No command-specific macOS React Native inspection evidence exists yet'),
macos: macos.accepted(
'No command-specific macOS React Native inspection evidence exists',
"react-native dismiss-overlay only answers meaningfully on a surface that carries a React Native overlay, the repository's React Native overlay evidence runs on iOS simulators and Android emulators, and no macOS lane invokes the command: neither the System Settings replay nor the desktop and recording provider scenarios carry a react-native step.",
),
tvos: tvos.gap('No tvOS-specific React Native inspection command evidence exists yet'),
// R61: no owner fact refuses this command on a browser — its whole device work is one bound
// `tapPoint` the web target admits — so it now runs and reports that no overlay is present.
Expand Down Expand Up @@ -800,7 +833,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'start web recording',
'web recording starts and stops through the scoped provider',
),
linux: linux.gap('No Linux-specific recording command evidence exists yet'),
linux: linux.accepted(
'No Linux-specific recording command evidence exists',
'The Linux runtime declares the screen-recording family unavailable (unsupported-platform-leaf): there is no Linux capture pipeline, so the command refuses at admission and no lane will produce a playable Linux recording.',
),
},
[C.trace]: {
androidEmulator: androidEmulator.live(
Expand All @@ -811,14 +847,20 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'full:observability-artifacts',
'typed start/stop lifecycle retains the requested path and captures non-empty diagnostics',
),
macos: macos.gap('No command-specific macOS trace evidence exists yet'),
macos: macos.accepted(
'No command-specific macOS trace evidence exists',
'trace never binds a device runtime: handleTraceCommand carries no bindDevice or inspectFacts call, its start/stop bookkeeping is pinned by the daemon trace-runtime tests, and no macOS lane invokes the command: neither the System Settings replay nor the desktop and recording provider scenarios carry a trace step.',
),
tvos: tvos.gap('No tvOS-specific trace command evidence exists yet'),
web: web.contract(
'packages/daemon/src/sessions/handlers/__tests__/trace-runtime.test.ts',
'starts and stops one trace through the session-owned trace slot on a web session',
'session-scoped trace start/stop bookkeeping works the same for a web session as any other platform',
),
linux: linux.gap('No Linux-specific trace command evidence exists yet'),
linux: linux.accepted(
'No Linux-specific trace command evidence exists',
'trace never binds a device runtime: handleTraceCommand carries no bindDevice or inspectFacts call and its start/stop bookkeeping is pinned by the daemon trace-runtime tests, and no Linux lane invokes the command: neither the replay script nor the command-evidence lane carries a trace step.',
),
},
[C.find]: {
androidEmulator: androidEmulator.live(
Expand Down Expand Up @@ -1386,7 +1428,10 @@ const COMMAND_COVERAGE_DECLARATIONS = {
'install-from-source reports the runtime-owned unavailable materialize and deploy facts',
'the web runtime fact rejects source-based app installation',
),
linux: linux.gap('No Linux-specific source-install command evidence exists yet'),
linux: linux.accepted(
'No Linux-specific source-install command evidence exists',
'Source installation runs through the app-deployment family the Linux runtime declares unavailable (unsupported-platform-leaf): the desktop has no materialized-app install route to source from, so the command refuses at admission and no lane will observe a Linux source install.',
),
},
} satisfies Record<PublicCommand, CommandCoverageDeclaration>;

Expand Down
32 changes: 20 additions & 12 deletions test/integration/command-coverage/entries.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,9 @@ import {
} from '../macos-e2e/live-scenarios.ts';
import {
LINUX_COMMAND_EVIDENCE,
LINUX_COVERAGE_GAP_ISSUE,
LINUX_REPLAY_EVIDENCE,
MACOS_COVERAGE_GAP_ISSUE,
TVOS_COVERAGE_GAP_ISSUE,
WEB_COVERAGE_GAP_ISSUE,
WEB_SMOKE_EVIDENCE,
type RepositoryEvidence,
} from './evidence.ts';
Expand Down Expand Up @@ -58,6 +56,11 @@ export type MacOsPlatformCoverageEntry =
assertion: string;
level: 'known-gap';
trackingIssue: number;
}
| {
assertion: string;
level: 'accepted-gap';
reason: string;
};

export type TvOsPlatformCoverageEntry =
Expand Down Expand Up @@ -91,8 +94,8 @@ export type WebPlatformCoverageEntry =
}
| {
assertion: string;
level: 'known-gap';
trackingIssue: number;
level: 'accepted-gap';
reason: string;
};

export type LinuxPlatformCoverageEntry =
Expand All @@ -103,8 +106,8 @@ export type LinuxPlatformCoverageEntry =
}
| {
assertion: string;
level: 'known-gap';
trackingIssue: number;
level: 'accepted-gap';
reason: string;
};

/** One command's coverage judgment on every platform that classifies the public catalog. */
Expand Down Expand Up @@ -170,6 +173,11 @@ export const macos = {
level: 'known-gap',
trackingIssue: MACOS_COVERAGE_GAP_ISSUE,
}),
accepted: (assertion: string, reason: string): MacOsPlatformCoverageEntry => ({
assertion,
level: 'accepted-gap',
reason,
}),
};

export const tvos = {
Expand Down Expand Up @@ -207,10 +215,10 @@ export const web = {
level: 'command-contract',
owner: { path, test },
}),
gap: (assertion: string): WebPlatformCoverageEntry => ({
accepted: (assertion: string, reason: string): WebPlatformCoverageEntry => ({
assertion,
level: 'known-gap',
trackingIssue: WEB_COVERAGE_GAP_ISSUE,
level: 'accepted-gap',
reason,
}),
};

Expand All @@ -230,10 +238,10 @@ export const linux = {
level: 'command-contract',
owner: { path, test },
}),
gap: (assertion: string): LinuxPlatformCoverageEntry => ({
accepted: (assertion: string, reason: string): LinuxPlatformCoverageEntry => ({
assertion,
level: 'known-gap',
trackingIssue: LINUX_COVERAGE_GAP_ISSUE,
level: 'accepted-gap',
reason,
}),
};

Expand Down
5 changes: 2 additions & 3 deletions test/integration/command-coverage/evidence.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,10 @@ export type RepositoryEvidence = {

const C = PUBLIC_COMMANDS;

// Tracking issues for the known-gap rows each platform still carries.
// The tracking issues that own the `known-gap` rows each platform still carries. An
// accepted-gap row states a `reason` instead of citing one of these.
export const MACOS_COVERAGE_GAP_ISSUE = 1916;
export const TVOS_COVERAGE_GAP_ISSUE = 1914;
export const WEB_COVERAGE_GAP_ISSUE = 1900;
export const LINUX_COVERAGE_GAP_ISSUE = 1915;

// Android contract evidence that only the coverage declarations cite. Evidence shared with a
// production test module keeps living next to that module and is imported by declarations.ts.
Expand Down
12 changes: 4 additions & 8 deletions test/integration/linux-e2e/coverage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,17 @@ import type { PublicCommand } from '../command-coverage/entries.ts';
import { LINUX_COMMAND_EVIDENCE, LINUX_REPLAY_EVIDENCE } from '../command-coverage/evidence.ts';
import { buildCoverageClassificationSummary } from '../support/coverage-classification.ts';

export {
LINUX_COMMAND_EVIDENCE,
LINUX_COVERAGE_GAP_ISSUE,
LINUX_REPLAY_EVIDENCE,
} from '../command-coverage/evidence.ts';
export { LINUX_COMMAND_EVIDENCE, LINUX_REPLAY_EVIDENCE } from '../command-coverage/evidence.ts';

/**
* One primary, observable owner for every public command on the Linux desktop.
*
* Live rows cite either the existing Linux replay or the separate command-evidence
* lane. The existing replay scope stays unchanged. Contract rows cite the existing
* provider scenario or dedicated Linux unit/runtime evidence; they do not turn
* provider scenario or dedicated Linux runtime-fact evidence; they do not turn
* mocked provider calls into live desktop claims. Capability denials are derived
* from the owning command-descriptor matrix. Known gaps are explicit follow-up
* work, not an implicit claim that a generic command works.
* from the owning command-descriptor matrix. Every remaining gap is an accepted
* limitation naming why no lane will cover it; Linux carries no open tracking issue.
*
* Projected from the command-owned declaration table; the rows are authored there.
*/
Expand Down
37 changes: 21 additions & 16 deletions test/integration/smoke-linux-coverage.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@ import test from 'node:test';

import { parseReplayScriptDetailed } from '@agent-device/ad-script';
import { PUBLIC_COMMANDS } from '@agent-device/command-registry/catalog';
import { assertCoverageClassificationSummaryWiredToManifest } from './support/coverage-classification.ts';
import {
LINUX_COVERAGE_GAP_ISSUE,
assertCoverageClassificationSummaryWiredToManifest,
assertCoverageGapRowsAreOwned,
} from './support/coverage-classification.ts';
import {
LINUX_COMMAND_EVIDENCE,
LINUX_PLATFORM_COVERAGE,
LINUX_PLATFORM_COVERAGE_CLASSIFICATION_SUMMARY,
Expand All @@ -24,6 +26,7 @@ const publicCommands = Object.values(PUBLIC_COMMANDS).sort();

test('Linux coverage exhaustively classifies the public catalog', () => {
assert.deepEqual(Object.keys(LINUX_PLATFORM_COVERAGE).sort(), publicCommands);
assertCoverageGapRowsAreOwned('Linux', LINUX_PLATFORM_COVERAGE);

for (const command of publicCommands) {
const entry = LINUX_PLATFORM_COVERAGE[command];
Expand All @@ -32,13 +35,6 @@ test('Linux coverage exhaustively classifies the public catalog', () => {
assert.ok(entry.owner.path.trim().length > 0, `${command} needs an evidence path`);
assert.ok(entry.owner.test.trim().length > 0, `${command} needs named evidence`);
}
if (entry.level === 'known-gap') {
assert.equal(
entry.trackingIssue,
LINUX_COVERAGE_GAP_ISSUE,
`${command} has the wrong Linux gap issue`,
);
}
}
});

Expand Down Expand Up @@ -117,11 +113,20 @@ test('Linux contract claims name existing executable evidence', () => {
}
});

test('Linux known gaps use one grouped tracking issue', () => {
const gapIssues = new Set(
Object.values(LINUX_PLATFORM_COVERAGE)
.filter((entry) => entry.level === 'known-gap')
.map((entry) => entry.trackingIssue),
);
assert.deepEqual([...gapIssues], [LINUX_COVERAGE_GAP_ISSUE]);
test('Linux accepted gaps name the permanent Linux limitation behind each row', () => {
const acceptedCommands = Object.entries(LINUX_PLATFORM_COVERAGE)
.filter(([, entry]) => entry.level === 'accepted-gap')
.map(([command]) => command)
.sort();
assert.deepEqual(acceptedCommands, [
'artifacts',
'install',
'install-from-source',
'logs',
'push',
'record',
'reinstall',
'shutdown',
'trace',
]);
});
Loading
Loading