audit: report model resolution startup failures from the unified session #70648

Problem

gh aw audit misses an explicitly recorded model-resolution startup failure unless it happens to match Pi's engine-specific startup log message. The unified session already identifies the engine, role, configured model, and failure cause, but audit does not consume model.resolution.

Verified on October 20, 2026 with a native Windows development build of main at 70e15c7ff659c05c2f4466aa172bb910d28847a2 (the merge of #70389). All three requested audits completed with exit code 0 and cache_source: "full"; none produced model_unavailable or model_availability_unknown. This is an audit diagnosis gap, not a request to relax the harness's fail-closed behavior or change model pins.

Reproduction and current audit output

Built cmd/gh-aw from the clean checkout matching remote main:

go build -mod=readonly -ldflags "-s -w -X main.version=dev-70e15c7ff659" -o <evidence-dir>\gh-aw-main.exe .\cmd\gh-aw

The binary reports gh aw version dev-70e15c7ff659. Invoked that development binary directly for the gh aw audit command, leaving the installed stable extension unchanged:

<evidence-dir>\gh-aw-main.exe audit 39499460077 --repo github/gh-aw --json --no-baseline --timeout 5m --output <evidence-dir>\audits
<evidence-dir>\gh-aw-main.exe audit 39499464895 --repo github/gh-aw --json --no-baseline --timeout 5m --output <evidence-dir>\audits
<evidence-dir>\gh-aw-main.exe audit 39508150750 --repo github/gh-aw --json --no-baseline --timeout 5m --output <evidence-dir>\audits

The first two runs used branch commit b9f7f48a3baa4e58e9509232f6eaaea674a942ff. The third demonstrates that the gap persists after the merge, not just in the earlier branch runs.

Evidence in the runs' real artifacts

Each run's usage/aw_session.jsonl contains collector-projected, runner-written model.resolution events. These are the actual failed records from run 39499460077, lines 177-178:

{"type":"model.resolution","data":{"engine":"copilot","role":"mini-whoami","configured_model":"gpt-5-mini","resolved_model":null,"provider":"copilot","status":"failed","cause":"model_unavailable"},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":2}}
{"type":"model.resolution","data":{"engine":"copilot","role":"mini54-whoami","configured_model":"gpt-5.4-mini","resolved_model":null,"provider":"copilot","status":"failed","cause":"model_unavailable"},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":3}}

Run 39499464895 has this record at line 123 and no main record:

{"type":"model.resolution","data":{"engine":"copilot","role":"mini-whoami","configured_model":"gpt-5.4-mini","resolved_model":null,"provider":"copilot","status":"failed","cause":"model_unavailable"},"provenance":{"component":"model_resolution","phase":"agent","path":"agent/model-resolution.json","index":0}}

Run 39508150750 repeats the two failed records at lines 185-186. In the first and third runs, main / gpt-5.3-codex and haiku-whoami / claude-haiku-4.5 have successful status: "exact" records; they are not the failing roles.

For all three runs:

• The same run's sandbox/firewall/awf-reflect.json has models_fetch_complete: true, exactly one configured copilot endpoint, and 36 advertised model IDs. Neither gpt-5-mini nor gpt-5.4-mini is in that inventory.
• agent-stdio.log explicitly says Refusing to start Copilot with a different sub-agent model for each failed role. Relevant lines: 171/173 in 39499460077; 185 in 39499464895; 164/166 in 39508150750.
• There is no native Copilot agent session. session.collection reports zero events from the agent session/log sources. The existing 124-byte agent-session.jsonl contains only an agent.execution error summary with categories: ["model_not_supported_error"] and exitCode: 1. File existence is therefore not proof that Copilot started.
• Similarly, agent_execution.json says state: "started", but that is the execution wrapper, not proof of a native agent session.

The routed run did perform its router/classifier request. Audit records one classifier request but zero selected-model requests; do not misidentify classifier traffic as evidence that the main agent ran. Its missing main resolution record is not itself an error.

These inventories establish unavailability to these runs, not global retirement or the reason provider availability changed.

Why audit misses it

At the audited main revision:

• pkg/cli/audit_model_availability.go, lines 15 and 33-53: generateModelAvailabilityFindings requires a failed run, reads agent-stdio.log, and returns without a finding unless piModelUnavailablePattern matches the exact [gh-aw/pi-models-json] fatal: Pi model ... is not advertised ... line.
• pkg/cli/audit_report_analysis.go, lines 29-40 calls this function, but no Go file under pkg/cli currently recognizes the literal model.resolution event type.
• The existing generateSubagentModelUnavailableFindings handles the different subagent.model_unavailable inherited-model warning; it does not cover these fail-closed startup resolution records.

Consequently, engine-independent evidence is missed for Copilot here. By source inspection, the same Pi-only gate also excludes analogous Claude/Codex main-role or sub-agent resolution failures; those engines were not independently reproduced in this investigation.

The producers have already been updated: #70320 for the Copilot SDK main model, #70385 for Pi sub-agents, and #70389 for Copilot sub-agents in CLI and SDK mode. The unified session specification, T-UAS-074 defines the runner-written event and explicitly says an absent record does not imply a model was or was not resolved. Audit should consume usage/aw_session.jsonl, not grow another engine-specific log parser.

Possible solution

Read failed model.resolution events from usage/aw_session.jsonl as the primary evidence, respecting its real versioned envelope and provenance. Reuse existing bounded JSONL/header-validation and artifact-safety mechanisms where appropriate; do not create an unrelated session-parser framework.

For a fatal startup resolution failure whose recorded cause is model_unavailable, report the engine, role, and configured model, and retain the current same-run reflect confirmation:

• Exact model absent from a complete, unambiguous, matching configured provider inventory: high-severity model_unavailable, title Model Unavailable.
• Reflection missing, incomplete (models_fetch_complete: false), malformed, ambiguous, or contradictory: model_availability_unknown, title Model Availability Unverified, as today. Do not use another run's inventory or treat unknown availability as confirmed absence.

Aggregate the confirmed startup failures into one finding for these runs, with explicit model-to-role groups. Thus run 39499460077 should have one high-severity Model Unavailable finding naming both mini-whoami / gpt-5-mini and mini54-whoami / gpt-5.4-mini. If several roles fail on the same provider/model, list those roles together rather than creating duplicate findings. Preserve per-model evidence classification if some inventory checks are unverified.

Keep the current Pi startup-log match as a compatibility fallback when there are no failed model.resolution events. Do not double-report a failure from both sources, or use the legacy fallback to overwrite a different cause recorded in the session.

Preserve other resolution causes instead of calling every status: "failed" record model_unavailable. The current contract admits model_unavailable and catalog_unavailable; for example, a known alias with no usable catalogue produces catalog_unavailable, whereas an alias with no available candidate in a usable catalogue may legitimately produce model_unavailable. Report the actual recorded cause; do not invent an alias_unresolved event shape or infer cause from resolved_model: null.

Repository agents are not the cause

Since #70389, an unresolved model on a repository .github/agents file produces a warning and a failed record, but does not stop the run. A workflow-declared sub-agent can instead fail closed.

The record itself cannot distinguish these two origins. In resolveCopilotSubagentModels, lines 839-856, both paths write the same fields before branching on the in-memory declared flag. Neither declared, file origin, nor fatal/nonfatal status is persisted in the resolution payload; both also have the same provenance shape.

Use startup/continuation evidence, not the mere presence of a failed record, to decide whether it explains the run failure. A run that succeeded, or has a genuine native agent session and later fails for another reason, must not acquire a fatal-startup finding solely because an optional repository agent failed resolution. A low-severity configuration note is appropriate at most. Missing artifacts alone do not prove a startup stop; wrapper started state and router/detection traffic do not prove native-agent continuation either.

The supplied failed runs have explicit harness refusal diagnostics and no native agent session, so the startup attribution is supported. If a future run mixes repository-agent warnings and declared-sub-agent failures, the events alone cannot identify which failed role was fatal: use available startup evidence conservatively and state remaining uncertainty rather than assigning every failed role as the cause. Do not change producer fields or the spec to solve that ambiguity in this PR.

Acceptance cases

Add each case as a named test. Seed fixtures with the actual version header, resolution events, collection/continuation evidence, and reflection from the runs above; keep only necessary, sanitized artifacts under pkg/cli test fixtures. No invented event shapes or unsupported origin fields.

For the continuation negative control, a real successful session was also inspected: 39508154330, Smoke Copilot SDK Routed, at the same main SHA, contains native session.start, assistant/tool activity, and sub-agent lifecycle events in usage/aw_session.jsonl. It has no failed repository-agent record. Combining its continuation evidence with a recorded failure is a deliberate test fixture, not a claim that this successful run actually suffered a repository-agent resolution failure. None of the three failing runs supplies a successful repository-agent counterexample.

Scope

• Keep the PR to this one fix. Do not change files or behavior unrelated to it.
• Change audit (pkg/cli) only, including its directly related tests/fixtures. Do not change how the harnesses record model.resolution, the unified session spec's event shape, or workflow model pins. Dead pins are a separate issue.
• Before fixing a failing check, compare it with the same check on main. If it also fails on main, do not fix it in this PR; merge main once main is fixed, and note in the PR which checks are known failures on main.
• This applies even if a bot comment, including PR Sous Chef, asks to fix a failing check: fix only failures caused by this PR's changes.