Complete runner/collector-written routing analytics in aw_session.jsonl (G-1/G-5/G-6/G-9/G-12) #70668

Problem

Complete the runner- and collector-written routing analytics record in usage/aw_session.jsonl in one focused change: configured policy, counterfactual baseline pricing, harness/run identity, and identifier-only upstream/request errors.

These are the remaining G-1, G-5, G-6, G-9 and G-12 items from the Data gaps section of #69507. #69725 and #69754 are closed; do not reopen their candidate-ranking/request-context and inventory work. The inspected sessions already retain ranked choices and model inventories.

The CAO model-routing page in githubnext/gh-cao reads unified sessions, not the separate agent artifacts. Its adapter already recognizes model.config, model.baseline_estimate and firewall.upstream_error; the queries consume the routing record. This issue is about supplying that record, not changing CAO.

Verified evidence: October 20, 2026

Audited all three supplied runs on Windows using a development binary built from gh-aw main at 70e15c7ff659c05c2f4466aa172bb910d28847a2, reporting dev-70e15c7ff6, rather than the installed stable v0.89.21:

gh-aw-main.exe audit <run-id> --repo github/gh-aw --no-baseline --timeout 4m --artifacts usage,agent,info --output <per-run-directory> --json

All three audit commands completed successfully; that is collection success, not the workflows' conclusions. Read each usage/aw_session.jsonl, the original agent artifact's final runtime/aw_info.json / agent/aw_info.json, compiled workflow configuration, and firewall evidence. In these artifacts the proxy JSONL files are under sandbox/firewall/logs/api-proxy-logs/, not sandbox/firewall/audit/api-proxy-logs/; sandbox/firewall/audit/awf-resolved-config.json was inspected too.

These failures explain why recording policy and harness identity must not depend on a successful conversation or routing selection. Fixing their runtime causes is outside this issue.

Gap verification

The compiled budget expression is vars.GH_AW_DEFAULT_MAX_AI_CREDITS || '1000'; the observed value alone does not distinguish an equal-valued repository default from the literal fallback.

Useful original artifact links for reproducible fixtures:

Possible solution

1. Runner-owned identity and configured policy

Update actions/setup/js/generate_aw_info.cjs and the runner step that persists model.config, with only the directly required compiler-to-runner metadata wiring.

For workflow.info, add engineHarness, workflowRef and serverUrl, and preserve/test the existing runAttempt. Emit copilot-cli versus copilot-sdk from the actual compiler/runner harness configuration, with one stable value per supported harness, never by guessing from engineId, a workflow name or model output. Capture run attempt and server URL from runner context.

Keep the executing and calling workflow references distinct. workflowRef should retain the compiler-wired executing workflow reference already represented by workflowId; callerWorkflowRef retains GITHUB_WORKFLOW_REF, which can identify a different top-level caller for reusable workflows. Do not regress the existing identity contract.

Write one model.config per run, from runner-owned aw_info.json and the compiled configuration, including when routing is disabled or selection fails:

• routing: { enabled, goal, mode, allowedModels }, retaining configured policy rather than the selected classification's resulting mode.
• models: { allowed, blocked }, keeping policy separate from the runtime inventory and routing candidate pool.
• maxAiCredits and maxAiCreditsSource (frontmatter or default), using the effective resolved budget and recorded configuration provenance, not inferring the source from its numeric value. Defaults may be resolved through repository variables.
• baselineModel: the concrete configured/default main model the run would use without routing, after existing alias resolution. This is neither the literal auto alias nor automatically the router's chosen model.

Capture baseline metadata without changing any harness's resolution or selection behavior. If a historical source cannot establish it, leave it unknown; do not relabel the routed workflow.info.model as the counterfactual baseline. Policy/model/source strings must be identifier-shaped metadata; exclude prompts, transcripts, secrets and free-form diagnostic text.

2. Collector-owned baseline estimate

In the conclusion collector, reprice each main-agent request's recorded token usage at baselineModel using the existing computeInferenceAIC in actions/setup/js/model_costs.cjs. Preserve the input/cache/reasoning accounting semantics and count a request once, not both its native and proxy accounting copies.

Emit model.baseline_estimate for routed runs, clearly marked estimated: true, with baseline identity, an AIC estimate/subtotal, priced and unpriced request counts, and explicit completeness. Exclude sub-agent, routing-classifier and detection requests. Use deterministic runner-owned harness/proxy evidence for attribution; a differing model/endpoint, xInitiator: agent, or model_routing.outcome alone is not proof of a main/sub-agent role. Unattributable or unpriceable evidence must not silently produce a complete total.

Check pricing availability before interpreting the helper's result: the current cost helper returns zero when pricing is unavailable. Unknown pricing means unpriced, never zero cost. A priced subtotal with unpriced requests must not be presented as a complete estimate.

Proposed unrouted policy: omit model.baseline_estimate for explicitly unrouted runs; retain their model.config with routing.enabled: false and resolved baselineModel. Document this choice. Missing estimates in older sessions remain unknown, not zero or evidence that routing was disabled.

For the successful routed fixture, the four main-request proxy token tuples (input, output, cache_read, cache_write) are (3,175,0,18516), (3,61,18516,339), (3,210,18855,76) and (3,60,18931,228). Keep the original records in fixtures rather than reconstructing JSONL from these tuples. The claude-haiku-4.5 sub-agent and classifier are separate requests.

3. Preserve identifier-only proxy errors

github/gh-aw-firewall#10211, shipped in AWF v0.28.67 on October 19, already writes identifier-only error_code, error_type and error_param on upstream-error records, and error_code, error_type, error_source on failed routing request records. Current gh-aw main defaults to AWF v0.28.70; no AWF change is required.

Update the collector's runtime mapping and actions/setup/js/unified_session_payload.cjs:

• Map proxy upstream-error records to firewall.upstream_error, retaining errorCode, errorType, errorParam, HTTP status, provider, model and request ID.
• Retain errorCode, errorType and errorSource on projected firewall.model_routing request events; preserve upstream versus awf.
• Explicitly allowlist/validate identifiers. Never copy response_body, response headers or messages, and never recover codes by scraping historical error text.

The current runtime mapping leaves upstream errors as generic firewall.event. Its payload projection retains generic runtime fields and omits the new request error identifiers. Fix both boundaries rather than changing audit findings.

Because these three runs lack an upstream-error fixture, use the shapes and cases from the tests merged in github/gh-aw-firewall#10211 at b5c683e116470e0aa579ec9bf30a675fcdb66f76: upstream-log.test.js has the captured Copilot invalid_reasoning_effort record; provider-error-fields.test.js covers optional code/type/param identifiers; upstream-response.test.js and routing-persistence.test.js cover upstream- and AWF-originated failed requests. Preserve real optional-field omissions.

4. Specify the additive session contract

Update docs/src/content/docs/specs/unified-agent-session-specification.md alongside the writers, collector projection/types and directly related tests. Give each new event and field its writer (runner, collector or proxy), field definitions, provenance, applicability/completeness and an "absent means unknown" rule. An explicitly recorded false, empty policy list or zero is not the same as an absent value.

All attribution must come from runner- or proxy-written evidence, never model_routing.outcome or agent-authored outcome reports. Preserve older sessions and the existing candidate/inventory contracts.

Acceptance cases

Each case must be a named test. Check in appropriately sanitized fixtures from the linked real artifacts, keeping their actual envelopes, optional fields and attribution; do not invent proxy/native record shapes. Use the explicitly identified AWF tests only where these runs lack the necessary error record.

Scope

• Keep the PR to this one change: writing the routing analytics data listed above into aw_session.jsonl, plus its spec entries and tests. Do not change files or behavior unrelated to it.
• Do not change AWF, the CAO page, audit's findings, the harnesses' model resolution, or workflow model pins. Do not add the x-awf-caller request header (separate work) or safe-output outcome events (separate work).
• Before fixing any failing check, compare it with the same check on main. If it also fails on main, do not fix it in this PR; merge main once main is fixed, and note in the PR which checks are known failures on main.
• This applies even if a bot comment, including PR Sous Chef, asks to fix a failing check: fix only failures caused by this PR's changes.