Repository navigation
[agent-job-health] Engine-conformance and Pi model-routing smoke tests flaking since 2026-10-15 #70673
Description
Summary
A recurring cluster of failures in the engine-conformance / Pi model-routing smoke tests has been observed in the Agent Job Health Monitor daily report since 2026-10-15, flagged again on 2026-10-17, and confirmed again today (2026-10-20). It has never crossed this report issue-creation threshold on any single day, but has now been visible across 3 separate daily windows spanning 5+ days without resolution.
Window analyzed
2026-10-19T23:31 UTC to 2026-10-20T23:18 UTC (530 runs, 184 distinct workflows, 91 agent-job failures, 17.17% run-weighted fleet failure rate).
Failure signature (this window)
5 unique runs affecting 4 distinct workflows:
| Step | Failure kind | Runs | Workflows |
|---|---|---|---|
| Assert engine conformance | driver_exit | 2 | engine-conformance-pi, engine-conformance-deepseek-harness |
| Execute Pi CLI | driver_exit | 3 | engine-conformance-pi, smoke-pi-routed |
| Assert model-routing and sub-agent evidence | driver_exit | 3 | smoke-copilot-sdk-routed-claude, smoke-pi-routed |
| Execute DeepSeek Harness | driver_exit | 1 | engine-conformance-deepseek-harness |
Representative run: https://github.com/github/gh-aw/actions/runs/39458068950 (smoke-pi-routed) failed at Execute Pi CLI and Assert model-routing and sub-agent evidence in the same run.
Prior occurrences
- 2026-10-15: Engine Conformance Test Matrix (claude+codex+deepseek-harness variants all failed within the same ~10min window), 3 runs, 3 workflows.
- 2026-10-17: Assert engine conformance step failure, same test matrix, 3 runs, 3 workflows, noted as recurring since 2026-10-15 and still untracked.
- 2026-10-20 (today): same family, now also touching the Pi CLI execution step and the model-routing assertion directly.
Suspected root cause
Not confirmed from structured data alone; no raw log excerpt was available in the downloaded cache for this report. The consistent pattern across engine-conformance-pi, engine-conformance-deepseek-harness, and smoke-pi-routed / smoke-copilot-sdk-routed-claude suggests a shared dependency such as the Pi engine driver or a shared model-routing assertion helper, rather than independent unrelated flakes.
Recommended action
- Investigate the shared engine-conformance / model-routing test harness for a common failure point (Pi CLI driver and the model-routing assertion look most implicated).
- Check PR Resolve Copilot sub-agent models in the harness (CLI and SDK) and fail closed #70389 (Resolve Copilot sub-agent models in the harness, merged 2026-10-20) for interaction, though most failures in this window predate that merge.
- If confirmed flaky/non-actionable, add a quarantine or retry so it stops surfacing in the daily fleet health report.
Filed automatically by the Agent Job Health Monitor workflow.
Generated by 🩺 Agent Job Health Monitor · claude · agent · 590.7 AIC · ⌖ 1.09 AIC · ⊞ 6.9K · ◷
- expires on Oct 27, 2026, 3:54 PM UTC-08:00
Activity
pelikhan commentedon Oct 21, 2026
Investigation: this report groups several deterministic failures under the same conformance/assertion step names. The sampled runs do not support a single recurring Pi-driver or routing-assertion flake.
unknown_model_ai_creditsfor implicitly selectedgpt-6.1-solclaim_handle: 'h1'ai_credits_limit_exceeded, 5.60425 / 5copilot/gpt-5.4fails same-run catalog resolution:model_unavailablecopilot/gpt-5.model_not_available_for_integratorforgpt-5.4copilot/claude-haiku-5.5.mini-whoamirequests retiredgpt-5.4-mini; model resolution failsmodel_unavailableThe missing result files, missing successful main requests, and absent sub-agent events on Oct 20 are downstream symptoms of model rejection, not independent evidence that the assertion helper is broken.
Existing evidence for #70652: both routed smokes already passed on its branch,
copilot/fix-stale-copilot-workflow-pins, at exact commit547a7345a84c66fcf915bd0a5f999f189f68cba9:gpt-5.6-luna /responses 200, Haiku childclaude-haiku-4.5 /v1/messages 200, GPT childgpt-5.3-codex /responses 200. Both children started once, completed, and have exact model-resolution records.claude-haiku-4.5 /chat/completions 200, childgpt-5.6-luna /responses 200, successful child lifecycle and exact resolution.These are feature-branch results, not recovery on main. #70652 remains open. No post-fix Pi/DeepSeek conformance execution was established in this investigation.
#70389 is not the origin: it merged Oct 20 at 22:26 UTC, after all four sampled Oct 20 failures. It makes Copilot sub-agent resolution fail closed and explicitly calls out the already-retired model pins.
Recommended disposition: link this issue to #70652 rather than duplicate its patch; retain it pending merge and fresh successful Pi/DeepSeek conformance evidence. Do not quarantine or retry deterministic model-policy rejections. Split the historical cluster into budget exhaustion, missing pricing, claim-scoped expectation mismatch, and retired-model pins.
Investigation only: no repository source changes and no workflow dispatches.