Skip to content

smoke-claude-sub-agents: check the served model with proxy evidence - #70759

Merged
SivaKesava1 merged 2 commits into
mainfrom
sivakesava1/fix-smoke-claude-sub-agents-served-model
Oct 21, 2026
Merged

SivaKesava1 merged 2 commits into
mainfrom
sivakesava1/fix-smoke-claude-sub-agents-served-model

Conversation

@SivaKesava1

Copy link
Copy Markdown
Collaborator

smoke-claude-sub-agents (added in #70671) fails on main on every run, although the fix it covers works. Its last post-step check expects the worker's assistant messages in Claude Code's stream-json output, but Claude Code 2.1.296 emits only the worker's prompt (a user record with parent_tool_use_id), never its assistant messages.

Evidence: run 39536972987 on main (15d31e5375)

  • model.resolution: marker-worker small → claude-haiku-5-5, resolved (Anthropic).
  • The worker is registered, invoked first, completes and returns the marker. subagent.configured is claude-haiku-5-5, with no general-purpose fallback.
  • The proxy served 1 × claude-haiku-5-5 (200) and 3 × claude-sonnet-5 (200).
  • The run fails anyway: marker-worker messages were not served by claude-haiku-5-5. The stream has 4 assistant records, all from the main agent.

Change: take the served model from the proxy instead of child stream messages.

  • At least one request on the resolved model, and all of them with status 200.
  • The main agent never used that model.

Detection and routing are off in this smoke, so those requests can only come from the worker. The test fixture now uses the record shape Claude Code actually emits, instead of an invented child assistant message. Its cases cover: the main agent on the worker's model (ambiguous), no request on the model, and a failed request on the model.

Verification

  • The new post-step replayed against run 39536972987's real artifacts passes. The old one fails with the error above.
  • smoke_claude_sub_agents.test.cjs: 6/6 pass.
  • The lock file was recompiled with a build of main, and only this workflow changed.

Only smoke-claude-sub-agents.md, its lock file and its test change.

Claude Code 2.1.296's stream-json output carries the worker's prompt
(`user` with `parent_tool_use_id`) but none of its assistant messages, so
the post-step's child-message check could never pass. Run 39536972987 on
main shows the worker resolving to claude-haiku-5-5, running and returning
the marker, and the proxy serving its request, yet failing that check.

Take the served model from the proxy instead: at least one request on the
resolved model, all with status 200, and the main agent never on that
model. Detection and routing are off in this smoke, so those requests can
only come from the worker. Update the test fixture to the record shape
Claude Code actually emits.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI max review requested due to automatic review settings October 21, 2026 03:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Failed proxy requests are absent from the usage log, so the new assertion and fixture do not verify the promised failure condition.

1 open issue
What changed in this PR

Updates gh-aw’s Claude sub-agent smoke test to verify the served model through proxy evidence rather than unavailable child assistant messages.

Changes:

  • Checks proxy requests and excludes ambiguous main-agent model usage.
  • Updates stream fixtures and negative cases.
  • Regenerates the compiled workflow.
File Description
actions/​setup/​js/​smoke_claude_sub_agents.test.cjs Updates fixtures and assertion cases.
.github/​workflows/​smoke-claude-sub-agents.md Uses proxy evidence for served-model checks.
.github/​workflows/​smoke-claude-sub-agents.lock.yml Synchronizes generated checks and metadata.

Review effort: Max

Comment thread .github/workflows/smoke-claude-sub-agents.md Outdated
AWF's token tracker skips non-2xx responses, so token-usage.jsonl only
holds successful requests and the status check there could never fail.
Check the sibling upstream-errors.jsonl, which AWF writes for failed
requests, for any failure on the resolved model, and take the fixture's
failure case out of the usage log.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SivaKesava1

Copy link
Copy Markdown
Collaborator Author

Verified at 01158e7268

  • Branch smoke: smoke-claude-sub-agents passes on 39539347303 (01158e7268) and 39537705319 (ee0482ef1a). marker-worker resolved small → claude-haiku-5-5 and passed every check.
  • Replay: the new post-step, run against the real artifacts of the failing main run 39536972987, passes. The old one fails there with marker-worker messages were not served by claude-haiku-5-5.
  • Copilot code review: its one finding was valid and is fixed in 01158e7268, with the thread resolved. AWF v0.28.70 writes only 2xx responses to token-usage.jsonl, so failed worker requests are now read from upstream-errors.jsonl.
  • Tests and CI: smoke_claude_sub_agents.test.cjs passes 6/6, and all 14 CI checks pass. Only smoke-claude-sub-agents.md, its lock file and its test change.

@SivaKesava1
SivaKesava1 merged commit d9813fa into main Oct 21, 2026
30 checks passed
@SivaKesava1
SivaKesava1 deleted the sivakesava1/fix-smoke-claude-sub-agents-served-model branch October 21, 2026 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants