[Copilot]

In smoke-copilot, the file-summarizer agent file is written with the literal model: ${{ experiments.subagent_model }}. The literal reaches AWF and gets HTTP 400. The run can still be green, because nothing checks that the sub-agent ran: one of the two audited runs never invoked it at all. The variant assignment and aw_info.json were already correct; only the written agent file was wrong.

Root cause

interpolate_prompt.cjs extracts and writes inline sub-agents at step 1.5. That is before variable interpolation (step 2) and experiment substitution (step 2.5), and those later passes only see the main prompt.

Changes

• Resolve experiment references when writing the agent file (extract_inline_sub_agents.cjs)
  • New resolveSubAgentExperimentReferences(agentName, content, env), called in writeInlineSubAgents just before the file is written.
  • Only rewrites the frontmatter; the agent body is left alone.
  • Handles the raw ${{ experiments.<name> }} form (runtime-import) and the __GH_AW_EXPERIMENTS_<NAME>__ placeholder form (inlined content).
  • Reads only GH_AW_EXPERIMENTS_<NAME>, the value already set from steps.pick-experiment.outputs.*. Nothing is re-picked and no other expression is evaluated.
  • Fails with ERR_CONFIG naming the sub-agent and field if the value is missing or empty. It also fails if the value doesn't match ^[A-Za-z0-9][A-Za-z0-9._/@+:-]*$, which keeps it from injecting YAML.
  • The alias (small/large) is written as is, with no alias resolution.
• Smoke check U4 (smoke_model_routing_assertions.cjs)
  • New requiredSubAgents option for evaluateSessionContract. For each named sub-agent the check fails if:
    • it never started;
    • any subagent.failed event matches it;
    • any invocation has no subagent.completed;
    • no model is recorded, or the recorded model contains ${{ or __GH_AW_.
  • The sub-agent lifecycle matching now lives in a subAgentLifecycle helper, shared with the existing S1/S2 routing checks.
• smoke-copilot.md: the post-step now passes requiredSubAgents: ["file-summarizer"].
  • The lock file diff is that script plus the workflow hash. The hash also changes GH_AW_HARNESS_VERSION.
• Docs: updated the sub-agent paragraph in experimental/experiments.md.

## agent: `file-summarizer`
---
model: ${{ experiments.subagent_model }}   # written as `model: small` or `model: large`
---

The agent file and the sub-agent model metadata read the same pick-experiment output, so they always agree. A Go test (TestSubagentMetadataMatchesSelectedExperimentVariant) covers this.

Tests

The named tests from the issue are included, except the compile-rejection test, which is only needed if this syntax is declined:

• writes_selected_experiment_model_before_agent_file_write
• both_subagent_model_variants_are_reachable
• preserves_plain_subagent_model_and_frontmatter
• subagent_metadata_matches_selected_experiment_variant
• smoke_fails_when_file_summarizer_fails_to_start
• smoke_fails_when_file_summarizer_is_not_invoked

The pipeline tests drive the real interpolate_prompt pipeline and fail if the fix is removed.

Known gaps / follow-ups

• End-to-end not run. I couldn't dispatch smoke-copilot from this agent, so it hasn't run on this branch and there is no run URL or tested commit yet. Expected result:

  • the agent file contains small or large, and the 400 for the literal expression is gone;
  • until Copilot sub-agent alias resolution lands (separate issue), file-summarizer will likely fail on the unresolved alias, and U4 should then fail the smoke on purpose.

  Please hold the merge until that fix lands, then rerun the smoke.

• inlined-imports: true mode. The compiler drops ## agent: blocks from the prompt in this mode, so no agent file is written at all. This existing gap is not fixed here.

• Shared imports. Experiment references in .github/workflows/shared/* imports are still rejected at compile time.

• Fixes Expand experiment model expressions before writing inline sub-agents; fail smoke on failed or missing delegation #70386

[Copilot]

🔵 Needs a closer look

Confirmed correctness gaps remain, and end-to-end smoke validation depends on a separate alias-resolution fix.

4 open issues

• Imported agent model placeholders are not resolved correctly · New
• Parameterized model selectors are rejected during validation · New
• CRLF inline agents fail model assignment parsing · New
• Cancelled subagent invocations incorrectly satisfy smoke checks · New

What changed in this PR

Tightens experiment-model handling and delegation checks in gh-aw’s inline sub-agent pipeline.

Changes:

• Resolves selected experiment variants before writing agent frontmatter.
• Adds required sub-agent lifecycle checks to smoke-copilot.
• Adds regression coverage and updates experiment documentation.

File	Description
pkg/​workflow/​compiler_experiments_test.go	Checks assignment and metadata consistency.
docs/​src/​content/​docs/​experimental/​experiments.md	Documents substitution and failure behavior.
actions/​setup/​js/​smoke_model_routing_assertions.test.cjs	Covers required delegation checks.
actions/​setup/​js/​smoke_model_routing_assertions.cjs	Adds U4 lifecycle assertions.
actions/​setup/​js/​interpolate_prompt.test.cjs	Exercises experiment expansion through interpolation.
actions/​setup/​js/​extract_inline_sub_agents.test.cjs	Tests substitution and validation.
actions/​setup/​js/​extract_inline_sub_agents.cjs	Resolves frontmatter experiment references.
.github/​workflows/​smoke-copilot.md	Requires file-summarizer delegation.
.github/​workflows/​smoke-copilot.lock.yml	Updates the compiled workflow and hashes.

_{Review effort: Max}

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Review (actions/setup/js/extract_inline_sub_agents.cjs:72): Imported agent blocks inlined at compile time pass through extractPromptChunksFromMarkdown and ExpressionExtractor. Those rewrite \${\{ experiments.subagent_model }} to steps.pick-experiment.outputs.subagent_model, then emit __GH_AW_EXPR_0498E4C3__, not the experiment placeholder matched here. This affects imported files under inlined-imports: true, not just the main-body gap noted in the PR. Because agent extraction precedes variable interpolation, this resolver writes the unresolved token into model even when the assignment is available, and activation succeeds. Align the compiler's experiment placeholder with this resolver, keeping lookup limited to experiment assignments, and reject unresolved model tokens before writing. Add a compiler-to-runtime case: the current "inlined" tests supply a synthetic placeholder rather than compiler output. - Resolve experiment references in inline sub-agent frontmatter before writing the agent file; fail smoke-copilot when file-summarizer doesn't run #70387 (comment)
3. Review (actions/setup/js/extract_inline_sub_agents.cjs:76): This rejects complete model selectors such as gpt-5.4?effort=low because ? and = are excluded. These parameters are supported by the model resolver (resolve_model_alias.cjs:458), and compiler validation directs authors to put the complete model selector in each experiment variant. Such workflows compile but then fail activation whenever that variant is selected. Permit YAML-safe query-parameter characters while retaining the whitespace and expression restrictions, and cover a parameterized variant. - Resolve experiment references in inline sub-agent frontmatter before writing the agent file; fail smoke-copilot when file-summarizer doesn't run #70387 (comment)
4. Review (actions/setup/js/extract_inline_sub_agents.cjs:103): Runtime imports preserve CRLF line endings, but this delimiter regex only accepts LF. A CRLF-authored inline agent therefore returns unchanged: the selected variant is not written, and a missing assignment does not throw, leaving the literal model expression for the runtime to reject. Accept optional \r at both delimiters and add CRLF pipeline cases for assigned and missing values. - Resolve experiment references in inline sub-agent frontmatter before writing the agent file; fail smoke-copilot when file-summarizer doesn't run #70387 (comment)
5. Review (actions/setup/js/smoke_model_routing_assertions.cjs:722): Copilot records stopped invocations as subagent.completed with data.cancelled: true; subagent_session_render.cjs:42 distinguishes these from successful completion. A start followed only by that event currently passes U4, so an aborted required delegation can still leave the smoke green. Require a non-cancelled completion for each invocation. Cover cancellation-only and success-then-cleanup lifecycles so a cleanup cancellation does not invalidate an earlier genuine completion. - Resolve experiment references in inline sub-agent frontmatter before writing the agent file; fail smoke-copilot when file-summarizer doesn't run #70387 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef state: 3caddd95d8c526431f46b86beb7b2da902cd21959fcdbbc5b38c0449e81dd5e3

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 12.4 AIC · ⌖ 7.06 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[gh-aw-bot]

Caution

agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.

Details

Potential security threats were detected in the agent output.

Review the workflow run logs for details.

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Fix failing check github-advanced-security (FAILURE): https://github.com/github/gh-aw/actions/runs/39468144692/job/118784860282.
3. Fix failing check JS Tests (shard 2/4) (FAILURE): https://github.com/github/gh-aw/actions/runs/39468125212/job/118785048173.
4. Fix failing check security (FAILURE): https://github.com/github/gh-aw/actions/runs/39468125439/job/118786200176.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 403070b
Sous-chef work: 043185828d65a534dd8b2b6b4d00e0d47bf21a2940d3f277c0ed15da7f668648 0d44b5dbbccb36360c4c148fdf80d42daaeb416739df2a1c54f1443259fcfaa9 960a6fe297781e0f1e1c8c03fa3740110735e1332e99a040b5aa7b1b6d14a3f2
Sous-chef state: 72858b08e4a28e10a9a0b61079cb23ba681c28e2cd50446fe73e20edf4524bbe

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 3.16 AIC · ⌖ 12.3 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[SivaKesava1]

End-to-end runs on 403070b2da: expansion works; U4 fails because the agent skips step 14

I dispatched smoke-copilot twice on this branch:

• 39469870442: caveman: yes, subagent_model: large
• 39469887119: caveman: no, subagent_model: large

The fix works. Activation succeeded in both runs, aw_info.json declares file-summarizer with model: "large", and the staged .github/agents/file-summarizer.agent.md is in the workspace (pre-agent-audit.txt). The literal ${{ experiments.subagent_model }} is gone.

Both runs fail the new U4 check: FAIL U4 sub-agent file-summarizer: never started (no subagent.started); observed events: none.

• In both runs the task tool was offered to the main agent, but it made no task call (5 and 7 turns). Its report says 14. file-summarizer: ❌ (skipped).
• This is not new. Of the last five scheduled main runs with session events, only 39181772988 invoked file-summarizer, and that call failed on the literal model. 38987143187, 38328775781, 38099607416 and 37866304925 made no task call, although their prompts contain step 14. The issue also noted this for 39325986042.

U4 does what the issue asked (fail on a missing invocation), but step 14 is one line in a 16-step list, and the agent usually skips it. As written, U4 would turn smoke-copilot red on most runs. The issue's requirement stays: make the invocation reliable rather than weaken U4.

Requested changes:

1. Make the sub-agent invocation a required first step in smoke-copilot.md. Add a short section before "Test Requirements" saying that the first action of the run must be a task tool call to the file-summarizer agent on README.md, before any numbered test; that this is mandatory and must not be skipped; and that the run reports the sub-agent's one-line summary. Change item 14 to report the result of that first step, so the numbering and the Output section's step references stay unchanged. Keep U4 as it is.
2. Recompile at the merged head. The "Update branch" merge at 17:40 left smoke-copilot.lock.yml with the old GH_AW_ADMISSION_WORKFLOW_HASH (from Add isolated OIDC admission and runtime credit allowances #70429). The Sous Chef session's 2516fcdfff refreshes it; I'll check that recompiling at the final head gives no diff.

Known failures on main, not caused by this PR:

• daily_report_portfolio.test.cjs ("keeps the dispatcher allowlist…") fails on main at a751cce353 too.
• The security job failed on a proxy.golang.org stream error; I re-ran it.
• github-advanced-security reports no open code-scanning alerts for this PR.

None of these three needs a change in this PR. Please don't change daily_report_portfolio or other unrelated files here.

After the round I'll dispatch smoke-copilot twice more to confirm U4 passes with the invocation step.

[Copilot]

[!CAUTION]
agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation....

Pushed b9996c4: make merge-main brought in c756711, which fixes the daily-report compiled-condition assertion behind the JS shard failure. Refreshed the smoke admission hash in 2516fcd. The final scoped gate, 262 focused JS tests, compiler/runtime regressions, and make test-security pass. Both security CI failures were Go proxy HTTP/2 INTERNAL_ERROR downloads, not security findings; the advanced-security reviewer reported no findings. Full shard recheck: 4,738 passed, two unchanged work-queue performance tests timed out at 30s (also reproduced alone). Independent review found no significant issues; automated CodeQL timed out. No CI triggered. Human review—including the warning on this comment—CI validation, and resolution of two already-answered review threads remain.

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Merged latest main via make merge-main in b9996c4 and preserved all four fixes from 1ff3981. Revalidated the compiler-to-runtime path and all 262 focused JS tests successfully. The listed threads already have substantive replies, so I did not duplicate them. Threads r4320334890 and r4320334950 still need manual resolution: no thread-resolution tool or authenticated GitHub CLI is available here.

[Copilot]

End-to-end runs on 403070b2da: expansion works; U4 fails because the agent skips step 14

I dispatched smoke-copilot twice on this branch:
...

Done in a1d2aac:

• smoke-copilot.md has a new "Required First Step" section before Test Requirements. The run's first action must be a task call to file-summarizer on README.md, before any numbered test. The section says the step is mandatory and the run reports the sub-agent's one-line summary.
• Item 14 now reports the result of that first step. The numbering, the Output section's step references and U4 are unchanged.
• make merge-main made no new merge commit because the branch already contains main at c756711. make recompile changed only smoke-copilot.lock.yml: the body hash, GH_AW_HARNESS_VERSION and GH_AW_ADMISSION_WORKFLOW_HASH. No other lock files changed.
• Passed: make fmt and make lint; the extract_inline_sub_agents, interpolate_prompt and smoke_model_routing_assertions JS tests (292 tests); the pkg/workflow Experiment tests.

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Fix failing check conclusion (FAILURE): https://github.com/github/gh-aw/actions/runs/39480083461/job/118831074180.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: a1d2aac
Sous-chef work: 51e2287c3d1ee41b7e13769862add61dcd292a623cf943abc5968c984e246bcf
Sous-chef state: 02defb763794df698e59669865d858d56e272c73f40da0b9f79df7c721f4450e

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 3.05 AIC · ⌖ 6.17 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[Copilot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Pushed edac572 after merging main at 24f43c2 via make merge-main (merge 7c32d27, no conflicts).

The failing conclusion correctly handled report_incomplete: Copilot denied URL approval for https://github.com. file-summarizer completed successfully with model=small, and U4 passed. Added host-scoped --allow-url github.com to this smoke workflow, regenerated its lock/hashes, and added a source/compiled-command regression. U4, incomplete-result handling, and AWF network policy are unchanged.

Passed: focused regression, 246 focused JS tests, make lint, make agent-report-progress, lock drift check, secret scan, and independent review. Dry-run source validation/shellcheck passed, but model-inventory refresh was unavailable. Automated review could not start due to an unavailable model; CodeQL timed out. No CI or workflow was triggered; the new head remains unverified end-to-end. All four earlier review threads are already resolved.

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: edac572
Sous-chef work:
Sous-chef state: 077fa6987612c02b744015546c0dacc3789cc8cc62e4b8abe9be14ebff8c149c

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 7.34 AIC · ⌖ 10.5 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[Copilot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Pushed d15824f, merging main at 44aab31 via make merge-main. Regenerated the sole conflicted smoke lock from merged sources; all existing fixes are preserved. All 345 workflows compile; 246 focused JS tests and focused/impacted Go tests pass. Independent review found no significant issues. Full/scoped lint gates are blocked by three unchanged-main modernize/embedlit findings and an unknown-linter warning; CodeQL timed out and automated review was unavailable. All four review threads were already resolved. No CI triggered; merged HEAD remains unverified by CI.