Repository navigation
Fix single-quoted experiment branches in sub-agent smoke prompts - #70747
Conversation
Recognize both quote styles and compare equality/inequality operators consistently so sub-agent smoke variants cannot retain conflicting control instructions. Exercise all three real smoke task branches through prompt interpolation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills... |
|
🔎 PR Code Quality Reviewer is reviewing code quality for this pull request... |
|
🔬 Test Quality Sentinel is analyzing test quality on this pull request... |
|
✂️ Ponytail Reviewer has started processing this pull request |
|
🔍 Design Decision Gate 🏗️ is checking for design decision records on this pull request... |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved experiment references in elseif conditions can select incorrect instructions or remove required prompt sections.
1 open issue
What changed in this PR
Adds single-quoted comparison support to gh-aw’s prompt renderer to correct sub-agent smoke experiment selection.
Changes:
- Handle both quote styles and all four comparison operators.
- Add comparison and smoke-workflow rendering regression tests.
- Isolate integration-test workspaces.
| File | Description |
|---|---|
actions/setup/js/is_truthy.test.cjs |
Covers quote styles, operators, and empty operands. |
actions/setup/js/is_truthy.cjs |
Unifies equality and inequality parsing. |
actions/setup/js/interpolate_prompt.test.cjs |
Tests smoke variants and isolates workspaces. |
Review effort: Max
| const neqMatch = trimmed.match(/^(.*?)\s*!==?\s*"([^"]*)"\s*$/); | ||
| if (neqMatch) { | ||
| return neqMatch[1].trim() !== neqMatch[2]; | ||
| const comparison = trimmed.match(/^(.*?)\s*(===?|!==?)\s*(?:"([^"]*)"|'([^']*)')\s*$/); |
|
🎉 This pull request is included in a new release. Release: |
Summary
Fix the remaining reproducible sub-agent smoke gap after #70652: the template truthiness helper treated single-quoted comparisons as nonempty truthy strings. All three
smoke-copilot-sub-agentsexperiment branches survived rendering, including the instruction not to invoke sub-agents during aninline_strictrun.!==is not consumed as an equality expression with!appended to the left-hand side.interpolate_prompt.main()for every variant and require exactly the assigned instructions. Isolate the existing integration tests' workspace in their temporary directory.Fixes #70670.
Existing-run evidence
assignments.jsonselectsinline_strict; activation logs showinline_strict == 'single_agent_control',inline_strict == 'delegated_sequential', andinline_strict == 'inline_strict'all retained. The final prompt includes both “Do not call any sub-agent” and delegation instructions. The post-step correctly rejects zero invocations (haiku-whoami must actually run exactly once,0 !== 1).single_agent_control; its assertion and overall run succeed. The issue's synthetic FAIL is expected by experiment design, not evidence of a model regression or successful positive-arm delegation.Scope and residuals
#70652 is already merged and #70649 closed. The repository pin regression passes against its recorded 36-model catalogue, with only the explicitly owner-deferred
video-analyzer/copilot/gemini-3.7-flashpin unresolved. That fixture is historical evidence, not a live catalogue availability claim. #70675 predates the pin-fix merge.No model pins, fail-closed model-resolution behavior, routing instrumentation, Claude/Codex parity, workflow configuration, or generated locks are changed. The literal variant interpolation in the smoke issue text is unchanged; the root cause fixed here is contradictory branch selection.
Validation
inline_strict.is_truthy,interpolate_prompt,render_template,smoke_copilot_sub_agents, andmodel_pin_check.PATH="/opt/homebrew/bin:$PATH" make agent-report-progresspasses formatting, JavaScript lint, schema freshness, build, and impacted tests. No Go changes; impacted Go tests and workflow recompilation correctly skip.Hosted verification
No workflow was dispatched. These changes are locally verified and published, not yet landed or hosted-verified. The latest main-branch smoke runs available at investigation time predate #70652; a passing negative-control run does not establish positive-arm delegation or wire-API success.
Draft per inherited user preference.