Add model-based fuzzing for the Copilot SDK shell parser by Copilot · Pull Request #70528 · github/gh-aw

Copilot · 2026-10-20T13:54:20Z

Extend shell-parser coverage with generated programs and independent extraction and permission oracles.

Generator: Bounded, deterministic, byte-driven programs combining comments, quoting, escapes, parameter expansions, assignments, redirections, and pipeline operators. Direct byte consumption supports shrinking; generated programs are never executed.
Assertions: Exact segments and executable names, ordered deduplication, SDK allow/deny decisions, fail-closed unsupported syntax, and robustness on malformed or truncated input.
Integration: Native Go fuzz target through the existing Node harness, deterministic Vitest regressions, and registration in make fuzz and the CI fuzz matrix.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-10-20T15:03:16Z

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · gpt56 · 6.85 AIC · ⌖ 5.4 AIC · ⊞ 14.7K · ◷

Generated by Ponytail Reviewer for #70528

github-actions · 2026-10-20T15:03:23Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

github-actions · 2026-10-20T15:03:30Z

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

github-actions · 2026-10-20T15:03:34Z

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · opus55 · 11.2 AIC · ⌖ 31.9 AIC · ◇ 13.4 AIC · ⊞ 1.9K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

github-actions · 2026-10-20T15:03:44Z

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · haiku45 · 5.59 AIC · ⌖ 7.12 AIC · ◇ 2.45 AIC · ⊞ 9.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

github-actions · 2026-10-20T15:09:30Z

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)

reviewed_at: 2026-10-20T15:05:34Z
review_event: COMMENT
top_themes: [fuzzing performance, subprocess startup overhead]
files_reviewed: [pkg/workflow/bash_command_parser_fuzz_test.go, actions/setup/js/fuzz_bash_command_model.cjs, actions/setup/js/fuzz_bash_command_model.test.cjs, actions/setup/js/fuzz_bash_command_parser_harness.cjs, Makefile, .github/workflows/cgo.yml]
comment_count: 1

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · 3.02 AIC · ⌖ 0.545 AIC · ⊞ 1.6K · ◷
_{Comment /review to run again}

github-actions

Verdict

One performance issue needs attention before relying on this fuzz target: every fuzz case launches a new Node process, so the configured 30-second run will spend most of its time on startup and achieve very low parser coverage.

Details

pkg/workflow/bash_command_parser_fuzz_test.go:57 creates a fresh exec.CommandContext for every input.
Reuse a persistent Node worker or move the model in-process, with per-request timeouts.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · 3.02 AIC · ⌖ 0.545 AIC · ⊞ 1.6K
_{Comment /review to run again}

github-actions · 2026-10-20T15:09:34Z

+		}
+		ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
+		defer cancel()
+		cmd := exec.CommandContext(ctx, node, harness)


This fuzz target starts a fresh Node process for every generated input, so the 30-second fuzz job spends most of its budget on process startup/module loading rather than exploring parser behavior and can provide almost no meaningful coverage.

💡 Why this matters

Each iteration invokes exec.CommandContext, starts Node, loads the harness and all required modules, and pays the fixed setup cost before one request is handled; the five-second timeout also makes a slow startup indistinguishable from a failing case. Reuse a persistent Node worker (or run the model in-process) and apply a per-request timeout so fuzzing can actually execute a useful number of cases.

github-actions

Skills-Based Review 🧠

Applied /tdd (test-only PR: model-based fuzzing infrastructure). No actionable issues found — approving.

📋 Verification performed

Ran FuzzBashCommandParserModel live (go test -fuzz=... -fuzztime=10s): passes, finds new coverage, no crashes.
Fuzzed the JS generator directly across 5000+ deterministic LCG seeds via checkShellProgramModel: all pass, including the line-continuation blank (backslash+newline) and zero-command edge cases.
Confirmed extractCommandName/splitOnPipelineOperators/permission-handler code paths referenced by the model (fail-closed quoting, env-assignment skipping, shell keyword handling) match the model's independent oracle.
Confirmed the Go test's CombinedOutput() + JSON-stdout pattern is consistent with sibling fuzz tests in pkg/workflow/.

Positive highlights

✅ Byte-driven (not hashed) choice consumption — enables real Go corpus shrinking.
✅ Independent oracle: expected segments/names are computed by the generator itself, not by re-parsing the module under test, so bugs in bash_command_parser.cjs can't silently pass.
✅ Explicit negative-space coverage: unsupported suffixes must invalidate the whole parse (not just a trailing segment) — a good regression guard against partial-trust bugs.
✅ Bounded, reproducible generator (≤8 commands, <5000 chars) keeps fuzzing fast and shrinkable.
✅ Proper CI/Makefile/.github/workflows/cgo.yml wiring so the new fuzz target actually runs in the matrix.

Minor, non-blocking observation

Makefile's new line uses -run='^$$' while the two existing fuzz lines above it (FuzzParseFrontmatter, FuzzExpressionParser) don't. Harmless either way (fuzz seed corpus still runs as subtests without it) and a pre-existing inconsistency, not a risk introduced here — no change requested.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 91.9 AIC · ⌖ 14.8 AIC · ⊞ 10.2K
_{Comment /matt to run again}

Copilot

🟢 Approval recommended

The test-only additions have no identified blocking issues; the remaining maintenance suggestion is optional.

No open issues 🎉

What changed in this PR

Adds model-based fuzzing for the Copilot SDK shell parser and permission handler through the existing Node harness.

Changes:

Generates bounded, deterministic programs with independent extraction and permission expectations.
Adds malformed-input checks and deterministic regressions without executing generated shell programs.
Registers the Go fuzz target in local and CI workflows.

File	Description
`pkg/workflow/bash_command_parser_fuzz_test.go`	Adds the native Go fuzz target.
`Makefile`	Includes the target in `make fuzz`.
`actions/setup/js/fuzz_bash_command_parser_harness.cjs`	Adds model-checking harness mode.
`actions/setup/js/fuzz_bash_command_model.test.cjs`	Adds deterministic generator regressions.
`actions/setup/js/fuzz_bash_command_model.cjs`	Implements generation and oracle assertions.
`.github/workflows/cgo.yml`	Registers the target in the CI fuzz matrix.

_{Review effort: Max}

gh-aw-bot · 2026-10-20T15:42:16Z

@copilot address the following outstanding work in one pass:

Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
Review (pkg/workflow/bash_command_parser_fuzz_test.go:57): This fuzz target starts a fresh Node process for every generated input, so the 30-second fuzz job spends most of its budget on process startup/module loading rather than exploring parser behavior and can provide almost no meaningful coverage. - Add model-based fuzzing for the Copilot SDK shell parser #70528 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: ebe5228
Sous-chef work: bdc208c15b66a66eb20f72be7d6362248d464448b4bd14e47689c583bfafb613
Sous-chef state: 98f23064a9c38b64ecf3ded862ba949d7a95c543b175ab1e883f732628e4999e

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 5.12 AIC · ⌖ 8.99 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

Add model-based Copilot SDK shell parser fuzz generator

ebe5228

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Oct 20, 2026

Copilot AI changed the title ~~Add model-based Copilot SDK shell parser fuzz tests~~ Add model-based fuzzing for the Copilot SDK shell parser Oct 20, 2026

Copilot AI requested a review from pelikhan October 20, 2026 13:55

Copilot created this pull request from a session on behalf of pelikhan October 20, 2026 13:55

View session

Copilot finished work on behalf of pelikhan October 20, 2026 13:55

pelikhan marked this pull request as ready for review October 20, 2026 15:01

Copilot AI max review requested due to automatic review settings October 20, 2026 15:01

Copilot started reviewing on behalf of pelikhan October 20, 2026 15:02

View session

github-actions Bot reviewed Oct 20, 2026

View reviewed changes

github-actions Bot approved these changes Oct 20, 2026

View reviewed changes

Copilot AI reviewed Oct 20, 2026

View reviewed changes

github-actions Bot mentioned this pull request Oct 20, 2026

[pr-sous-chef] PR Sous Chef: PR #70345 nudged #70400

Closed

Copilot started work on behalf of gh-aw-bot October 20, 2026 15:43

View session

pelikhan merged commit ecdb146 into main Oct 20, 2026
128 of 138 checks passed

pelikhan deleted the copilot/add-fuzz-generator-for-shell-programs branch October 20, 2026 15:51

Copilot AI restored the copilot/add-fuzz-generator-for-shell-programs branch October 20, 2026 15:52

Copilot stopped work on behalf of gh-aw-bot due to an error October 20, 2026 15:53
The session was cancelled by the user.

Copilot AI requested a review from gh-aw-bot October 20, 2026 15:53

This was referenced Oct 20, 2026

Smoke Test: MAI-Code-1.1-Flash - 39458045303 #70564

Closed

Smoke Test: MAI-Code-1.1-Flash - 39518935420 #70681

Closed

Navigation Menu

Repository navigation

Add model-based fuzzing for the Copilot SDK shell parser - #70528

Add model-based fuzzing for the Copilot SDK shell parser#70528
pelikhan merged 1 commit into
mainfrom
copilot/add-fuzz-generator-for-shell-programs

Copilot AI commented Oct 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

github-actions Bot Oct 20, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

gh-aw-bot commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Copilot AI commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ponytail Reviewer

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Design Decision Gate 🏗️

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Quality Sentinel

Uh oh!

github-actions Bot commented Oct 20, 2026

Comment Memory

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Verdict

Uh oh!

github-actions Bot Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Positive highlights

Minor, non-blocking observation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

gh-aw-bot commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Oct 20, 2026 •

edited

Loading

github-actions Bot commented Oct 20, 2026 •

edited

Loading

github-actions Bot commented Oct 20, 2026 •

edited

Loading

github-actions Bot commented Oct 20, 2026 •

edited

Loading

github-actions Bot commented Oct 20, 2026 •

edited

Loading

github-actions Bot commented Oct 20, 2026 •

edited

Loading