Skip to content

fix(workflow): disambiguate "workflow" to mean a swamp workflow#1478

Merged
stack72 merged 2 commits into
mainfrom
worktree-workflow-collision-qw1
May 30, 2026
Merged

fix(workflow): disambiguate "workflow" to mean a swamp workflow#1478
stack72 merged 2 commits into
mainfrom
worktree-workflow-collision-qw1

Conversation

@stack72
Copy link
Copy Markdown
Contributor

@stack72 stack72 commented May 30, 2026

Summary

Claude Code ships a built-in Workflow tool that auto-triggers on the bare keyword "workflow" at the harness input layer — before any skill is evaluated. The swamp repo has its own first-class workflow concept (a declarative YAML DAG of model-method steps, authored via swamp workflow create). These collide: when a user says "create a workflow that runs model X then Y" (dropping "swamp"), the harness can win and spin up agent task lists / worktrees / cron scheduling instead of producing a durable workflows/*.yaml DAG.

This PR closes the gap with two layers of pure-documentation/guidance, no behavior change to the CLI:

1. Generated CLAUDE.md rule (repo_service.ts) — the user-facing CLAUDE.md written during repo_init / repo_update gains a rule pinning "workflow" / "automate" / "orchestrate" / "automated job" to the swamp meaning, gating agent-orchestration mechanisms behind explicit opt-in vocabulary. Phrasing is tool-agnostic since the same template serves Cursor, opencode, codex, and kiro alongside Claude. Existing repos pick it up via the managed-section migration on swamp repo update. The swamp-dev CLAUDE.md mirrors the guidance (Claude-Code-only, so it names the literal harness tools).

2. swamp-workflow skill hardening — the skill's trigger list previously listed the same bare verbs the harness cues on (automate, orchestrate, create/run/execute workflow), actively contributing to the ambiguity. Those are removed and every trigger namespaced to the swamp concept, with an explicit NOT-clause in the description and a ## Skill boundary section in the body. Adversarial should_trigger: false eval seeds lock the boundary in.

The two layers are intentionally paired: namespacing the skill triggers trades "false-positives-to-harness" for "false-negatives-to-nothing", and the CLAUDE.md rule gives the model in-context authority to still route un-namespaced phrasings to the skill.

Caveat

This is soft model guidance. The harness workflow keyword fires above the instruction/skill tier, so on a borderline prompt the model can still be pulled toward the orchestration reading. A hard guarantee would require disableWorkflows / permissions.deny: ["Workflow"], which is deliberately out of scope here.

Test Plan

  • deno fmt --check, deno lint — clean
  • deno run test6512 passed, 0 failed (includes a new repo_service_test.ts case asserting the workflow-disambiguation rule lands in the generated CLAUDE.md)
  • tessl skill review on swamp-workflow — 0.925 avg, both judges above the 0.90 threshold
  • promptfoo trigger evals (deno run eval-skill-triggers --model sonnet) — all swamp-workflow seeds pass, including the 3 new adversarial negatives ("create a workflow to run these three steps", "schedule a recurring agent to run nightly", "set up a worktree workflow"). The 3 unrelated suite failures (swamp-data, swamp-vault) are pre-existing on main and out of scope.

🤖 Generated with Claude Code

stack72 and others added 2 commits May 30, 2026 01:29
The Claude Code harness ships a built-in Workflow tool that auto-triggers
on the bare keyword "workflow" at the input-parsing layer. That collides
with swamp's own first-class workflow concept (a declarative YAML DAG of
model-method steps authored via `swamp workflow create`). When a user
drops the word "swamp" the harness can win and spin up agent task lists,
worktrees, or cron/remote scheduling instead of authoring a durable
workflow.

Pin the term in the generated user-facing CLAUDE.md (written during
repo_init / repo_update) with a new rule that maps "workflow" / "automate"
/ "orchestrate" / "automated job" to the swamp meaning and gates agent
orchestration behind explicit opt-in vocabulary. Phrasing stays
tool-agnostic since the same template serves Cursor, opencode, codex, and
kiro alongside Claude. Mirror the guidance into the swamp-dev CLAUDE.md
(Claude-Code-only, so it names the literal harness tools).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rd clash

Claude Code's built-in Workflow tool auto-triggers on the bare keyword
"workflow" at the harness input layer, before any skill is evaluated. The
swamp-workflow skill's trigger list made this worse by listing the same
bare verbs the harness cues on — automate, automation, orchestrate,
create/run/execute workflow.

Remove those bare collision tokens and namespace every trigger to the
swamp concept so the two-word phrase "swamp workflow" becomes the
discriminator. Add an explicit NOT-clause to the description and a
"## Skill boundary" section to the body stating the skill is unrelated to
the Claude Code Workflow tool, agent task lists, worktrees, and
cron/remote-agent scheduling. Lock the boundary into the eval suite:
re-namespace the cron-shaped positive seed and add adversarial
should_trigger:false seeds for the generic-task, schedule-agent, and
worktree phrasings.

Pairs with the CLAUDE.md rule from the previous commit, which gives the
model in-context authority to still route un-namespaced phrasings to the
skill — covering the under-triggering risk this trade introduces.

Verified: tessl skill review 0.925 (both judges above threshold);
promptfoo trigger evals pass for all swamp-workflow seeds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial Review

Critical / High

None found.

Medium

  1. Positive trigger eval may be fragile after trigger namespacingtrigger_evals.json line 29: "Execute the nightly-sync workflow and show me the results" has should_trigger: true, but the SKILL.md description no longer lists bare "execute workflow" as a trigger — it now requires "swamp workflow run", "swamp workflow validate", etc. The word "nightly-sync" doesn't contain "swamp", so trigger matching depends on whether the eval harness matches the broader description text ("running workflows") rather than the explicit trigger list. The PR body says promptfoo evals all pass, so this may be fine empirically, but the gap between the tightened trigger phrases and the unchanged positive evals could make the eval suite less reliable as a regression guard. Similar concern applies to lines 9 ("Show me the run history for the deployment workflow") and 23 ("Inspect the execution logs from the last workflow run") — neither uses "swamp" as a prefix.

  2. Inconsistency between CLAUDE.md and generated CLAUDE.md on "RemoteTrigger" — The dev CLAUDE.md (line 19) references RemoteTrigger in the list of harness tools to avoid. The generated repo_service.ts template (line 819) also references "remote agent" as an explicit opt-in keyword but doesn't mention RemoteTrigger by tool name. The SKILL.md boundary section (new lines 28-32) also references RemoteTrigger. Since the generated CLAUDE.md is tool-agnostic (serves Cursor, codex, etc.), mentioning RemoteTrigger in the dev CLAUDE.md but omitting it from the generated CLAUDE.md is consistent with intent — but worth confirming this asymmetry is deliberate.

Low

  1. Eval seed "Create a swamp workflow that runs the data-sync model every night." changed from generic to namespacedtrigger_evals.json line 19: The original eval "Create an automated job that runs the data sync every night" tested the skill's ability to match the ambiguous phrasing that motivated this PR. Replacing it with the swamp-prefixed version removes the one positive eval that exercised the "bare verb → swamp workflow" routing the CLAUDE.md rule is meant to enable. This weakens the eval suite's coverage of the intended behavior, though it's consistent with the trigger-narrowing strategy.

  2. SKILL.md line count — The SKILL.md body is already at ~640 lines (the ## Skill boundary section adds 9 lines to an already-over-500-line file). CLAUDE.md says "Keep SKILL.md body under 500 lines." This pre-dates this PR but the PR makes it slightly worse. Non-blocking since it's pre-existing.

Verdict

PASS — This PR is pure documentation/guidance with no runtime code changes beyond a template string addition that has test coverage. The changes are internally consistent and well-tested. The trigger eval coverage concern is real but the PR body confirms promptfoo evals pass, and the CLAUDE.md rule provides a second routing layer for bare "workflow" phrasings. No blocking issues.

Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

Blocking Issues

None.

Suggestions

  1. Pre-existing: SKILL.md exceeds 500-line body limit. The swamp-workflow SKILL.md is already ~642 lines before this PR (which adds 9 more). CLAUDE.md says to keep it under 500 lines and split into references/ files. Not introduced by this PR, but worth addressing in a follow-up.

Summary

Clean, well-scoped PR. The two-layer approach (CLAUDE.md rule + skill trigger namespacing) is a sensible defense-in-depth strategy for the keyword collision. The repo_service.ts changes stay within the Domain Service's responsibility of generating CLAUDE.md content, and the new test follows existing patterns exactly. No DDD, security, or convention concerns.

@stack72 stack72 merged commit 1c11711 into main May 30, 2026
11 checks passed
@stack72 stack72 deleted the worktree-workflow-collision-qw1 branch May 30, 2026 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant