From a8197b10e7c8cc144c29f7c21a9f397dbb158f68 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 4 Oct 2026 20:15:36 -0400 Subject: [PATCH 1/3] docs(evals): document plugin eval's linked-worktree refusal Adds the CLI's own refusal for a plugin inside a repository with many linked worktrees, its cause, the copy-outside-git workaround, and a dated record row. Bumps evals to 0.5.7. Co-Authored-By: Claude Opus 5.5 --- plugins/evals/.claude-plugin/plugin.json | 2 +- plugins/evals/CHANGELOG.md | 6 ++++++ plugins/evals/skills/plugin-eval/SKILL.md | 21 ++++++++++++++++++++- 3 files changed, 27 insertions(+), 2 deletions(-) diff --git a/plugins/evals/.claude-plugin/plugin.json b/plugins/evals/.claude-plugin/plugin.json index 84d7cc7732..a8619c5791 100644 --- a/plugins/evals/.claude-plugin/plugin.json +++ b/plugins/evals/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "evals", - "version": "0.5.6", + "version": "0.5.7", "description": "LLM evaluation methodology and eval-suite design, based on Anthropic's evaluation guidance. /evals:methodology answers questions on success criteria, eval design, and grading. /evals:design interviews for measurable success criteria and scaffolds a graded eval suite for an LLM app or Claude Code skill. /evals:plugin-eval preflights the CLI and target, prices a suite before running, and reads the with-versus-without delta. /evals:validate checks case files with no model call.", "author": { "name": "Melodic Software", diff --git a/plugins/evals/CHANGELOG.md b/plugins/evals/CHANGELOG.md index c49f51347e..6101514452 100644 --- a/plugins/evals/CHANGELOG.md +++ b/plugins/evals/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog: evals +## [0.5.7] - 2026-10-04 + +### Changed + +- **`plugin-eval` documents the CLI's linked-worktree refusal ([#6374](https://github.com/melodic-software/claude-code-plugins/issues/6374)).** `claude plugin eval` refuses a plugin inside a git repository that "registers more linked worktrees than can be screened"; the new "Linked-worktree refusal" section gives the cause and the workaround (copy the plugin outside any git repository, run against the copy, write `--json` back to the plugin's gitignored `evals/results/`), with a dated record row and recheck trigger. The description gains a trigger phrase for it. + ## [0.5.6] - 2026-10-04 ### Fixed diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index 9356ea5492..1fb397ae80 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Guided practice around the `claude plugin eval` CLI, which runs and scores a plugin's eval suite: preflight (version floor, sandbox backend, target type), static validation with no model call, a cost estimate under the configured ceiling, the run, and the with-versus-without delta read correctly. Use when: 'run my plugin evals', 'plugin eval', 'evaluate this plugin', 'eval my skill', 'does my skill actually fire', 'what is the delta', 'read my eval results', 'compare two eval runs', 'did my change make the skill better', 'is this gain real', 'aggregate-result.json', 'eval CI gate', 'can this machine run evals', 'how much will this eval cost', 'Bash refuses claude plugin eval', 'plugin eval blocked in a worktree', 'can plugin eval measure CLAUDE.md or rules' (it names the route that can). Not for designing success criteria (use /evals:design) or the skill-creator evals.json format (use /skill-quality:check validate-evals when installed)." +description: "Guided practice around the `claude plugin eval` CLI, which runs and scores a plugin's eval suite: preflight (version floor, sandbox backend, target type), static validation with no model call, a cost estimate under the configured ceiling, the run, and the with-versus-without delta read correctly. Use when: 'run my plugin evals', 'plugin eval', 'evaluate this plugin', 'eval my skill', 'does my skill actually fire', 'what is the delta', 'read my eval results', 'compare two eval runs', 'did my change make the skill better', 'is this gain real', 'aggregate-result.json', 'eval CI gate', 'can this machine run evals', 'how much will this eval cost', 'Bash refuses claude plugin eval', 'plugin eval blocked in a worktree', 'plugin eval refuses: too many worktrees', 'can plugin eval measure CLAUDE.md or rules' (it names the route that can). Not for designing success criteria (use /evals:design) or the skill-creator evals.json format (use /skill-quality:check validate-evals when installed)." argument-hint: "[preflight|validate|run|read |ci|init] [target]" user-invocable: true disable-model-invocation: false @@ -103,6 +103,25 @@ result from any directory; for `init` it is `claude plugin eval init --bare / //`. +2. Run `claude plugin eval` against the copy, with an absolute `` and an absolute `--json` path. +3. Point `--json` at the original plugin's gitignored `evals/results/`, so the result lands where the + `read` action and the repository expect it. The copy is a scratch artifact; edits go to the original. + +| Fact | Basis and as-of | Recheck trigger, and what to do when it fires | +|---|---|---| +| `claude plugin eval ` refuses a plugin inside a git repository with many linked worktrees (`registers more linked worktrees than can be screened`); copying the plugin outside any git repository and running against the copy works | Our own observation, not upstream text: melodic-software/claude-code-plugins#6374, Claude Code 2.1.289, Linux/WSL2, observed 2026-10-04, while running the `user-interface` suite | Recheck trigger: a Claude Code release note changes the linked-worktree limit or `plugin eval`'s repository screening. Then re-run `claude plugin eval` on a plugin in a repository with many linked worktrees and refresh this row with the outcome | + ### Tested model and judge `same_model` compares the model under test with the judge model. Resolve each from the planned From 241745a74446e41ce21f34725d081dda9203ffea Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sun, 4 Oct 2026 20:19:26 -0400 Subject: [PATCH 2/3] docs(evals): drop unverified scope claim from the linked-worktree refusal Co-Authored-By: Claude Opus 5.5 --- plugins/evals/skills/plugin-eval/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index 1fb397ae80..8004926dc2 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -108,7 +108,7 @@ result from any directory; for `init` it is `claude plugin eval init --bare Date: Sun, 4 Oct 2026 20:38:19 -0400 Subject: [PATCH 3/3] docs(evals): compress the linked-worktree refusal into a record row Co-Authored-By: Claude Opus 5.5 --- plugins/evals/CHANGELOG.md | 2 +- plugins/evals/skills/plugin-eval/SKILL.md | 23 +++-------------------- 2 files changed, 4 insertions(+), 21 deletions(-) diff --git a/plugins/evals/CHANGELOG.md b/plugins/evals/CHANGELOG.md index 6101514452..8e704820ba 100644 --- a/plugins/evals/CHANGELOG.md +++ b/plugins/evals/CHANGELOG.md @@ -4,7 +4,7 @@ ### Changed -- **`plugin-eval` documents the CLI's linked-worktree refusal ([#6374](https://github.com/melodic-software/claude-code-plugins/issues/6374)).** `claude plugin eval` refuses a plugin inside a git repository that "registers more linked worktrees than can be screened"; the new "Linked-worktree refusal" section gives the cause and the workaround (copy the plugin outside any git repository, run against the copy, write `--json` back to the plugin's gitignored `evals/results/`), with a dated record row and recheck trigger. The description gains a trigger phrase for it. +- **`plugin-eval` documents the CLI's linked-worktree refusal ([#6374](https://github.com/melodic-software/claude-code-plugins/issues/6374)).** `claude plugin eval` refuses a plugin inside a git repository that "registers more linked worktrees than can be screened"; a new record row in the worktree table gives the cause and the workaround (copy the plugin into a new empty directory outside git, excluding `.git` and results, run against the copy, write `--json` back to the plugin's gitignored results directory), with a dated basis and recheck trigger. The description gains a trigger phrase for it. ## [0.5.6] - 2026-10-04 diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index 8004926dc2..9520e6d1d6 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -97,30 +97,13 @@ print the exact command this skill would have run and tell the user to paste it outside Claude Code. For `run` that is `claude plugin eval ...` with an absolute `` and an absolute `--json` path inside the isolated worktree, so the command gives the same result from any directory; for `init` it is `claude plugin eval init --bare `, which takes no -`--json`. After a `run`, read the `--json` file back with the `read` action. +`--json`. After a `run`, read the `--json` file back with the `read` action. The table also records +the CLI's own linked-worktree refusal. | Fact | Basis and as-of | Recheck trigger, and what to do when it fires | |---|---|---| | In a worktree-isolated session the built-in Bash guard refuses any command containing `eval`, including `claude plugin eval --help`, with `this command runs a string through eval, which can't be verified to stay inside the worktree`; the user's own `!` command is refused the same way | melodic-software/claude-code-plugins#5696 repro on Claude Code 2.1.285, Linux/WSL2, verified 2026-10-01 | Recheck trigger: a Claude Code release note touches worktree isolation or `plugin eval`. Then re-run `claude plugin eval --help` from an isolated worktree session and refresh this row with the outcome | - -### Linked-worktree refusal - -A different refusal, from the CLI itself and not the Bash guard: when the plugin sits inside a git -repository that has many linked worktrees, `claude plugin eval` stops before any case runs, saying the -repository "registers more linked worktrees than can be screened". A repository with dozens of linked -worktrees, as this one routinely has, trips it from any checkout. - -The workaround copies the plugin outside any git repository and evaluates the copy: - -1. Copy the plugin to a directory with no git repository above it, leaving out its gitignored results: - `rsync -a --exclude evals/results/ / //`. -2. Run `claude plugin eval` against the copy, with an absolute `` and an absolute `--json` path. -3. Point `--json` at the original plugin's gitignored `evals/results/`, so the result lands where the - `read` action and the repository expect it. The copy is a scratch artifact; edits go to the original. - -| Fact | Basis and as-of | Recheck trigger, and what to do when it fires | -|---|---|---| -| `claude plugin eval ` refuses a plugin inside a git repository with many linked worktrees (`registers more linked worktrees than can be screened`); copying the plugin outside any git repository and running against the copy works | Our own observation, not upstream text: melodic-software/claude-code-plugins#6374, Claude Code 2.1.289, Linux/WSL2, observed 2026-10-04, while running the `user-interface` suite | Recheck trigger: a Claude Code release note changes the linked-worktree limit or `plugin eval`'s repository screening. Then re-run `claude plugin eval` on a plugin in a repository with many linked worktrees and refresh this row with the outcome | +| `claude plugin eval` refuses a plugin inside a git repository with many linked worktrees (`registers more linked worktrees than can be screened`). This is the CLI's own check, not the Bash guard. Workaround: copy the plugin into a newly created empty directory outside any git repository, leaving out git metadata and results, e.g. `rsync -a --exclude .git --exclude 'evals/results' / /`; run against the copy with an absolute target; point `--json` at the original plugin's gitignored results directory | Our own observation: melodic-software/claude-code-plugins#6374, Claude Code 2.1.289, Linux/WSL2, observed 2026-10-04 | Recheck trigger: a Claude Code release note changes the linked-worktree limit or `plugin eval`'s repository screening. Then re-run on a plugin in a repository with many linked worktrees and refresh this row with the outcome | ### Tested model and judge