diff --git a/plugins/evals/.claude-plugin/plugin.json b/plugins/evals/.claude-plugin/plugin.json index 84d7cc7732..a8619c5791 100644 --- a/plugins/evals/.claude-plugin/plugin.json +++ b/plugins/evals/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "evals", - "version": "0.5.6", + "version": "0.5.7", "description": "LLM evaluation methodology and eval-suite design, based on Anthropic's evaluation guidance. /evals:methodology answers questions on success criteria, eval design, and grading. /evals:design interviews for measurable success criteria and scaffolds a graded eval suite for an LLM app or Claude Code skill. /evals:plugin-eval preflights the CLI and target, prices a suite before running, and reads the with-versus-without delta. /evals:validate checks case files with no model call.", "author": { "name": "Melodic Software", diff --git a/plugins/evals/CHANGELOG.md b/plugins/evals/CHANGELOG.md index c49f51347e..8e704820ba 100644 --- a/plugins/evals/CHANGELOG.md +++ b/plugins/evals/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog: evals +## [0.5.7] - 2026-10-04 + +### Changed + +- **`plugin-eval` documents the CLI's linked-worktree refusal ([#6374](https://github.com/melodic-software/claude-code-plugins/issues/6374)).** `claude plugin eval` refuses a plugin inside a git repository that "registers more linked worktrees than can be screened"; a new record row in the worktree table gives the cause and the workaround (copy the plugin into a new empty directory outside git, excluding `.git` and results, run against the copy, write `--json` back to the plugin's gitignored results directory), with a dated basis and recheck trigger. The description gains a trigger phrase for it. + ## [0.5.6] - 2026-10-04 ### Fixed diff --git a/plugins/evals/skills/plugin-eval/SKILL.md b/plugins/evals/skills/plugin-eval/SKILL.md index 9356ea5492..9520e6d1d6 100644 --- a/plugins/evals/skills/plugin-eval/SKILL.md +++ b/plugins/evals/skills/plugin-eval/SKILL.md @@ -1,5 +1,5 @@ --- -description: "Guided practice around the `claude plugin eval` CLI, which runs and scores a plugin's eval suite: preflight (version floor, sandbox backend, target type), static validation with no model call, a cost estimate under the configured ceiling, the run, and the with-versus-without delta read correctly. Use when: 'run my plugin evals', 'plugin eval', 'evaluate this plugin', 'eval my skill', 'does my skill actually fire', 'what is the delta', 'read my eval results', 'compare two eval runs', 'did my change make the skill better', 'is this gain real', 'aggregate-result.json', 'eval CI gate', 'can this machine run evals', 'how much will this eval cost', 'Bash refuses claude plugin eval', 'plugin eval blocked in a worktree', 'can plugin eval measure CLAUDE.md or rules' (it names the route that can). Not for designing success criteria (use /evals:design) or the skill-creator evals.json format (use /skill-quality:check validate-evals when installed)." +description: "Guided practice around the `claude plugin eval` CLI, which runs and scores a plugin's eval suite: preflight (version floor, sandbox backend, target type), static validation with no model call, a cost estimate under the configured ceiling, the run, and the with-versus-without delta read correctly. Use when: 'run my plugin evals', 'plugin eval', 'evaluate this plugin', 'eval my skill', 'does my skill actually fire', 'what is the delta', 'read my eval results', 'compare two eval runs', 'did my change make the skill better', 'is this gain real', 'aggregate-result.json', 'eval CI gate', 'can this machine run evals', 'how much will this eval cost', 'Bash refuses claude plugin eval', 'plugin eval blocked in a worktree', 'plugin eval refuses: too many worktrees', 'can plugin eval measure CLAUDE.md or rules' (it names the route that can). Not for designing success criteria (use /evals:design) or the skill-creator evals.json format (use /skill-quality:check validate-evals when installed)." argument-hint: "[preflight|validate|run|read |ci|init] [target]" user-invocable: true disable-model-invocation: false @@ -97,11 +97,13 @@ print the exact command this skill would have run and tell the user to paste it outside Claude Code. For `run` that is `claude plugin eval ...` with an absolute `` and an absolute `--json` path inside the isolated worktree, so the command gives the same result from any directory; for `init` it is `claude plugin eval init --bare `, which takes no -`--json`. After a `run`, read the `--json` file back with the `read` action. +`--json`. After a `run`, read the `--json` file back with the `read` action. The table also records +the CLI's own linked-worktree refusal. | Fact | Basis and as-of | Recheck trigger, and what to do when it fires | |---|---|---| | In a worktree-isolated session the built-in Bash guard refuses any command containing `eval`, including `claude plugin eval --help`, with `this command runs a string through eval, which can't be verified to stay inside the worktree`; the user's own `!` command is refused the same way | melodic-software/claude-code-plugins#5696 repro on Claude Code 2.1.285, Linux/WSL2, verified 2026-10-01 | Recheck trigger: a Claude Code release note touches worktree isolation or `plugin eval`. Then re-run `claude plugin eval --help` from an isolated worktree session and refresh this row with the outcome | +| `claude plugin eval` refuses a plugin inside a git repository with many linked worktrees (`registers more linked worktrees than can be screened`). This is the CLI's own check, not the Bash guard. Workaround: copy the plugin into a newly created empty directory outside any git repository, leaving out git metadata and results, e.g. `rsync -a --exclude .git --exclude 'evals/results' / /`; run against the copy with an absolute target; point `--json` at the original plugin's gitignored results directory | Our own observation: melodic-software/claude-code-plugins#6374, Claude Code 2.1.289, Linux/WSL2, observed 2026-10-04 | Recheck trigger: a Claude Code release note changes the linked-worktree limit or `plugin eval`'s repository screening. Then re-run on a plugin in a repository with many linked worktrees and refresh this row with the outcome | ### Tested model and judge