Skip to content

docs(evals): document plugin eval's linked-worktree refusal - #6410

Merged
kyle-sexton merged 3 commits into
mainfrom
docs/6374-plugin-eval-linked-worktrees
Oct 5, 2026
Merged

kyle-sexton merged 3 commits into
mainfrom
docs/6374-plugin-eval-linked-worktrees

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Closes #6374

Summary

claude plugin eval refuses a plugin inside a git repository with many linked worktrees ("registers more linked worktrees than can be screened"). This repository routinely has dozens, so every session rediscovered it. /evals:plugin-eval documented only the Bash guard's separate worktree refusal.

Fix

  • plugins/evals/skills/plugin-eval/SKILL.md: a "Linked-worktree refusal" section after "Worktree isolation" with the cause, the copy-outside-git workaround (rsync excluding the gitignored results, run against the copy with absolute paths, write --json back to the original's evals/results/), and a dated record row with a recheck trigger. The description gains the trigger phrase 'plugin eval refuses: too many worktrees'.
  • evals 0.5.6 → 0.5.7 with a CHANGELOG entry.

Verification

  • scripts/check-changelog-parity.sh --check-bump origin/main, node scripts/generate-catalog.mjs --check, node scripts/generate-cheatsheet.mjs --check: pass.
  • markdownlint-cli2 and typos --config _typos.toml on SKILL.md and CHANGELOG.md: clean.
  • Description length about 993 of 1024 characters.
  • The workaround is the one used for the user-interface suite in feat(user-interface): add the user-interface plugin #6318 (Phase 6).

Related

🤖 Generated with Claude Code

kyle-sexton and others added 2 commits October 4, 2026 20:15
Adds the CLI's own refusal for a plugin inside a repository with many linked worktrees, its cause, the copy-outside-git workaround, and a dated record row. Bumps evals to 0.5.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…usal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 5, 2026 00:33
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-05T00:37:16.277469Z 241745a Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@claude

claude Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 2m 9s —— View job


Code review complete

  • Gather context (PR diff, changed files, CLAUDE.md/REVIEW.md)
  • Review plugins/evals/skills/plugin-eval/SKILL.md changes
  • Review plugins/evals/CHANGELOG.md and version bump
  • Check PR body contract and cross-file conventions
  • Post findings

No findings to report. This is a small, self-contained docs change and it holds up well under review:

  • The new "Linked-worktree refusal" section (SKILL.md#L106-L123) follows the repo's upstream-drift convention correctly for a probed behavior: it states the observation in the author's own words, points at the originating issue (docs(evals): document that plugin eval refuses a plugin inside a repo with many linked worktrees #6374) as the probe evidence, carries an as-of date, and names an observable recheck trigger — matching the format of the existing "Worktree isolation" row directly above it.
  • The second commit (241745a) dropped an unverified scope claim ("It applies to run only") before this review ever ran — good self-correction, and consistent with the project's bar for not asserting what wasn't actually confirmed.
  • Cross-checked the author's claimed description length: the frontmatter description string is 993 characters (measured directly), matching the PR body's claim exactly.
  • plugin.json version (0.5.6 → 0.5.7) and the new CHANGELOG.md entry are consistent with each other and with the file's existing entry format. evals isn't in scripts/fragment-plugins.txt, so a standalone changelog entry (rather than a shared fragment) is the right move.
  • PR body satisfies the PR body contract: Closes #6374 plus non-empty ## Summary/## Fix/## Verification/## Related sections. Title is valid Conventional Commits form.

Not independently verified (author-claimed only, per this lane's rule against restating unverified claims as confirmed): scripts/check-changelog-parity.sh --check-bump, generate-catalog.mjs --check, generate-cheatsheet.mjs --check, markdownlint-cli2, and typos — this lane's sandboxed Bash only permits git/gh-style commands, so node/script invocations were denied before I could run them. Worth having CI's own lanes confirm these rather than relying solely on the PR description.

@claude

claude Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 56s —— View job


Security review complete

  • Gather context (PR diff, changed files)
  • Review changed files for security-relevant surface
  • Check instruction-surface deletions lens (SKILL.md edits)
  • Post findings or clear result

No security issues found.

This PR only touches documentation and a version bump:

  • plugins/evals/.claude-plugin/plugin.json: version bump 0.5.6 → 0.5.7.
  • plugins/evals/CHANGELOG.md: new changelog entry describing the doc addition.
  • plugins/evals/skills/plugin-eval/SKILL.md: adds a "Linked-worktree refusal" section documenting a claude plugin eval CLI behavior and a workaround (copy the plugin outside any git repository via rsync, run the eval against the copy, write --json results back to the original's gitignored evals/results/), plus a dated record row and an added trigger phrase in the frontmatter description.

Checked against the instruction-surface-deletions lens required by this skill: the diff is additive only (a new section, a new table row, an appended trigger phrase) — nothing is deleted, narrowed, or softened, so that lens finds nothing to flag. The documented workaround is advice for a human operator running shell commands against their own checkout; it does not interpolate untrusted input into a command, execute code, or handle secrets/tokens, so there's no injection or credential-handling surface here.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review has reviewed this pull request through 56c7959; a later push is reviewed from there.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

claude-review has reviewed this pull request through 56c7959; a later push is reviewed from there.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 241745a744

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/evals/skills/plugin-eval/SKILL.md Outdated
Comment thread plugins/evals/skills/plugin-eval/SKILL.md Outdated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 50s —— View job


Security review complete

  • Gather context (incremental diff since 241745a, changed files)
  • Review plugins/evals/CHANGELOG.md changes
  • Review plugins/evals/skills/plugin-eval/SKILL.md changes
  • Check instruction-surface-deletions lens
  • Post findings

No security issues found.

This increment (reviewing only what changed since 241745a, per the lane's incremental scope) touches:

  • plugins/evals/CHANGELOG.md: rewords the entry to match the reworked SKILL.md section.
  • plugins/evals/skills/plugin-eval/SKILL.md: folds the standalone "Linked-worktree refusal" section into a single new row in the existing worktree-isolation table (SKILL.md#L106-L107), and updates the workaround text in two ways that happen to resolve both findings Codex raised against the prior revision:
    • The rsync example now also --exclude .git (previously it only excluded evals/results/), so the scratch copy no longer carries git metadata that could re-trip the same linked-worktree screening.
    • The destination is now described as "a newly created empty directory," replacing the prior reusable <dir outside git>/<plugin-name>/ path, so a stale file from a deleted/renamed case or component can't silently linger across runs.

This remains documentation only: advice for a human operator to run shell commands by hand against their own checkout. No command is constructed from untrusted input, no code is executed by this skill, and no secrets/tokens are handled, so there's no injection or credential-handling surface.

Instruction-surface-deletions lens: the diff deletes the standalone ### Linked-worktree refusal section and merges its content into the table (docs/conventions/instruction-exception-register/README.md was present and consulted). The one sentence dropped outright — "The copy is a scratch artifact; edits go to the original." — doesn't match any of the six Gate 0 classes (irreversible-action, secret-handling, data-integrity, external-publication, legal-compliance, agent-authority); it's operator-facing documentation style, not an enforced guardrail, and the diff is otherwise a compression that doesn't change what the workaround forbids or requires. No finding.

@claude

claude Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 3m 5s —— View job


Code review complete

  • Gather context (incremental diff, custom instructions)
  • Load /review:code-review skill and follow it
  • Review incremental changes to plugins/evals/CHANGELOG.md and plugins/evals/skills/plugin-eval/SKILL.md since 241745a
  • Post findings

No findings to report. Commit 56c7959 compresses the "Linked-worktree refusal" section into a single row of the existing worktree-isolation table and, in doing so, resolves both findings from Codex's prior review of 241745a (#6410 (review) comments):

  • P1 (.git left in the scratch copy): the rsync line now adds --exclude .git alongside the existing results exclusion — rsync -a --exclude .git --exclude 'evals/results' <plugin-dir>/ <new empty dir>/ (SKILL.md#L122). Since .git has no slash, rsync's exclude matches it as a basename at any depth, so it correctly drops both a full .git directory and a linked worktree's .git pointer file, addressing the exact failure mode Codex flagged.
  • P2 (stale files in a reused destination): the instruction now says to copy into "a newly created empty directory outside any git repository" rather than a directory that might be reused across runs, which is the alternative Codex's suggestion offered (require a fresh empty destination instead of adding --delete).

The CHANGELOG wording (CHANGELOG.md#L7) matches the SKILL.md change in substance. No new correctness issues in this incremental diff.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 5, 2026
Merged via the queue into main with commit 62869cc Oct 5, 2026
15 checks passed
@kyle-sexton
kyle-sexton deleted the docs/6374-plugin-eval-linked-worktrees branch October 5, 2026 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs(evals): document that plugin eval refuses a plugin inside a repo with many linked worktrees

1 participant