fix(evals): remove synthetic system prompt hints, install skills natively per harness (AI-1034) by claude[bot] · Pull Request #180 · supabase/evals

claude · 2026-08-07T16:34:52Z

The harness writes its own system prompt into every eval run, so evals measure our framing as much as the agent's out-of-the-box behaviour. Two strings do most of the damage: You are an agent solving a Supabase eval task... and When you are done, end your turn with a short summary.

Problem

The base prompt goes to all four harnesses. Three of them (claude-code, codex, opencode) ship their own system prompt already, and codex/opencode have no system-prompt flag at all, so whatever we pass them lands on the user prompt.
It describes bash and files_read tools that only the ai-sdk harness has.
end your turn with a short summary feeds agentReport, which the LLM judge reads. Coaching stopping behaviour is a scoring change, not a formatting one.
Skills only installed to .claude/skills. Codex reads .agents/skills and cannot read .claude/skills, so the injected ## Available skills listing was the only reason skills ever worked there. The flagless skills add fallback also created dozens of stray agent directories inside the scored workspace.

Fix

Prompt assembly moves to apps/framework/harness/system-prompt.ts and gates on the harness. ai-sdk keeps a short factual prompt because it has none of its own. The three CLI harnesses get '', and the engine then stages no system-prompt file rather than an empty one.
end your turn with a short summary is gone for every harness, including ai-sdk.
The tool-surface description is ai-sdk only.
Skills install natively per harness: skills add <dir> --agent claude-code codex opencode, writing .claude/skills and .agents/skills rather than a directory per known agent. The injected listing is gone for CLI harnesses. The installer also drops a skills-lock.json in the workspace root, which it did before this change too.
buildSystemPrompt drops caller-supplied blocks for a CLI harness rather than trusting call sites to gate them. Only executorMcpServer carries a promptAddendum and only ai-sdk experiments use it, so nothing changes today. A future CLI experiment paired with it would have silently reintroduced injected text.
Raw run artifacts record systemPrompt, so this class of regression shows up in results/*.json rather than only in code. export-results.ts drops the field, so it does not reach the web export.

Testing

Local suites on 0ef478c: framework 7/7, core 122/122, sandbox unit 49/49, sandbox docker 4/4, both typechecks exit 0, pnpm format:check clean.
The docker suite covers the multi-scope install (skill tree lands in both directories, no stray agent dirs). It had never executed before. It passes against a real container.
Single-pair dispatch on this branch, run 32785480269: claude-code-sonnet-5 against build-cli-001-bootstrap-app. systemPrompt in the run artifact is "", and the eval passed.
The apps/framework smoke script fails at the first judge-backed scenario for lack of root .env credentials. Byte-identical failure on main.

Comparability

No recorded numbers change here. The raw results gain a systemPrompt field and the schema is loose, so old files still parse.

fix(core): default the judge to gpt-5.6-sol at medium reasoning effort #222 changed the default judge to gpt-5.6-sol at medium effort. Baselines on main were refreshed after it, so both sides of this comparison are post-fix(core): default the judge to gpt-5.6-sol at medium reasoning effort #222.
feat(evals): prompt-injection security scenarios (AgentDojo-style) #202's prompt-injection evals recorded baselines with the injected prompt still in place. Those are the numbers most likely to move.

Open questions

Should #202's baselines be annotated at merge, or left to the nightly regression refresh to re-baseline?

AI-1034

vercel · 2026-08-07T16:34:59Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
evals	Ignored	Preview	Aug 24, 2026 10:50pm

The eval harness handed every agent a synthetic system prompt. For the three CLI harnesses that prompt described a tool surface they do not have (`bash`, `files_read`) and coached them on how to end a turn — both of which bias exactly what an eval is supposed to measure: out-of-the-box behaviour. The CLI engine now treats the system prompt as optional. When it is empty, nothing is staged and nothing is passed: - `engine` writes `$HOME/.eval/system-prompt.txt` only for a non-empty prompt, and leaves `RunnerExecArgs.systemPromptPath` undefined otherwise. - claude-code omits `--append-system-prompt-file` entirely. - codex and opencode, which have no system-prompt flag, stop prepending a block (and its blank-line separator) to the *user* prompt. `LocalStackSessionArgs` gains a required `agent: AgentHarnessId` so the sandbox layer can tell a CLI harness from the in-process ai-sdk one. Refs AI-1034, #164 Co-Authored-By: Claude <noreply@anthropic.com>

…listing (AI-1034) Skills were installed with a flagless `skills add`, which — finding no agent CLI installed yet — falls back to every one of the 71 agents the CLI knows. That scattered ~53 stray roots across the workspace (`.adal`, `.factory`, and non-dotted `data/` and `skills/` among them), and the workspace is exported into run artifacts and scored. It is also order-dependent: had an agent CLI been installed first, the fallback would have quietly stopped producing `.claude/skills` altogether. `skills add` now names the three CLI harnesses explicitly, which installs into exactly the two project scopes they discover natively: - `.claude/skills/` — Claude Code - `.agents/skills/` — Codex and OpenCode This is the actual fix for Codex, which does not read `.claude/skills` at all and therefore saw no skills in any eval. All three are installed unconditionally: the ids collapse to two directories, an unused copy costs a few kilobytes, and no agent id has to be threaded through `createAgentEnvironment` for correctness. Argument order matters — `--agent` is variadic, so the source directory must precede it and `--skill` terminates the list. `--copy` stays: symlink mode skips agents whose top-level directory does not already exist. With each CLI discovering, advertising and loading skills itself, `buildSkillsPrompt` becomes ai-sdk-only, like `buildToolSurfaceAddendum`. The block it rendered told agents to read `.claude/skills/<name>/SKILL.md` with `files_read` — a path Codex cannot see and a tool no CLI harness has. The post-install check now verifies every agent scope, so a skill missing from one of them fails loudly instead of leaving that harness silently skill-less. Refs AI-1034, #164 Co-Authored-By: Claude <noreply@anthropic.com>

… (AI-1034) Prompt assembly moves out of `run-eval.ts` (an entry script that runs `main()` on import, so it cannot be unit-tested) into `harness/system-prompt.ts`, keyed on `exp.agent.id`. Every block is now ai-sdk-only — the task framing, the tool-surface addendum, the skills listing — so a CLI harness assembles to `''` and the engine stages no system prompt file. The two "end your turn with a short summary" sentences are gone from both modes: stopping behaviour is part of what is measured. `runOne` now returns the exact assembled `systemPrompt`, so it lands in `results/<experiment>/<eval>.json` and what an agent was told is verifiable from the artifacts. It was previously unrecorded for every CLI harness. `export-results.ts` builds an explicit whitelist, so it does not reach the published web data. `apps/framework` gains a `test` script (`vitest run harness`), wired into `check`, so the prompt-assembly tests have a runner. Refs AI-1034, #164 Co-Authored-By: Claude <noreply@anthropic.com>

Brings in a week of upstream work, most notably #192 (eval execution moved from GitHub Actions to Vercel Sandboxes), #178 (claude-code MCP tool-name normalization), #168 (RLS docs guide eval) and several results refreshes. One conflict, in apps/framework/package.json: both sides extended the `check` script. Upstream added `test:vercel-runner` for the new Vercel runner tests; this branch added `test` to give the prompt-assembly tests a runner. Both are kept: check: pnpm typecheck && pnpm test && pnpm test:framework && pnpm test:vercel-runner The dependency block in that file merged cleanly (upstream re-sorted it and added @vercel/sandbox, p-limit, p-retry and zod). run-eval.ts, packages/core/src/agents/engine.ts and packages/core/src/index.ts merged without conflict and both intents are intact: - run-eval.ts now takes readFlag/positiveInteger from lib/cli-args.js (upstream hoisted the local readFlag out) while still assembling its prompt through buildSystemPrompt() and recording systemPrompt in the result JSON. - engine.ts still stages no system-prompt file when the assembled prompt is empty, and additionally passes upstream's mcpServerNames to parseTranscript. - index.ts keeps upstream's parser/transcript type changes alongside this branch's AgentRunArgs.systemPrompt and LocalStackSessionArgs.agent docs. No new CLI harness landed upstream (still claude-code, codex, opencode), so the `skills add --agent` list is unchanged. The new Vercel runner shells out to `pnpm eval`, so it inherits this branch's prompt behaviour with no change. apps/web/src/data/*eval-results.json and results/ are untouched by this branch and the merge takes main's versions wholesale. Refs AI-1034, #164 Co-Authored-By: Claude <noreply@anthropic.com>

Second resolution against upstream; the previous one merged main up to d5db72e. Five commits landed since: the SUPABASE_MCP_SERVER_PATH local MCP build support (#193), the AgentDojo-style prompt-injection scenarios (#202), and three eval-result refreshes (#208, #209, #213). Two conflicts, both from #193 threading a new `mounts` option through the sandbox layer alongside this branch's new `agent` option: - packages/sandbox/src/bare-sandbox.ts — union of both. This branch's named `BareSandboxOptions` (required, because `agent` is mandatory) gains upstream's `mounts?: readonly SandboxMount[]`, and the import list takes `AgentHarnessId` and `SandboxMount`. - apps/framework/harness/run-eval.ts — the `createBareSandbox` call passes both `agent: exp.agent.id` and `mounts: supabaseMcpServerMounts()`. No new harness landed, so the no-synthetic-prompt gating is unchanged. apps/web/src/data/ and results/ are taken from main verbatim. Co-Authored-By: Claude <noreply@anthropic.com>

…t-hints

…aller buildSystemPrompt gated its base framing on the agent but passed the addendum and skills blocks straight through. Both are ai-sdk-only today, but that's enforced by their producers across three files rather than by the assembler, and a block reaching a CLI harness fails silently: no error, no red test, just an eval measuring our prompt instead of the agent's own behaviour. An MCP server carrying a promptAddendum is the live path in. Only executorMcpServer has one, and only ai-sdk experiments use it, so nothing changes today — a new CLI-harness experiment paired with it would.

seanoliver · 2026-08-25T22:18:03Z

Closing in favour of a two-PR stack:

fix(sandbox): install skills natively per harness (AI-1034) #240 makes the native skills install deterministic. Bottom of the stack, merges to main.
fix(framework): remove the harness system prompt for CLI agents (AI-1034) #241 removes the harness system prompt for the CLI agents. Stacked on fix(sandbox): install skills natively per harness (AI-1034) #240.

Same change, split along package lines so each half can be reviewed on its own rationale. Every commit here is carried over, plus review fixes and verification runs. Nothing from this branch is lost.

One correction worth recording, since this PR's description got it wrong: removing the injected skills listing would not have stripped Codex of skills. The old flagless skills add fell back to installing for every agent it knows, and that set already included .agents/skills. The install fix is about determinism and workspace pollution, not about skills reaching an agent.

## What's inside - ~215 lines of skills install changes ([skills.ts](https://github.com/supabase/evals/pull/240/changes#diff-badfb2869f8854f19b00a8e02d6e9d94ef4e1cea66004399b290c03b8ecfb1fa), [local-stack-runtime.ts](https://github.com/supabase/evals/pull/240/changes#diff-366863d3b55e1541a2a038a7af0efb7952e345a84cdcdcce85727ca5dd0aef7f)) - ~5 lines of plumbing that does nothing yet ([index.ts](https://github.com/supabase/evals/pull/240/changes#diff-9a4ceebe7c6f86856371906c3f061d3b56b7457022b05179884a113e7ced67e8)), for the PR stacked on this one - ~195 lines of tests, skimmable ([unit.test.ts](https://github.com/supabase/evals/pull/240/changes#diff-d9864aa0981cd32280dc8a1cca41ada618e91ef3684f52c4c49234e378aee648), [docker.test.ts](https://github.com/supabase/evals/pull/240/changes#diff-216a9cc0506fc98aac7b13fd029e70581f3e089c35dcde73fac0f2217e93b00f)) Bottom of a 2-PR stack under #241, which removes the system prompt we write for the CLI agents. Claude Tag wrote the first pass at this (previously #180, now closed). ## Problem Installing a skill means copying its folder somewhere an agent will look, and each agent reads a different directory. - `skills add` takes a flag naming which agents to install for, but we weren't not passing it. - Without that flag, the CLI tries to detect an installed agent, and when it cannot, it installs for all 71 it knows about. - So every sandbox got ~52 folders it had no use for: `.aider-desk`, `.factory`, `.kilocode`, `.windsurf`, `.zencoder`, and non-dotted `data/` and `skills/` among them. Each holds a copy of the same two Supabase skills. - It also didn't install Codex skills into the native `.agents/skills` folder, which required us to include an addendum to the Codex prompt to get it to look in the Claude skills folder (`.claude/skills`). ## Changes ### Install Name the three agents we run in this repo: ```bash skills add '<dir>' --agent claude-code codex opencode --skill '*' --copy --yes ``` That writes two directories, `.claude/skills` and `.agents/skills`, instead of 52. `--copy` is required so that `.claude/skills` gets created in an empty eval workspace. Without it only `.agents/skills` shows up. After installing, `installSkills` confirms every skill is actually present in both directories (`.claude/skills` and `.agents/skills`). It used to check `.claude/skills` only. ### Plumbing for the next PR The prompt builders now take the agent as an argument and ignore it. This is used by #241 to skip the prompt text for the CLI agents. ## How to review Confirm the new install works: ```bash pnpm --filter @supabase-evals/sandbox test:docker ``` If interested, run both install strategies side by side in a temp folder to see the difference: ```bash mkdir -p /tmp/probe/stage/skills/demo /tmp/probe/old /tmp/probe/new printf -- '---\nname: demo\ndescription: Demo.\n---\n\nBody.\n' > /tmp/probe/stage/skills/demo/SKILL.md # old, no --agent cd /tmp/probe/old env -u CLAUDECODE -u CLAUDE_CODE_ENTRYPOINT npx -y skills@1.5.11 add /tmp/probe/stage --skill '*' --copy --yes ls -A # new, three agents named cd /tmp/probe/new env -u CLAUDECODE -u CLAUDE_CODE_ENTRYPOINT npx -y skills@1.5.11 add /tmp/probe/stage --agent claude-code codex opencode --skill '*' --copy --yes ls -A ``` You should see 53 entries (old) vs. 3 (new). ## Follow up tasks - Remove `skills-lock.json` from the workspace after installing. The CLI drops it in the root and it ends up in the exported workspace, same as before this change. - Assert the full set of workspace directories in the docker test. It currently samples five of the ~52 the old fallback created, so it catches that fallback firing but does not prove nothing else was written. - Add a check that each agent actually surfaces the skills it finds. Both this PR and #241 assume it, and nothing tests it. Ref AI-1034 --------- Co-authored-by: Matt Rossman <22670878+mattrossman@users.noreply.github.com>

claude Bot force-pushed the fix/remove-eval-prompt-hints branch from f2952bb to ecfee71 Compare August 7, 2026 16:37

claude added 3 commits August 7, 2026 16:43

claude Bot force-pushed the fix/remove-eval-prompt-hints branch from ecfee71 to a2873a2 Compare August 7, 2026 16:46

vercel Bot deployed to Preview August 14, 2026 17:41 View deployment

vercel Bot deployed to Preview August 18, 2026 08:16 View deployment

seanoliver added 2 commits August 24, 2026 15:18

Merge remote-tracking branch 'origin/main' into fix/remove-eval-promp…

61b9461

…t-hints

seanoliver force-pushed the fix/remove-eval-prompt-hints branch from 0ef478c to 1120e9b Compare August 24, 2026 22:50

seanoliver mentioned this pull request Aug 25, 2026

fix(framework): remove the harness system prompt for CLI agents (AI-1034) #241

Merged

seanoliver closed this Aug 25, 2026

seanoliver mentioned this pull request Aug 25, 2026

fix(sandbox): install skills natively per harness (AI-1034) #240

Merged

Navigation Menu

Uh oh!

fix(evals): remove synthetic system prompt hints, install skills natively per harness (AI-1034) - #180

fix(evals): remove synthetic system prompt hints, install skills natively per harness (AI-1034)#180
claude[bot] wants to merge 7 commits into
mainfrom
fix/remove-eval-prompt-hints

claude Bot commented Aug 7, 2026 •

edited by seanoliver

Loading

Uh oh!

vercel Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

seanoliver commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

claude Bot commented Aug 7, 2026 • edited by seanoliver Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Testing

Comparability

Open questions

Uh oh!

vercel Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seanoliver commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

claude Bot commented Aug 7, 2026 •

edited by seanoliver

Loading

vercel Bot commented Aug 7, 2026 •

edited

Loading