chore: refresh eval results - #239
Draft
Draft
Conversation
The eval harness handed every agent a synthetic system prompt. For the three CLI harnesses that prompt described a tool surface they do not have (`bash`, `files_read`) and coached them on how to end a turn — both of which bias exactly what an eval is supposed to measure: out-of-the-box behaviour. The CLI engine now treats the system prompt as optional. When it is empty, nothing is staged and nothing is passed: - `engine` writes `$HOME/.eval/system-prompt.txt` only for a non-empty prompt, and leaves `RunnerExecArgs.systemPromptPath` undefined otherwise. - claude-code omits `--append-system-prompt-file` entirely. - codex and opencode, which have no system-prompt flag, stop prepending a block (and its blank-line separator) to the *user* prompt. Refs AI-1034, #164
…034) With each CLI discovering, advertising and loading skills itself, `buildSkillsPrompt` becomes ai-sdk-only, like `buildToolSurfaceAddendum`. The block it rendered told agents to read `.claude/skills/<name>/SKILL.md` with `files_read` — a path Codex cannot see and a tool no CLI harness has, duplicating and contradicting what the agent's own harness already tells it. `buildToolSurfaceAddendum` gets the same gate: `createCliAgent` ignores `args.tools`, so a CLI agent works the workspace through its own built-in tools and that text names tools it does not have. Refs AI-1034, #164
… (AI-1034) Prompt assembly moves out of `run-eval.ts` (an entry script that runs `main()` on import, so it cannot be unit-tested) into `harness/system-prompt.ts`, keyed on `exp.agent.id`. Every block is now ai-sdk-only — the task framing, the tool-surface addendum, the skills listing — so a CLI harness assembles to `''` and the engine stages no system prompt file. The two "end your turn with a short summary" sentences are gone from both modes: stopping behaviour is part of what is measured. `runOne` now returns the exact assembled `systemPrompt`, so it lands in `results/<experiment>/<eval>.json` and what an agent was told is verifiable from the artifacts. It was previously unrecorded for every CLI harness. `export-results.ts` builds an explicit whitelist, so it does not reach the published web data. `apps/framework` gains a `test` script (`vitest run harness`), wired into `check`, so the prompt-assembly tests have a runner. Refs AI-1034, #164
…aller buildSystemPrompt gated its base framing on the agent but passed the addendum and skills blocks straight through. Both are ai-sdk-only today, but that's enforced by their producers across three files rather than by the assembler, and a block reaching a CLI harness fails silently: no error, no red test, just an eval measuring our prompt instead of the agent's own behaviour. An MCP server carrying a promptAddendum is the live path in. Only executorMcpServer has one, and only ai-sdk experiments use it, so nothing changes today — a new CLI-harness experiment paired with it would.
buildSystemPrompt discarded a non-empty addendum for a CLI harness. That is as silent as injecting it, and the block can be load-bearing: executorMcpServer's addendum is the pause/resume protocol its tools require, not a tool description. A CLI harness paired with it would get the tools and none of the protocol, then stall on the first paused execution with a recorded prompt of '' explaining nothing. Throw instead. The producers already gate their output, so anything arriving here means an experiment is misconfigured. Also drop BareSandboxHandle.promptAddendum. Its only caller is guarded by agentRunsInSandbox, true only for CLI harnesses, so buildSkillsPrompt could only ever return '' for it.
run-eval.ts called main() at module scope, so importing it dispatched a run and then called process.exit. Nothing exercised that until this branch added `vitest run harness`, which sweeps in run-eval.test.ts, and that file imports assertLocalMatchesInterface from run-eval.js. The suite passes only because main() loses the race with vitest teardown; with credentials in the environment it would start real sandbox work inside the test worker and leak containers past process.exit. Guard the invocation on argv[1]. Importing the module is now inert, and running it directly is unchanged. Also name the offending argument in the buildSystemPrompt error and point at the runtime or MCP server that produced it, rather than calling it a "harness addendum" when local-stack hands over a pre-joined blob.
Dropping promptAddendum left createBareSandbox with an agent option it never reads and a comment claiming it decides whether skills are advertised in the prompt. Nothing in that function advertises anything. The local-stack session still needs its agent; this one does not.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
seanoliver
force-pushed
the
sean/ai-1034-harness-native-prompts
branch
from
September 8, 2026 20:51
5bed078 to
3cefa60
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refreshes
apps/web/src/data/eval-results.jsonfrom the latest automated eval run.