Repository navigation
feat(repo): Codex and Copilot CLI harness profiles, native review mode #116
+2,868
−127
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| ### Added | ||
|
|
||
| - **Codex CLI and Copilot CLI harness profiles for `repo run` and `repo reach`** — `--harness codex`, `codex-review`, `copilot` and `copilot-review` run oneDAL-style repository tasks under `codex exec`, Codex's native review mode (`codex exec review --commit HEAD`), `copilot -p` and `copilot -p` with a `/review`-prefixed prompt, so guidance arms can be compared per harness, not only under Claude Code. Their JSON traces convert to `events.jsonl` (`trace: codex-json`, `copilot-json`). A profile can now run the harness's own prompt (`native_prompt`) and convert its answer before grading (`answer_format: codex-review` appends a findings block parsed from Codex's review text: the final message decides when it holds a finding or a clean verdict, and the earlier messages are read only when it has neither); `repo reach` refuses a `native_prompt` profile. `bin_companions` mounts the helper files an executable needs beside it (Codex's `codex-code-mode-host`). | ||
|
|
||
| ### Fixed | ||
|
|
||
| - **`repo reach` writes the profile's `config_files`** — only `repo run` wrote them, so a profile whose endpoint or hooks live in a config file (a Codex `config.toml` naming a provider) probed a different setup than it ran. | ||
| - **A profile's `trace` is part of the execution id** — a profile edited only in its trace converter kept its id, so `repo run` reused cells graded from another converter's answer and read events; a `claude-stream-json` trace adds no key, so Claude Code ids are unchanged. | ||
| - **A trace record holding U+2028, U+2029 or U+0085 is one record** — the Claude Code, Codex and Copilot converters split their JSON-lines stdout with `str.splitlines`, which also breaks at those characters, legal unescaped in a JSON string; such a record was dropped as not JSON, and an answer in it was lost. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -31,7 +31,7 @@ SPDX-License-Identifier: Apache-2.0 | |
| > Slice S3 adds harness profiles (§6) and the reach probe (§7): | ||
| > | ||
| > ```bash | ||
| > python cli.py repo reach <source> --manifest <path> --model <id> [--harness claude-code|codex|copilot|copilot-review|<profile.yaml>] | ||
| > [--bin <cli>] [--scenarios root,cwd,read] [--only <scenario>] [--dry-run] | ||
| > ``` | ||
| > | ||
|
|
@@ -417,8 +417,11 @@ reads the upstream base (`base_sha`) and the workspace start commit | |
| `cache_read_tokens`, `cache_write_tokens`) over the same scope as `cost_usd`. | ||
| `usage_scope` is `tree` when they include subagents and helper models (Claude | ||
| Code's `modelUsage`), `main` when the harness reported the parent's only, and | ||
| `null` when it reported none. Codex counts `cached_input_tokens` inside its | ||
| `input_tokens`; the converter moves them to `cache_read_tokens`, so | ||
| `input_tokens` is the uncached part in every cell that reports tokens, as with | ||
| Claude. `subagent_calls` counts the parent's and subagents' `Agent`/`Task` | ||
| calls; `null` when the harness wrote no result record. | ||
| `first_prompt_tokens` is what the first main-thread model call was sent | ||
| (`input + cache_creation + cache_read` tokens of that call, counted once per | ||
| message id, as `context trace` reads it): everything start-up loaded plus the | ||
|
|
@@ -855,7 +858,7 @@ env_allow: [PATH, HOME, AWS_REGION, AWS_BEARER_TOKEN_BEDROCK, ANTHROPIC_API_KEY, | |
| env_set: {CLAUDE_CODE_DISABLE_BACKGROUND_TASKS: "1", DISABLE_AUTOUPDATER: "1"} | ||
| egress: ["bedrock-runtime.*.amazonaws.com", "api.anthropic.com"] # the only hosts the agent container reaches (§2) | ||
| endpoint_env: ANTHROPIC_BASE_URL # used by payload capture, if set | ||
| trace: claude-stream-json # converter to events.jsonl; or: codex-json, copilot-json, text | ||
| guidance: # DECLARED; verified by the reach probe, never trusted | ||
| names: [CLAUDE.md, CLAUDE.local.md, AGENTS.md] | ||
| resolution: ancestors-of-cwd + lazy-on-subtree-access | ||
|
|
@@ -904,8 +907,62 @@ args_if_present: # argv added after the "{tools}" element wh | |
| Fix or remove the file in the task base; `--tools read` or `none` do not | ||
| pass it, so `repo run` does not check it there. | ||
|
|
||
| - `trace` names the converter of the harness's stdout to `events.jsonl`: | ||
| `claude-stream-json`, `codex-json` (`codex exec --json`), `copilot-json` | ||
| (`copilot -p --output-format json`) or `text` (stdout is the answer, no | ||
| events). Codex has no read tool, so its `read` events come from shell | ||
| commands that only print files (`cat`, `head`, `tail`, `sed -n`, `nl`). | ||
| Neither Codex nor Copilot reports a cost: `cost_usd` is `null` and a batch | ||
| budget counts each cell's cap. | ||
| - `bin_companions` names files the executable starts from its own directory. | ||
| `repo run` mounts the executable alone at `/eval/harness/<name>`, so each | ||
| companion is mounted read-only beside it, from the directory of the file | ||
| `--bin` resolves to (a symlink is followed); a missing one | ||
| stops the run before any agent starts. Codex 0.160 needs | ||
| `codex-code-mode-host`: every tool call goes through it. | ||
| - `repo reach` writes `config_files` into each scenario's config directory as | ||
| `repo run` does, with `{config_dir}` and `{support_dir}` as host paths. | ||
| - `native_prompt: true` marks a harness that runs its own built-in prompt, such | ||
| as a native review mode. `invoke` then passes no `{prompt}` (and there is no | ||
| `prompt_suffix`), the agent never sees the task prompt, and `repo reach` | ||
| refuses the profile. Use such a profile only on tasks whose grader reads what | ||
| it answers, which `answer_format` provides. | ||
| - `answer_format` converts the final answer before grading. `codex-review` | ||
|
There was a problem hiding this comment.
Contributor
Author
There was a problem hiding this comment. Fixed in 8ad1cb7: the contract now says a missing findings block means no review rather than a clean one, and that a priority outside P0-P3 gets
antonwolfy marked this conversation as resolved.
Resolved
|
||
| keeps Codex's review text and appends a ```` ```json ```` block of findings | ||
| `{file, line, end_line, severity, priority, rule, message}` parsed from its | ||
| `- [P1] title — path:12-14` lines (a failed run, an empty answer or a `[Pn]` | ||
| header that does not parse gets no block, so no block means no review, not a clean one; a priority outside P0–P3 gets `severity` null): `file` relative to the workspace, | ||
| `severity` `blocker` (P0), `major` (P1) or `minor` (P2, P3), `rule` the title | ||
| and `message` the body. The final message decides when it holds a finding or | ||
| a clean-verdict sentence (`No issues found.`): an earlier draft it revises or | ||
| takes back is not graded. A final message with neither (`Review finished.`) | ||
| points back at the review, so the findings of every message are read, each | ||
| once as its last version. | ||
|
|
||
| Built-in profiles besides `claude-code`: | ||
|
|
||
| | profile | runs | trace | notes | | ||
| |---|---|---|---| | ||
| | `codex` | `codex exec --json … -- {prompt}` | `codex-json` | `CODEX_HOME` is the per-run config dir; no turn or spend flag, the timeout bounds a run; `none`/`read` get Codex's read-only sandbox, which needs unprivileged user namespaces | | ||
| | `codex-review` | `codex exec review --commit HEAD --json` | `codex-json` | `native_prompt`, `answer_format: codex-review`; review tasks only (the change under review is HEAD) | | ||
| | `copilot` | `copilot --prompt={prompt} --output-format json --allow-all-tools …` | `copilot-json` | `COPILOT_HOME` per run; a GitHub token as `COPILOT_GITHUB_TOKEN` (`GH_TOKEN` and `GITHUB_TOKEN` are not forwarded: Copilot leaves them readable by the agent's commands), or BYOK through the `COPILOT_PROVIDER_*` variables; `none` names a tool that does not exist, because an empty `--available-tools` leaves every tool on | | ||
| | `copilot-review` | as `copilot`, with the prompt `/review {prompt}` | `copilot-json` | a prompt variant of `copilot`, not Copilot's review agent: under `-p`, `/review` reaches the model as plain text, with the same system prompt and tools; the task prompt still asks for its answer format | | ||
|
|
||
| npm installs `codex` and `copilot` as Node launchers. A run mounts one file as | ||
| the harness, so `--bin` names the native binary the launcher starts | ||
| (`@openai/codex-<platform>/vendor/<triple>/bin/codex`, | ||
| `@github/copilot-<platform>/copilot`). To route a CLI to another endpoint, | ||
| copy its profile: for Copilot, set the `COPILOT_PROVIDER_*` variables (and | ||
| `COPILOT_OFFLINE=true` to skip GitHub sign-in); for Codex, add a `config.toml` | ||
| under `config_files` that defines a `model_providers` entry with the endpoint's | ||
| `base_url`, `wire_api = "responses"` and `env_key = "CODEX_API_KEY"` (the variable that | ||
| holds its key; without it Codex sends no `Authorization` header) and selects it with `model_provider`. | ||
| Add the endpoint's host to `egress` in both cases. The hosted pull-request | ||
| reviewers (Copilot code review, `@codex review`, CodeRabbit) are not CLIs and | ||
| have no profile. | ||
|
|
||
| Profiles for other harnesses (Gemini CLI, Cursor agent, Aider, OpenCode, | ||
| in-house agents) follow the same shape. Their `guidance` | ||
| blocks start as the vendor's documented behaviour and are corrected by the | ||
| reach probe. Loading changes between versions of one harness, so | ||
| reach results are keyed by version. | ||
|
|
@@ -975,7 +1032,12 @@ per scenario). | |
| - `class` is one of: | ||
| - `loaded`: both tokens reported; | ||
| - `truncated`: one token reported; | ||
| - `fetched`: reported only after the agent read the file with a tool. For the | ||
| Codex and Copilot profiles a shell read counts only as a bare `cat`, `head`, | ||
| `tail`, `nl` or `sed -n`; for Codex, a command that is one such read of one | ||
| file and exits non-zero is not a read (the exit code of `cat a missing` does | ||
| not say which file failed, so both stay reads). A read through a pipe, `grep` | ||
| or `rg` is not seen, so such a file can be classed `loaded`; | ||
| - `not_loaded`; | ||
| - `error`: the scenario produced no usable answer: no answer, a non-zero | ||
| exit, or an error result. | ||
|
|
@@ -1308,9 +1370,10 @@ python cli.py repo compare repo-runs/replay --baseline guidance:none --metric re | |
| under `none`) and an | ||
| `id` hashing the profile's name, `invoke`, tool arguments, `env_set`, | ||
| `env_allow`, `config_dir_env`, `config_files`, `prompt_suffix`, `egress`, | ||
| `bridge`, `trace` (unless `claude-stream-json`), `answer_format`, the SHA-256 of the harness executable (`--bin`), | ||
| the SHA-256 of each `bin_companions` file and harness version. | ||
| A rerun whose `id` differs (another `--tools`, a `--delegation` that changes the tool arguments, profile, | ||
| harness binary or harness version, or an edited `invoke`, `env_set`, `env_allow`, `config_dir_env`, `config_files`, `prompt_suffix`, `egress`, `bridge`, `trace` or `answer_format`) is refused before any agent starts, and | ||
| `--force` or another `--batch` runs it. `--force` replaces the cells of the | ||
| run's plan only: if the batch dir holds finished cells outside that plan | ||
| whose `id` differs from this run's (e.g. `--reps 2`, then `--reps 1 | ||
|
|
@@ -1325,7 +1388,8 @@ python cli.py repo compare repo-runs/replay --baseline guidance:none --metric re | |
| names a `dokimi:` grader (or a task directory that cannot be read) also stays on the built-in graders' sources. | ||
| - The agent's container mounts the workspace, the per-run `home`, `config` and | ||
| scratch directories (each `image.env` value under `{run_dir}/`), the caches, | ||
| the harness executable (`--bin`, read-only at `/eval/harness`, with its | ||
| `bin_companions` beside it, §6) and the | ||
| framework's helper scripts (read-only at `/eval/support`). Before a batch, | ||
| the harness must report its version inside the image, and a probe container | ||
| with the same mounts must see neither `/eval/content` nor `/eval/source`, no | ||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Must-fix: This reach-probe guidance lists
codex-reviewalongside the supported profiles without noting thatrepo reach --harness codex-reviewalways fails: the profile setsnative_prompt: true, andprobe_files()rejects native-prompt profiles. Add the exception here and direct users to--harness codexfor the probe, as the checklist later in this document already does.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in 83e61f5: the reach section now says
repo reachrefusescodex-reviewand to probe withcodex, as the checklist below it does.