Add codex-setup and docs/faq, and correct the --agent list - #17
Merged
Merged
Conversation
The fork gained these after the earlier import, so they were missing here. docs/codex-setup.md -- codex is fully wired in the harness (68 references in run-swe-headless.py, and run-e2e-benchmark.sh accepts --agent codex) but had no setup page, so a user had no way to learn it was an option. docs/faq/ (4 files) -- one page per agent for wiring it to a model, same format throughout, covering Claude Code, omp and codex. Corrected the --agent list in two places, both of which omitted codex: - README.md: --agent claude|pi|omp|kiro -> claude|pi|omp|kiro|codex - run-e2e-benchmark.sh line 84: the inline comment still read "claude (Claude Code) or pi (pi agent)", predating omp, kiro and codex. The script's own validation at line 167 already listed all five. README documentation map gained rows for docs/faq/, docs/codex-setup.md, and docs/omp-setup.md (the last was imported earlier but never listed). Still not imported: docs/release-notes/0.2.0.md, like 0.1.0.md, sources its figures from docs/metrics/pareto-frontier-*.json and quotes per-model scores and per-tier breakdowns directly. The per-harness guides (harness-pi-swe2.md and friends) are results reports, not onboarding docs -- harness-pi-swe2.md opens "# Results: pi harness (swe2)" and is a per-model table of scores, costs and token counts. Onboarding for those harnesses is already covered by omp-setup.md, kiro-cli-setup.md, codex-setup.md, docs/faq/ and benchmarks/docs/path-*.md. Verified: every relative link in README.md resolves, and every internal link across docs/ including docs/faq/ resolves.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this is a separate PR
These changes were pushed to
docs/harness-docsafter PR #16 had already been merged at48866a3, so they never landed. Cherry-picked onto currentmainhere.What was missing
docs/codex-setup.mdanddocs/faq/(4 files) appeared upstream after the first docs import, so the earlier PR predated them.The
--agentlist omitted codexcodexhas been fully supported all along — 68 references inrun-swe-headless.py, andrun-e2e-benchmark.sh's own validation at line 167 already accepts all five agents. But two user-facing places said otherwise:README.md—--agent claude|pi|omp|kiro→claude|pi|omp|kiro|codexrun-e2e-benchmark.sh:84— the inline comment read "claude (Claude Code) or pi (pi agent)", predatingomp,kiroandcodexentirelySo a user reading either one would not know
codexwas an option, despite the code accepting it.Added
docs/codex-setup.mdcodex execheadless use, the provider block an endpoint run needs, and why the sandbox must be bypassed on a benchmark hostdocs/faq/README.mddocs/faq/wiring-claude-code-to-models.mddocs/faq/wiring-codex-to-models.mddocs/faq/wiring-omp-to-models.mdREADME documentation map gained rows for
docs/faq/,docs/codex-setup.md, anddocs/omp-setup.md— the last was imported in #16 but never listed.Still left out
docs/release-notes/0.2.0.md— like0.1.0.md, it sources its figures fromdocs/metrics/pareto-frontier-*.jsonand quotes per-model scores plus per-tier breakdowns.The six per-harness guides (
harness-pi-swe2.mdand friends) — these are results reports, not onboarding docs.harness-pi-swe2.mdopens# Results: pi harness (swe2)and is a per-model table of scores, costs and token counts. Onboarding for those harnesses is already covered byomp-setup.md,kiro-cli-setup.md,codex-setup.md,docs/faq/andbenchmarks/docs/path-*.md.Verified
README.mdresolvesdocs/includingdocs/faq/resolvesbash -nclean on the edited scriptaarora79refs, no references to excluded results docs