A reproducible benchmark framework for evaluating coding agents on real software-engineering execution, reliability, efficiency, and cost-to-accepted-result.
Experimental and evolving. License: Apache-2.0.
Coding agents do more than generate code: they inspect a repository, use tools, make decisions, manage context, recover from failures, validate changes and deliver a result. This framework organizes that execution around independent acceptance, reproducible metadata and explicit scope.
The primary questions are whether an agent succeeds on its first pass, how reliably it reaches acceptance, and how much time, token usage and cost it consumes. Critical failures, scope discipline, hallucinated success, self-report accuracy and work amplification matter alongside functional scores.
A plausible patch is not the same as a verified engineering result. Long-horizon work exposes failures in persistence, concurrency, compatibility, deployment, recovery and product judgment that a short code-generation task may miss. This project provides task families and an execution ledger for studying those behaviors. It is not a static set of 15 questions or a universal model-intelligence ranking.
A family defines a capability domain. Public development resources illustrate that domain; rotating evaluation generations can change the concrete problem. The table describes task objectives, not capabilities already implemented by every seed.
| Family | Capability focus | v0.1 public resources |
|---|---|---|
| 01 Snapshot | Smoke, delivery, baseline execution discipline. | Family definition only |
| 02 LedgerRace | Persistence, concurrency, migration, idempotency and rollback. | TASK only |
| 03 ResilientSync | Retries, cancellation, corruption prevention and restart persistence. | TASK + seed |
| 04 SafeBundle | Archive safety, normalization, rollback and resource limits. | TASK + seed |
| 05 RED LIGHT | Product judgment, safety boundaries and contradictory requirements. | TASK only |
| 06 Greenfield (Durable Job Queue) | Queue architecture, persistence, leases, concurrency and crash recovery. | TASK + seed |
| 07 MysterySync | Debugging, root-cause isolation, distractors and minimal patches. | TASK + seed |
| 08 Behavior-Preserving Refactor | API compatibility, ordering, serialization, concurrency and exceptions. | TASK + seed |
| 09 Hot Path | Performance optimization while preserving correctness and memory limits. | TASK + seed |
| 10 Black Box Compatibility | Behavioral inference and legacy CLI compatibility. | TASK + seed |
| 11 Bug Hunter | Reproduction, root-cause evidence and false-positive control. | TASK only |
| 12 Fire Drill | Incident triage, mitigation, recovery verification and rollback decisions. | TASK + seed |
| 13 Ship It | Production reliability, deployment, migration, health and backup design. | TASK + seed |
| 14 Pixel Contract | Deterministic UI behavior, state handling and accessibility. | TASK + seed |
| 15 Product Judgment | Ambiguity, safe assumptions, compatibility and stop-versus-proceed decisions. | TASK + seed |
There are 14 sanitized task statements and 11 original .NET 8 development seeds. Snapshot is documented at family level only; LedgerRace, RED LIGHT and Bug Hunter currently publish their tasks but no seeds. Seeds may deliberately lack the requested behavior. No complete official evaluation variant is distributed.
- Core5: LedgerRace, RED LIGHT, Greenfield / Durable Job Queue, MysterySync, Black Box Compatibility.
- Core7: Core5 plus SafeBundle and Product Judgment.
- Full15: all 15 capability families above.
The suite definitions preserve the framework's existing membership. They describe coverage; they are not runnable official suites in this release because their evaluation assets are omitted. The separate demo suite is a synthetic framework exercise and is not part of Full15.
Public materials are development and methodology resources.
Official leaderboard evaluation uses private rotating graders, fixtures, references, and/or variants that are intentionally not distributed in the current public release.
All current official graders, hidden fixtures, references, mutants, secret generation state and private results remain outside this repository. The small checker under examples/synthetic-demo/ was newly authored for the public demo; it is not a leaderboard grader. Future delayed disclosure may release retired generations only after a new sealed generation replaces them.
Requirements: Python 3.10 or newer and Git. No Python packages, .NET SDK, model API, account or credential is needed for the framework demo. The optional development seeds use .NET 8.
Clone the published repository, or use your local release checkout:
git clone https://github.com/ex32013/agent-benchmark.git
cd agent-benchmarkRun these commands from the repository root (PowerShell or a POSIX shell):
python tools/BenchmarkOrchestrator/orchestrator.py --help
python tools/BenchmarkOrchestrator/orchestrator.py health 00-synthetic-demo
python scripts/validate_schema.py
python scripts/smoke.pyThe smoke prepares a fresh temporary workspace, starts a synthetic run, executes the public inventory seed, finishes, independently checks the result, archives JSON/JSONL/CSV and builds a report. It verifies metadata and schema, then cleans up its own temporary directory. Expect PASS and SYNTHETIC EXAMPLE — NOT A MODEL RESULT. No model is evaluated and no official score is reproduced.
For a local run you inspect yourself, the orchestrator's smoke command prepares the demo and prints its run ID and workspace. See the CLI guide for the lifecycle. Local runs and raw logs are private until separately sanitized.
| Measure | Interpretation |
|---|---|
| First-pass success | Accepted valid tasks before repair or intervention |
| First-pass cost | Resource use through the first submitted result |
| Cost to Accepted Result | Cost across attempts and declared repair through acceptance |
| Time / Tokens to Accepted Result | Elapsed time and non-overlapping token usage through acceptance |
| Accepted tasks/hour | Accepted unique tasks divided by included execution hours |
| Accepted tasks/$ | Accepted unique tasks divided by a named monetary cost basis |
| Critical failure rate | Valid tasks with at least one critical gate failure |
| Scope discipline | Compliance with the task's authorized change boundary |
| Hallucinated success / self-report accuracy | Claimed outcome compared with independent evidence |
| Work amplification | Work consumed relative to a declared baseline in the same units |
Report valid-task denominators, failed attempts, unreached acceptance and missing measurements. These are methodology definitions; not every metric is automatically measured or aggregated by v0.1. See scoring for implementation boundaries.
Keep three independent concepts:
- Equivalent API Cost: token usage priced under a declared price card and date.
- Actual Marginal Cash Cost: additional cash actually paid for the execution.
- Subscription / Quota Consumption: entitlement consumed, with its unit and measurement basis.
Free promotional tokens can have actual marginal cash cost $0, nonzero equivalent API cost and nonzero quota consumption. A subscription price divided by tokens is not a reliable marginal-cost measure; record quota burn and accepted throughput. A zero or unknown denominator does not justify an infinite efficiency claim. Missing measurements remain null. See economics.
The orchestrator records a fresh workspace, run ID, task/seed/grader hashes, lifecycle timing, model and harness identity, exposure, repair linkage, validity and usage source/confidence. Public metadata also records declared network and context modes. Git status/diff and JSONL/CSV ledgers support local inspection.
See the run metadata template, schemas, and synthetic result. The example is SYNTHETIC EXAMPLE — NOT A MODEL RESULT. There are no real model results or public leaderboard scores in v0.1.
Metadata is evidence, not enforcement. The runner does not sandbox agents, disable networking, launch models, enforce hard timeouts or prove that supplied usage is accurate. Local manifests include machine/workspace details and must be reviewed before sharing. Reproducibility policy explains the boundary.
Native or representative coding harnesses are allowed in the main evaluation methodology because the user experience and cost to acceptance are part of the question. Record model, snapshot when known, provider, effort, harness and version, context-management mode, network mode, exposure, date and first-pass/repair phase. The same model with a changed harness is a new run configuration. Do not silently pool those results. See methodology and leaderboard policy.
Public development variants WILL eventually enter model training data. This is acceptable: learning engineering skills is useful; memorizing exact solutions undermines evaluation. Official generations rotate schemas, fixtures, failure timing, APIs and surface forms, and where appropriate language or framework.
Leaderboard execution requires externally enforced network_access = false, or a separately declared strictly controlled track. Otherwise an agent could retrieve TASK files, checkers or previous solutions from public repositories. Current evaluation material must never be fully disclosed. See contamination policy.
This release provides a usable framework and deterministic demonstration, not a public model ranking. Official evaluation internals are intentionally absent, and public tasks may require you to author your own local tests/fixtures. health checks public asset integrity; it is not certification of benchmark difficulty or grader quality.
The inherited orchestrator remains a manual workflow tool. Its report is a local summary, several discipline fields are placeholders, adaptive replication is a recommendation, and suite/budget options are not an automated scheduler. Economic aggregation and self-report comparisons require additional reviewed analysis. Provider usage adapters operate only on explicitly supplied local data and are not used in CI. Only the synthetic example is a complete public prepare/run/grade/reproduce package.
Start with CONTRIBUTING.md and adding a benchmark. Evaluation readiness requires Design → Seed → Grader → Reference → Mutants → Repeatability → Anti-cheat → Freeze. Public contributions must respect the sealed boundary. Follow SECURITY.md for sensitive reports and the Code of Conduct.
Offline validation:
python scripts/check_syntax.py
python -B tools/BenchmarkOrchestrator/tests/test_orchestrator.py
python -B tools/BenchmarkOrchestrator/tests/test_zcode_usage_collector.py
python -B -m unittest discover -s tests -v
python scripts/validate_schema.py
python scripts/smoke.py
python scripts/audit_public.pyLicense: Apache-2.0 for original project code, documentation, schemas and development resources authorized for release. See LICENSE, NOTICE, and the provenance notes. Third-party works retain their own licenses; this release does not vendor third-party dependencies or relicense unrelated source trees.