testbench

Neutral multi agent workloads for testing agentplane releases without bias.

Why this repo exists

Two agentplane products watch agent systems from the outside:

• tokenops meters and caps token spend.
• chronicle records runs so you can replay them.

To trust a new release of either product, you run it against realistic multi agent load and check the result. But that load has to be honest. If a workload imported its own tooling, it could be shaped (even accidentally) to make that tooling look good, and the release check would be biased.

So every workload here is tool agnostic. The agents depend only on two small, vendor neutral interfaces (an LLM client and an event hook). An external tool plugs into those interfaces from the outside. Nothing in this repo imports tokenops or chronicle, and a CI test (tests/test_agnostic.py) fails the build if that ever changes.

What the two workloads do

MAS 1: Orchestrator and Workers (built on LangGraph)

Turns one question into a short research brief using four agents.

Input: a question, for example "What is token governance and why does it matter for agents?"

1. A supervisor agent reads the question and decides which worker runs next.
2. The researcher worker pulls relevant facts from a static knowledge set.
3. The analyst worker turns those facts into a few key points.
4. The writer worker composes the final brief.
5. Control returns to the supervisor after each worker, and it finishes once all three workers have run.

Output: a titled brief with bullet point findings.

Brief: What is token governance and why does it matter for agents?

- Tokens are the unit of both cost and latency in LLM systems.
- Uncapped agent loops can spend unbounded tokens before a human notices.
- Governance means measuring, attributing, and capping spend per run.
- A control plane can enforce budgets without changing agent code.

MAS 2: Evaluator and Optimizer (plain Python, no framework)

Refines a short answer until it is good enough, using two agents in a loop.

Input: a prompt, for example "Explain token governance to a new engineer in two sentences."

1. The generator agent writes a draft answer.
2. The evaluator agent scores the draft from 0 to 1 against a fixed rubric and writes one line of feedback.
3. If the score is below 0.8, the feedback goes back to the generator and it tries again.
4. The loop stops when the score reaches 0.8 or after 3 attempts, whichever comes first.

Output: the best draft, its score, and how many attempts it took.

result (score=0.90, iterations=3)
Answer to 'Explain token governance...': a concise, well scoped response. [addressed:1] [addressed:2]

How a run stays honest

Both workloads run fully offline by default. A deterministic stub client returns fixed text and fixed token counts for a given input, so the same run always produces the same output and the same trace. That is what makes one release comparable to the next. No API key and no network are needed for the default run or for the tests.

To run against real models instead, install the live extra and pass --live. The agent code does not change; only the client behind the interface does.

How an external tool attaches

A tool supplies its own implementation of one or both interfaces and passes it in when building the workload. The agents never see the tool.

• LLM client interface: wrap the model call to meter spend (tokenops) or record it (chronicle).
• Event hook interface: receive one event per model, tool, or delegate step.

See docs/attaching-external-tools.md for short examples.

Quickstart

pip install -e ".[dev]"     # offline dependencies only, no API keys

make run-mas1               # run the orchestrator and workers demo
make run-mas2               # run the evaluator and optimizer demo
make test                   # run the test suite (offline, deterministic)
make lint                   # ruff and mypy

Run against real providers:

pip install -e ".[dev,live]"
OPENAI_API_KEY=... make run-mas1-live

Layout

src/testbench/
  core/                  the two interfaces, the stub client, live providers, run config
  orchestrator_workers/  MAS 1 (LangGraph)
  evaluator_optimizer/   MAS 2 (plain Python)
tests/                   test suite, including the tool agnostic import guard
docs/                    architecture and integration notes

MIT licensed.