Problem
docker agent eval runs a suite and prints a summary. There is no way to compare that summary
against a previous run, so the question people actually have about an eval suite cannot be answered:
I edited the system prompt / bumped the model / changed a tool description. Is the agent better or
worse than before?
Today that means eyeballing two terminal scrollbacks. In CI it means nothing at all — the eval
command's exit code reflects whether the run executed, not whether quality held.
The data is already saved
SaveRunJSON (pkg/evaluation/save.go:403) already writes an EvalRun — name, timestamp, and every
Result with its expectations and scores — to <output>/<run>-eval.json. That file is already a
baseline in everything but name; nothing reads it back.
Proposal
--baseline <run.json>: after the run, compare against the saved run, print the deltas, and exit
non-zero on regression.
$ docker agent eval ./agent.yaml --baseline results/2026-08-01-eval.json
…
Baseline comparison (tolerance 0.000)
METRIC BASELINE CURRENT DELTA
! size pass rate 1.000 0.750 -0.250
tool F1 mean 0.910 0.925 +0.015
failure rate 0.000 0.000 +0.000
· total cost 0.412 0.older +0.031
Changed evaluations
! evals/refactor.json pass → fail
❌ Regression against baseline
Plus --regression-tolerance so judge variance does not fail a build on noise.
Design questions that need answers, not defaults
Judge variance will masquerade as regression. An LLM judge does not return the same score twice.
Without a tolerance, a gate built on this will flap, get disabled, and then be ignored. A tolerance
is not a nice-to-have; it is what makes the feature usable.
But a tolerance must not be able to hide a real breakage. An evaluation that passed and now fails
is precisely the signal a gate exists for. If the tolerance can absorb that, the gate is worthless.
So the tolerance should govern aggregate rates only, and a pass → fail transition should gate
regardless.
A metric that did not exist in the baseline must not read as a regression from zero. Adding the
first size_expected to a suite would otherwise look like a catastrophic drop.
Cost should be reported but should not gate. A provider price change is not a quality
regression, and a gate that fires on it will be turned off.
Out of scope for a first version
A model matrix (--models a,b,c) to answer "which model should this agent use?" is the natural
companion, but evals run each agent in a container (pkg/evaluation/Dockerfile.template,
ContainerRuntime), so a matrix needs per-run model override plumbed through container
orchestration. That is a much larger change and should be argued separately, on top of a working
baseline comparison.
Problem
docker agent evalruns a suite and prints a summary. There is no way to compare that summaryagainst a previous run, so the question people actually have about an eval suite cannot be answered:
Today that means eyeballing two terminal scrollbacks. In CI it means nothing at all — the eval
command's exit code reflects whether the run executed, not whether quality held.
The data is already saved
SaveRunJSON(pkg/evaluation/save.go:403) already writes anEvalRun— name, timestamp, and everyResultwith its expectations and scores — to<output>/<run>-eval.json. That file is already abaseline in everything but name; nothing reads it back.
Proposal
--baseline <run.json>: after the run, compare against the saved run, print the deltas, and exitnon-zero on regression.
Plus
--regression-toleranceso judge variance does not fail a build on noise.Design questions that need answers, not defaults
Judge variance will masquerade as regression. An LLM judge does not return the same score twice.
Without a tolerance, a gate built on this will flap, get disabled, and then be ignored. A tolerance
is not a nice-to-have; it is what makes the feature usable.
But a tolerance must not be able to hide a real breakage. An evaluation that passed and now fails
is precisely the signal a gate exists for. If the tolerance can absorb that, the gate is worthless.
So the tolerance should govern aggregate rates only, and a pass → fail transition should gate
regardless.
A metric that did not exist in the baseline must not read as a regression from zero. Adding the
first
size_expectedto a suite would otherwise look like a catastrophic drop.Cost should be reported but should not gate. A provider price change is not a quality
regression, and a gate that fires on it will be turned off.
Out of scope for a first version
A model matrix (
--models a,b,c) to answer "which model should this agent use?" is the naturalcompanion, but evals run each agent in a container (
pkg/evaluation/Dockerfile.template,ContainerRuntime), so a matrix needs per-run model override plumbed through containerorchestration. That is a much larger change and should be argued separately, on top of a working
baseline comparison.