No way to tell whether an agent's behaviour changed between two runs

[dwin-gharibi]

Problem

When a task that worked yesterday does not work today, the first question is which variable moved:
the model's decisions, or the environment it ran in. Nothing in docker-agent answers it.

Sessions are recorded in full — every assistant turn with its tool calls is persisted — but there is
no way to compare two of them. The existing recording features solve different problems:

Feature	What it does
pkg/recording, docker agent record	records a TUI session for playback/demos
e2e/testdata/cassettes	replays HTTP for the e2e suite
pkg/evaluation	scores runs against declared expectations

None of them compares two real runs of the same task.

So triage is manual: open two sessions, scroll both, try to spot where they parted ways.

Proposal

docker agent replay <session-a> <session-b> — report the first point where the agent behaved
differently.

$ docker agent replay a1b2c3d4 e5f6a7b8
Comparing a1b2c3d4 (7 turns) against e5f6a7b8 (9 turns)

❌ First divergence at turn 3 (after 3 matching turn(s)).
   a1b2c3d4 called:
     read_file({"path":"pkg/cache/cache.go"})
   e5f6a7b8 called:
     search_files_content({"query":"persistToDisk","path":"."})

Everything after this point is downstream of the divergence and is not compared.

Two design points that make or break it

Compare tool calls, not prose. Model output is nondeterministic: two runs of the same task almost
always word things differently while doing exactly the same work. Diffing assistant text would report
a difference on essentially every comparison, which is no signal at all. The tool calls are what
changed the world, so they are what should be compared — assistant text is worth carrying for context
around a divergence, but must not decide whether one occurred.

Stop at the first divergence. Once two runs differ, every later turn is downstream of that
difference — different tool results, different context. Comparing the tail produces noise. The first
divergence is the whole answer.

Scope

The ambitious version is to re-run a recorded session against a different model, holding the
environment fixed with recorded tool results, to isolate a model or prompt change from environment
drift. That is worth having but is substantially larger, and it has an unresolved design question: the
replay diverges the moment the new model calls a tool the recording has no result for, and the choice
between "stop there" and "start executing live" changes what the feature is.

Comparing two already-recorded sessions delivers the triage answer, needs no provider in the loop, and
is the natural substrate for that follow-up: the comparison logic is the same either way.