AI record(): capture full per-turn interaction (messages + latency + usage + tool calls)

[tmgbedu]

Summary

Agent.record() cassettes previously persisted a flat, lossy shape keyed by the hash of the user input:

{"<hash>": {"content": "...", "tool_calls": [], "usage": {"input": N, "output": N}}}

A tool-calling turn even dropped its tool_calls entirely — the plain Runner returned only the final tool-result message, whose .tool_calls is empty. This is why the example/agents router tests were red on the base.

Each recorded turn is now an ordered transcript keyed by the hash of the user input, capturing the whole exchange:

{"<hash-of-user-input>": [
  {"type": "human", "content": "suggest me jobs"},
  {"type": "ai", "tool_calls": [...], "uses": {"input_token": N, "output_token": N, "cache_token": N, "total_token": N}, "response_time": 12.0},
  {"type": "tool_response", "content_type": "json", "content": "[...]", "response_time": 5.0},
  {"type": "ai", "content": "Here is a job...", "uses": {...}, "response_time": 8.0}
]}

• AIMessage records token uses (input/output/cache/total), response_time (ms), and content.
• tool_response records the tool's response, inferred content_type (json/text), and its own response_time.
• Multi-turn is supported — each turn is a separate hash-keyed entry.

What changed

• ai/recording.py (new): ordered message types (HumanMessage / AIMessage / ToolCallMessage), usage_metadata → uses mapping, and to_response() to reconstruct an AgentResponse from a transcript.
• ai/runner.py (Runner + shared BaseRunner): capture the interaction as it happens — preserve the model's requested tool_calls, record each tool's response and per-call latency, and expose it via AgentResponse.transcript / .tool_events. Fixes the dropped-tool_calls bug.
• ai/graph.py (GraphRunner): capture the full tool loop (ai → tool_response → ai) across the graph nodes.
• ai/response.py: add tool_events and transcript fields to AgentResponse.
• ai/testing.py (AgentRecordFake): write the new ordered-list cassette; reconstruct on replay. The hash-key inputs are unchanged, so existing keys stay valid. Backward compatible: the reader still accepts the legacy flat shape.
• example/agents unit cassettes migrated to the new format (regenerated through the real record path with a faked model, so keys are identical).

Testing

• New framework tests (TDD): test_recording.py, test_runner_recording.py, test_graph_recording.py, test_record_cassette_format.py (incl. a backward-compat legacy-cassette test).
• Full AI suite: 300 passed.
• example/agents suite: 9 passed, 3 skipped — including the two previously-red router tests (test_the_router_agent, test_the_router_with_initial_messages).
• Full framework suite: 1908 passed. (Two unrelated test_sqlite_polymorphic ORM failures are pre-existing on the base — a like.log → like.record test bug — and are out of scope here.)

Notes

• Depends on ai-package-fix and targets it (not main): the record() / GraphAgent refactor, the flat cassette format, and the example cassettes exist only on that branch. The first commit folds in the pending ai-package-fix working-tree baseline so the suite runs green before the format change.

[tmgbedu]

Code Reviewer verdict — REQUEST CHANGES (1 blocker: CI red)

Logic is correct and well-tested; only a CI gate is failing.

Blocker

1. Ruff format check FAILING (CI not green). Two new test files need reformat (line-length reflow only, no logic change): tests/ai/test_recording.py, tests/ai/test_record_cassette_format.py. Fix: uv run ruff format tests/ai/test_recording.py tests/ai/test_record_cassette_format.py and push. (ruff check passes, Pyright passes, Pytest still pending.)

Verified good

• Hash keying deterministic (sort_keys=True); 4 migrated cassettes keep byte-identical keys (basic_job_search 921847ca… unchanged) → old recordings still resolve.
• Schema ordered typed transcript; is_transcript() discriminates new vs legacy safely.
• Backward-compat reader handles transcript / legacy flat dict / bare string / legacy stream shapes; covered by test_old_flat_cassette_still_replays.
• Latency/usage per-message; usage last-write-wins consistent record↔replay; runtime sums each entry once — no double counting.
• Runner refactor additive side-effect; only intended change is tool_calls/usage now populated on tool turns (the dropped-tool_calls fix); structured path preserved.
• Tests AI 300 passed; example/agents 9 passed/3 skipped incl. the 2 previously-red router tests. Base = ai-package-fix (not main). ✓

Approve on my end once the two files are formatted and CI is green. DO NOT MERGE.