feat(ai)!: agents speak the LangChain create_agent contract - #190
Open
tmgbedu wants to merge 16 commits into
Open
feat(ai)!: agents speak the LangChain create_agent contract#190tmgbedu wants to merge 16 commits into
tmgbedu wants to merge 16 commits into
Conversation
* feat(ai): add TestJudgeAgent trajectory-match evaluator + ModelBuilder fake registry
Add a deterministic evals harness for testing AI agents, modeled on
LangChain's agentevals create_trajectory_match_evaluator.
TestJudgeAgent(trajectory_match_mode=...).record() yields a callable
evaluator that compares an actual message trajectory against a reference
trajectory with no live LLM call — pure structural comparison, typically
driven by Agent.fake() output:
with TestJudgeAgent(trajectory_match_mode="strict").record() as evaluator:
evaluation = evaluator(outputs=response["messages"], reference_outputs=reference)
assert evaluation["score"] is True
Supported modes mirror agentevals semantics:
- strict: same messages in order, same role and tool calls per position
(message content is not compared)
- unordered: same set of tool calls, any order
- subset: output tool calls all appear in the reference
- superset: output tool calls cover all reference tool calls (extras ok)
The evaluator returns a subscriptable result with key/score/comment,
following the LangSmith evaluator-result convention.
Also add a model-level fake registry to ModelBuilder: ModelBuilder.fake()
registers a GenericFakeChatModel for an agent (by class name or instance),
and get_model_for() returns it when present, otherwise builds a real
provider model as before. Agent._build_model() now routes through
get_model_for(), so a faked agent runs the same message-building /
pipeline / tool-execution path as a real one with only the underlying
model swapped. The existing Agent.fake()/FakeAgent/AgentBinding path is
unchanged.
* refactor(ai): rename ModelBuilder to Ai, rebuild fake()/record() around model-level fakes
Ai (formerly ModelBuilder) drops the per-instance agent binding: fake_agent_models/
fake_agent_responses are class-level registries keyed by agent class name, and
get_model_for()/build() take the agent as an argument instead of storing it in
the constructor.
Agent.fake() now registers a fixed, ordered list of replies as a deterministic
chat model (via Ai.fake()) instead of binding a whole FakeAgent stand-in into
the container. Replies flow through the real message-building/pipeline/tool-
execution path, so faked tool calls actually execute — only the model at the
bottom is swapped. Agent.record()/RecordingAgent/AgentBinding are unchanged.
Drops the dead _match_fake()/self._fakes machinery on Agent (never populated
by any code path) along with the now-unreachable FakeAgent/NoFakeResponse
testing helpers.
* feat(ai): add TestJudgeAgent trajectory-match evaluator + ModelBuilder fake registry
Add a deterministic evals harness for testing AI agents, modeled on
LangChain's agentevals create_trajectory_match_evaluator.
TestJudgeAgent(trajectory_match_mode=...).record() yields a callable
evaluator that compares an actual message trajectory against a reference
trajectory with no live LLM call — pure structural comparison, typically
driven by Agent.fake() output:
with TestJudgeAgent(trajectory_match_mode="strict").record() as evaluator:
evaluation = evaluator(outputs=response["messages"], reference_outputs=reference)
assert evaluation["score"] is True
Supported modes mirror agentevals semantics:
- strict: same messages in order, same role and tool calls per position
(message content is not compared)
- unordered: same set of tool calls, any order
- subset: output tool calls all appear in the reference
- superset: output tool calls cover all reference tool calls (extras ok)
The evaluator returns a subscriptable result with key/score/comment,
following the LangSmith evaluator-result convention.
Also add a model-level fake registry to ModelBuilder: ModelBuilder.fake()
registers a GenericFakeChatModel for an agent (by class name or instance),
and get_model_for() returns it when present, otherwise builds a real
provider model as before. Agent._build_model() now routes through
get_model_for(), so a faked agent runs the same message-building /
pipeline / tool-execution path as a real one with only the underlying
model swapped. The existing Agent.fake()/FakeAgent/AgentBinding path is
unchanged.
* refactor(ai): rename ModelBuilder to Ai, rebuild fake()/record() around model-level fakes
Ai (formerly ModelBuilder) drops the per-instance agent binding: fake_agent_models/
fake_agent_responses are class-level registries keyed by agent class name, and
get_model_for()/build() take the agent as an argument instead of storing it in
the constructor.
Agent.fake() now registers a fixed, ordered list of replies as a deterministic
chat model (via Ai.fake()) instead of binding a whole FakeAgent stand-in into
the container. Replies flow through the real message-building/pipeline/tool-
execution path, so faked tool calls actually execute — only the model at the
bottom is swapped. Agent.record()/RecordingAgent/AgentBinding are unchanged.
Drops the dead _match_fake()/self._fakes machinery on Agent (never populated
by any code path) along with the now-unreachable FakeAgent/NoFakeResponse
testing helpers.
* refactor(ai): rename model_builder.py to ai.py
* feat(ai): fluent Agent.record() testing DSL
Bind record() with `as agent` to get a synchronous prompt() plus
assert_text_response(), assert_tool_called()/assert_tool_not_called(),
assert_response_time_lt(), and assert_response_judged() (LLM-as-judge,
verdict cached in the cassette) that all judge the most recent turn.
record(cassette, messages=...) seeds a session's prior history; cassette
keys now fold in the conversation so far (not just the literal message
text) so two sessions with different histories but the same follow-up
text don't collide. The pre-existing bare context-manager usage from
task #327 is unchanged.
Removes TestJudgeAgent/evals.py: its deterministic structural trajectory
matching is unused anywhere outside its own tests, and this DSL now
covers response judging via assert_response_judged.
* fix(ai): fluent record() prompt() is async, not sync
RecordingAgent.prompt() no longer wraps the real async call with
asyncio.run() — it's now the same async method the bare context-manager
path already awaited internally. Wrapping every call in a fresh event
loop was unnecessary and would break inside any already-running loop.
Fluent tests now use IsolatedAsyncioTestCase + await agent.prompt(...).
* refactor(ai): drop dead non-streaming fallback in Agent.stream()
AgentBinding only ever wraps a RecordingAgent (from record()), which
always implements stream() — the hasattr(swapped, "stream") check and
its buffered-response fallback could never actually run.
* style(ai): remove stale comments on Ai's fake registries
* refactor(ai): back assert_response_judged with a JudgeAgent Replace RecordingAgent._judge_live's hand-rolled init_chat_model()/invoke() call with JudgeAgent, an Agent subclass. The judge now goes through the same provider/model resolution pipeline as any other agent, and is fakeable via JudgeAgent.fake() and replayable via JudgeAgent.record() instead of only being testable by mocking langchain directly. assert_response_judged() is now async (the judge call is a real Agent.prompt() under the hood) and accepts an optional provider kwarg, forwarded through to JudgeAgent and folded into the cached verdict's cache key. * refactor(ai): drop JudgeAgent's constructor, use plain Agent attributes model/provider are never set via constructor args anywhere else in the framework -- always class attributes or the @model()/@Provider() decorators. Match that: JudgeAgent has no __init__ override, and _judge_live() sets .model/.provider directly on the instance, same as any other Agent. * refactor(ai): parse JudgeAgent verdicts via schema(), not hand-rolled JSON Give JudgeAgent a Verdict pydantic schema() and put the grading rubric in instructions() -- the JSON reply is now turned into a typed result through the same structured-output path any Agent gets from schema()/response.parsed. Drops the bespoke _build_prompt()/_parse_verdict() helpers. * feat(ai): enforce schema() via with_structured_output on real model calls When an Agent declares schema() and has no tools, build the model with chat_model.with_structured_output(schema, include_raw=True) so the provider enforces the shape, instead of relying on prompt instructions + post-hoc JSON parsing. include_raw keeps the raw message so content/usage/cassettes still work; the Runner passes the structured result through without executing the synthetic tool call, and _to_agent_response unwraps it into response.parsed. Streaming opts out (needs raw token chunks), tools take precedence over a schema in a single call, and the fake/record paths are unchanged (they keep exercising the JSON-string parse path for deterministic replay). Also drops two stale docstrings from Ai. * feat(ai): pass tools and schema in one payload; model picks per turn When an agent declares both tools() and schema(), bind them together via bind_tools([*tools, schema]) so a single model call offers both. The model returns either a real tool call (which the Runner executes) or the schema as its structured answer (which the Runner parses into response.parsed). Schema without tools still uses with_structured_output() for enforcement. _apply_schema no longer coerces content when the agent has tools, so a tool's plain-text output is not force-parsed into the schema. * refactor(ai): always bind schema as a tool, drop with_structured_output branch schema() is just appended to tools() and the whole set is bound in one call. This collapses the schema-only special case: with_structured_output only added tool_choice="any" enforcement, which is redundant now that _apply_schema parses plain JSON text for tool-less agents -- so a schema-only agent gets its parsed result whether the model emits the schema tool call or replies in JSON text. Also drops the now-dead structured-dict passthrough in Runner.run. * refactor(ai): use with_structured_output for schema; leave tool binding as-is Bind tools exactly as before, then wrap the model with with_structured_output(schema, include_raw=True) when a schema is declared. The wrapped model returns {raw, parsed, parsing_error}; the Runner passes it through and _to_agent_response unwraps it into response.parsed. Reverts the schema-as-a-tool detection and the _apply_schema tools guard.
utils/structures.py was used (Configuration.data(), Loader.load()), so per the foundation/support consolidation it belongs in support/ alongside the rest of the framework's internal helpers. The utils/ directory had no __init__.py and nothing else referenced it, so it's gone entirely now that its one file has moved. Updated the two import sites accordingly.
…ures-1133 refactor: move utils/structures.py into support/
…kes directly Agent.fake()/record() now return AgentFake/AgentRecordFake directly instead of wrapping the latter in an AgentBinding indirection layer. AgentRecordFake gains the container-binding, cassette-resolution, and decorator behavior AgentBinding used to provide, so `with Agent.record(...) as agent:` and the `@Agent.record(...)` decorator form both keep working unchanged. AgentBinding is removed entirely, along with every reference to it (imports, __init__ exports, type hints, tests).
Two of the example/agents record() cassettes were recorded against message text that no longer matches the tests' current wording, so their cache keys never hit and every run fell through to a live Gemini call. The new units/agents/record_stream.json fixture had the same problem in the other direction: it only captured the first turn, so the second prompt() call and the assert_response_judged() verdict both missed the cassette too. Re-recorded all three cassettes against the real code path (mocking only the network boundary and the judge, same seam test_agent_record_fluent.py uses) so every prompt/stream/judge call in test_router_agent.py and test_chat_controller.py now replays from disk with no live model calls. uv.lock refreshes the editable fastapi-startkit metadata, which was stale at 0.45.0 against the package's actual 0.50.0.
…1141 refactor(ai): simplify Agent.fake()/record(), remove AgentBinding
…usage + tool calls) (#199) * chore(ai): sync working baseline for record() format work Fold in the pending ai-package-fix working-tree state so the example/agents suite runs green before extending the cassette format: fix the ToolCallView import, carry usage in the flat cassette, skip the not-yet-implemented log() tests, and regenerate the unit cassettes. * feat(ai): record full per-turn interaction as an ordered message list Agent.record() cassettes previously stored a flat, lossy shape keyed by the hash of the user input ({content, tool_calls, usage}); a tool-calling turn even dropped its tool_calls entirely, because the runner returned only the final tool-result message. Each recorded turn is now an ordered transcript keyed by the hash of the user input, capturing the whole exchange: [ human, ai(tool_calls, uses, response_time), tool_response(content_type, content, response_time), ai(content, uses, response_time) ] - AIMessage records token uses (input/output/cache/total), response_time, and content; tool calls record the tool response and their own response_time. - The plain Runner and the GraphRunner both capture the interaction while it happens (preserving the model's requested tool_calls and per-call latency). - Multi-turn is supported: each turn is a separate hash-keyed entry. - Replay reconstructs the AgentResponse from the transcript; the reader still accepts the legacy flat shape for backward compatibility. Existing example/agents unit cassettes are migrated to the new format. * test(agents): remove HTTP chat controller feature test Drops example/agents/tests/features/test_chat_controller.py. Its stream/no-stream assertions duplicated the unit coverage, and its record-and-replay cases depended on a global-model-swap decorator that does not exist yet, so they were skipped. * refactor(ai): rename ToolCallView to ToolCallAssert The class is a predicate helper for assert_tool_called(); ToolCallAssert names its assertion role more clearly than the generic View. * refactor(ai): rename ToolCallAssert to AssertToolCall * refactor(ai): build recording entries with functions, not classes Replace the HumanMessage/AIMessage/ToolCallMessage dataclasses with plain recording.human()/ai()/tool_response() builder functions. The persisted cassette JSON is unchanged (identical keys/values, and _save sorts keys).
* feat(ai): add agent.assert_tokens() for accumulated token limits
agent.assert_tokens(lambda x: x.where('input', '<=', 5000).where('output', '<=', 5000))
Tokens accumulate across every prompt()/stream() in a session (summing each
ai message's token uses from the transcript, with a usage fallback), and the
fluent TokenQuery evaluates where-clauses over input/output/cache/total. Works
on both live-record and replay; to_response() now carries the transcript so
replayed turns expose their per-message uses.
* style(ai): ruff-format assert_tokens docstring
* style(ai): ruff-format runner.py and test_record_cassette_format.py
CI runs 'ruff format --check .' repo-wide; these two files (carried in from
#199) were unformatted and failed the check.
* test(orm): fix polymorphic relation tests to use the record relationship
The Like model's MorphTo relationship is named 'record'; the tests referenced
a nonexistent 'like.log' attribute, which returned None and failed to await /
assert. Aligns with the eager-load case that already uses with_('record').
Main added tests/utils/test_structures.py (task #1214) importing from fastapi_startkit.utils.structures, but this branch relocated the module to fastapi_startkit.support.structures. On the PR merge the old module was gone, breaking collection with ModuleNotFoundError. Move the test to tests/support/test_structures.py to mirror the source layout and point the import at fastapi_startkit.support.structures. Test content and coverage are preserved unchanged.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Delete AgentResponse/AgentSnapshot and make the ai package's agents return
and stream exactly what langchain's create_agent produces:
- prompt() returns the state dict {"messages": [HumanMessage, AIMessage,
ToolMessage, ...]}, plus "structured_response" for schema agents. AI/tool
messages carry their response_time (ms) in additional_kwargs.
- stream() yields astream_events-shaped StandardStreamEvent dicts
(on_chat_model_start/stream/end, on_tool_start/end); GraphRunner.stream
delegates to the compiled graph's astream_events verbatim.
- New fastapi_startkit.ai.state helpers (text, tool_calls, tool_events,
usage, runtime, structured) replace the response wrapper's accessors.
- Testing harness (AgentFake/AgentRecordFake) operates on states; every
assert_* keeps its API. Cassette on-disk format unchanged — old
cassettes (including legacy flat shapes) still replay.
- usage now sums every model call in a turn instead of only the last.
- Example app (remember mixin, router/search/summarizer graph nodes,
SSE routes) migrated to the state helpers; frontend SSE frames
unchanged via a route-level event→frame mapping.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tmgbedu
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the
fastapi_startkit.aipackage's agents drop-in interchangeable with LangChain'screate_agent: same return shape, same stream events, no wrapper classes.prompt()returns thecreate_agentstate dictAgentResponseandAgentSnapshotare deleted. Every agent (plain, graph, fake, cassette replay) now returns:{"messages": [HumanMessage, AIMessage(tool_calls=[...]), ToolMessage, ...]} # plus "structured_response": <parsed> for schema agents (LangChain's key)AI/tool messages carry their
response_time(ms) inadditional_kwargs.stream()yieldsastream_events-shaped eventsStandardStreamEventdicts:on_chat_model_(start|stream|end)with eachAIMessageChunkindata.chunk, andon_tool_(start|end)around each tool execution.GraphRunner.streamdelegates to the compiled graph'sastream_eventsverbatim; the root chain-end supplieslast_response.Reading turn data
New
fastapi_startkit.ai.statehelpers replace the wrapper's accessors:text(),tool_calls(),tool_events(),usage(),runtime(),structured().Compatibility
assert_*onAgentFake/AgentRecordFakekeeps its API.Behavior change
usagesums every model call in a turn (graph turns with multiple LLM calls report combined tokens) instead of only the last call; per-call numbers remain on eachAIMessage.usage_metadata.Testing
fastapi_startkit: 1396 passed (uv run pytest tests/ --ignore=tests/masoniteorm)example/agents: 24 passed, including record/replay against previously recorded cassettes🤖 Generated with Claude Code