feat(ai): Pest-evals-style assertions on the agent test double - #203
Open
tmgbedu wants to merge 3 commits into
Open
feat(ai): Pest-evals-style assertions on the agent test double#203tmgbedu wants to merge 3 commits into
tmgbedu wants to merge 3 commits into
Conversation
Add convenience assertions to AgentFake/AgentRecordFake, modelled on Pest's eval expectations: - assert_json() — response content is valid JSON (toBeJson) - assert_follow_trajectory(expected) — tools called across the session match the expected ordered sequence (toFollowTrajectory) - assert_satisfy(expectation) — LLM judge: response meets an expectation (toSatisfy) - assert_relevant() — LLM judge: response is on-topic for the last prompt (toBeRelevant) - assert_safe() — LLM judge: response is free of harmful content (toBeSafe) - assert_prompt_judged(expectation) — LLM judge on the prompt The judged assertions reuse the existing JudgeAgent. Their provider/model now default to the agent under test (overridable per call), so the no-arg forms work out of the box. assert_response_judged now grades the whole response (answer plus tool calls, when present) instead of only the final text. Judge verdicts are cached in a sidecar <cassette>.judge.json file, keeping recorded conversations free of grading noise. Covered by tests/ai/test_eval_assertions.py.
…ests Apply ruff format so the whole-tree 'ruff format --check .' CI gate passes. Covers the new testing.py plus test_agent.py, test_graph_agent.py and test_assert_response_time.py, which the create_agent-contract refactor left unformatted. Formatting only — no behavior change.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
The sidecar verdict cache and whole-response grading both change where/how a verdict is keyed, which would strand verdicts recorded by earlier cassettes (inline in the interaction file, keyed on the plain response text) and force a live judge call on replay. Look verdicts up in both the sidecar and the interaction cassette, and try the plain-text key as a fallback to the whole-response key. New verdicts are still written to the sidecar. Existing example/agents cassettes (router, summarizer) replay again without hitting the network. Add a regression test for the legacy-inline fallback.
Contributor
Author
CI status
The judged example tests that my first push briefly broke ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Pest-evals-inspired assertions to the agent test double (
AgentFake/AgentRecordFake), on top of thecreate_agentstate-dict contract.Deterministic
assert_json()— response content parses as valid JSON (toBeJson)assert_follow_trajectory(expected)— the tools called across the whole session match the expected ordered sequence (toFollowTrajectory)LLM judge (reuse the existing
JudgeAgent)assert_satisfy(expectation)— response satisfies a natural-language expectation (toSatisfy)assert_relevant()— response is on-topic for the last prompt (toBeRelevant)assert_safe()— response is free of harmful content (toBeSafe)assert_prompt_judged(expectation)— grade the prompt itselfNotes
provider/modelnow default to the agent under test (overridable per call), so the no-arg forms work without repeating model config.assert_response_judgednow grades the whole response — answer text plus tool calls when present — instead of only the final text. Text-only responses are graded as before, so existing behaviour/tests are unchanged.<cassette>.judge.jsonfile, keeping recorded conversations free of grading noise.Tests
tests/ai/test_eval_assertions.py(11 tests) covering each assertion, the model/provider defaulting, whole-response grading, and sidecar caching.340 passed(tests/ai/+tests/support/), ruff clean.Base
Targets
ai-package-fix(notmain) because it builds on the refactoredai.statehelpers / state-dict contract that only exist on this branch.