Skip to content

feat(ai): Pest-evals-style assertions on the agent test double - #203

Open
tmgbedu wants to merge 3 commits into
ai-package-fixfrom
task/ai-eval-assertions
Open

feat(ai): Pest-evals-style assertions on the agent test double#203
tmgbedu wants to merge 3 commits into
ai-package-fixfrom
task/ai-eval-assertions

Conversation

@tmgbedu

@tmgbedu tmgbedu commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

What

Adds Pest-evals-inspired assertions to the agent test double (AgentFake / AgentRecordFake), on top of the create_agent state-dict contract.

Deterministic

  • assert_json() — response content parses as valid JSON (toBeJson)
  • assert_follow_trajectory(expected) — the tools called across the whole session match the expected ordered sequence (toFollowTrajectory)

LLM judge (reuse the existing JudgeAgent)

  • assert_satisfy(expectation) — response satisfies a natural-language expectation (toSatisfy)
  • assert_relevant() — response is on-topic for the last prompt (toBeRelevant)
  • assert_safe() — response is free of harmful content (toBeSafe)
  • assert_prompt_judged(expectation) — grade the prompt itself

Notes

  • The judged assertions' provider/model now default to the agent under test (overridable per call), so the no-arg forms work without repeating model config.
  • assert_response_judged now grades the whole response — answer text plus tool calls when present — instead of only the final text. Text-only responses are graded as before, so existing behaviour/tests are unchanged.
  • Judge verdicts are cached in a sidecar <cassette>.judge.json file, keeping recorded conversations free of grading noise.

Tests

  • New tests/ai/test_eval_assertions.py (11 tests) covering each assertion, the model/provider defaulting, whole-response grading, and sidecar caching.
  • Full AI suite green: 340 passed (tests/ai/ + tests/support/), ruff clean.

Base

Targets ai-package-fix (not main) because it builds on the refactored ai.state helpers / state-dict contract that only exist on this branch.

tmgbedu added 2 commits July 30, 2026 15:49
Add convenience assertions to AgentFake/AgentRecordFake, modelled on Pest's
eval expectations:

- assert_json() — response content is valid JSON (toBeJson)
- assert_follow_trajectory(expected) — tools called across the session match the
  expected ordered sequence (toFollowTrajectory)
- assert_satisfy(expectation) — LLM judge: response meets an expectation (toSatisfy)
- assert_relevant() — LLM judge: response is on-topic for the last prompt (toBeRelevant)
- assert_safe() — LLM judge: response is free of harmful content (toBeSafe)
- assert_prompt_judged(expectation) — LLM judge on the prompt

The judged assertions reuse the existing JudgeAgent. Their provider/model now
default to the agent under test (overridable per call), so the no-arg forms work
out of the box. assert_response_judged now grades the whole response (answer plus
tool calls, when present) instead of only the final text. Judge verdicts are
cached in a sidecar <cassette>.judge.json file, keeping recorded conversations
free of grading noise.

Covered by tests/ai/test_eval_assertions.py.
…ests

Apply ruff format so the whole-tree 'ruff format --check .' CI gate passes.
Covers the new testing.py plus test_agent.py, test_graph_agent.py and
test_assert_response_time.py, which the create_agent-contract refactor left
unformatted. Formatting only — no behavior change.
@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.50746% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...astapi_startkit/src/fastapi_startkit/ai/testing.py 98.50% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

The sidecar verdict cache and whole-response grading both change where/how a
verdict is keyed, which would strand verdicts recorded by earlier cassettes
(inline in the interaction file, keyed on the plain response text) and force a
live judge call on replay.

Look verdicts up in both the sidecar and the interaction cassette, and try the
plain-text key as a fallback to the whole-response key. New verdicts are still
written to the sidecar. Existing example/agents cassettes (router, summarizer)
replay again without hitting the network.

Add a regression test for the legacy-inline fallback.
@tmgbedu

tmgbedu commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

CI status

  • Run tests (fastapi_startkit): ✅ green — the framework package this PR changes, including all tests/ai (new eval assertions + regression tests).
  • config-app ✅, database-app ✅, Ruff ✅, Pyright ✅, codecov/patch ✅.
  • Run tests (agents): ❌ — all 11 failures are ConnectionError: PostgreSQL server at "127.0.0.1:3306" rejected SSL upgrade, a pre-existing CI database-environment misconfiguration (asyncpg pointed at :3306). The same job fails identically on the base ai-package-fix branch (run 30494801363), so it is not introduced here.

The judged example tests that my first push briefly broke (test_router_agent::test_the_router_with_initial_messages, test_job_search_integration summarizer) are fixed by the backward-compat commit — pre-existing inline/plain-text judge verdicts replay again without a live call. Verified locally: those tests pass; the only remaining example failures are the DB-env ones above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant