DecisionTarget review: 4 small gaps
Reviewed simpleaudit/targets/decision.py + decision.py + judges/choice_match.py + the ModelAuditor integration (v0.4.0). The overall design is clean — answer-key separation (public_decision via TargetContext.extra, full block only to the judge), fail-fast limit checks before any request, max_turns = 1, structured answer surfaced through TargetResponse.decision, and a genuinely code-only choice_match (no judge client created). The wire format is compatible with Ollama, vLLM, and OpenRouter's System One endpoints (verified against all three specs).
Four small gaps:
1. OpenRouter factory points at the legacy alpha URL
OPENROUTER_DECISIONS_URL = "https://openrouter.ai/api/alpha/decisions". OpenRouter's currently documented endpoint is POST https://openrouter.ai/api/v1/systemone (same request/response shape; accepts bare jev-… ids mapped onto typesafe/). The alpha route is likely to be deprecated — the factory should target /v1/systemone (or accept a base_url).
2. No retry/backoff
ModelTarget inherits max_retries/retry_backoff via the AnyLLM client; DecisionTarget.send has none, so a transient 5xx from the endpoint fails the scenario outright.
3. Fresh httpx.AsyncClient per call
When no client is injected, send creates and closes an AsyncClient for every request — no connection reuse across a run's scenarios. Minor at audit cadence, but a shared client (or a client= convention) would be cleaner.
4. state must be a mapping
validate_decision rejects a string state, while both vLLM and OpenRouter explicitly accept plain strings (or JSON sent as text). build_request already falls back to the prompt text when state is empty; accepting a string would just pass it through.
Happy to send PRs for 1 and 4 (one-line changes + tests); 2 and 3 are larger and can be follow-ups.
DecisionTargetreview: 4 small gapsReviewed
simpleaudit/targets/decision.py+decision.py+judges/choice_match.py+ theModelAuditorintegration (v0.4.0). The overall design is clean — answer-key separation (public_decisionviaTargetContext.extra, full block only to the judge), fail-fast limit checks before any request,max_turns = 1, structured answer surfaced throughTargetResponse.decision, and a genuinely code-onlychoice_match(no judge client created). The wire format is compatible with Ollama, vLLM, and OpenRouter's System One endpoints (verified against all three specs).Four small gaps:
1. OpenRouter factory points at the legacy alpha URL
OPENROUTER_DECISIONS_URL = "https://openrouter.ai/api/alpha/decisions". OpenRouter's currently documented endpoint isPOST https://openrouter.ai/api/v1/systemone(same request/response shape; accepts barejev-…ids mapped ontotypesafe/). The alpha route is likely to be deprecated — the factory should target/v1/systemone(or accept abase_url).2. No retry/backoff
ModelTargetinheritsmax_retries/retry_backoffvia the AnyLLM client;DecisionTarget.sendhas none, so a transient 5xx from the endpoint fails the scenario outright.3. Fresh
httpx.AsyncClientper callWhen no client is injected,
sendcreates and closes an AsyncClient for every request — no connection reuse across a run's scenarios. Minor at audit cadence, but a shared client (or aclient=convention) would be cleaner.4.
statemust be a mappingvalidate_decisionrejects a stringstate, while both vLLM and OpenRouter explicitly accept plain strings (or JSON sent as text).build_requestalready falls back to the prompt text whenstateis empty; accepting a string would just pass it through.Happy to send PRs for 1 and 4 (one-line changes + tests); 2 and 3 are larger and can be follow-ups.