Skip to content

Document the on_turn callback contract - #101

Open
avalyset wants to merge 1 commit into
SimulaMet:devfrom
avalyset:docs/on-turn-contract
Open

avalyset wants to merge 1 commit into
SimulaMet:devfrom
avalyset:docs/on-turn-contract

Conversation

@avalyset

@avalyset avalyset commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

on_turn had a type in each signature, a three-line note in the run_scenario_reps docstring, and nothing in the README. Which roles fire when, what turn_index and max_turns mean, and what happens on a failure could only be recovered by reading run_scenario and SingleTurnAuditor side by side. I mentioned this on #95.

What changes

OnTurn in model_auditor.py names the callback type and documents the contract as the code behaves today, measured on both runners:

  • "auditor" only when the auditor wrote the probe, so not on a test_prompt turn
  • "target" per turn
  • "judge" once, at max_turns - 1, including when the judgment is a parse-failure ERROR
  • nothing from a call that raised, and nothing after it
  • no scenario name in the arguments, so calls interleave under max_workers > 1, and a retried rep in run_scenario_reps fires again

The nine on_turn annotations use the alias, the README parameter table gets an on_turn row, and the run_scenario_reps docstring points to the alias. OnTurn is not exported from simpleaudit/__init__.py.

No behaviour change: with docstrings stripped, the AST of the three changed modules differs only in the nine annotations, the alias and the imports that bring it in.

Testing

TestOnTurnContract in tests/test_single_turn.py runs ModelAuditor and SingleTurnAuditor and asserts the full sequence for each, including the target-down and judge-down paths. It passes on the unchanged code too, since it pins existing behaviour. Shifting the single-turn judge index from (0, 1, …) to (1, 1, …) fails it.

1349 passed, 19 skipped on Python 3.13.15, against 1348 on e5ec692 in the same environment.

Two edges, documented, not changed

  • ModelAuditor(max_turns=0) runs no turns, fires (-1, 0, "judge") and judges an empty conversation as pass. The contract states max_turns is assumed to be at least 1.
  • An async def callback is called but not awaited, so its body never runs. The contract says the callback must be a plain function.

Either could be its own PR if wanted.

A note from a trial merge

Merging main (9783293) into this branch gives no text conflicts, and the same holds for e5ec692 alone. With or without this PR, the merged tree fails test_single_turn.py::TestEntryPoints::test_run_sync_does_not_fall_back_to_the_turn_loop with TypeError: SingleTurnAuditor.run_async() got an unexpected keyword argument 'evidence_resolver', since main now passes evidence_resolver from run(). Worth knowing before the next sync.

on_turn had a type in each signature and a three-line note in one
docstring (AuditExperiment.run_scenario_reps), and nothing in the README.
Which roles fire when, what turn_index and max_turns mean, and what
happens on a failure were only recoverable by reading run_scenario and
SingleTurnAuditor side by side.

OnTurn in model_auditor.py now names the callback type and documents the
contract as the code behaves today, measured on both runners: "auditor"
only when the auditor wrote the probe, so not on a test_prompt turn;
"target" per turn; "judge" once, at max_turns - 1, including on a
parse-failure judgment; nothing from a call that raised, and nothing
after it. It also records what the arguments do not carry: no scenario
name, so calls interleave under max_workers > 1, and a retried rep in
run_scenario_reps fires again. The signatures use the alias, the README
parameter table gets an on_turn row, and the run_scenario_reps docstring
points to the alias.

No behaviour change: with docstrings stripped, the AST of the three
changed modules differs only in the nine annotations, the alias and the
imports that bring it in.

One test runs ModelAuditor and SingleTurnAuditor and asserts the full
sequence for each, including the target-down and judge-down paths. It
passes on the unchanged code too, since it pins existing behaviour.
@avalyset
avalyset requested a review from kelkalot as a code owner October 7, 2026 13:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant