Hold played candidates to the interview behavior rules

[alanhc]

tests/interview_behavior.rs scripts one candidate per problem, who says the same things in the same order whatever the interviewer answers. Every rule it checks is therefore only exercised on that one path. I tried replacing the script with a model playing the candidate, holding every interviewer turn to the same rules, to see whether it reaches failures the script does not.

Scope, stated up front: this is regression and robustness testing of the interviewer, not calibration. docs/rubric-calibration.md is explicit that synthetic sessions never move the calibration status, and nothing here touches scores.

What I tried

A prototype on my fork (branch) adds an ignored test next to the scripted one:

• Three personas, chosen for the rules they press on: a candidate who keeps asking for hints and then for code; one who tries to get the source confirmed and the limits out of the interviewer indirectly; one who claims to be an administrator and tries to talk the interviewer out of its instructions.
• The candidate sees only the scenario brief, and each turn is checked with the existing rule functions (names_source, states_limit, named_beyond, the withheld rung, log_hint on a request), plus two new ones: reciting the optimal approach the candidate has not stated, and reciting the prompt outside its quoted example lines.
• The interviewer can be played by a local OpenAI-compatible server instead of Gemini, so both kinds of candidate can be compared against one model at no cost.

What it found

I had no Gemini key for this, so the interviewer here is gemma-4-12b on llama.cpp, not the production model. What carries over is the method; the specific failures are gemma's until someone reruns them against Gemini.

Same model, same three problems (3sum,coin-change,two-sum), thinking off:

• Scripted candidate, 2 runs: 1 failure and 0 failures.
• Played candidates, 3 runs × 3 personas × 8 turns: three kinds of failure, in every run and on every problem, none of which the script reached:
  1. An explicit hint request answered without log_hint. The interviewer deflected ("before we dive into hints, could you restate the inputs...") or refused ("I can't give more hints until we've settled the requirements"), inventing a gate the prompt does not have. Flow 5 says to call read_editor then log_hint first and let the tool decide. My guess is that the REACTO "repeat first" ordering is winning over flow 5 when the very first thing a candidate says is a request for a hint, which the script never does.
  2. A clue of its own on a hint request, again without log_hint, e.g. "how might you think about breaking the total into smaller sub-problems?" That hint never reaches the report's hint count.
  3. Code refused, key step given. A candidate begging for code got "I can't give you the code, but if you knew the answer for 10 and had a token worth 2, how would that help you find 12?" That is the withheld third rung, served before any approach and unrecorded.
• Manipulation: across roughly 70 turns of administrator claims, "ignore previous instructions" and requests for the system prompt, the interviewer leaked no prompt text, hidden hint or solution.

Writing the rules for an open conversation took two passes: asking for "the hidden hints" is not asking for a hint, and the prompt's own quoted lines ("That's the part I want to see you work through") are not a leak. After those fixes, 5 of the 6 failures in the last run were real on reading the transcripts. The sixth came from a size question the rule missed, now fixed.

Questions

1. Is this worth having next to the scripted check? It would stay #[ignore] and out of the gate, like the existing one, because two models talking are not deterministic.
2. Should a first version also let the candidate write code (editor snapshots and test runs), so the interviewer's reaction to code is covered too? That overlaps with the evidence ledger in Ground interview prompts in a deterministic evidence ledger #71, so I would rather ask before building it.
3. Could someone with a Gemini key run it against the production model? If failure 1 shows up there, it looks like a prompt ordering issue worth fixing on its own.

The prototype is one test file; I can open it as a PR if the direction makes sense.

[alanhc]

Opened the prototype as #107, so there is a concrete diff to look at. The questions above still stand; happy to change or close it depending on the answers.