Run CodeTrial on a local model

[alanhc]

CodeTrial needs Gemini for two things: the live interviewer (Gemini Live) and the text calls (report, quiet-pause reviews). This issue collects what it takes to run each on a model on the operator's own GPU, what has been tried on one machine (RTX 5070 Ti, 16 GB), and what is still open. The goal is an option for operators who cannot or would rather not send interviews to Google, not a replacement for Gemini.

Where Gemini is used

Use	Transport	Local status
Final report	generateContent, JSON schema	Works, #99
Quiet-pause (interim) review	generateContent, plain text	Works, #99
tests/interview_behavior.rs (text model held to the live prompt and tools)	generateContent with function calling	Works on a branch, not yet a PR
Live interviewer	BidiGenerateContent WebSocket, audio in and out	Not started

Face presence already runs in the browser. LiveKit Cloud and Compiler Explorer are not models, but both can be self-hosted, which together with the above would make a fully offline deployment possible apart from the voice.

Approach so far

Rather than add a second provider to the Rust side, a small shim (scripts/gemini-shim.py, standard library only) answers Gemini's REST API from llama-server's OpenAI-compatible endpoint, and CODETRIAL_GEMINI_REST_BASE points the Rust calls at it. The Rust change is the base URL, plus repair prompts that name what a smaller model got wrong, which Gemini benefits from too.

The branch alanhc/codetrial:shim-tools, stacked on #99, adds:

• function declarations, calls and responses carried across as OpenAI tools, so tests/interview_behavior.rs runs against a local model through the same variable;
• a default output limit when a request names none, and a stop on Gemma 4's empty thought-channel loop when thinking is off (see below);
• --thinking off, which turns thinking off for every request, including the behaviour check's, which names no thinking config;
• tests/test_gemini_shim.py in the gate.

Findings

Model choice matters more than size. On the Two Sum report prompt:

Model	Reports passing validation
Qwen3.5-9B Q6_K	4 of 16 (before the repair changes in #99)
Qwen3-14B Q4_K_M	1 of 6 on a mid-level candidate prompt
gemma-4-12b Q4_K_M	10 of 10 with #99

Nearly every failure was the improvementPlan rule (one item per improvement, copied exactly), which repairs could not fix until they named the offending items.

Scores separate but may be lenient. gemma-4-12b gave a stub solution 10/20–40 NO_HIRE, a clean solution 100/100 HIRE, and a correct solution that needed a hint and misstated space complexity 85/70 HIRE, the same on all five runs. That last one arguably should not clear the mid-level bar; it has not been compared with Gemini on the same prompt.

Interim reviews take 0.1 to 1.1 seconds against a 12-second budget, stay grounded, and record a prompt-injection attempt as a note instead of obeying it.

The behaviour check passed 18 of 21 problem runs (3sum, coin-change, two-sum, seven runs each) with thinking off, in about 24 seconds a run; each miss was a second hint request answered without calling log_hint. With thinking on it passed 6 of 9 in about 240 seconds: the model sometimes answered a hint request without calling log_hint, and once spoke its own reasoning aloud, naming "greedy" while the ladder was withheld.

Gemma 4 behaves better with thinking off, with one quirk to handle:

• With thinking on, 5 interviewer turns in 117 ran to the 4096-token limit repeating themselves (often right after a tool response), and 2 put their reasoning in the reply instead of the thought channel.
• With thinking off, none of the 116 turns was capped, empty or carried markup, and the longest was 93 tokens.
• The quirk: with thinking off and nothing to say, it opens and closes empty thought channels until the output limit. In the interim review this came back as three garbage notes bound for the report. Stopping at the first channel opener is too eager, because after a tool response the model legitimately writes one or two empty channels before its reply; that stop left 27 of 28 such turns empty. Stopping at the fourth consecutive opener lets all 28 reply and still ends the empty case in about 13 tokens.

All of this is one prompt per case on one machine, so it shows the path works end to end, not that a local model matches Gemini.

The live interviewer

No open model that fits in 16 GB replaces Gemini Live on its own; the realistic shape is a cascade:

Silero VAD → faster-whisper large-v3-turbo → gemma-4-12b with tools → Kokoro TTS

The same shim idea applies: a local server that speaks BidiGenerateContent (setup, realtimeInput, serverContent, toolCall, interrupted, turnComplete, sessionResumptionUpdate) would leave src/livekit.rs untouched apart from the endpoint, since open_live_session_at already takes a URL. What it has to rebuild: server-side endpointing with the configured silence and sensitivity, barge-in that cancels TTS and the pending reply, both transcripts, the four tools, and sliding-window context. Estimated end-of-speech to first audio is about 0.8 to 1.5 s, which is slower than Gemini Live but has not been measured.

Questions for maintainers

1. Is a local-model option in scope for this repository, or better kept as an out-of-tree recipe?
2. If in scope, should the shim live in scripts/, or should the Rust side talk to an OpenAI-compatible endpoint directly and drop the shim?
3. Is it worth building the live-interviewer cascade here, or only the text paths?
4. Would a Gemini run of the mid-level candidate prompt be useful as a calibration reference? I can add it as a fixture.

[alanhc]

Updated the description with a second behaviour-check round. With thinking turned off for every request (--thinking off, now on shim-tools), gemma-4-12b passes 9 of 9 problem runs in about 24 seconds each, against 6 of 9 in about 240 seconds with thinking on, and no turn out of 116 was capped, empty or carried markup.

Getting there needed the empty-channel stop moved from the first opener to the fourth in a row: after a tool response the model writes one or two empty thought channels before its reply, and the first-opener stop had left those replies empty.

[alanhc]

Correction to the comment above: 9 of 9 was three runs. Across seven runs with thinking off it is 18 of 21, each miss a second hint request answered without calling log_hint. The description and the README on shim-tools now say so.