🤖 feat: Auto model and thinking-level routing via AI SDK evaluation models

[ibetitsmike]

Summary

Adds an experimental Auto choice to the composer's model picker and, independently, to its thinking-level picker. With either Auto on, the backend asks an AI SDK evaluation model (experimental_evaluate) how hard the prompt is, using difficulty tiers the user defines in Settings (easy / medium / hard / extreme by default), and runs the turn on the model and/or thinking level mapped to the chosen tier. The evaluation model is the user's choice (typesafe:jev-latest by default; any typesafe, openai, anthropic, or google model that ships an evaluationModel() factory works), so nothing is hardcoded to TypeSafe. The composer's model and level stay the fallback whenever routing cannot happen. A thinking level Auto chose also raises itself mid-turn when the turn looks stuck (repeated tool failures or a replayed call), one level at a time, unless the user moves the slider.

Background

Switching models per message by hand is tedious: cheap fast models suit most prompts, expensive reasoning models only some. This experiment lets the user describe the tiers once, map each to a model, a thinking level, or both, and let a classifier pick per prompt. Gated behind the auto-model-routing experiment.

The redesign in a795a1715c replaced the original hand-rolled TypeSafe HTTP client with the AI SDK evaluation API (see Evaluation and the TypeSafe AI provider), split Auto into two dimensions, and made the router an Effect service. That API requires ai >= 7.0.103, so 20bec3032b bumps ai to 7.0.107 and the AI SDK providers to the matching @ai-sdk/provider 4.0.17 line and adds @ai-sdk/typesafe-ai (@ai-sdk/xai stays on the patched 4.0.37; a bump silently dropped patches/@ai-sdk%2Fxai@4.0.37.patch). That commit is self-contained and can be split into its own PR.

Implementation

• Two flags, not model strings. SendMessageOptions.autoModelRouting (model) and autoThinkingLevel (thinking) ride alongside a concrete model and thinkingLevel, so every existing validator keeps seeing real values. The backend classifies once when either flag is set and a mapped tier exists, applies tier.model only under the model flag and tier.thinkingLevel only under the thinking flag, and records which dimensions Auto set (AutoModelRoutingRecord.thinkingLevel). A tier whose mapping covers none of the enabled dimensions yields an unmapped-tier fallback.
• One resolution point. AgentSession.sendMessage resolves routing before modelForStream, strips both flags before any snapshot, and skips classification for synthetic or agent-initiated turns. The routed model passes the budgeted-goal pricing gate and the attachment/vision checks (against the attachments the provider request actually carries); whether it can be built at all (credentials, disabled or removed provider, policy on the route-resolved identity, catalog) is decided where every request is prepared: TurnRequestBuilder.prepare retries resolveAndCreateModel with the composer's model when a routed tier model fails and records the factory's own reason. The session adopts that prepared record on stream-start, so live pricing, compaction thresholds and mid-stream compaction follow-ups follow the model that actually streams. Every model-only fallback reverts just the model and keeps the tier's thinking level, and the record's thinking level is stamped with the level the request runs at. Explicit per-send choices (one-shot /model, /+2, Implement the plan) drop the matching flag. /compact follow-ups (including attachment-only ones) are routed once and carry the record through redispatch; a resume validates the persisted record with AutoModelRoutingRecordSchema before reusing it.
• Router as an Effect service. AutoModelRouterTag / AutoModelRouterLive (src/node/services/di) replace ad-hoc construction in WorkspaceService and the oRPC handlers. AutoModelRouter.classifyEffect runs experimental_evaluate with one choice question keyed by tier id (the SDK validates the verdict against the offered choices and the probability distribution), maxRetries: 0, an 8 s timeout, and the prompt plus up to three prior user prompts from the latest durable boundary. TypeSafe confidence is read from providerMetadata.typesafe when present; other providers simply return no confidence.
• Evaluation model factory (src/node/services/evaluationModelFactory.ts). Builds the Experimental_EvaluationModelV4 from the user's provider:model string: credentials come from providers.jsonc (typesafe is a reserved id whose key resolves config, then apiKeyFile, then TYPESAFE_API_KEY / TYPESAFE_AI_API_KEY / JEV_API_KEY), disabled providers and the enforced policy (provider_access, model_access, forced base_url) are honored, and every failure is a typed reason (invalid_model, policy_denied, provider_disabled, missing_api_key) that the Settings status line shows.
• Config. config.json autoModelRouting.{tiers, evaluationModel}; tiers and evaluator heal independently on load. oRPC: config.updateAutoModelRouting, config.getAutoModelRoutingEvaluationStatus({ evaluationModel? }) (probes the typed value before saving), config.previewAutoModelRouting({ prompt, workspaceId, config? }) (a paid evaluator call billed to the named workspace's usage ledger as auto_model_routing_preview; the Settings panel bills the last selected workspace and says so).
• Renderer. ModelSelector and ThinkingSelector each gain a pinned, keyboard-reachable Auto row with its own per-workspace flag (copied from the creation composer to the new workspace). A concrete pick leaves Auto for that dimension: setWorkspaceModelWithOrigin for models, ThinkingContext.setThinkingLevel (selector and keybinds) plus the palette setter for thinking. Settings: evaluation model field with live availability status, TypeSafe key field only while the evaluator is a typesafe model, tier editor (model, thinking level, or both; text fields commit on blur or Enter, an invalid draft reverts), "Test routing" preview. The transcript badge shows the routed tier, model, and thinking level.
• Mid-turn thinking escalation (src/node/services/autoThinkingEscalation.ts). Only for turns whose thinking level Auto set. StreamManager.prepareStep reads the step transcript it already has (no evaluator call per step): after AUTO_THINKING_ESCALATION_WINDOW_STEPS (3) consecutive tool steps that all failed, or that replayed the same tool calls with the same results, it requests one level up the ladder through the same ActiveTurnThinkingOverride the slider uses, so the rebuild's per-model clamp, the first-step message rebuild, and the continuous-compaction swap check apply unchanged. At most AUTO_THINKING_ESCALATION_MAX_PER_TURN (2) raises per turn; a raise the model's ceiling clamps away ends further attempts; a slider move during the turn (manual) disables escalation for the rest of it. Each applied raise is persisted as AutoModelRoutingRecord.escalations[] (step, from, to, reason); the badge labels the level the turn finished at and the tooltip lists one line per raise. The replay signal compares each step's complete call/result set: a step counts only when every call and every result match the step before (a repeated status check beside new edits is progress), and a step of nothing but wait tools (AUTO_THINKING_ESCALATION_WAIT_TOOLS) never counts. Raises follow the turn: the holder's onLiveRoutingChanged sink reports every mid-turn change to what an Auto-routed stream runs on (an applied raise or slider move, a refusal fallback), so the session's live stream context (model, record, options.thinkingLevel) stays current and mid-stream compaction follow-ups resume on what the stream ran on, a resume under thinking Auto reads the raises off the interrupted assistant row, and a new stream seeds its per-turn cap from carried raises. A slider move mid-turn withdraws Auto's thinking claim (level and raises) from the record.

Validation

• Unit: autoModelRouter.test.ts drives the real experimental_evaluate against a fake Experimental_EvaluationModelV4 (verdict mapping, criteria keyed by tier id, context trimming, missing confidence, thrown/aborted evaluator, unbuildable evaluator) plus factory resolution for all four providers, policy denial, forced base URL, disabled provider, and credential precedence. agentSession.autoModelRouting.test.ts covers model-only, thinking-only, and both-dimension Auto, unmapped tiers, corrupted resume records, and /compact follow-ups. tests/ipc/config/autoModelRouting.test.ts covers the evaluator round-trip and IPC rejection of unsupported evaluators. aiService.test.ts covers the preparation-time fallback (factory verdict recorded, the composer model's own failure when both fail, no retry for a record that is not a live routed swap, record thinking level following the request). autoThinkingEscalation.test.ts covers step collection, both stuck signals, the per-turn cap and the ladder top; streamManager.test.ts drives the real prepareStep with a stuck transcript (raise applied through the rebuild and recorded, slider move disables it, a raise clamped away is not provenance, a raise a sparse ladder clamps upward is recorded at the level that applied) and startStream wiring (armed only for an Auto-set thinking level, raises land on the stream-end record, a slider move withdraws the claim, the session's live-routing sink sees each change), and the refusal-fallback swap reports the fallback model and clamped level to the session; agentSession.autoModelRouting.test.ts covers the resume after a raise, the resume on a refusal fallback model, and mid-stream compaction follow-ups after a raise, a slider move, and a refusal fallback; autoThinkingEscalation.test.ts covers the judged-step boundary carried into a resume and reset by a compaction follow-up. AutoModelRoutingExperimentConfig.test.tsx covers blur and Enter commits, duplicate-label rejection, cross-window config spreading, and the rejected-write revert.
• Earlier live UAT rounds (below) exercised the pre-redesign TypeSafe client; the evaluation-model path has unit coverage only so far and should get a live round with a real key before the experiment is promoted.

Known limits and follow-ups

• The Auto flags are browser-local composer state (usePersistedState, per workspace); another browser or device starts with the concrete model and level. Making them follow the workspace means a field on the persisted WorkspaceAISettings, deferred.
• A routed tier's thinking level goes through the existing per-model floor (getDefaultMinimumThinkingLevel), so a low tier shows as medium on models with an explicit thinking policy. Same rule as picking that model by hand.
• Mid-turn escalation is a request-level thinking change, like the slider: on OpenAI it invalidates the cached prompt prefix. GPT-6 Astra's cache-preserving configuration_update input item (single-agent mode, incompatible with automatic compaction) is a provider-specific optimization the repo's @ai-sdk/openai does not expose; deferred as a follow-up.
• The stuck signal is deterministic and local (failed tool results, identical replays); it does not judge text-only loops, and the live badge shows a raise only once the turn ends because the stream-start record predates it.

Risks

• Send path. The only shared-path change is in AgentSession.sendMessage, inert unless the experiment is on and a flag is set. With Auto on, the user message is appended after classification (about 2 s later than usual, up to 8 s on a timeout). Every evaluator failure falls back to the composer model and level and shows a fallback badge.
• Privacy. With Auto on, the prompt and up to three prior user prompts are sent to the configured evaluation model's provider. The Settings panel states this next to the evaluator field.
• Dependency bump. ai 7.0.19 -> 7.0.107 and all first-party AI SDK providers move together. Typecheck, make static-check, and the AI suites are green; streamManager*.test.ts only fail under bun 1.2.15 (eager mockRejectedValueOnce), not the pinned 1.3.5.
• Compatibility. New optional config and message-metadata fields only; older builds ignore them.

Delivery record: redesign (2026-09-21)

• Owner direction: do not hardcode a dependency on TypeSafe's Jev; follow the AI SDK evaluation docs and keep the evaluation model the user's choice; split auto reasoning effort from auto model selection; make the router an Effect service.
• 20bec3032b: dependency bump only. a795a1715c: the redesign. Backend, common, node tests, and the IPC test by the parent; the composer/transcript and Settings renderer work by two exec children in the shared checkout with disjoint file ownership. The children's storybook test-runner run for ExperimentsSection.stories.tsx passed 6/6 including the new phone play.
• The three Codex threads open on 3673cd234d (record validation on resume, attachment-only /compact provenance, phone-viewport story) are fixed in a795a1715c with tests that were red on the previous head; replies posted and threads resolved.
• Codex review round 12 on a795a1715c (automatic "New commits" pair; security pass added nothing): 5 P2 findings, all confirmed and fixed in 973691a671 with tests that were red on the previous head: a resume with one Auto dimension restored both saved values (now per dimension, record kept only when the resume still runs on the routed model), the OpenAI evaluator dropped the configured organization, Implement-the-plan left thinking Auto on, non-timeout evaluator failures persisted the raw SDK message (now a status/name category), and evaluator usage bypassed cost accounting (now recordHeadlessUsage with analyticsSource: auto_model_routing). Threads resolved after the push. 112ef36374 merges origin/main and f6aa2916a4 sets the flake hash CI computed for that merged lockfile.
• Codex review round 13 on 973691a671 (automatic pair; security pass added nothing, both earlier advisories now marked Resolved): 1 P1 and 1 P2, both confirmed and fixed in b923ee7d78 with a test that was red on the previous head: the typesafe providers.jsonc entry honored only baseUrl (now resolveConfigBaseUrl, both spellings), and configured proxy headers were dropped for every evaluator provider (now seeded into buildAIProviderRequestHeaders). Threads resolved after the push. Flake Hash Check passed on 973691a671.
• Codex review round 14 on b923ee7d78 (automatic pair; security pass added nothing): 1 P1 and 3 P2, all confirmed and fixed in abb8d9bf9d with tests that were red on the previous head: an id shadowed by a custom chat provider fell through to native env credentials (now an unavailable custom_provider reason), a budgeted goal could pay for an unpriced evaluator (now gated like the tier model), a resume that left thinking Auto kept the record's routed thinking level (now dropped), and context-budget-rejected prompts reached the evaluator (now the provider-eligibility filter). Threads resolved after the push. Every CI job other than the Codex gate was green on b923ee7d78.
• Codex review round 15 on abb8d9bf9d (automatic pair; security pass added nothing): 2 P2, both confirmed and fixed in b6e663c6d4 with tests that were red on the previous head: workflow-trigger display rows reached the evaluator context (now the same isWorkflowDisplayOnlyMessage exclusion provider assembly uses) and a recovered compaction follow-up forwarded an unvalidated autoModelRouting record (now safeParse, dropped when malformed). d66117390b types the hand-edited fixture that the first push left red in typecheck. Threads resolved after the push.
• Codex review round 16 on d66117390b (automatic pair; security pass added nothing): 1 P2, confirmed and fixed in a55638eeab with three tests that were red on the previous head: routing checked attachments against the pre-compaction context, so an on-send compaction or a /compact follow-up fell back over an image or PDF the summary was about to fold away (now gateRoutedModelAgainstAttachments runs in sendMessage on the request that streams, fresh or carried by a follow-up, against this turn's parts plus the post-boundary context; the on-send follow-up carries the pre-gate decision and the compaction request keeps the gated options). Thread resolved after the push.
• Codex review round 17 on a55638eeab (automatic pair; security pass added nothing): 3 P2. The refusal-fallback record patch dropped the clamped thinking level, fixed in 9ee07c80b5 with the existing swap test extended (red on the previous head). The other two ask for further pre-flight gates (route availability, route-resolved policy identity) that would copy more of createModel; the review loop was paused on those two threads with options posted on the PR (land with follow-ups, or a preparation-time fallback); the owner chose the preparation-time fallback in this PR (round 18).
• Round 18 (owner direction on the round-17 options): 76b0fbb60f removes the hand-rolled policy pre-flight from resolveAutoModelRouting (and the policyService dependency of AgentSession) and adds the preparation-time fallback in TurnRequestBuilder.prepare: when resolveAndCreateModel fails for a routed tier model, the composer's model is prepared instead and the record carries the factory's own reason, which closes both open round-17 threads (removed credential/provider/catalog entry, policy on the route-resolved identity) with one mechanism instead of pre-flight copies. Codex's automatic pair on 9ee07c80b5 (5 P2, security pass added nothing) is addressed in cba615b28a with tests that were red on the previous head: display-only rows (context-budget-rejected, workflow display) no longer feed the attachment gate; the record's thinking level is stamped once, in TurnRequestBuilder, with the level the request runs at (covers the attachment and preparation fallbacks); a pricing fallback keeps the tier's thinking level; explicit agent switches clear thinking Auto the way they clear model Auto (setWorkspaceThinkingLevelWithOrigin); the badge tooltip distinguishes a fallback after a verdict from a failed classification. All seven threads replied to and resolved after the push.
• Codex review round 19 on cba615b28a (automatic pair; security pass added nothing): 3 P2, all confirmed and fixed in a5bf5ad0a2 with tests that were red on the previous head: a review-only /compact follow-up skipped routing (now classified on the text the redispatch sends, reviews included), the resume lookup searched only the last 20 rows so a routed turn behind more report rows lost its provenance on Continue (the bounded tail is now classifier-only), and evaluator proxy base URLs were not normalized like chat requests (now normalizeOpenAICompatibleBaseURL / normalizeAnthropicBaseURL). Threads replied to and resolved after the push.
• Codex review round 20 on a5bf5ad0a2 (automatic pair; security pass added nothing): 2 P2, both confirmed and fixed in 3c3f17eb3a with an it.each test that was red on the previous head for both cases: the session's live stream context kept the unavailable tier model after the preparation-time fallback (usage pricing, goal accounting and compaction thresholds ran as the tier model), now adoptPreparedRouting takes the record and model from the non-replay stream-start; and mid-stream summarize/continuous compaction built the Continue follow-up without the routing record, now both forward the stream context's record like the on-send path. Threads replied to and resolved after the push.
• Codex review round 21 on 3c3f17eb3a (automatic pair; security pass added nothing): 3 P2 in three unrelated areas (the count went 2 to 3; each is a local defect with a small fix, so the loop continued, and a next round that does not shrink pauses for direction), all confirmed and fixed in 7c51497883 with tests that were red on the previous head: evaluator spend reached the workspace ledger but never a budgeted goal's cap (now charged through recordStreamAccounting as a zero-turn user-origin stream); the Settings preview classified against the persisted config while an optimistic save was in flight (the request now carries the tiers on screen); a thinking-only one-shot (/+2 hello) still disabled model Auto under a pre-split rule (only a model one-shot pins the model now). Threads replied to and resolved after the push.
• Codex review round 22 on 7c51497883 (automatic pair; security pass added nothing): 3 P2, the count did not shrink and the findings returned to areas earlier rounds touched, so the loop paused with a PR comment; the owner chose to fix them here and accept further rounds. Fixed in 8335fbe27b with tests that were red on the previous head: the classifier context sliced 20 rows before filtering user prompts (now filters first), a thinking-only routing resumed at a hand-picked level kept a routed record (the record now survives with a dimension still on Auto, or with matching concrete picks), and the Settings panel's debounced tier commit spread a stale config (now one latest-config ref for every write). Threads replied to and resolved after the push.
• Owner direction (mid-turn reasoning): Astra-style raising of the reasoning effort when the model is stuck, in this PR, with the proposed defaults. bfdc658a10 adds the escalation described under Implementation; reading A (Auto's own thinking level, request-level change) was chosen over the provider-specific cache-preserving item.
• Codex review round 23 on bfdc658a10 (automatic pair; security pass added nothing): 4 P2, all on the new escalation surface, all confirmed and fixed in 554759c244 with tests that were red on the previous head: a resume after a raise dropped back to the tier's level (now reads the raises off the interrupted assistant row and continues at the last one); mid-stream compaction follow-ups were built from a stream context that never learned about raises (now the holder's onEscalated sink updates it, and the follow-up stream seeds its cap from carried raises); a slider move after Auto set or raised the level left the record claiming Auto's level (now withdrawn); identical successful wait calls counted as a stuck replay (now the replay must repeat the same result and wait tools are excluded). Threads replied to and resolved after the push.
• Codex review round 24 on 554759c244 (automatic pair; security pass added nothing): 2 P1 (AGENTS.md rules) and 2 P2, all confirmed and fixed in 8113cdbbf3: useAutoModelRouting memoized its callbacks by hand (now the fetch closure lives inside the subscription effect and a rejected write re-runs it); the Settings tier text fields persisted through a 500 ms timer with refs and an unmount flush (now they commit on blur or Enter like the evaluation-model field); a slider move and a refusal fallback changed what the stream ran on without reaching the session's stream context, so a mid-stream compaction follow-up resumed on the tier's model and level and re-armed a withdrawn claim (now one onLiveRoutingChanged sink replaces onEscalated and reports every change; the follow-up is covered for a raise, a slider move, and a fallback by tests that were red on the previous head). Threads replied to and resolved after the push.
• Codex review round 25 on 8113cdbbf3 (automatic pair; security pass added nothing): 2 P2 (the count went 4 to 2). One fixed in c78937c2ca with a test that was red on the previous head: a hand-edited or damaged autoModelRouting tier failed the whole config.json schema (now .optional().catch(undefined), the settingsBackup pattern; the normalizing read path owns the repair). One declined with evidence: the creation composer's PDF preflight validates the composer's concrete model under Auto by design, because that model is the fallback for every model-only gate and AgentSession.sendMessage rejects a PDF the composer model cannot take before routing runs (agentSession.ts:3869 vs. :4174); skipping the preflight would recreate the empty-workspace failure it prevents. Threads replied to and resolved after the push.
• Codex review round 26 on c78937c2ca (automatic pair; security pass added nothing): 2 P2 on the escalation resume path, both confirmed and fixed in 71ad9e1fc5 with tests that were red on the previous head: a resume after a raise re-judged the same stuck window and spent the second raise at once (the escalation state now seeds its judged-step boundary from the last carried raise and clamps it to the steps the transcript still holds, so a compaction follow-up counts fresh); a resume under model Auto went back to the tier model after a refusal fallback (the resume now continues on the interrupted assistant row's record, the model that was answering, and keeps that provenance). Threads replied to and resolved after the push.
• Codex review round 27 on 71ad9e1fc5 (automatic pair; security pass added nothing): 2 P2. One fixed in 21d74a3762 with a test that was red on the previous head: a raise a sparse thinking ladder clamped upward (low requesting medium on a low/high model) applied but was rejected as provenance and ended escalation (now recorded at the level that applied). One declined with evidence plus a pinning test: the tier's thinking level is clamped against the model that streams in streamWithHistory (the same enforceThinkingPolicy call a composer pick gets, agentSession.ts:8208) before request assembly; a tier level the tier model's ladder lacks streams at the clamped level, like a composer pick passes without a production change. Threads replied to and resolved after the push.
• Codex review round 28 on 21d74a3762 (automatic pair; security pass added nothing): 1 P2, the backend twin of the round-25 finding (move the PDF preflight in sendMessage after routing). Declined with evidence and resolved, no code change: the composer's concrete model is the fallback for every model-only gate (attachment re-gate, pricing gate, unbuildable tier model, refusal chain), so the pre-routing check is what keeps that fallback viable and keeps the evaluator from being billed for a send the provider would reject mid-turn. @codex review requested on the same head for a verdict; the head's Test / Unit job died in a bun allocator panic while loading an unrelated test file (exit 133, no failing assertion) and was rerun.
• Codex review round 29 on 21d74a3762 (manual request for a verdict; security pass added nothing): 1 P1 + 2 P2, none on the escalation surface. All fixed in 5dca6adbdd with tests that were red on the previous head. The routing settings hook fetched its config before its change subscription was live, so a save landing in that window was never seen (P1; it now subscribes first). The evaluator's goal charge could tip a budgeted goal into budget_limited before the response streamed, after which the response's own user-origin accounting was skipped; the spend now rides the routed turn's own stream accounting (and its live preview), and compaction streams leave it for the turn behind the boundary. A fallback badge after a verdict hid the tier and the thinking level Auto had applied; it now shows both, marked as a model fallback. Threads replied to and resolved after the push.
• Codex review round 30 on 5dca6adbdd (manual request; security pass added nothing): 1 P1 + 2 P2, again on three unrelated areas. All fixed in 612c282ea4 with tests that were red on the previous head. The palette gained Toggle Auto Model Routing and Toggle Auto Thinking Effort (experiment-gated, workspace and creation scopes) so both Auto dimensions have a keyboard path, since the model-cycle and thinking-step shortcuts can only leave Auto (P1). The round-29 deferred evaluator charge is now bound to the send's preparation attempt: it reaches stream accounting only when that attempt's stream is delivered, an attempt that never streams settles it immediately as the zero-turn user charge, and a terminal stream error discarded it with the stream's own cost (round 33 changed this case to charge the evaluator spend, which had been billed), so nothing owed leaks into an unrelated later turn. An explicit agent switch whose resolved model or level already matched the stored value skipped the setters and left Auto armed; the switch now leaves both Auto dimensions regardless. Threads replied to and resolved after the push.
• Codex review round 31 on 612c282ea4 (manual request; security pass added nothing): 2 P2 (the count went 3 to 2). Both fixed in c3c73d403a with tests that were red on the previous head. A compaction the user stops before its follow-up dispatches now settles the evaluator spend it carried against the originating goal instead of leaving it to the next unrelated stream. The Settings key controls for the TypeSafe evaluator stay hidden while the typesafe providers.jsonc entry is a legacy custom chat provider, so Save/Clear cannot overwrite that provider's credential (the evaluator refuses such an entry, and the status line already says so). Threads replied to and resolved after the push.
• Codex review round 32 on c3c73d403a (manual request; security pass added nothing): 1 P2 (the count went 2 to 1), a refinement of the round-25 fix. Fixed in 2e9c17dc03 with tests that were red on the previous head: the on-disk schema now keeps the routing block's shape instead of validating it strictly and collapsing the whole block on one bad tier, so the normalizing read repairs per tier and per field and a hand-edited tier no longer costs the other tiers and the evaluator; a non-object block still reads as absent. Thread replied to and resolved after the push.
• Codex review round 33 on 2e9c17dc03 (manual request; security pass added nothing): 4 P2 (the count went 1 to 4), on four separate surfaces. All fixed in 345f02f189 with tests that were red on the previous head. A routed stream that ended in a terminal provider error discarded the evaluator spend it carried (already billed) along with the failed stream's cost; that spend is now settled as the zero-turn user charge, reversing the round-30 discard for this case. A resume that hand-picks a different model while thinking stays on Auto still ran at Auto's level but dropped the record, so escalation never seeded and the badge vanished; the record now survives as a thinking-only routing restamped on the picked model. A refusal fallback that clamped Auto's thinking level left the escalation state at the refused model's level, so the next raise was a no-op that retired escalation; the swap now rebases it (and clears a ceiling the refused model hit). AutoModelRoutingRecordSchema now requires provider:model format for model and requestedFallbackModel, both re-read from history to build requests, so a damaged record is treated as absent on resume. Threads replied to and resolved after the push.
• Codex review round 34 on 345f02f189 (manual request; security pass added nothing): 1 P2 (the count went 4 to 1). Fixed in 11ae296224 with tests that were red on the previous head: the Settings panel's Classify preview is a paid evaluator request that was never recorded in the cost ledger or analytics. Ledgers are per-workspace, so the preview now names the workspace it bills (the last selected one, shown next to the button; Classify stays disabled without one), the handler refuses a workspace the config does not know before calling the evaluator, and records the verdict's usage through recordHeadlessUsage with source auto_model_routing_preview before tier mapping. Goal accounting is untouched (a preview is not a turn). Thread replied to and resolved after the push.
• Codex review round 35 on 11ae296224 (manual request; security pass added nothing): 1 P2 + 1 P3 (the count went 1 to 2), both on the round-34 panel. Fixed in b0c7d9d131 with tests that were red on the previous head. The panel trusted the persisted workspace selection's shape and a legacy id-only entry crashed the Experiments panel when its label rendered; it now reads only the stored id and resolves the workspace through live metadata (a removed workspace disables Classify instead of being billed). The evaluator availability check did not re-run when provider credentials changed from elsewhere; the providers config snapshot is now a dependency of that effect. Threads replied to and resolved after the push.
• Codex review round 36 on b0c7d9d131 (manual request; security pass added nothing): 2 P2 (the count stayed at 2, the second round without shrinkage). One fixed in 31491d194b with a test that was red on the previous head: the live-routing sink updated the active context's model and record but not the send options that the context-window rollover and the compaction retry replay through streamWithHistory, so a recovery after a raise, a slider move, or a refusal fallback re-armed the pre-change provenance; the sink now mirrors the live model and record onto those options too. One declined and resolved: deferring the budgeted-goal pricing gate on the composer's model until after routing, the third instance of the pre-routing gate question after the round-25 and round-28 attachment twins; the composer's model is the fallback of every model-only gate (the tier model's own pricing fallback, attachments, an unbuildable tier model) and must itself be priceable. Threads replied to and resolved after the push.
• Codex review round 37 on 31491d194b (manual request; security pass added nothing): 1 P2 (down from 2). Confirmed and fixed in c44ac4db2c with a test that was red on the previous head: the replay signal looked for one call whose input and result repeated in every step of the window, so three steps that each repeated a status check beside distinct successful edits counted as a loop and raised the thinking level; the signal now compares each step's complete call/result set, so only a step whose every call and result match the step before is a replay. Thread replied to and resolved after the push.
• Deleted as dead weight after probing the SDK: the router's own tier-id filtering of probabilities and the unknown-choice check (experimental_evaluate rejects unknown choices and non-summing distributions itself).

Delivery record: original TypeSafe-client design (superseded)

• Design: deep-codebase-exploration (5 lanes) established that Auto must be a flag beside a concrete model, that AgentSession.sendMessage is the single resolution point, and that the key belongs in providers.jsonc outside ProviderName.
• Static review: read-only reviewer pass over 3e1241488b (2 P1, 2 P2, 9 P3). P1/P2 fixed in 7f865f194c; declined: clearing Auto before a cancellable model switch (there is no cancel path) and the aggregator writing undefined (matches adjacent metadata lines).
• Remote UAT round 1: dogfood.cdr.dev chat 00d5e851-fd0d-438d-8e44-390e90234ada on 3e1241488b, BLOCKED for the classifier (no key in Coder Agents workspaces), 6 defects found (creation composer ignored Auto, resume lost the routed model/badge, tier edits committed only on blur, no validation feedback, raw error bodies surfaced, missing accessible names). Fixed in 7f865f194c and 804751eb68.
• Local UAT round 2 on 804751eb68 with the real key: PASS, all six round 1 defects re-verified as fixed; three P3 notes adjudicated above (browser-local Auto, thinking floor, confidence 99% next to a probability rounded to 100% is the API's own numbers).
• Polish: simplify + deslop pass, behavior-neutral (ec6087b4f6).
• Codex review round 1 on ec6087b4f6: 6 findings. Fixed in a0b41b28de, each with a test that failed on the previous head: explicit per-send models bypassed by Auto (P1), thinking-only tiers skipped (P1), attachments not revalidated against the tier model (P2), TypeSafe key counted as a chat provider (P2), disclosure missing prior prompts (P2). Declined with evidence: atomic multi-draft unmount flush (unreachable, both fields commit on blur and only one can hold focus).
• Codex review round 2 on a0b41b28de: 5 findings, all confirmed and fixed in b51e13b7f6 with tests that failed on the previous head: classifier ignored provider policy (P1), typesafe could collide with a custom provider id (P1, now reserved plus shape guards), thinking-only one-shot commands kept Auto (P2), on-send compaction dropped the routing record (P2), refusal fallback left the record on the refused model (P2). Evidence replies posted on all five threads and the threads resolved after the fix.
• Merge of origin/main (7791a46625): one conflict in ProposePlanToolCall.tsx, where main consolidated both Implement sends into runPlanAction; the autoModelRouting: false opt-out moved into that single shared send. Typecheck, 2583 targeted tests, and make static-check green on the merged tree.
• Codex review round 3 on 7791a46625, triggered automatically by the push ("New commits"), not requested: 3 findings. Fixed in 64c8167c98 with tests that were red on the previous head: typesafe rejected by PolicyProviderIdSchema so an enforced provider_access could never authorize the classifier (P1), and a policy-denied tier model failed the turn instead of falling back (P2). Declined with evidence: history loss when Stop cancels classification during an edit (P1) is unreachable, because the only cancelSignal producer is the bash-monitor wake, which routing skips.
• Codex review round 4 on 64c8167c98, triggered by marking the PR ready at 20:25Z (code and security passes 8 and 9 against the six-pass cap): 4 P2 findings, all confirmed and fixed in acac7524ca with tests that were red on the previous head: a user-built compaction request (/compact plus follow-up, compact-and-retry) bypassed Auto for the follow-up (now classified once and stored on parsed.followUpContent), a resume naming the direct twin of a Coder-routed tier model kept the record (now modelSelectionEqualityKey), unbounded tier labels/descriptions (32/400-char caps in schema and UI), and an unbounded badge label (truncated, tooltip keeps the full label). Evidence replies posted on all four threads and the threads resolved.
• Local UAT round 3 on 64c8167c98 with the real key (fresh sandbox root, plus policy-file runs): PASS on every headline flow and on the reachable Codex-round fixes, including the enforced-policy paths (classifier authorized when typesafe is listed, denied tier model falls back with a legible reason, policy without typesafe skips the classifier), thinking-only tiers, attachment revalidation, plan Implement on the concrete model, 375 px layout, zero console errors. One new P2: a text-only Auto send routed to a tier model whose provider rejects a PDF sent earlier in the conversation failed the turn and kept retrying. Fixed in 16c5679970: the tier-model attachment check now covers every user attachment still in the context window (test red on the previous head). Not reachable in that round: on-send compaction record carry-over (threshold is backend-owned), refusal fallback. Design note confirmed against the code: Implement-from-plan sends on the concrete model but does not clear the composer's Auto choice; only the send drops the flag. Evidence: /home/coder/.xum-uat-evidence/auto-model-routing/round-3/.
• Local UAT round 3 delta on 16c5679970 with the real key (fresh sandbox root): PASS. Verified the history-attachment fix (PDF in history, Easy tier on xai:grok-4.6: classified, fell back to the composer model with a legible reason, no provider error, no retry loop), /compact plus follow-up under Auto (summary unrouted, follow-up routed once with badge and record), bare /compact unchanged, the 32/400 caps in the UI and on disk, badge truncation at 375 px with the full label in the tooltip, and easy/extreme regression sends; zero console errors at the end of the run. Skipped: Coder-route resume (no direct twin model configured in the sandbox; covered by the unit test). One P3 found: doubled period in the attachment fallback tooltip, fixed in 80684e010a (unit test red on the previous head). One-off React dev warning during creation-composer setup that did not reproduce in four controlled repeats, not attributed to routing code. Evidence: /home/coder/.xum-uat-evidence/auto-model-routing/round-3-delta/. The final head 80684e010a differs from the UAT-tested 16c5679970 only by that tooltip punctuation change.
• Codex review round 5 on 80684e010a, triggered by the owner marking the PR ready at 08:59Z on 2026-09-21 after deciding to spend the passes (owner instruction: "work until it's done and then merge"): 5 findings. Two confirmed and fixed with tests that were red on 80684e010a: the classifier only checked isProviderAllowed("typesafe"), so an enforced policy listing typesafe with a model_access that excludes jev-latest still sent the prompt (P1, now isModelAllowed, one check instead of two), and min(1) on tier label/description accepted whitespace-only text from a hand-edited config (P2, now trimmed before the length checks). Declined with evidence replies: stale tier provider should fall back rather than fail (P2, scope growth, recorded above under follow-ups), an unawaited tier write racing Test routing (P2, requests are issued in order on one connection, mirrors useModelFallbacks, no reproduction), and Continue after a refusal fallback restarting on the refused tier model (P2, the product-wide resume contract: retrySendOptions is written at send time for every model selection). Threads resolved after the push.
• Codex review round 6 on be00f62d69 (automatic "New commits" pair on the round 5 fix push): 2 P2 code findings plus 1 P1 security finding. The security finding was confirmed and fixed with a test that was red on be00f62d69: an enforced policy base_url for typesafe was ignored, so the classifier always posted the prompt and bearer credential to api.typesafe.ai (now resolved from getForcedBaseUrl("typesafe") when policy is enforced, shared by the Auto send and the Settings preview). The code findings were both confirmed and fixed with tests that were red on be00f62d69: an attachment-only send (empty text plus files, a valid send) still paid for a classifier call with an empty prompt (now falls back to the composer model with a legible reason before the round-trip), and a START_WORKSPACE_CREATION prefill carrying a concrete model left the project-scoped Auto flag on, so the creation send treated the explicit model as a mere fallback (the prefill now clears Auto, the same explicit-pick rule setWorkspaceModelWithOrigin applies). Threads resolved after the push.
• Codex review round 7 on a16db9163a (automatic "New commits" pair on the round 6 fix push; security review completed with no new findings): 1 P2, declined with evidence: an optimistic experiment toggle racing an immediate Auto send is the pre-existing ExperimentsContext fire-and-forget pattern shared by every experiment gate, the override request precedes any later send on the same connection, and carrying the experiment value on the send would be a new experiments-plumbing contract outside this diff. Thread resolved.
• Codex review round 8 on c476f0ff8c (automatic "New commits" pair on the round 7 fix push): both passes completed with zero findings. The board still listed the fixed security advisory without Codex's Resolved marker (the pass started 2 s before the thread was resolved), so the Codex Comments gate stayed red; a manual @codex security review (one pass, explicit "No security issues were found" comment) did not add the marker either.
• Codex review round 9 on the same head c476f0ff8c (manual @codex review, requested only to obtain the Resolved marker): the security pass added the marker but raised a new Medium advisory, and the code pass returned 4 new findings on a head it had passed clean 35 minutes earlier. Confirmed and fixed with tests that were red on c476f0ff8c: TYPESAFE_API_KEY/JEV_API_KEY missing from providerSecretEnvVarNames so repo-automation-off subprocesses could inherit the classifier credential (P1), classifier context read through getLastMessages across the latest durable boundary so a /clear or compaction did not hide earlier prompts from Jev (P1, now getHistoryFromLatestBoundary), a persisted routing record with a non-string model throwing on Continue (P2, now read as absent), and unbounded classifier probabilities persisted per row (security Medium, now filtered to the configured tiers). Declined with evidence: byte-capping the classifier response body (no backend fetch bounds bodies, endpoint trusted by configuration) and per-tier thinking choices derived from the mapped model (documented per-model floor, shared selector behavior). Threads resolved before the push.
• Codex review round 10 on 045a0754f9 (automatic "New commits" pair on the round 9 fix push; security pass added no advisory but also did not mark the persisted-probabilities advisory resolved despite its thread being resolved before the pass began): 2 P2 code findings, both confirmed and fixed with tests that were red on 045a0754f9: the routing-record lookup picked the last user row by role, so a completed subagent report card appended after an interrupted routed turn shadowed the record (now reuses findLastRetryUserMessage, the resume path's own predicate), and the tier-model attachment gate covered PDFs only, so an image in the send or the active context could be routed to a non-vision tier model (now falls back with a legible reason; the pre-existing send-time PDF check is unchanged). Threads resolved before the push.
• Review budget (cap: six passes per PR, code and security counted separately): 26 of 6 consumed, namely the in-house reviewer pass, Codex code review x12, Codex security review x13. The round 5 pair, the automatic "New commits" pairs on the round 5, 6, 7, 9, and 10 fix pushes, the manual security-only pass, and the manual round 9 pair were authorized by the owner on 2026-09-21 ("work until it's done and then merge"). Convergence note: round 8 passed c476f0ff8c clean and round 9 found 5 issues on the identical head. Before that: 9 of 6, namely the in-house reviewer pass, Codex code review x4, Codex security review x4. Rounds 1 and 2 were requested; round 3 ran automatically on the merge push; round 4 ran because the PR was marked ready during the merge loop, which was a cap violation and has been corrected by a supervisor stop: no further Codex, security, or advisory review may be initiated for this PR. The PR was converted back to draft before pushing acac7524ca, 16c5679970, and 80684e010a so the ready-PR auto-review could not fire (verified after each push: no new board row, no reaction). That block was lifted by the owner on 2026-09-21 (marking the PR ready fired round 5; each fix push fired a "New commits" pair).

---

Generated with xum • Model: anthropic:claude-fable-5-1 • Thinking: xhigh • Cost: $567.86

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-22T03:36:57.712444Z	c44ac4d	Manual request
🔒 Security Review	✅ Completed 2026-09-22T03:37:47.625830Z	c44ac4d	Manual request

Security findings

Advisory findings (2)

• 🟡 Honor the policy-forced TypeSafe base URL · High · Resolved
• 🟡 Bound classifier responses before buffering them · Medium · Resolved

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[ibetitsmike]

@codex review

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: ec6087b4f6

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ec6087b4f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ibetitsmike]

@codex review

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: a0b41b28de

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a0b41b28de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7791a46625

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 64c8167c98

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 80684e010a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: be00f62d69

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Here are some automated security review suggestions for this pull request.

Reviewed commit: be00f62d69

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[ibetitsmike]

@codex review

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: b0c7d9d131

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0c7d9d131

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ibetitsmike]

@codex review

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 31491d194b

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31491d194b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ibetitsmike]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: c44ac4db2c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: c44ac4db2c

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[ThomasK33]

FYI @ibetitsmike — while a stacked PR (#4321) was briefly compared against a pre-#4307 base after a rebase, Codex reviewed this PR's code as part of it and left five findings that are about the auto-model-routing change, not the stack: #4321 (comment) (non-array tiers dropping a valid evaluationModel on load; context.options not updated after the tier-model fallback; startup recovery losing the Auto record; evaluator construction failures surfacing as defects in classify(); optimistic routing edits racing sends). Not verified by me — surfacing them so they aren't lost when the comment is hidden on #4321.