🤖 feat(workflows): evaluate() primitive wired through runner, service and CLI - #4329
Merged
ThomasK33 merged 9 commits intoSep 23, 2026
Conversation
ThomasK33
added this pull request to stack #4326
September 22, 2026 07:06
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 04223b3064
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 67fdbe3dd1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ThomasK33
added a commit
that referenced
this pull request
Sep 22, 2026
…tries past the attempt cap Codex round 2 on #4329: - A resumed or retried evaluate step that fails before admission (revoked key, changed endpoint, elapsed deadline) now writes that failure onto its persisted record (admission kept, so the budget is untouched). Previously only admission-mismatch did, leaving a `started`/stale record that neither resume nor checkpoint retry could recover once credentials returned. - A failed evaluation step whose admission already sits at EVALUATION_MAX_ATTEMPTS is no longer eligible for checkpoint retry; the shared predicate (UI, task_await, workflow_resume, service) reports a distinct reason instead of offering a retry that can only record attempts-exhausted again. Tests: runner (unauthorized on resume → failed record with attempt 1 → retry completes at attempt 2) and service (three provider failures → retry refused with the exhausted reason, no fourth dispatch), both red→green. _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$441.23`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=441.23 -->
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2cc851fcd5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ThomasK33
added a commit
that referenced
this pull request
Sep 22, 2026
…dmission retry from checkpoint Codex round 3 on #4329: a first attempt that fails before admission (no configured model, missing key, unsupported route, …) deliberately writes no step record, so the shared checkpoint-retry gate found nothing to match and every user-facing retry path refused the run even after the configuration was fixed. The runner's exact error text is the run's only trace of that failure, so its format now lives in common/types/evaluation.ts next to the error name (`formatWorkflowEvaluationStepError`) with a matching anchored recogniser, and the gate admits a run whose latest error is that text (bare or sandbox-prefixed) when no failed evaluation record matched. A matched record still takes precedence, so the attempt cap keeps applying, and an author-thrown error that merely embeds the text still does not qualify. Service test: first resolution finds no model → run failed with zero step records → gate admits → checkpoint retry completes at attempt 1 (red→green); the author-caught case stays refused. _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$441.23`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=441.23 -->
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: faa77ed2a3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ThomasK33
added a commit
that referenced
this pull request
Sep 22, 2026
…ilures; verify cached receipts on replay Codex round 4 on #4329: - The round-3 recogniser admitted every formatted evaluation error as a retryable pre-admission failure, including deterministic ones (invalid-spec/state, size bounds) and `admission-missing` (a failed record without an admission), so the UI/service could advertise a retry that only reproduces the failure. The gate now parses the reason/code pair (`parseWorkflowEvaluationStepError`) and admits only failures that can change between attempts without touching the source: no-model, unsupported-provider/route, unknown-model, runtime-unavailable, unauthorized, deadline. - `replayCompletedStep` accepted a cached result on answer shape alone. It now also requires the result's state receipt and (when present) the admission's state/questions receipts to match the input being evaluated, so a record corrupted or misassociated under an unchanged key fails closed instead of returning answers produced for another state. Tests: runner fixtures carry real receipts; new receipt-mismatch replay case (result and admission variants); admission-missing and invalid-state now assert no retry is offered. All red→green. _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$466.44`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=466.44 -->
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 76d50f5ee4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ThomasK33
added a commit
that referenced
this pull request
Sep 22, 2026
… admission bound to its input Codex round 5 on #4329 consolidated the receipt checks added in round 4 into one trust decision at lookup: `parseTrustedAdmission` accepts an existing record's admission only when it parses and its state/questions receipts and request-shaping provider options match the input being evaluated now. A completed record without such an admission fails the run untouched (`cached result invalid`) instead of replaying on answer shape and state receipt alone; a started/failed record without one takes the existing admission-missing path before any selection or attempt advancement, so a corrupted or misassociated admission can no longer dispatch a billable request. Replay receives the trusted admission (no second parse, no fallback attempt) and keeps only the result's own state receipt check. The pinned model selection is deliberately not compared with the spec or Settings — carrying it across attempts is the admission's purpose. Test: table-driven boundary (missing admission, state receipt, questions receipt, provider options × completed and started records) asserting no resolution/dispatch, completed-record immutability, admission-missing on resume, and no retry offered; red against the previous head. _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$469.98`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=469.98 -->
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b4993cdf0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Member
Author
Paused for non-convergence (review round 6 of 6)Head Ledger (all fixed rounds are red→green with
Why it does not converge. The gate ( Proposal (maintainer decision): have the runner persist an explicit failure classification at throw time — e.g. a small CI on Remote UAT: round 1 ( Generated with |
Member
Author
|
Resumed under an explicitly bounded exception: one assessment of the reduced head Scope reduced as proposed: the error-text inference for checkpoint retry (rounds 3–4) is removed; the record-based core stays (admission failures on the record, attempt cap, trusted admission at lookup, receipt-verified replay), and thrown resolvers now take the typed path at both admitted resolution points. First-attempt pre-admission recovery and nested-workflow propagation are stated limitations with follow-ups (see body). The evaluate() suites moved to CI's isolated-process list after two Bun segfaults on the previous head. If this assessment finds another systemic issue in the same area, the PR stays paused; no further loop is intended. Generated with |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bbfa4f2ce0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Member
Author
|
Bounded assessment of The assessment returned four findings, three of them in the same admission-binding / recoverability area as rounds 2–6 ( CI on this head: Generated with |
Member
Author
|
Remote UAT round 2 (exact head Loopback-fixture run on a fresh dogfood workspace (chat
This does not change the hold on this PR (4 acked threads, see the pause comment); it records the exact-head evidence for the reduced contract. |
ThomasK33
added a commit
that referenced
this pull request
Sep 22, 2026
…recording and resolution bounds Codex round-8 findings on #4329, all in workflowEvaluationStep.ts: - An explicit per-call model is part of the admission trust check: a record whose admission pinned another model is untrusted (replay refused, resume fails closed as admission-missing). Selections made from the mutable Settings/CLI default stay pinned as before. - A cached result must name the admitted model; otherwise replay fails closed like a state-receipt mismatch. - The evaluation{failed} event append after the durable failed record is guarded (logged with a fixed code) so a transient reject cannot replace the typed error and strand a checkpoint-retryable run. - Model resolution is bounded like dispatch: Stop interrupts a stalled resolver, the attempt deadline turns it into a deadline failure. Five runner tests, each red against the previous implementation. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$565.98`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=565.98 -->
Member
Author
|
Resumed to bring the stack to green CI. The four round-8 threads left open under the hold are fixed in Codex re-reviews automatically on this push. Any further non-blocking finding in the admission area will be tracked as a follow-up rather than patched here, to keep this PR the change it started as. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 43f6d20544
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
github-merge-queue
Bot
removed this pull request from the merge queue because a pull request earlier in the stack was removed
Sep 23, 2026
… and CLI Adds the sandbox-facing `evaluate(state, options)` step and its host lifecycle (validate → hash → lookup/replay → select → admit → recheck → dispatch → commit → account) in `workflowEvaluationStep.ts`, registered by `WorkflowRunner` as `__workflowEvaluate`. `WorkflowService` takes a shared `evaluationAdapter`; the ORPC path, tool-started runs (turnRequestBuilder bindings) and the headless CLI construct `WorkflowEvaluationAdapter`, and `xum workflow run` gains `--evaluation-model` (per-call model > flag > Settings default, read from the real config). Every step record of an evaluate step carries its admission so resumes and checkpoint retries reuse the pinned selection and fail closed on a changed endpoint fingerprint. Tests: runner lifecycle matrix with a fake adapter (replay, resume, retry, deadlines, abort, lease loss, revocation, post-commit irreversibility, sentinel discipline), CLI precedence, and a TEST_INTEGRATION loopback wire test through the real OpenAI/Anthropic/Google adapters. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$326.41`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=326.41 -->
Two evaluate() calls with the same id/state/questions/model but different provider options previously shared a replay key, so the second could return the first's cached result without dispatching under its own request settings. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$326.41`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=326.41 -->
…valuate() step Remote UAT (C2) showed a run whose evaluate() step failed could neither be retried from checkpoint nor resumed, although the lifecycle re-attempts a failed record with its persisted admission. Checkpoint-retry eligibility now also admits runs whose latest error is exactly the failed evaluation step's error (bare or sandbox-prefixed with the error name); author-thrown errors that embed it stay non-retryable. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$417.19`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=417.19 -->
…tries past the attempt cap Codex round 2 on #4329: - A resumed or retried evaluate step that fails before admission (revoked key, changed endpoint, elapsed deadline) now writes that failure onto its persisted record (admission kept, so the budget is untouched). Previously only admission-mismatch did, leaving a `started`/stale record that neither resume nor checkpoint retry could recover once credentials returned. - A failed evaluation step whose admission already sits at EVALUATION_MAX_ATTEMPTS is no longer eligible for checkpoint retry; the shared predicate (UI, task_await, workflow_resume, service) reports a distinct reason instead of offering a retry that can only record attempts-exhausted again. Tests: runner (unauthorized on resume → failed record with attempt 1 → retry completes at attempt 2) and service (three provider failures → retry refused with the exhausted reason, no fourth dispatch), both red→green. _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$441.23`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=441.23 -->
…dmission retry from checkpoint Codex round 3 on #4329: a first attempt that fails before admission (no configured model, missing key, unsupported route, …) deliberately writes no step record, so the shared checkpoint-retry gate found nothing to match and every user-facing retry path refused the run even after the configuration was fixed. The runner's exact error text is the run's only trace of that failure, so its format now lives in common/types/evaluation.ts next to the error name (`formatWorkflowEvaluationStepError`) with a matching anchored recogniser, and the gate admits a run whose latest error is that text (bare or sandbox-prefixed) when no failed evaluation record matched. A matched record still takes precedence, so the attempt cap keeps applying, and an author-thrown error that merely embeds the text still does not qualify. Service test: first resolution finds no model → run failed with zero step records → gate admits → checkpoint retry completes at attempt 1 (red→green); the author-caught case stays refused. _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$441.23`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=441.23 -->
…ilures; verify cached receipts on replay Codex round 4 on #4329: - The round-3 recogniser admitted every formatted evaluation error as a retryable pre-admission failure, including deterministic ones (invalid-spec/state, size bounds) and `admission-missing` (a failed record without an admission), so the UI/service could advertise a retry that only reproduces the failure. The gate now parses the reason/code pair (`parseWorkflowEvaluationStepError`) and admits only failures that can change between attempts without touching the source: no-model, unsupported-provider/route, unknown-model, runtime-unavailable, unauthorized, deadline. - `replayCompletedStep` accepted a cached result on answer shape alone. It now also requires the result's state receipt and (when present) the admission's state/questions receipts to match the input being evaluated, so a record corrupted or misassociated under an unchanged key fails closed instead of returning answers produced for another state. Tests: runner fixtures carry real receipts; new receipt-mismatch replay case (result and admission variants); admission-missing and invalid-state now assert no retry is offered. All red→green. _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$466.44`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=466.44 -->
… admission bound to its input Codex round 5 on #4329 consolidated the receipt checks added in round 4 into one trust decision at lookup: `parseTrustedAdmission` accepts an existing record's admission only when it parses and its state/questions receipts and request-shaping provider options match the input being evaluated now. A completed record without such an admission fails the run untouched (`cached result invalid`) instead of replaying on answer shape and state receipt alone; a started/failed record without one takes the existing admission-missing path before any selection or attempt advancement, so a corrupted or misassociated admission can no longer dispatch a billable request. Replay receives the trusted admission (no second parse, no fallback attempt) and keeps only the result's own state receipt check. The pinned model selection is deliberately not compared with the spec or Settings — carrying it across attempts is the admission's purpose. Test: table-driven boundary (missing admission, state receipt, questions receipt, provider options × completed and started records) asserting no resolution/dispatch, completed-record immutability, admission-missing on resume, and no retry offered; red against the previous head. _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$469.98`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=469.98 -->
…recoverability Scope reduction after six review rounds converged on the same area. The checkpoint-retry gate no longer infers recoverability from run error text (the round-3/4 parser and reason/code allowlist are removed); it trusts only what the runner recorded: a failed evaluate step whose admission is bound to the input (attempt cap via that admission), matched against the run's error. Consequence, stated as a known limitation: a FIRST attempt that fails before admission (no configured model, revoked key) writes no record on purpose and is therefore not checkpoint-retryable in this layer — the run is started again. Persisting a runner-decided failure classification for that case is the tracked follow-up. The retained contract is completed for thrown resolvers: a resolveSelection that throws (lazy provider import, model construction) at either admitted resolution point — resume selection or the post-admission recheck — now takes the typed path with a fixed identity (provider-failure/unknown), so the admitted record receives its failure and the run stays retryable instead of stranding at `started` with a raw exception as the run error. Only the error name is logged. CI: the evaluate() lifecycle suites create a QuickJS runtime per case; the shared coverage process segfaulted Bun twice on the previous head (exit 132, zero failing assertions), so they join pr.yml's isolated-suite list next to WorkflowRunner.test.ts. Tests: service no-model case now asserts the retry is refused (reduced contract); runner covers thrown resolvers on recheck and on resume (record failed at the admitted attempt, retry offered, no sentinel text). _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$511.72`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=511.72 -->
…recording and resolution bounds Codex round-8 findings on #4329, all in workflowEvaluationStep.ts: - An explicit per-call model is part of the admission trust check: a record whose admission pinned another model is untrusted (replay refused, resume fails closed as admission-missing). Selections made from the mutable Settings/CLI default stay pinned as before. - A cached result must name the admitted model; otherwise replay fails closed like a state-receipt mismatch. - The evaluation{failed} event append after the durable failed record is guarded (logged with a fixed code) so a transient reject cannot replace the typed error and strand a checkpoint-retryable run. - Model resolution is bounded like dispatch: Stop interrupts a stalled resolver, the attempt deadline turns it into a deadline failure. Five runner tests, each red against the previous implementation. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$565.98`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=565.98 -->
ThomasK33
force-pushed
the
thomask33/eval-l3b-workflow-evaluate
branch
from
September 23, 2026 05:52
43f6d20 to
a9cc5e2
Compare
12 tasks
An error occurred while trying to automatically change base from
thomask33/eval-l3a-evaluation-adapter
to
thomask33/eval-l2b-evaluation-service
September 23, 2026 07:57
yermakoffivan
pushed a commit
to yermakoffivan/mux
that referenced
this pull request
Sep 23, 2026
… and tool card (coder#4330) ## Summary Fourth layer of the workflow `evaluate()` primitive (stacked on coder#4329), **reduced to timeline rendering**: `evaluate()` steps in the right-sidebar workflow timeline and the workflow tool card (badge, model/usage metadata, coalesced rows). The Settings evaluation-model card that this PR originally also carried was removed after review round 5 (see Scope below). ## Background Plan layers: L2a coder#4321 → L2b coder#4322 → L3a coder#4324 → L3b coder#4329 (runner + CLI) → **L4 (this PR)** → L5 TypeSafe provider → L6 example + docs. coder#4324 deferred the "coalesce `evaluation` events in the tool card" finding to this layer. ## Implementation - `projectWorkflowRun`: `WorkflowStepView.evaluation` = admission selection + attempt, plus `responseModelId`/`usage` from the committed `EvaluationStepResult` and `cached` from the replay event. `WorkflowTimeline` shows an `evaluation` / `evaluation · cached` badge and a metadata line (model, `↳ responseModelId` only when it differs from the **bare** requested id — providers report `gpt-5` while the admitted string is `openai:gpt-5` —, `attempt n` when > 1, tokens); evaluate steps have no task, so no Open action. - `WorkflowRunToolCall.getWorkflowDisplayRows` coalesces `evaluation` events per `stepId + inputHash` (all attempts of one step) into one row: latest status, `started` title preserved when a terminal event lacks one, merged details. - Storybook: `App/Workflows` → `EvaluationSteps` (cached / failed / running steps). ## Scope Review rounds 2–5 all found asynchronous-ownership defects in the Settings card (route/policy gating, then load race, reconnect load, superseded saves, stale verdict) — see the [pause comment](coder#4330 (comment)). The card, its `config.checkEvaluationModel` ORPC method, TasksSection wiring, story plumbing and IPC test were removed from this PR and preserved on `thomask33/eval-l4c-card` for a cohesive load/save lifecycle design (single owner of the displayed value, per-API-identity loads, one in-flight operation) in its own PR, which carries the card's review history and the TypeSafe card additions from L5. Until then `evaluationDefaults.model` is set via the existing `config.updateEvaluationDefaults` endpoint / `xum workflow run --evaluation-model`. ## Validation - `WorkflowTimeline.test.tsx` (+1): the `↳ responseModelId` line appears only when the response id differs from the requested bare id (same id → no line; dated snapshot id → shown). - `projectWorkflowRun.test.ts` (+1): evaluation facts from admission + result + cached event; running attempt exposes admission only; agent steps stay `undefined`. - `WorkflowRunToolCall.test.tsx` (+1): two attempts + a title-less terminal event collapse into one row labelled with the started title; other steps stay separate. - Storybook plays green for `App.workflows.stories.tsx` (`bun x test-storybook … --verbose`); `make static-check` green; Workflows/Tools suites 170/170. ## Risks - UI-only; no persisted formats change. Timeline/tool-card changes touch only `evaluation` rows/steps; agent, patch and nested-workflow rendering is untouched. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$511.72`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=511.72 -->
yermakoffivan
pushed a commit
to yermakoffivan/mux
that referenced
this pull request
Sep 23, 2026
…#4332) ## Summary Fifth layer of the workflow `evaluate()` primitive (stacked on coder#4330): the evaluation resolver, CLI and wire path learn the reserved evaluation-only `typesafe` credential key that main already ships for auto model routing, so an `evaluate()` step can run on TypeSafe AI with a TypeSafe-only configuration. No new `PROVIDER_DEFINITIONS` entry and no chat-picker exposure. ## Background Plan layers: L2a coder#4321 → L2b coder#4322 → L3a coder#4324 → L3b coder#4329 (runner + CLI) → L4 coder#4330 (timeline rendering) → **L5 (this PR)** → L6 example + docs. `typesafe` is not a `ProviderName`: it is the credential key auto model routing uses for its evaluation model (`DEFAULT_AUTO_MODEL_ROUTING_EVALUATION_MODEL`), which is exactly what `evaluate()` needs. ## Implementation - `EVALUATION_PROVIDERS` gains `typesafe` (typed as `ProviderName | typeof TYPESAFE_PROVIDER_KEY`); `isEvaluationEligibleModelString` accepts `typesafe:<id>`. - `resolveTypeSafeCredentials` (shared with auto model routing, `providerRequirements.ts`) reads the `typesafe` providers entry or `TYPESAFE_API_KEY_ENV_VARS`; `createEvaluationModel` gets a `createTypeSafeAi` branch. The pinned selection's `routeKind` is `direct`, and the route comparison is done as strings because `resolveRoute` returns the key through an unchecked cast. - CLI: the ephemeral run config's providers copy keeps evaluation-only entries (previously only chat `ProviderName`s were copied), so `xum workflow run --evaluation-model typesafe:jev-latest` works with only a TypeSafe key configured. - Offering `typesafe:jev-latest` in the Settings evaluation card moves with the card to its own PR (see coder#4330's scope note). ## Validation - `providerModelFactory.test.ts` (+3): `typesafe:` resolves on the direct route from the providers entry and from the env var; missing key → `unauthorized`; a chat model string is unaffected. - `evaluationModels.test.ts`: eligibility of `typesafe:<id>` and rejection of other non-`ProviderName` prefixes. - `src/cli/workflow.test.ts` (+1): a TypeSafe-only run config reaches `evaluate()` (red → green: before the copy fix the run failed `unsupported-provider`). - `tests/ipc/workflows/evaluateWire.test.ts` (`TEST_INTEGRATION=1`, +2 lanes): real `@ai-sdk/typesafe-ai` adapter against the loopback fixture through ORPC — booleans travel as `noul`, rounded distributions validate, confidence never reaches the result; a 401 is a provider failure and an interrupted hang aborts the request. - `make static-check` green. ## Risks - Additive: only `typesafe:`-prefixed model strings take the new branch; chat providers and routing are untouched. Secrets: the key is read through the same credential resolver auto model routing already uses; nothing new is persisted. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$511.72`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=511.72 -->
yermakoffivan
pushed a commit
to yermakoffivan/mux
that referenced
this pull request
Sep 23, 2026
…ng example (coder#4333) ## Summary Sixth and last layer of the workflow `evaluate()` primitive (stacked on coder#4332): authoring documentation for `evaluate()` in the built-in `workflow-authoring` skill and a runnable GitHub-issue screening example shipped as a skill asset (`skill://workflow-authoring/screen-github-issue.js`), exercised end to end through `WorkflowService` with fake evaluation and agent adapters. ## Background Plan layers: L2a coder#4321 → L2b coder#4322 → L3a coder#4324 → L3b coder#4329 (runner + CLI) → L4 coder#4330 (timeline rendering) → L5 coder#4332 (TypeSafe provider) → **L6 (this PR)**. Docs live inline in the skill Markdown (no free-floating docs), and `builtInSkillContent.generated.ts` is regenerated from it. ## Implementation - `workflow-authoring.md` gains an `evaluate(state, options)` section: signature and accepted `state` types (string / JSON array / JSON object), question/answer shapes and bounds, the canonical-JSON state digest, model precedence (per-call `model` → `--evaluation-model` → persisted `evaluationDefaults.model`; the Settings card for that default is a separate PR), supported direct routes, the explicit replay key (`id` + `state`, `questions`, `model`, `providerOptions`; not `title`/`timeoutMs`), attempt semantics including `retry_from_checkpoint` eligibility for admitted failures, failure vs. interruption (fixed-template error text), the sequential-only rule, and what is *not* guaranteed. `evaluate()` is described as tool-free classification, not a sanitizer or security boundary. - `screen-github-issue.js`: a screening workflow with flat args (`repo`, `issueNumber`, `title`, `body`). One tool-free `evaluate()` call answers `injection`, `kind` and `severity` (impact scale valid for every kind). `not_detected` returns the evaluator's triage answers and starts **no agent** — issue text is never placed in an agent prompt, not even after a false-negative screen (built-in Explore keeps `bash`/`web_fetch`). This is prompt isolation, not read isolation: the labeling agent keeps its tools, and "do not read the issue" stays guidance. `suspected`/`uncertain` hand only `{ repo, issueNumber, label, reasonCode, stateSha256 }` to an exec agent that applies the `needs-human-review` label and reports `{ labeled, detail? }` through a schema; the run fails when the label was not applied. The runnable command passes `--evaluation-model` (for `evaluate()`) and `--model` (for the labeling agent) and creates the `needs-human-review` label once per repository (`--add-label` does not create labels). - `evaluateScreening.test.ts` (7): runs the example via `WorkflowService` with fake evaluation and agent adapters — `not_detected` with a sentinel-bearing body starts no agent task and echoes only triage + digest; `suspected`/`uncertain` route to the labeling agent with identifiers and digest only (never the text) and a `labeled` schema; a labeling agent reporting `labeled: false` fails the run (no result event, fixed-template error without body bytes); an evaluator failure fails the run before any agent step; resuming after an interrupt in the labeling step replays the evaluation without re-dispatching; a malformed `repo` is rejected before evaluating. ## Validation - `evaluateScreening.test.ts` 7/7 (labeling-failure case red against the previous example); `agentSkills` suites green after regeneration; `make static-check` green (includes docs link check). ## Follow-ups (non-blocking review findings deferred at the round budget) Tracked in coder#4363. - Label setup line: create `needs-human-review` only when absent (or pin `--color`) so a re-run of the advertised `gh label create --force` cannot recolor an existing label. - Docs: describe `score` answers as a continuous value in `[0, levels − 1]` (probability-weighted mean when a distribution is returned; an integer only when the provider selects an exact level) and warn against indexing/switching on it. ## Risks - Docs + a skill asset + tests; no runtime code changes. - The example's not_detected branch is intentionally agent-free (data-flow restriction, not host-enforced tool isolation); a host-enforced GitHub label action does not exist and is not claimed. --- _Generated with `xum` • Model: `coder:anthropic/claude-fable-5-1` • Thinking: `xhigh` • Cost: `$537.54`_ <!-- mux-attribution: model=coder:anthropic/claude-fable-5-1 thinking=xhigh costs=537.54 -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Third layer of the workflow
evaluate()primitive (stacked on #4324): the sandbox-facingevaluate(state, options)step, its host-side lifecycle, and the construction sites that give every workflow entry point (ORPC/desktop, tool-started runs, headless CLI) aWorkflowEvaluationAdapter. After this PR a workflow script can callevaluate()end to end against OpenAI, Anthropic or Google.Background
Plan layers: L1 SDK upgrade (superseded by #4307) → L2a #4321 schemas → L2b #4322 resolver +
EvaluationService→ L3a #4324 adapter + schemas + config → L3b (this PR) runner + construction + CLI → L4 Settings UI/timeline → L5 TypeSafe provider → L6 example + docs.Implementation
workflowEvaluationStep.tsowns the lifecycle contract (validate → hash → lookup/replay → select → admit → recheck → dispatch → commit → account).WorkflowRunneronly registers__workflowEvaluateand lends lease-scoped journal wrappers; the sandbox glue__muxEvaluateadds theparallel()/pipeline()collector guard and exportsevaluate.Every step record of an evaluate step carries its
evaluationadmission (selection + fingerprint, timeout, deadline, state/questions digests, provider options). The runner trusts an existing record only through that admission: it must parse and be bound to the current input (parseTrustedAdmission), otherwise a completed record fails the run untouched (cached result invalid) and a started/failed record fails closed asadmission-missing. Resume andretry_from_checkpointreuse the pinned selection (attempt + 1, capped atEVALUATION_MAX_ATTEMPTS) and fail closed on a changed endpoint fingerprint. Once a record exists, every failure — typed selection failure, elapsed deadline, or a thrown resolver (fixed identityprovider-failure/unknown) — is written onto it, so the shared checkpoint-retry gate (getWorkflowCheckpointRetryEligibility) can match the run's error to a failed evaluation record with budget left. First-attempt pre-admission failures (no model, revoked key) write no record on purpose; see Known limitations.A pre-dispatch re-resolution catches credential revocation / endpoint change during preparation; the re-resolved instance is the one dispatched.
Post-commit boundary: once the completed record is durable, a rejected
evaluation{completed}event append or a throwing usage ledger is logged with a fixed code and never turns the step into a failure (the adapter already swallows ledger failures; the lifecycle guards the contract).WorkflowEvaluationPort(Pickof the adapter's three methods) is the runner/service option type so tests fake the adapter without casts.WorkflowServiceOptions.evaluationAdapteris a plain shared instance — the adapter holds no run state.Construction:
resolveWorkflowContext(ORPC) builds the adapter fromevaluationService/sessionUsageServicenow exposed onORPCContext/ServiceContainer; tool-started runs getbindings.evaluationService+bindings.requestAnalyticsIngestfrom the core DI layer;xum workflow rungains--evaluation-model <model>(per-callmodel> flag > Settings default, read from the real config since the ephemeral run config only copies providers/secrets/trust).Audit (plan item 6):
stripWorkflowRunRecordForModelkeeps evaluate records/admissions andevaluationevents out of agent prompts; the thrown failure message is the runerrorforwarded to the parent chat, so it stays template-only. Documented at both boundaries.Trust and recovery hardening (
43f6d2054): an explicit per-callmodelis part of the admission trust check; a cached result must name the admitted model; theevaluation{failed}event append is guarded so a transient rejection cannot replace the typed failure; model resolution is bounded by Stop and the attempt deadline like dispatch.Validation
WorkflowRunner.evaluate.test.ts(35) +WorkflowService.evaluate.test.ts(4): happy path/replay/checkpoint retry, resume with persisted selection vs. changed default, fingerprint mismatch, trusted-admission boundary (missing admission / state receipt / questions receipt / provider options × completed and started records: no resolve, no dispatch, record immutable, no retry offered), result state-receipt mismatch, attempt cap and exhausted-budget refusal, preparation vs. post-admission vs. dispatch deadlines, unauthorized on resume → failed record → retry completes at attempt 2, thrown resolver on recheck/resume → typed failure on the record, runtime abort (started record kept), completion-before-Stop replay, lease loss before dispatch, revocation/endpoint change on the recheck, pre-admission no-record (retry refused by the gate; runner-level retry at attempt 1), author-caught error not retryable, post-commit irreversibility,parallel()guard, sandbox arg validation, missing adapter, and sentinel discipline.04223b306, chataf513830): scenarios A–I PASS except C2 (failed evaluation not retryable from checkpoint) — fixed in67fdbe3dd; the later review rounds hardened that fix. Evidence:.mux-uat/l3/round-1/(fixture request log, receipts, casts). Deferred observations: ORPC wraps refusals asINTERNAL_SERVER_ERRORvia the CLIapiproxy (pre-existing); CLI ephemeral-workspace runs write no headless-usage row (intentional ephemeral root); a v0.29.0 reader dropsevaluationfields (forward-compat, see Downgrade). A short exact-head re-confirm on the final head is in progress.src/cli/workflow.test.ts:--evaluation-modelprecedence via three key-independent failure codes (no-model→unsupported-providerfrom the real-config Settings default →unsupported-routefrom the flag → per-call wins) plus blank-flag rejection.tests/ipc/workflows/evaluateWire.test.ts(TEST_INTEGRATION=1 bun x jest …, 6): real@ai-sdk/openai|anthropic|googleadapters against a loopback fixture through ORPCworkflows.start— one non-streaming, tool-free request per attempt, mapped answers, usage sidecar row (workflow_evaluation) written only after the completed record exists, 401 → one request + no usage row, invalid option →invalid-output, hang +workflows.interruptcloses the connection and leaves the attempt resumable.make static-checkgreen on every head. CI: the evaluate suites create a QuickJS runtime per case and segfaulted Bun twice in the shared coverage process (exit 132, zero failing assertions); they now run inpr.yml's isolated-suite list next toWorkflowRunner.test.ts.Risks
ORPCContext.evaluationServiceis a new required field (test fixtures constructing partial contexts cast already).evaluate()become unreadable on an older build.Known limitations (follow-ups)
Tracked in #4363.
Review rounds 2–6 all concerned how a failed
evaluate()step becomes checkpoint-retryable; the contract was reduced to what the runner records (see the pause and resume comments).evaluate()step to the parent's nested-workflow step; needs the nested-workflow resume design.evaluation{started}progress-event append after the durable started record like the other appends, so a transient events-file rejection cannot strand an admittedstartedrecord behind a failed run; and re-clamp a persisted admission'stimeoutMson resume/retry (or reject an out-of-range value as invalid persisted state) so a corrupted record cannot extend an attempt beyond the advertised bounds.Generated with
xum• Model:coder:anthropic/claude-fable-5-1• Thinking:xhigh• Cost:$326.41