Conversation
…xperimental_evaluate Layer L1 of the workflow evaluate() plan: a pure dependency prerequisite, no product code changes. - Pin exactly (no caret) ai 7.0.107, @ai-sdk/openai 4.0.71, @ai-sdk/anthropic 4.0.58, @ai-sdk/google 4.0.76 — the first releases shipping experimental_evaluate / Experimental_Evaluation* / .evaluationModel(). Experimental APIs may change in patch releases, hence exact pins. - Declare @ai-sdk/provider 4.0.17 and @ai-sdk/provider-utils 5.0.45 as direct dependencies. Xum imports spec types from both packages in ~40 files but never declared them, so they resolved to whichever copy Bun hoisted. After the upgrade the hoisted root copy stayed at 4.0.3 (kept by bedrock/deepseek/openai-compatible) while ai/openai/anthropic/google nested 4.0.17, and 4.0.17 changed JSONValue to accept Readonly<JSONObject> / readonly arrays; tsgo reported 23 structural mismatches at the wrapLanguageModel / LanguageModelV4 boundaries. Declaring the packages at ai's exact versions hoists 4.0.17/5.0.45 to the root so Xum's types match the runtime it drives. Zero source changes were needed. - Older providers keep their own nested provider copies (bedrock/deepseek/ openai-compatible/moonshotai 4.0.3, xai 4.0.7, zai 4.0.8). Xum's SDK error checks use marker-based isInstance (no instanceof on SDK error classes), so duplicate copies are not a correctness hazard. xai stays on 4.x with its existing patch.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…hange The fixed-output offlineCache derivation hashes bun.lock inputs; CI's Flake Hash Check reported the new hash after the dependency pins.
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
Superseded by #4307 (merged to The rest of the stack was rebased directly onto |
This branch has not been deployed
1 participant
Summary
Layer L1 (prerequisite) of the workflow
evaluate()plan: pinai7.0.107,@ai-sdk/openai4.0.71,@ai-sdk/anthropic4.0.58,@ai-sdk/google4.0.76 exactly, and declare@ai-sdk/provider4.0.17 /@ai-sdk/provider-utils5.0.45 as direct dependencies. No product code changes. Stack: L1 (this) → L2 evaluation resolver +EvaluationService→ L3 workflowevaluate()→ L4 Settings/timeline → L5 TypeSafe provider → L6 screening example.Background
The AI SDK's
experimental_evaluate(evaluation models,choice|score|booleanquestions) first shipped inai@7.0.106with@ai-sdk/openai@4.0.70,@ai-sdk/anthropic@4.0.57,@ai-sdk/google@4.0.75. Xum pinnedai7.0.19, so later layers cannot import it. Experimental APIs may change in patch releases, hence exact pins (no caret) for the four upgraded packages.Implementation
@ai-sdk/xaistays on 4.x with its existing patch (5.xremovedprovider.chat, whichproviderModelFactory.tsuses for non-frontier Grok); bedrock, openai-compatible, deepseek, moonshotai, zai are untouched. Nooverrides/resolutions.@ai-sdk/provider/provider-utilsbecame direct deps: Xum imports spec types from both packages in ~40 files without declaring them, so they resolved to whichever copy Bun hoisted. After the upgrade the hoisted root copy stayed at 4.0.3 (kept by bedrock/deepseek/openai-compatible) whileai/openai/anthropic/google nested 4.0.17, whoseJSONValueacceptsReadonly<JSONObject>/ readonly arrays → 23 tsgo structural mismatches. Declaring the versionsaiuses hoists 4.0.17/5.0.45 to the root so Xum's types match the runtime it drives; zero source changes were needed.bun.lock(recorded per plan): root@ai-sdk/provider4.0.17 /provider-utils5.0.45; nested copies remain for amazon-bedrock, deepseek, openai-compatible (provider 4.0.3 / utils 5.0.7), moonshotai (4.0.3 / 5.0.11), xai (4.0.7 / 5.0.27), zai (4.0.8 / 5.0.33), andollama-ai-provider-v2(provider 3.0.10 / utils 4.0.26). Xum's SDK error checks use marker-basedisInstance(19 sites, 0instanceofon SDK error classes), so duplicate copies are not a correctness hazard; this was also probed at runtime (see Validation).flake.nix: the offline-cache fixed-output hash is derived frombun.lock, so it is updated alongside the lockfile (value reported by CI's Flake Hash Check; no local nix here).ai@7.0.107onlyexperimental_evaluateandExperimental_EvaluationUnsupportedQuestionTypeErrorare runtime values; the otherExperimental_Evaluation*names are type-only exports.Validation
bun install --frozen-lockfileon the committed state; export probe:experimental_evaluateandExperimental_EvaluationUnsupportedQuestionTypeErrorexported fromai,Experimental_EvaluationMockModelV4fromai/test,.evaluationModelis a function oncreateOpenAI/createAnthropic/createGoogleGenerativeAIinstances.make static-checkgreen. Mock-routed unit suites (streamManager*,providerModelFactory,tools,gatewayStreamNormalization,common/utils/ai) 2285 pass / 0 fail, identical pass set tomain.tests/ipcfiles usingmockAiRouter/MockLanguageModelV3/enableMockMode, excluding anything callingvalidateApiKeys()or reading provider keys):acp.toolRouter,historyFingerprintParity,streaming/queuedMessages.completing,streaming/queuedMessages.starting,streaming/stopAdmission.mock,tasks/persistentSubagentCompaction,config/mcpPrompts,streaming/compactionReplacement.mock— all pass. Real-provider suites (validateApiKeys(),tests/ipc/providers) were not run (no spend authorization).make test-unit: the same 6 deterministic failures + 1 error (taskGitPatchEngine×2,gitNoHooksEnv×3,backup/gitRepo×1, SIGTERM) reproduce onmainin isolated reruns — pre-existing, unrelated to the SDK; the remaining differences were load flakes that pass isolated.9bc3b23a6— the head before the flake-hash-only follow-up commit, which changes no runtime code; labeled prerequisite compatibility check, endorsed PASS by an independent runner): fresh clone + Bun 1.3.5 frozen install, static-check, the suites above; headless dev-server smoke against loopback fixtures with no real provider env — OpenAI Responses wire (function_call_output) and openai-compatible Chat Completions wire (role:"tool") both streamed a tool call + final answer; a fixture 401 surfaced through the nested openai-compatibleprovider-utilscopy intoai@7.0.107asAPICallError401 (isRetryable:false) with a visible provider error and a successful recovery send; runtime probe of all 7 provider/provider-utils copies:isInstancemarkers resolve,instanceofdoes not (as expected). All 101 fixture requests usedmodel=fixture-model; zero external hosts.Risks
Low. Dependency-only change; chat/tool/streaming paths are exercised by the mock-routed suites and the remote smoke. The main residual risk is provider-specific wire behavior for Anthropic/Google/Bedrock/xAI/zai/DeepSeek/Ollama, which only real-provider suites cover (not run here); those providers keep their own nested SDK copies and were not upgraded.
Generated with
xum• Model:coder:anthropic/claude-fable-5-1• Thinking:xhigh• Cost:$65.95