🤖 fix: recover once from rejected OpenAI reasoning replay

[ThomasK33]

Summary

Recover once when OpenAI Responses rejects replayed reasoning. Strip only the request's OpenAI reasoning parts, preserve visible text and tool history, and retry inside the existing stream. If the rejection repeats—or repair would be unsafe—classify it as terminal so automatic retries and delegated tasks cannot loop forever.

Second layer of #4335. The first layer avoids stale server-side references. This layer handles rejected encrypted blobs and stale reasoning IDs in intra-turn SDK steps.

Resumed for merge preparation: both CI fixture repairs are now published. Local static checks and 1,526 targeted tests pass; full CI and final-head reviews remain pending. The maintainer requested continuation and merge, so a bounded final review cycle will proceed with the prior six assessments retained in the count. Remote UAT remains halted pending human adjudication of the audit event; merge is still gated on accepted final-head UAT evidence.

Implementation

1. Match resolved OpenAI Responses models and narrowly recognized HTTP 400/404 reasoning-replay errors. Also recognize SDK stream-error 500s: StreamProviderError, or APICallError with SSE content type and an error/response.failed frame. Ordinary HTTP 500s retain their policy.
2. Inspect only the final error when the SDK exhausts its internal retry budget. An earlier reasoning rejection must not taint a later unrelated error.
3. Allow one in-stream repair per attempt, before current-step output. Reuse the prepared transcript even at step 0, preserve completed steps and cumulative usage, and do not re-execute completed tools.
4. Use reasoning_rejected for a final matching failure. Shared retry eligibility, delegated workspace turns, and running tasks treat it as terminal. Manual continuation gets a fresh attempt.

No history rewrite, sticky flag, migration, global retry cap, or UI change. Read the diff in this order: reasoningProviderOptions.ts → streamManager.ts → terminal-error consumers → tests. Most of the diff is regression coverage; repair and terminal handling remain one cohesive guarantee.

Validation

Current head: b398e7834ad9a0ec7eabec51440717d2aec3e84a.

• Pinned Bun 1.3.5: make static-check and 1,526 targeted tests passed (776 recovery/notification tests plus the full 750-test TaskService suite). Red-first receipts cover repair and terminal classification. The stream-error regression uses the real OpenAI SDK with a local HTTP-200 SSE fixture, including SDK retry exhaustion; it is not a live WebSocket-network test.
• Layered component tests cover both rejection forms, a second rejection, outer retry abandonment, task/turn settlement, persisted-error eligibility, fresh manual attempts, safety guards, nonmutation, prior-step preservation, and ordinary 503/401/429 policy. These are not a full Electron/socket end-to-end proof.
• CI fixture repairs supply required stream request state and keep cancellable metadata mocks read-only. Two strengthened no-write regressions failed before the mock fix and pass afterward; config snapshots now retain project Map entries and abandoned reads settle before teardown. The prior CI lockfile error is consistent with this migration/teardown race; the new full-process CI run remains the acceptance gate.
• The initial clean-context readiness advisory found no code blockers. Follow-up fixes require final exact-head review; the overall delivery remains blocked by remote-UAT qualification.

Historical real-provider UAT and media — not evidence for the current head

Round 3 passed on predecessor eae709006 through the Coder AI gateway. A persisted blob was changed to blob-broken. The provider rejected it with HTTP 400 invalid_encrypted_content; one repair log was emitted; the captured successful request contained no reasoning or item references and retained prior text. Existing history rows remained byte-identical. A second user turn repeated successful repair. Screenshots and independently decoded video frames showed no error/retry barrier. Four isolation audits found no shared-deployment mutations.

Round 4 tested predecessor 006ce021e, including two prepared-first-step unit tests and two successful real-provider repair turns. Its endorsed verdict is BLOCKED, not PASS: an API-key deletion under the test user's identity appeared in the audit window and remains unattributed. The serving process also retained unused Anthropic provider variables; this is recorded as a same-environment run, not clean-provider evidence. The corrective run was cancelled without being sent. Owned server/browser processes were cleaned up; the shared workspace was left untouched.

Evidence limits: failed pre-stream requests have no raw HTTP body in existing devtools. Marker transmission is corroborated by recorded SDK input and the provider's truncated marker echo, not by a captured rejected wire body. Exact network-request count is inferred from step records plus repair logs. The second-rejection terminal path is automated-test coverage only. This was AI-gateway-backed traffic, not direct api.openai.com.

Independent UAT chat. Local evidence: .mux-uat/round-3/ and .mux-uat/round-4/; validation pointers: .mux-uat/recovery-delivery/phase2/review-fix-2/.

r3-uat.webm

Risks and follow-up

The repaired request loses prior reasoning context and can lose prefix-cache hits; visible history is retained. Corrupt persisted reasoning is intentionally not rewritten, so later turns may incur rejected requests until compaction or new context. One in-stream repair does not disable the SDK's own retry budget.

Downgrade limitation: terminal reasoning-rejection handling is guaranteed only by versions containing this change. The parent version's history/abandon-marker readers accept the transient metadata, but its retry policy does not recognize reasoning_rejected and may resume its existing retry loop. This PR does not backport recovery into older binaries or add a persistence compatibility alias.

Failed-request observability is a pre-existing, non-blocking limitation tracked in #4343, with redaction requirements and a follow-up trigger. No logging expansion is included here. Composer model selection on reload was also observed outside this diff; defect versus intended behavior was not established.

---

📋 Implementation Plan

Auto-heal the Item with id 'rs_…' not found retry loop

Result

Yes — there is a small, automatic fix. The error is deterministic, so the retry loop can never succeed. The elegant fix is to stop sending the server-side reference at all: replay OpenAI reasoning as its self-contained encrypted_content blob (which Xum already persists) instead of item_reference: rs_…. That is a ~6-line, request-only change in the existing replay seam. It also repairs already-stuck workspaces on their next auto-retry, because every retry rebuilds the request from chat.jsonl through that seam — no manual /compact needed.

A second, bounded layer (one-shot in-stream repair when OpenAI rejects a reasoning replay for any reason) is proposed as a stacked follow-up so a related cross-org failure cannot turn into the same loop.

What is happening (verified)

1. Xum persists providerOptions.openai.{itemId: "rs_…", reasoningEncryptedContent} on reasoning parts only (src/node/services/streamManager.ts:4164-4271). Text and tool-call parts never carry an itemId, so this error class is confined to reasoning replay.
2. On every send, attachReasoningReplayMetadata (src/node/utils/messages/reasoningProviderOptions.ts:130-176) bridges that metadata into providerMetadata. It drops a bare itemId (interrupted stream) but keeps itemId whenever encrypted content is present.
3. store is not set for OpenAI, so @ai-sdk/openai defaults store: true and — when an itemId is present — emits { type: "item_reference", id: "rs_…" } and ignores the encrypted content (node_modules/@ai-sdk/openai/dist/index.js:5676-5718). The request therefore depends on OpenAI's server-side item store.
4. The lookup fails whenever that store does not hold the item: route/credential switch (Coder AI gateway ↔ direct OPENAI_API_KEY; a Codex‑OAuth store:false turn followed by a direct turn), multi-instance gateways, or eviction. OpenAI returns HTTP 400 Item with id 'rs_…' not found.
5. StreamManager.categorizeError maps any other 400 to "api" (streamManager.ts:5406); "api" is not in NON_RETRYABLE_STREAM_ERRORS (src/common/utils/messages/retryEligibility.ts:49-60); RetryManager has no attempt cap (retryManager.ts:107-133, retryState.ts:53-67). Each retry re-reads history and rebuilds the identical request → infinite loop (attempt 11…). Sub-agent tasks loop the same way (taskService.ts:903-909 only terminates on refusal/auth/quota/model/runtime errors).
6. /compact fixed it only because the summary row carries no provider metadata and all later turns slice from the compaction boundary (compactionHandler.ts:986-1034, compactionBoundary.ts:157-176) — i.e. it threw away the context to get rid of one stale id.

Why encrypted replay is safe and already supported

• @ai-sdk/openai CHANGELOG (4.0.0-beta.3): reasoning parts without itemId are emitted as { type: "reasoning", encrypted_content, summary }; "The OpenAI Responses API accepts reasoning items without an id when encrypted_content is supplied". The converter path is index.js:5719-5734 and emits no id field.
• Xum already requests include: ["reasoning.encrypted_content"] for every Responses reasoning model (src/common/utils/ai/providerOptions.ts:588-593), so the blob is present on every completed reasoning part.
• Codex OAuth (store:false, item_reference stripped in providerModelFactory.ts:988-993) and Grok (store:false default, 🤖 feat: default Grok Responses to store=false for ZDR parity #3807) already run every turn on inline encrypted reasoning.
• previous_response_id is never used (providerOptions.ts:485-488), so the existing retryStreamWithoutPreviousResponseId one-shot repair (streamManager.ts:5223-5314) never fires for this path.
• Encrypted blobs are org-bound (OpenAI data-controls docs; LiteLLM multi-region incident report, Feb 2026). Same-org route switches (gateway ↔ direct on the same org, gateway instance flapping, eviction) are fully healed. A cross-org switch would instead produce invalid_encrypted_content — handled by Phase 2.

Phase 1 — Make OpenAI reasoning replay self-contained (recommended; net ≈ +6 LoC product)

File: src/node/utils/messages/reasoningProviderOptions.ts, inside attachReasoningReplayMetadata right after the existing bare-itemId drop loop (lines 158-163).

// OpenAI Responses: never replay by server-side reference. With store=true the
// SDK turns itemId into `item_reference` and ignores the encrypted blob; that
// reference is unresolvable after a route/credential change (gateway<->direct,
// Codex store=false turns) and OpenAI answers 400 "Item with id 'rs_…' not
// found" forever. Encrypted content is self-contained, so send only that.
if (replayMetadata.openai?.itemId != null) {
  const { itemId: _omit, ...rest } = replayMetadata.openai;
  replayMetadata.openai = rest;
}

Notes for the implementer:

• Scope to openai only. Leave xai untouched (its converter behaviour without ids is not established; Grok already runs store:false).
• Request-only: persisted chat.jsonl keeps itemId (debuggability, downgrade safety). No migration.
• Update the stale wording in the module comments that describe itemId as required for replay (lines ~8, ~82, ~151-157) so the "why" stays accurate.
• No store default change, no retry-policy change, no UI change.

Effect on existing stuck workspaces: the next scheduled auto-retry (or startup auto-retry / user "continue") re-runs streamWithHistory → prepareMessagesForProvider → this seam → request succeeds. Nothing else needed.

Tests (Phase 1)

1. src/node/utils/messages/reasoningProviderOptions.test.ts (describe("attachReasoningReplayMetadata")):
   • openai itemId + reasoningEncryptedContent → providerMetadata.openai equals { reasoningEncryptedContent } (no itemId); input part not mutated.
   • xai itemId + encrypted → unchanged (pins the scope).
   • Existing bare-itemId drop case still passes.
2. Real-SDK request-body test (new describe in src/node/services/providerModelFactory.test.ts, following the capturedBody mock-fetch pattern at ~774-841, or a focused sibling test next to messagePipeline.ts):
   • Build history with a completed reasoning part { itemId: "rs_stale", reasoningEncryptedContent: "blob" } + assistant text, run prepareMessagesForProvider → createOpenAI({ fetch: mockFetch }).responses(...) doGenerate/streamText.
   • Assert for default store (unset) and explicit store: true: body.input contains { type: "reasoning", encrypted_content: "blob", summary: [...] }, contains no item_reference, and the string rs_stale appears nowhere. Assert store: false yields the same shape.
   • Fragmented reasoning: two consecutive reasoning parts sharing the id, encrypted content only on the first → exactly one reasoning input item carrying the blob (Pass 0 coalescing in transformModelMessages must still group them); summary text retained.
3. Multi-step continuity: existing prepareStep/tool-loop tests stay green (intra-turn SDK step messages are untouched by this change — they carry fresh same-route ids).

Phase 2 — One-shot in-stream repair for rejected reasoning replay (stacked follow-up; net ≈ +80 LoC product)

Purpose: after Phase 1 the only remaining deterministic 400s from reasoning replay are (a) invalid_encrypted_content after a cross-org route switch and (b) Item with id 'rs_…' not found from intra-turn SDK step messages (fresh ids under a flapping gateway). Both would still hit the unbounded "api" retry loop. Mirror the existing retryStreamWithoutPreviousResponseId pattern:

1. Detect (src/node/services/streamManager.ts, next to extractPreviousResponseIdFromError): isOpenAIReasoningReplayRejection(error) — an OpenAI Responses request with HTTP 400/404 (or an SDK stream-error 500: StreamProviderError, or APICallError with an SSE content type and an error/response.failed frame) and either an explicit Item with id 'rs_…' not found message or error.code === "invalid_encrypted_content" (also match "encrypted content" + "could not be verified"). Reuse extractErrorCode/extractStatusCode. Determine OpenAI Responses provenance from the resolved route/SDK model, not only the requested model prefix. Do not classify unrelated rs_ mentions, other item types, xAI, or non-Responses requests.
2. Strip (src/node/utils/messages/reasoningProviderOptions.ts): stripOpenAIReasoningReplay(messages: ModelMessage[]) — remove only reasoning parts whose providerOptions.openai is set. Preserve string-content assistants, other-provider reasoning, all visible text and tool history. Drop only array-content assistant messages newly emptied by this removal. Do not mutate inputs; return the same array when nothing changed. SDK step messages retain the OpenAI namespace, so no broad reasoning removal or user-message coalescing is needed.
3. Retry once (retryStreamWithoutOpenAIReasoningReplay, sibling of streamManager.ts:5223-5314, wired at the call site ~4805-4824 with its own didRetry… flag): same guards (not aborted, no soft interrupt, hasParts && currentStepStartIndex !== parts.length → bail), apply the strip to streamInfo.request.messages (and stepTracker.latestMessages for step scope), resetStreamStateForRetry, createStreamResult. Bail if the strip changed nothing. Log Retrying stream without OpenAI reasoning replay with errorCode/statusCode/retryScope.
4. Bound the outer retry as well. Send the final failure through handleStreamFailure. A final error that still matches the narrow OpenAI Responses reasoning rejection is reasoning_rejected, including when repair was unsafe or a no-op. Add this value to the existing StreamErrorTypeSchema, shared NON_RETRYABLE_STREAM_ERRORS, and RUNNING_TASK_TERMINAL_STREAM_ERRORS. This prevents the outer automatic retry loop and settles delegated workspace turns and running tasks. Preserve abort precedence. A subsequent 503, 401, or 429 retains its ordinary classification and retry/auth policy; merely attempting repair must not make other errors terminal. Manual continuation remains possible with a fresh attempt. No new persisted field, module, or subsystem.

Accepted trade-off: the retried step/turn loses prior reasoning context (visible text/tool history is kept). No sticky flag — each turn may incur rejected requests until compaction/new context, including any SDK-internal retries. The single in-stream repair is invisible to the UI (no stream-error event is emitted for that repair). Add a sticky flag only if telemetry shows it matters.

Tests: mirror describe("StreamManager - previousResponseId recovery") for both error shapes and one repair without an intermediate stream-error. Prove second rejection does not schedule an outer retry, delegated turns fail, running tasks settle, and persisted terminal errors do not auto-resume at startup. Cover normal policy after subsequent 503/401/429; abort, soft interrupt, emitted current-step parts, missing step messages, and no-op guards; preservation of prior-step text/tools/usage without re-execution; nonmutation and unchanged-array identity; negative matching and xAI; and manual continuation after a terminal attempt. Prefer existing fixtures and parameterized behavior tests over new harnesses.

Rejected alternatives

• Default store:false for OpenAI — changes dashboard/storage semantics for all users; unnecessary because only reasoning parts carry ids.
• Reactive-only repair (Phase 2 without Phase 1) — history is never repaired, so every later turn fails first, then retries.
• A global retry cap or terminal-only handling without repair — stops retrying but does not recover. Phase 2 first attempts safe in-stream repair, then treats only a final, strictly matched reasoning rejection as terminal. Unrelated transient errors retain the existing retry policy.

Accepted trade-offs / residual risks

• Request bodies grow (one base64 blob per prior reasoning item instead of a ~40-byte reference). Same as today's Codex/Grok paths; bounded by compaction. Measure once in dogfooding (body size for a long session).
• Prompt-cache hit rate, latency and answer quality are expected to be unchanged (the server resolves both forms to the same reasoning tokens) but are not proven by the converter code; confirm cachedInputTokens > 0 on turn 2 in dogfooding.
• Encrypted content is org-bound. Phase 1 heals same-org route switches and eviction; cross-org switches need Phase 2.

Acceptance criteria

1. Unit + real-SDK body tests above pass; make static-check and bun test src/node/utils/messages src/node/services/providerModelFactory.test.ts src/node/services/streamManager.test.ts green.
2. A workspace whose chat.jsonl reasoning part carries a bogus itemId (real encrypted content) continues successfully on a real OpenAI reasoning model without /compact; before the fix the same workspace loops with Item with id … not found.
3. devtools.jsonl (API Debug Logs on) for turn 2+ shows "type":"reasoning" items with encrypted_content and zero item_reference entries.
4. No behaviour change for Anthropic/Google/xAI histories (existing tests).

Dogfooding (screenshots and video; label as real-route evidence)

Use the dev-server-sandbox skill (fresh XUM_ROOT, free ports) and agent-browser with an owned --session; start agent-browser record start before step 3, record stop after step 6, and attach_file the .webm plus each PNG.

1. make dev-server in the sandbox; enable API Debug Logs; select an OpenAI reasoning model available in this environment (coder:openai/gpt-5.x via the gateway or openai:gpt-5.x direct). Screenshot: model picker.
2. Turn 1: ask for a short reasoning-heavy task. Screenshot: completed turn with a reasoning block.
3. Turn 2: "continue". Verify in <XUM_ROOT>/sessions/<ws>/devtools.jsonl that the request input contains type:"reasoning" + encrypted_content and no item_reference; record cachedInputTokens. Screenshot: terminal grep output.
4. Stale-id repro: stop the dev server; edit chat.jsonl and change the turn-1 reasoning providerOptions.openai.itemId to rs_000stale; restart with the same XUM_ROOT.
   • Control (optional, on main before the change): send "continue" → observe Stream Error [API] Item with id 'rs_000stale' not found + Retrying in … barrier. Screenshot.
   • With the fix: send "continue" → turn completes; no error, no compaction. Screenshot of the transcript + the devtools.jsonl request line showing the blob and no rs_000stale.
5. Model-switch check: switch to a different GPT‑5.x variant and send one more turn; confirm success (blob accepted across model variants).
6. Long-session size check: note the request body byte size from devtools.jsonl for the longest available history; report it in the PR.
7. (Phase 2 only) On the exact pushed Phase 2 head, corrupt the encrypted blob of one reasoning part ("blob-broken") → send → expect one rejected OpenAI Responses request, one Retrying stream without OpenAI reasoning replay log line, and a successful repaired turn; no Retrying in … barrier or intermediate stream-error. Capture screenshots, video, request/log evidence, and unchanged persisted history. Automated integration tests separately prove that a second matching rejection terminates instead of starting the outer retry loop.

Quality gates

1. Phase 1 code + tests → make static-check + targeted bun test green → dogfood steps 1–6 → PR (stacked base for Phase 2).
2. Phase 2 code + tests → same checks → dogfood step 7 → stacked PR.
3. Do not start Phase 2 until Phase 1's stale-id repro (step 4) passes on a real route.

---

Generated with xum • Model: coder:openai/gpt-6-astra • Thinking: high • Cost: $194.25

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-22T20:02:38.391747Z	b398e78	New commits
🔒 Security Review	✅ Completed 2026-09-22T20:03:19.246487Z	b398e78	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eae7090063

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 006ce021e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

Paused: not ready

1. Published head: 13495f4ee16b2b6f30e01c0078d4e08ad61dd4ea. Automatic code and security reviews completed on this head, followed by the Codex thumbs-up at 18:51:29 UTC. Zero unresolved review threads. Combined assessment count: 6/6.
2. CI remains red: run 35768889805 reported two failing notification tests and one separate unhandled TaskService ENOENT while writing locks/project-registration.lock.tmp-*. The two test failures reproduce locally with pinned Bun 1.3.5; their fixtures omitted the required request object.
3. Local-only repair: a23eed7cc3e1f19c1e57b626132de9e8d4fc5980 supplies typed request state and asserts successful completion in those fixtures. One test file, 25 added lines, no production changes. Static checks and 783 targeted tests passed for the same source tree; the two repaired tests also passed after commit. This commit is unpublished and unreviewed. Existing approvals do not cover it.
4. Separate CI error: 21 cancellation tests repeated three times passed (63 executions). That does not explain or fix the full-process TaskService teardown error. Follow-up trigger: investigate/account for it on the next authorized full CI attempt; it remains a blocker, not a waived flake.
5. Remote UAT remains blocked: round 4 on 006ce021e has functional evidence but no accepted isolation pass because a shared-deployment mutation remains unattributed. Its corrective run was cancelled. No remote testing resumed, and the current head has not passed live UAT.

The independent advisory recommendation was to repair the bounded fixture issue locally, then stop rather than trigger assessments seven and eight automatically. Another push requires review-budget authorization. The repository readiness helper also still requires a manual trigger; that requirement needs an explicit decision rather than fabricating a request or silently exceeding the cap. The deployment owner must separately adjudicate the audit event before remote UAT resumes.

Decision: blocked, not ready with follow-ups. No merge or auto-merge. Logs, media, red/green receipts, the local commit, and the preserved remote chat remain available for continuation.

---

Generated with xum • Model: coder:openai/gpt-6-astra • Thinking: high • Cost: $201.04