[Fix] Sessions and tasks keep retrying when the inference provider is out of credits

[roomote-roomote]

​Created by Roomote. Follow up by mentioning @roomote-roomote, in the web UI, or in Slack.

What changed

When an inference provider says the account is out of credits or quota, sessions and tasks used to show "The inference provider returned a temporary error. Retrying automatically…" and keep retrying a request that could not succeed. They now stop right away and say:

You seem to have run out of credits for {provider}. Choose another provider/model or reset your subscription to continue.

{provider} is the provider's display name. For openai/ models it is "ChatGPT (subscription)" when a connected ChatGPT subscription is serving them; otherwise it is the catalog label ("OpenAI", "OpenRouter", "Anthropic", "Roomote inference", …).

Detection. A shared, worker-safe detector in @roomote/types (isInferenceCreditsExhaustedError) is built from what the providers actually return, not guessed:

Provider	Signal treated as out of credits
ChatGPT subscription (Codex backend)	429 with error.type: "usage_limit_reached" ("The usage limit has been reached") or usage_not_included, as the Codex CLI's api_bridge.rs treats them as fatal
OpenAI API / Codex spend caps	insufficient_quota (type or code), credit_balance_exhausted, organization_spend_limit_exceeded, project_spend_limit_exceeded, organization_usage_limit_exceeded; OpenCode's rewrite of the streamed code ("Quota exceeded. Check your plan and billing details.")
Anthropic	400 "Your credit balance is too low…" and "You have reached your specified (workspace) API usage limits…"; 429 rate_limit_error with details.error_code: "enforced_spend_limit_reached" (monthly cap); 402 billing_error
OpenRouter	402 with metadata.limit_source openrouter_credits / openrouter_key_limit, error_type: "payment_required", "This request requires more credits…"
Any provider	HTTP 402

Structured signals are checked first. Provider-specific phrases are used only because OpenCode's retry status carries just the message text.

Stays retryable: plain 429 rate limits; unidentifiable errors; Gemini's per-minute RESOURCE_EXHAUSTED 429 (it reuses OpenAI's "You exceeded your current quota…" sentence, so the OpenAI phrase also requires OpenAI's docs link); and OpenRouter's in-flight budget 402 (limit_source: openrouter_in_flight_budget). OpenRouter documents that one as a temporary hold sent with Retry-After, so it never reads as out of credits.

Sessions (Fast and other non-task inference):

• An OpenCode retry status whose message matches ends the prompt and aborts OpenCode's own backoff loop. Every non-task prompt now watches OpenCode's event stream for this, including provider-key validation and helper calls such as routing and titles that pass no callbacks. Before this, OpenCode retried usage limits as rate limits: each retry surfaced as the "temporary error" notice, and calls without callbacks sat through the whole backoff or timed out.
• classifyNonTaskInferenceError checks credits before the "explicitly non-retryable" and 429 rules. Exhausted credits therefore no longer receive the one fresh-session retry granted to generic rejections, and no longer receive rate-limit backoff.
• insufficient_credits was already a terminal reason, so platform events settle once instead of being re-delivered.

Tasks (worker OpenCode harness): the same errors are terminal for OpenCode's internal retry, Roomote's 6× provider-error recovery, and the 429 rate-limit recovery. The terminal transcript message and terminal_provider_error event use the new text. The provider is taken from the providerID OpenCode reports for the failing session. The gateway markers decide whether openai/ and xai/ models run on a ChatGPT or Grok subscription.

Saving provider keys (product decision). Running out of credits or quota no longer blocks saving or adding a provider. Only rejected credentials (invalid_credentials) block, so operators can add the provider and top up the account in parallel.

• Settings > Models and the setup wizard: the key is saved and a non-blocking warning toast appears: "The {provider} account seems to be out of credits or quota. Add credits before using it." Validation recognizes the credits signal even while OpenCode is retrying a usage limit, so it returns this warning instead of timing out.
• Endpoint providers (vLLM, LiteLLM, Ollama, OpenAI-compatible): a 402 discovery probe no longer blocks. The existing post-save discovery error reports it.
• OpenRouter OAuth connect: this uses the same save path, so it no longer fails for a zero-credit account. That redirect path does not show the warning.
• invalid_endpoint still blocks endpoint connections.

apps/docs/models.mdx now describes the save behavior and adds a "You seem to have run out of credits" entry under common issues. A changeset is included.

How it was tested

New and updated Vitest coverage:

• @roomote/types: 40 fixture cases for the detector. They use real provider response bodies, and the negatives include plain rate limits, Gemini's look-alike quota 429, the OpenRouter in-flight 402, an invalid key, and credit wording that appears only in the echoed request.
• @roomote/cloud-agents non-task usage: a credits retry status ends the prompt without onProviderRetry and aborts the OpenCode session, including for a helper call and a validateNonTaskInference call that pass no callbacks (both previously sat through OpenCode's backoff until the deadline); classification ordering is also covered.
• @roomote/cloud-agents Fast service: an exhausted-credits closeout names ChatGPT (subscription), OpenAI, or OpenRouter, runs exactly one inference attempt, and writes no retry notice; a platform-event variant settles once.
• @roomote/worker harness: a ChatGPT usage-limit retry status and OpenAI/Anthropic/OpenRouter session errors fail the task without any continue prompt or retry notice.
• @roomote/web: provider validation only blocks on credentials and returns the credits warning; the Models command saves an out-of-credits key and an endpoint provider that returns 402; the wizard shows the warning and continues.

Whole-suite runs of the touched files pass: types, non-task usage, Fast service, worker harness, recovery and rate-limit units, and the web command, validation, and UI suites. oxlint --deny-warnings, format:check, web and worker ESLint, tsc for the four touched packages, the pre-push check-types:fast/knip, and the docs check (pnpm --filter @roomote/docs check) all pass.

Not exercised end to end against real exhausted provider accounts; the provider shapes come from the OpenCode v1.18.30 and Codex CLI sources and the Anthropic and OpenRouter error docs. No browser proof: the user-visible changes are transcript text and a warning toast that needs a real out-of-credits key to reach.

Checklist

• The PR title follows the repo convention: [Fix], [Feat], [Improve], [Refactor], [Docs], or [Chore] followed by a user-facing description
• This PR is small and scoped to one change
• pnpm lint and pnpm check-types pass locally
• I added tests or included a clear manual validation note above
• I removed secrets, tokens, private keys, and customer data from code, logs, and screenshots
• If this change should appear in the changelog, I ran pnpm changeset

[roomote-community]

No new code issues found. See task

• packages/cloud-agents/src/server/non-task-provider-usage.ts:1737 observes retry statuses for no-callback non-task and validation calls, so out-of-credits retries are stopped immediately.

_{Reviewed 427dafb}

[roomote-roomote]

Fixed the one outstanding finding behind the failed Roomote code review check in 427dafba.

• Finding: retry statuses were only observed for non-task prompts that passed event callbacks. Provider-key validation and plain helper calls never subscribed, so an out-of-credits retry loop was not stopped: validation waited for its deadline and reported a timeout instead of the credits warning.
• Fix: every non-task prompt now watches OpenCode's event stream. An out-of-credits retry status ends the prompt and aborts the OpenCode session regardless of callbacks, and a failed subscription is still non-fatal.
• Tests: added regression tests for a no-callback helper call and a no-callback validateNonTaskInference. Both failed before the change and pass now; the non-task suite passes 101/101. Locally, one cloud-agents test in opencode-runtime.test.ts fails the same way without this change and passes once XDG_CONFIG_HOME is unset. Typecheck, oxlint, format and pre-push checks pass.

Result: the review thread is resolved. The automatic re-review of 427dafb passed: the Roomote code review check succeeded, the review summary reports no new code issues, and all other checks are green, including Test. No visual proof was captured for this backend-only fix because the proof step timed out.