feat(ai-openrouter): capture per-request cost from chat responses#469
Open
season179 wants to merge 3 commits intoTanStack:mainfrom
Open
feat(ai-openrouter): capture per-request cost from chat responses#469season179 wants to merge 3 commits intoTanStack:mainfrom
season179 wants to merge 3 commits intoTanStack:mainfrom
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds OpenRouter per-request USD cost capture by cloning/parsing SSE chat completions in an HTTPClient response hook, buffering results in a CostStore, and attaching optional Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Adapter as OpenRouter Adapter
participant Hook as HTTPClient Hook
participant SSE as SSE Stream
participant CostStore as CostStore
participant App
Client->>Adapter: chatStream(request)
Adapter->>Hook: construct client with cost-capture hook
Adapter->>SSE: open chat completion stream (HTTP response)
SSE->>Hook: HTTP Response (text/event-stream)
Hook->>Hook: clone Response body and parse clone (out-of-band)
Hook->>CostStore: store extracted usage.cost & usage.cost_details for response id
par streaming
SSE->>Adapter: yield content/token usage chunks
Adapter->>Adapter: accumulate finalUsage and finishReason
and
Hook->>CostStore: store parsed cost when found
end
SSE->>Adapter: trailing usage-only chunk (finalUsage)
Adapter->>CostStore: take(responseId)
CostStore-->>Adapter: {cost, costDetails} | undefined
Adapter->>Adapter: buildRunFinishedUsage(finalUsage, costInfo)
Adapter->>App: emit RUN_FINISHED with usage (+ cost when present)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
packages/typescript/ai/src/types.ts (1)
796-806:costDetailscomment says "loosely typed" but the shape is actually locked to OpenRouter's two fields.The JSDoc argues the type must be loose to accommodate provider divergence (BYOK upstream, cache discounts, per-tier rates, ...), but
costDetailsonly declaresupstreamInferenceCostandcacheDiscount. Any future adapter that reports, say, a tier-specific rate would be forced to either omit it oras any-cast — exactly what the comment says should be avoided.If the intent is genuinely "loose", consider adding an index signature so additional provider-specific keys are type-legal without a cast. Otherwise, tighten the comment to match the actual (narrow, OpenRouter-shaped) contract.
♻️ Suggested adjustment
costDetails?: { upstreamInferenceCost?: number | null cacheDiscount?: number | null + [key: string]: number | null | undefined }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai/src/types.ts` around lines 796 - 806, The JSDoc promises a "loosely typed" costDetails but the declared shape only allows upstreamInferenceCost and cacheDiscount; update the type for costDetails (in packages/typescript/ai/src/types.ts, symbol: costDetails) to include an index signature (or use Record<string, number | null>) while keeping the existing named keys so provider-specific numeric fields are allowed without casting; this ensures adapters can add tier/rate/BYOK fields legally without changing the comment.packages/typescript/ai-openrouter/src/adapters/cost-capture.ts (1)
161-167: Minor: content-type check is case-sensitive.
content-typeheader values are case-insensitive per RFC; OpenRouter today uses lowercasetext/event-stream, but a proxy on the path could legitimately return e.g.Text/Event-Streamand cost capture would silently skip. Cheap to harden:♻️ Proposed tweak
- const contentType = res.headers.get('content-type') ?? '' + const contentType = (res.headers.get('content-type') ?? '').toLowerCase() // Cost capture is only wired for streaming chat completions. Non-SSE // responses on `/chat/completions` (e.g. `structuredOutput()` which // calls `chat.send({ stream: false })`) never consume `costStore` — // skipping them here avoids cloning the response and second-parsing // potentially large JSON bodies for no downstream consumer. if (!contentType.includes('text/event-stream')) return🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openrouter/src/adapters/cost-capture.ts` around lines 161 - 167, The content-type check in the cost capture branch is case-sensitive and may miss valid SSE responses; update the check around the contentType variable so it compares case-insensitively (e.g. normalize contentType with toLowerCase() or use a case-insensitive regex) before testing for 'text/event-stream' and keep the early return behavior unchanged to avoid extra parsing.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/typescript/ai-openrouter/tests/openrouter-adapter.test.ts`:
- Around line 6-19: Merge the two separate type imports from
'../src/adapters/text' into one line (exported types OpenRouterTextAdapter and
OpenRouterTextModelOptions) to remove import/no-duplicates, and replace the
typed vi.importActual call in the vi.mock helper (currently using await
vi.importActual<typeof import('@openrouter/sdk')>('@openrouter/sdk')) with an
untyped runtime import (await vi.importActual('@openrouter/sdk')) or cast the
result to any so you avoid the banned import() type annotation; keep the other
type imports (CostStore, StreamChunk, Tool) and the module-scope mockSend
variable unchanged.
---
Nitpick comments:
In `@packages/typescript/ai-openrouter/src/adapters/cost-capture.ts`:
- Around line 161-167: The content-type check in the cost capture branch is
case-sensitive and may miss valid SSE responses; update the check around the
contentType variable so it compares case-insensitively (e.g. normalize
contentType with toLowerCase() or use a case-insensitive regex) before testing
for 'text/event-stream' and keep the early return behavior unchanged to avoid
extra parsing.
In `@packages/typescript/ai/src/types.ts`:
- Around line 796-806: The JSDoc promises a "loosely typed" costDetails but the
declared shape only allows upstreamInferenceCost and cacheDiscount; update the
type for costDetails (in packages/typescript/ai/src/types.ts, symbol:
costDetails) to include an index signature (or use Record<string, number |
null>) while keeping the existing named keys so provider-specific numeric fields
are allowed without casting; this ensures adapters can add tier/rate/BYOK fields
legally without changing the comment.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 90bbcd91-81f2-44c8-9463-74a171a180c3
📒 Files selected for processing (9)
.changeset/openrouter-cost-tracking.mdREADME.mddocs/adapters/openrouter.mdpackages/typescript/ai-openrouter/src/adapters/cost-capture.tspackages/typescript/ai-openrouter/src/adapters/text.tspackages/typescript/ai-openrouter/tests/cost-capture.test.tspackages/typescript/ai-openrouter/tests/openrouter-adapter.test.tspackages/typescript/ai/src/activities/chat/middleware/types.tspackages/typescript/ai/src/types.ts
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 18, 2026
Merge the two separate type imports from '../src/adapters/text' into
one statement (import/no-duplicates) and replace the inline `typeof
import('@openrouter/sdk')` type annotation with a top-level
`import type * as OpenRouterSDK` (@typescript-eslint/consistent-type-imports).
Picked up in CodeRabbit review of TanStack#469.
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 18, 2026
Content-Type header values are case-insensitive per RFC 9110. OpenRouter today serves lowercase `text/event-stream` but a proxy on the path could return a different casing, which would make cost capture silently skip a real SSE response. Lowercase the header before the substring match. Picked up in CodeRabbit review of TanStack#469.
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 18, 2026
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 20, 2026
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
season179
force-pushed
the
feat/openrouter-cost-tracking
branch
from
ab571ba to
7a0d1ff
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/typescript/ai-openrouter/src/adapters/cost-capture.ts`:
- Around line 64-90: The fallback in take(id) uses this.pendingParses which
still contains parses that have already announced a different id, causing
head-of-line blocking; fix by removing the parse from the pending set when it
announces an id. In announceId(id, parse) call this.pendingParses.delete(parse)
(e.g., right after creating the ParseEntry / this.idToParse.set) so announced
parses are no longer awaited by the Promise.allSettled fallback; apply the same
removal change to the other announce-path (the similar block referenced at lines
~115-124) so all announce flows remove their parse from pendingParses.
In `@packages/typescript/ai-openrouter/src/adapters/text.ts`:
- Around line 110-111: The adapter narrows finish reasons and drops
'content_filter'; change the declared finalFinishReason variable type and the
Generator return type to use RunFinishedEvent['finishReason'] instead of the
union 'stop'|'length'|'tool_calls', and update the mapping logic that maps
OpenRouter finish reasons to RunFinishedEvent to include an explicit case for
'content_filter' (preserving that value rather than converting it to 'stop'),
ensuring finalFinishReason is assigned the mapped 'content_filter' where
appropriate.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e23b2f7b-821f-4b1d-97a8-f2262c11a3c4
📒 Files selected for processing (9)
.changeset/openrouter-cost-tracking.mdREADME.mddocs/adapters/openrouter.mdpackages/typescript/ai-openrouter/src/adapters/cost-capture.tspackages/typescript/ai-openrouter/src/adapters/text.tspackages/typescript/ai-openrouter/tests/cost-capture.test.tspackages/typescript/ai-openrouter/tests/openrouter-adapter.test.tspackages/typescript/ai/src/activities/chat/middleware/types.tspackages/typescript/ai/src/types.ts
✅ Files skipped from review due to trivial changes (2)
- README.md
- docs/adapters/openrouter.md
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/typescript/ai/src/types.ts
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 20, 2026
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
season179
force-pushed
the
feat/openrouter-cost-tracking
branch
from
7a0d1ff to
e98120e
Compare
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 20, 2026
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
season179
force-pushed
the
feat/openrouter-cost-tracking
branch
from
e98120e to
e6b1aab
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/typescript/ai-openrouter/src/adapters/cost-capture.ts`:
- Around line 326-344: extractCostFromUsage currently allows returning
costDetails when usage.cost is missing; change it so costDetails are only
included when the authoritative total cost (usage.cost) is present. Update
extractCostFromUsage to treat undefined cost as a reason to return undefined (or
at minimum only attach costDetails when cost !== undefined), so that the
returned object never contains costDetails without a numeric cost; refer to
extractCostFromUsage, the local variables cost/details/upstream/cacheDiscount,
and helper pickNumberOrNull to implement this conditional.
- Around line 219-223: The isChatCompletionsRequest function currently tests the
full URL string which can match query params like "?next=/chat/completions";
instead parse the url into a URL object and run the regex (or a pathname
comparison) against the URL.pathname only (i.e., use new URL(url).pathname and
then test that string) so the hook only triggers for actual /chat/completions
pathnames; update isChatCompletionsRequest to use URL.pathname for the check.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 9dae5fc2-0ed4-4a5c-b2d3-4c751d20543d
📒 Files selected for processing (9)
.changeset/openrouter-cost-tracking.mdREADME.mddocs/adapters/openrouter.mdpackages/typescript/ai-openrouter/src/adapters/cost-capture.tspackages/typescript/ai-openrouter/src/adapters/text.tspackages/typescript/ai-openrouter/tests/cost-capture.test.tspackages/typescript/ai-openrouter/tests/openrouter-adapter.test.tspackages/typescript/ai/src/activities/chat/middleware/types.tspackages/typescript/ai/src/types.ts
✅ Files skipped from review due to trivial changes (2)
- README.md
- docs/adapters/openrouter.md
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/typescript/ai-openrouter/tests/openrouter-adapter.test.ts
- packages/typescript/ai-openrouter/src/adapters/text.ts
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 20, 2026
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
season179
force-pushed
the
feat/openrouter-cost-tracking
branch
from
e6b1aab to
3d50ef4
Compare
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 21, 2026
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
season179
force-pushed
the
feat/openrouter-cost-tracking
branch
from
3d50ef4 to
168d495
Compare
Author
|
Hi @AlemTuzlak, any chance you could take a look at this? Happy to address any feedback. |
Contributor
|
@season179 can you check if this is still an issue on the latest version of tanstack ai, I've fixed a bug in our chunk processor that indeed did stop usage from going through properly, this fix might not be needed at all |
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 24, 2026
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
season179
force-pushed
the
feat/openrouter-cost-tracking
branch
from
ebdff27 to
1d5579b
Compare
Author
|
Thanks @AlemTuzlak, I checked this against the latest TanStack AI changes. The recent chunk processor fix does help token usage flow through correctly, but this PR is still needed for a different reason: it exposes OpenRouter’s provider-reported The goal here is not to compute cost from token usage. OpenRouter’s actual cost can vary based on routing, fallback providers, BYOK, caching, and provider-specific pricing details, so token × price-table math can be wrong. This PR preserves the exact cost amount and cost breakdown returned by OpenRouter itself and attaches it to I’ve updated the PR to make that scope clearer. |
Capture OpenRouter's per-request cost from the SSE chat response and surface it on UsageTotals.costDetails so callers can track spend. Implementation notes: - Cost capture is isolated per-request so concurrent chats don't cross-contaminate; SSE parsing matches separators directly (handles LF and CRLF) and matches the content-type case-insensitively. - Skipped on non-streaming chat responses to avoid needless overhead. - Omits usage entirely when token counts are missing, even if a cost was captured, so downstream consumers never see partial usage. - Does not downgrade a RUN_FINISHED event to RUN_ERROR on a late stream abort; the deferral trade-off is documented in the adapter. - Flushes the trailing SSE frame and resolves no-cost takes fast.
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
season179
force-pushed
the
feat/openrouter-cost-tracking
branch
from
1d5579b to
71d9c30
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Capture OpenRouter's provider-reported per-request cost from chat completion responses and expose it on
RUN_FINISHED.usage.OpenRouter returns
usage.costandusage.cost_detailsinline in chat responses (docs), but the@openrouter/sdkchat-completion parser strips fields that are not declared in its schema. This PR recovers those exact provider-returned fields from the streamed response and forwards them through TanStack AI.Closes #468
Why This Uses OpenRouter's Exact Cost
This does not compute cost locally from token counts.
OpenRouter can route the same model id to different upstream providers, fallback providers, BYOK providers, and cached-token paths. Those paths can have different effective prices and provider-specific breakdowns. A local token × price-table calculation would drift and could silently report the wrong amount.
This PR only forwards the cost amount and cost details OpenRouter actually returned.
Implementation
HTTPClient./chat/completionsresponses:text/event-streamcase-insensitivelyURL.pathname, not the full URL stringResponse.clone()and parses the cloned SSE body out-of-band, so the SDK's normal stream consumer remains untouched.usage.costas the authoritative total.usage.cost_details, camelCased underusage.costDetails.cost_detailswhenusage.costis absent, because a breakdown without the total cannot be reconciled.httpClientinstances before appending the hook, preserving custom fetchers, retries, tracing, and existing hooks without mutating the caller's client.RUN_FINISHEDuntil the upstream stream fully drains so trailing usage-only chunks are included.If cost is absent or parsing fails, the stream still completes normally;
RUN_FINISHED.usagesimply omits cost fields.API Surface
Adds shared
UsageTotalsfields:usage.cost?: numberusage.costDetails?: Record<string, number | null | undefined>RunFinishedEvent.usage, middlewareUsageInfo, andFinishInfo.usagenow share that shape so cost can be read consistently from runtime events and middleware callbacks.costDetailsis intentionally generic. OpenRouter's current fields, such asupstreamInferenceCost,upstreamInferenceInputCost,upstreamInferenceOutputCost, andcacheDiscount, are preserved at runtime without hard-coding OpenRouter-specific keys into the shared@tanstack/aitype.Docs + Release
docs/adapters/openrouter.md.@tanstack/ai-openrouterand@tanstack/ai.Validation
pnpm exec eslint packages/typescript/ai-openrouter/src/adapters/cost-capture.ts packages/typescript/ai-openrouter/src/adapters/text.ts packages/typescript/ai-openrouter/tests/cost-capture.test.ts packages/typescript/ai-openrouter/tests/openrouter-adapter.test.ts packages/typescript/ai/src/activities/chat/middleware/types.ts packages/typescript/ai/src/types.ts pnpm --filter @tanstack/ai-openrouter test:lib -- cost-capture.test.ts openrouter-adapter.test.ts git diff --check@tanstack/ai-openrouterunit tests pass: 4 files, 69 tests.Known unrelated typecheck failures remain on latest
main:packages/typescript/ai-openrouter/src/adapters/image.ts@tanstack/aiChecklist