Skip to content

feat: emit OTel GenAI token and model attributes on LLM spans - #125

Open
johndemic wants to merge 1 commit into
DEVtheOPS:mainfrom
johndemic:feat/gen-ai-semconv-attributes
Open

johndemic wants to merge 1 commit into
DEVtheOPS:mainfrom
johndemic:feat/gen-ai-semconv-attributes

Conversation

@johndemic

@johndemic johndemic commented Sep 19, 2026

Copy link
Copy Markdown

Problem

LLM spans carry token counts and the model only under OpenInference keys
(llm.token_count.*, llm.model_name). Backends that read the OpenTelemetry
GenAI conventions identify the span as GenAI — gen_ai.provider.name is
already set — but then find no usage or model data on it, so they cannot
price the call.

Change

Mirror the existing values onto the GenAI keys. Nothing is renamed or removed,
so OpenInference consumers (Arize/Phoenix) are unaffected:

gen_ai.request.model
gen_ai.response.model
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
gen_ai.usage.cache_read_input_tokens
gen_ai.usage.cache_creation_input_tokens

Four come from @opentelemetry/semantic-conventions (already a dependency).

Why the cache keys are string literals

semconv spells these gen_ai.usage.cache_read.input_tokens (dotted). Anthropic's
API and the GenAI consumers this targets read the underscored form, so the
constants would emit attributes nothing reads. The new tests assert the literal
strings for the same reason — a constant renamed upstream should fail the suite
rather than silently change the wire format.

Why cache tokens matter here

Not just completeness. A coding agent against a prompt-cached Anthropic model
routinely reports thousands of cache-creation tokens against single-digit input
tokens — one real session showed cache_write=16739 against prompt=3. A
consumer mapping only input/output understates cost by orders of magnitude.

Verification

Tested against a live GenAI backend with a control, 14 seconds apart, identical
token counts — the only difference being attribute naming:

span shape model resolved tokens cost computed
patched (gen_ai.*) yes 3 in / 5 out / 16739 cache-create $0.06285525
unpatched (OpenInference only) unknown 0 / 0 $0.00

The backend derived the cost from token counts alone (no cost attribute was
sent) and its figure matched this plugin's own llm.cost.total to eight
decimal places.

Tests

Three added to tests/handlers/spans.test.ts; all fail against unpatched source.

baseline main: 326 pass
this branch:   329 pass, 0 fail

bun run lint, bun run check:jsdoc-coverage, bun run typecheck all pass.

End-to-end verification

Since opening this, I ran the patched plugin against a live GenAI backend and
captured the wire payload through a logging proxy.

The span the plugin emits:

SPAN opencode.llm  kind=CLIENT  scope=com.opencode
  gen_ai.provider.name                     = anthropic
  gen_ai.request.model                     = claude-sonnet-4-6
  gen_ai.response.model                    = claude-sonnet-4-6
  gen_ai.usage.input_tokens                = 3
  gen_ai.usage.output_tokens               = 5
  gen_ai.usage.cache_read_input_tokens     = 0
  gen_ai.usage.cache_creation_input_tokens = 16919
  llm.cost.total                           = 0.06353025      <- unchanged
  llm.token_count.prompt                   = 3               <- unchanged
  llm.token_count.prompt_details.cache_write = 16919         <- unchanged

The resulting record on the backend:

model                    claude-sonnet-4-6
provider                 anthropic
inputTokenCount          3
outputTokenCount         5
cacheReadTokenCount      0
cacheCreationTokenCount  16919
totalCost                $0.06353025

Every field round-trips, cache tokens included. The backend derived the cost
from token counts alone — no cost attribute is read from the span — and its
figure matched this plugin's own llm.cost.total to eight decimal places.

Without the patch the same traffic records as model: unknown, 0 tokens, $0,
matching the control in the table above.

LLM spans carried token counts and the model only under OpenInference keys
(llm.token_count.*, llm.model_name). Backends that read the OpenTelemetry
GenAI conventions therefore saw a span they could identify as GenAI, via
the gen_ai.provider.name already set, but found no usage or model data on
it and could not price the call.

Mirror the existing values onto the GenAI keys. Nothing is renamed or
removed, so OpenInference consumers are unaffected:

  gen_ai.request.model
  gen_ai.response.model
  gen_ai.usage.input_tokens
  gen_ai.usage.output_tokens
  gen_ai.usage.cache_read_input_tokens
  gen_ai.usage.cache_creation_input_tokens

The cache keys are spelled out rather than taken from semconv, which emits
gen_ai.usage.cache_read.input_tokens. Anthropic's API and the GenAI
consumers this targets read the underscored form. The new tests assert the
literal strings for the same reason: a constant renamed upstream must fail
the suite rather than silently change the wire format.

Cache tokens matter for correctness here, not just completeness. A coding
agent against a prompt-cached Anthropic model routinely reports thousands
of cache-creation tokens against single-digit input tokens, so a consumer
mapping only input and output understates cost by orders of magnitude.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant