diff --git a/develop-docs/sdk/telemetry/traces/modules/ai-agents.mdx b/develop-docs/sdk/telemetry/traces/modules/ai-agents.mdx index 29c514daf12e97..c531f83269e1ba 100644 --- a/develop-docs/sdk/telemetry/traces/modules/ai-agents.mdx +++ b/develop-docs/sdk/telemetry/traces/modules/ai-agents.mdx @@ -10,9 +10,9 @@ This page lists every attribute SDKs send on AI spans. Send all of them. The des The [Sentry Conventions](https://github.com/getsentry/sentry-conventions/) are the single source of truth for every `gen_ai.*` attribute and span op. They follow the [OpenTelemetry Semantic Conventions for Generative - AI](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/README.md) where possible. - Sentry doesn't support span events, so message content that OpenTelemetry - stores in events is stored as span attributes instead. + AI](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/README.md) + where possible. Sentry doesn't support span events, so message content that + OpenTelemetry stores in events is stored as span attributes instead. ## Span Hierarchy @@ -126,15 +126,118 @@ Describes a decision on generated content against a set of questions, such as gr - Span `name`: `"decide {gen_ai.request.model}"` (e.g. `"decide typesafe/jev-1.13"`) - `gen_ai.operation.name`: `"decide"`. Decisions share the `ai_client` operation type with model calls, so this attribute is what tells them apart. -| Attribute | Type | Description | Example | -| :--------------------------- | :----- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------- | -| `gen_ai.request.model` | string | The evaluation model the SDK asked for. Drives model breakdowns, cost, and the span title. Without a model, Relay doesn't classify the span as `ai_client`. | `"typesafe/jev-1.13"` | -| `gen_ai.response.model` | string | The concrete model that responded. Omit it when the provider doesn't report one. | `"typesafe/jev-1.13-20260917"` | -| `gen_ai.usage.input_tokens` | int | Input tokens used by the evaluation. Drives token widgets and cost. | `154` | -| `gen_ai.usage.output_tokens` | int | Output tokens used by the evaluation. Drives token widgets and cost. | `28` | -| `gen_ai.agent.name` | string | The agent the decision ran under. Omit it when the decision runs outside an agent. | `"Support Agent"` | -| `gen_ai.input.messages` | string | The content being evaluated and the questions asked about it, as a one-element array holding an evaluation object. Shown in the decision **Input** view. **[3]** | `'[{"type": "evaluation", "state": "...", "questions": {"accuracy": {"type": "score", "criteria": ["incorrect", "correct"]}}}]'` | -| `gen_ai.output.messages` | string | The answers, as a one-element array holding an evaluation object. Shown in the decision **Output** view and as the one-line result in the transcript and timeline. **[4]** | `'[{"type": "evaluation", "answers": {"accuracy": {"type": "score", "score": 1}}}]'` | +| Attribute | Type | Description | Example | +| :--------------------------- | :----- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------------------------------- | +| `gen_ai.request.model` | string | The evaluation model the SDK asked for. Drives model breakdowns, cost, and the span title. Without a model, Relay doesn't classify the span as `ai_client`. | `"typesafe/jev-1.13"` | +| `gen_ai.response.model` | string | The concrete model that responded. Omit it when the provider doesn't report one. | `"typesafe/jev-1.13-20260917"` | +| `gen_ai.usage.input_tokens` | int | Input tokens used by the evaluation. Drives token widgets and cost. | `154` | +| `gen_ai.usage.output_tokens` | int | Output tokens used by the evaluation. Drives token widgets and cost. | `28` | +| `gen_ai.agent.name` | string | The agent the decision ran under. Omit it when the decision runs outside an agent. | `"Support Agent"` | +| `gen_ai.input.messages` | string | The content being evaluated and the questions asked about it, as a one-element array holding one object. Shown in the decision **Input** view. See [Decision Input Messages](#decision-input-messages). | `'[{"state": "...", "questions": {"accuracy": {"type": "score", "criteria": ["incorrect", "correct"]}}}]'` | +| `gen_ai.output.messages` | string | The answers, as a one-element array holding one object. Shown in the decision **Output** view and as the one-line result in the transcript and timeline. See [Decision Output Messages](#decision-output-messages). | `'[{"answers": {"accuracy": {"type": "score", "score": 1}}}]'` | + +### Decision Input Messages + +`gen_ai.input.messages` holds a one-element array: `[{"state", "questions"}]`. Copy `state` and the questions as the caller passed them. If the provider uses a different layout, convert it to this layout. `` is the question key. + +| Field | Type | Required | Description | Example | +| :------------------------------ | :--------------------- | :----------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------ | +| `state` | any | Yes | The content being evaluated: a string, an object, or a message array, as the caller passed it. Replace binary data, including `data:` URLs, with `"[Blob substitute]"`. | `"I was charged twice and want a refund."` | +| `questions` | object | Yes | The questions, keyed by question name. | `{"urgency": {...}}` | +| `questions..type` | string | Yes | `"boolean"`, `"predicate"`, `"noul"`, `"choice"`, or `"score"`. Copy other values as they are. | `"choice"` | +| `questions..instructions` | string | Yes | The question text. | `"Which team should handle this?"` | +| `questions..criteria` | object or string array | For `choice` and `score` | For `choice`, an object that maps each option to its description. For `score`, an ordered array of labels. The index is the score. | `{"billing": "Charges and refunds"}`, `["low", "high"]` | +| `questions..descriptions` | object | No | Only for `score`. Maps a score index (as a string) to the description of that level. Omit the levels that have no description. | `{"1": "Blocks the customer"}` | + +### Decision Output Messages + +`gen_ai.output.messages` holds a one-element array: `[{"answers"}]`. `answers` uses the same keys as `questions`, and each answer has a `type` that matches its question. + +| Field | Type | Required | Description | Example | +| :----------------------------- | :----- | :---------------------------- | :---------------------------------------------------------------------------------------------------- | :------------------------------------- | +| `answers` | object | Yes | The answers, keyed by question name. | `{"urgency": {...}}` | +| `answers..type` | string | Yes | The type of the question. | `"score"` | +| `answers..probability` | number | For `boolean` and `predicate` | The probability that the answer is yes. Shown as Yes when it's 0.5 or higher. | `0.97` | +| `answers..noul` | number | For `noul` | The probability that the answer is yes. Jev's variant of `boolean`. | `0.98` | +| `answers..choice` | string | For `choice` | The selected option. | `"billing"` | +| `answers..score` | number | For `score` | The score. It can be a fraction between two levels. | `1.58` | +| `answers..probabilities` | object | No | Only for `choice` and `score`. Maps each option, or each score index as a string, to its probability. | `{"billing": 0.64, "technical": 0.36}` | +| `answers..legend` | object | No | Only for `score`. Maps each score index (as a string) to its label. | `{"0": "low", "1": "high"}` | +| `answers..confidence` | number | No | Only for `choice` and `score`. The confidence of the answer. | `0.28` | + +Omit optional fields when the provider doesn't report them. A field with the wrong type makes that question or answer unreadable, and Sentry shows it raw. + +## Extraction Span + +Describes an extraction of typed fields from an input, such as the total and due date of an invoice, for example from Cloudflare Clairic. The model can extract values that are in the input, compute values from them, and estimate or forecast values that the input doesn't determine. + +- Span `op`: `"gen_ai.extract"` +- Span `name`: `"extract {gen_ai.request.model}"` (e.g. `"extract @cf/cloudflare/clairic"`) +- `gen_ai.operation.name`: `"extract"` + +| Attribute | Type | Description | Example | +| :--------------------------- | :----- | :------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------- | +| `gen_ai.request.model` | string | The extraction model the SDK asked for. Drives model breakdowns, cost, and the span title. | `"@cf/cloudflare/clairic"` | +| `gen_ai.response.model` | string | The concrete model that responded. Omit it when the provider doesn't report one. | `"clairic"` | +| `gen_ai.usage.input_tokens` | int | Input tokens used by the extraction, including all passes. Drives token widgets and cost. | `655` | +| `gen_ai.usage.output_tokens` | int | Output tokens used by the extraction. Drives token widgets and cost. | `0` | +| `gen_ai.agent.name` | string | The agent the extraction ran under. Omit it when the extraction runs outside an agent. | `"Billing Agent"` | +| `gen_ai.input.messages` | string | The input and the field specs, as a one-element array holding the request body. See [Extraction Input Messages](#extraction-input-messages). | `'[{"state": "Invoice INV-2291 ...", "fields": {"total": {"type": "number", "unit": "USD"}}}]'` | +| `gen_ai.output.messages` | string | The answers, as a one-element array holding the response body. See [Extraction Output Messages](#extraction-output-messages). | `'[{"answers": {"total": {"type": "number", "value": 1080}}, "truncated": false}]'` | + +Don't send the other usage counts that the provider reports (such as Clairic's `state_tokens`, `image_tokens`, `schema_tokens`, and `passes`). `gen_ai.usage.input_tokens` already includes them. + +### Extraction Input Messages + +`gen_ai.input.messages` holds a one-element array with the request body as the caller passed it, without `model`. Don't change the values, and leave out the keys that the caller didn't set. `` is the field key. + +| Field | Type | Required | Description | Example | +| :----------------------------- | :--------------- | :---------------------------- | :------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------- | +| `state` | string | When there are no `images` | The text the model reads. Text answers are spans of this string. | `"Invoice INV-2291 from Northwind Supply, issued ..."` | +| `fields` | object | Yes | The field specs, keyed by field name. Names match `^[A-Za-z_][A-Za-z0-9_.\-]{0,63}$`. | `{"total": {...}}` | +| `fields..type` | string | Yes | `"text"`, `"number"`, `"date"`, `"time"`, `"datetime"`, `"duration"`, `"list"`, `"estimate"`, or `"forecast"`. | `"number"` | +| `fields..instructions` | string | No | What the field must contain. | `"Amount due including shipping"` | +| `fields..unit` | string | No | The unit label. For `duration`, one of `seconds`, `minutes`, `hours`, `days`, `weeks`, `months`, or `years`. | `"USD"` | +| `fields..integer` | boolean | No | Only for `number` and `duration`. Makes the value a whole number. | `true` | +| `fields..decimals` | int | No | Only for `number`, `duration`, and number lists. From 0 to 6. | `2` | +| `fields..items` | string | For `list` | The type of the list items: `"text"`, `"number"`, or `"date"`. | `"text"` | +| `fields..min` | number or string | For `estimate` and `forecast` | The lowest value, or an ISO date for date estimates. | `0` | +| `fields..max` | number or string | For `estimate` and `forecast` | The highest value, or an ISO date for date estimates. | `120` | +| `fields..step` | number or string | No | The grid of an estimate: a positive number (days for date estimates), `"month"`, or `"year"`. | `1` | +| `fields..scale` | string | No | `"linear"` or `"log"`. | `"linear"` | +| `fields..null_threshold` | number | No | The probability of null above which the value is `null`. Greater than 0 and up to 1. | `0.5` | +| `fields..horizon` | int | No | Only for `forecast`. The number of steps, from 1 to 64. | `12` | +| `fields..interval` | string | No | Only for `forecast`. `"step"`, `"minute"`, `"hour"`, `"day"`, `"week"`, `"month"`, `"quarter"`, or `"year"`. | `"month"` | +| `images` | string array | No | One `"[Blob substitute]"` string for each image. Never send the image data. | `["[Blob substitute]"]` | +| `reference_date` | string | No | The ISO date that anchors dates written without a year. | `"2026-09-01"` | +| `options` | object | No | Request options. | `{"include_quantiles": true}` | +| `options.include_quantiles` | boolean | No | Adds 99 percentiles to estimates. | `true` | +| `options.return_state_text` | boolean | No | Returns the text that spans index into. | `true` | + +### Extraction Output Messages + +`gen_ai.output.messages` holds a one-element array with the response body as the provider returned it, without `model` and `usage`. Don't change the values. `answers` has one answer for each field in the input, keyed by field name. + +| Field | Type | Required | Description | Example | +| :-------------------------- | :------------------ | :------- | :----------------------------------------------------------------------------------------------------------------------------------- | :------------------------- | +| `answers` | object | Yes | The answers, keyed by field name. | `{"total": {...}}` | +| `answers..type` | string | Yes | The type of the field. | `"number"` | +| `answers..value` | any | Yes | The answer, typed by the field: a string, a number, an ISO date, or an array for `list`. `null` when the input doesn't determine it. | `1080` | +| `answers..guess` | any | No | The best guess. It is there even when `value` is `null`. | `"INV-2291"` | +| `answers..p_null` | number | No | The probability that the value is `null`. | `2.3e-08` | +| `answers..confidence` | number | No | The confidence of the answer. | `0.9999` | +| `answers..span` | int array | No | Only for `text`. `[start, end]` character offsets of the value in `state` (or in `state_text`). `end` is exclusive. | `[8, 16]` | +| `answers..spans` | array of int arrays | No | Only for `list`. One `[start, end]` pair for each item. | `[[120, 129], [131, 141]]` | +| `answers..mean` | number or string | No | Only for `estimate`. The mean of the distribution. | `29.535` | +| `answers..median` | number or string | No | Only for `estimate`. The median of the distribution. | `30` | +| `answers..p10` | number or string | No | Only for `estimate`. The 10th percentile. | `21` | +| `answers..p90` | number or string | No | Only for `estimate`. The 90th percentile. | `38` | +| `answers..p_below` | number | No | Only for `estimate`. The probability that the value is below `min`. | `5.9e-06` | +| `answers..p_above` | number | No | Only for `estimate`. The probability that the value is above `max`. | `1.2e-05` | +| `answers..quantiles` | array | No | Only for `estimate`, when `options.include_quantiles` is set. 99 percentiles. | | +| `answers..steps` | array | No | Only for `forecast`. One estimate for each step. | | +| `truncated` | boolean | Yes | Copy it as the provider returned it. | `false` | +| `state_text` | string | No | Only when `options.return_state_text` is set. The text that spans index into. | | ## Memory Span @@ -151,7 +254,7 @@ Describes a read from, write to, or management operation on an agent's persisten | `gen_ai.memory.record.count` | int | The number of records returned by `search_memory`, or written or deleted by other operations. Shown as the result in the transcript. | `3` | | `gen_ai.agent.name` | string | The agent the operation ran under. Omit it when the operation runs outside an agent. | `"Support Agent"` | | `gen_ai.memory.query.text` | string | The search query, shown as the preview for `search_memory`. Only for `search_memory`. It can contain sensitive data, so only send it when the user opts in. | `"user dietary preferences"` | -| `gen_ai.memory.records` | string | The records stored or retrieved. **[5]** They can contain sensitive data, including PII, so only send them when the user opts in. | `'[{"id": "mem_123", "content": "User prefers dark mode", "score": 0.95}]'` | +| `gen_ai.memory.records` | string | The records stored or retrieved. **[3]** They can contain sensitive data, including PII, so only send them when the user opts in. | `'[{"id": "mem_123", "content": "User prefers dark mode", "score": 0.95}]'` | ## Notes @@ -167,17 +270,4 @@ Replace binary data in positions where the provider schema expects it, including **[2]** Send every message the model received on that call, including the earlier turns of the conversation. Don't drop messages that previous spans already carried or keep only the most recent user message: that loses data, for example when two user messages arrive in a row. -**[3]** The input is `[{"type": "evaluation", "state", "questions"}]`. `state` is the content being evaluated. `questions` maps each question key to an object with a `type` and an `instructions` string. A `score` question has `criteria` as an ordered string array whose index is the score (`["low", "medium", "high"]`). A `choice` question has `criteria` that maps each label to its description (`{"positive": "...", "negative": "..."}`). - -**[4]** The output is `[{"type": "evaluation", "answers"}]`. `answers` is keyed by the same question keys, and each answer has a `type` that matches its question: - -| Answer `type` | Fields | -| :--------------------- | :------------------------------------------------------------------------- | -| `boolean`, `predicate` | `probability`. Shown as Yes when it's 0.5 or higher. | -| `noul` | `noul` holds the probability. Jev's variant of `boolean`. | -| `score` | `score`, plus `legend` (`{"0": "low"}`), `probabilities`, and `confidence` | -| `choice` | `choice`, plus `probabilities` and `confidence` | - -`legend`, `probabilities`, and `confidence` can be omitted when the evaluator doesn't report them. A field with the wrong type makes that question or answer unreadable, and Sentry shows it raw. - -**[5]** A JSON array of records. Each record has a `content` (a string or JSON object) and can also carry an `id`, `metadata`, and a `score`. +**[3]** A JSON array of records. Each record has a `content` (a string or JSON object) and can also carry an `id`, `metadata`, and a `score`.