From 83fdd75cfdf5318b1195756815384fe6d2bc5663 Mon Sep 17 00:00:00 2001 From: Andrei Borza Date: Tue, 20 Oct 2026 15:01:11 +0200 Subject: [PATCH 1/6] docs(develop): Document decision and extraction message structure Co-Authored-By: Claude Opus 5.5 --- .../telemetry/traces/modules/ai-agents.mdx | 154 +++++++++++++++--- 1 file changed, 127 insertions(+), 27 deletions(-) diff --git a/develop-docs/sdk/telemetry/traces/modules/ai-agents.mdx b/develop-docs/sdk/telemetry/traces/modules/ai-agents.mdx index 29c514daf12e9..84c309836b433 100644 --- a/develop-docs/sdk/telemetry/traces/modules/ai-agents.mdx +++ b/develop-docs/sdk/telemetry/traces/modules/ai-agents.mdx @@ -10,9 +10,9 @@ This page lists every attribute SDKs send on AI spans. Send all of them. The des The [Sentry Conventions](https://github.com/getsentry/sentry-conventions/) are the single source of truth for every `gen_ai.*` attribute and span op. They follow the [OpenTelemetry Semantic Conventions for Generative - AI](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/README.md) where possible. - Sentry doesn't support span events, so message content that OpenTelemetry - stores in events is stored as span attributes instead. + AI](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/README.md) + where possible. Sentry doesn't support span events, so message content that + OpenTelemetry stores in events is stored as span attributes instead. ## Span Hierarchy @@ -126,15 +126,128 @@ Describes a decision on generated content against a set of questions, such as gr - Span `name`: `"decide {gen_ai.request.model}"` (e.g. `"decide typesafe/jev-1.13"`) - `gen_ai.operation.name`: `"decide"`. Decisions share the `ai_client` operation type with model calls, so this attribute is what tells them apart. -| Attribute | Type | Description | Example | -| :--------------------------- | :----- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------- | -| `gen_ai.request.model` | string | The evaluation model the SDK asked for. Drives model breakdowns, cost, and the span title. Without a model, Relay doesn't classify the span as `ai_client`. | `"typesafe/jev-1.13"` | -| `gen_ai.response.model` | string | The concrete model that responded. Omit it when the provider doesn't report one. | `"typesafe/jev-1.13-20260917"` | -| `gen_ai.usage.input_tokens` | int | Input tokens used by the evaluation. Drives token widgets and cost. | `154` | -| `gen_ai.usage.output_tokens` | int | Output tokens used by the evaluation. Drives token widgets and cost. | `28` | -| `gen_ai.agent.name` | string | The agent the decision ran under. Omit it when the decision runs outside an agent. | `"Support Agent"` | -| `gen_ai.input.messages` | string | The content being evaluated and the questions asked about it, as a one-element array holding an evaluation object. Shown in the decision **Input** view. **[3]** | `'[{"type": "evaluation", "state": "...", "questions": {"accuracy": {"type": "score", "criteria": ["incorrect", "correct"]}}}]'` | -| `gen_ai.output.messages` | string | The answers, as a one-element array holding an evaluation object. Shown in the decision **Output** view and as the one-line result in the transcript and timeline. **[4]** | `'[{"type": "evaluation", "answers": {"accuracy": {"type": "score", "score": 1}}}]'` | +| Attribute | Type | Description | Example | +| :--------------------------- | :----- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------- | +| `gen_ai.request.model` | string | The evaluation model the SDK asked for. Drives model breakdowns, cost, and the span title. Without a model, Relay doesn't classify the span as `ai_client`. | `"typesafe/jev-1.13"` | +| `gen_ai.response.model` | string | The concrete model that responded. Omit it when the provider doesn't report one. | `"typesafe/jev-1.13-20260917"` | +| `gen_ai.usage.input_tokens` | int | Input tokens used by the evaluation. Drives token widgets and cost. | `154` | +| `gen_ai.usage.output_tokens` | int | Output tokens used by the evaluation. Drives token widgets and cost. | `28` | +| `gen_ai.agent.name` | string | The agent the decision ran under. Omit it when the decision runs outside an agent. | `"Support Agent"` | +| `gen_ai.input.messages` | string | The content being evaluated and the questions asked about it, as a one-element array holding an evaluation object. Shown in the decision **Input** view. See [Decision Input Messages](#decision-input-messages). | `'[{"type": "evaluation", "state": "...", "questions": {"accuracy": {"type": "score", "criteria": ["incorrect", "correct"]}}}]'` | +| `gen_ai.output.messages` | string | The answers, as a one-element array holding an evaluation object. Shown in the decision **Output** view and as the one-line result in the transcript and timeline. See [Decision Output Messages](#decision-output-messages). | `'[{"type": "evaluation", "answers": {"accuracy": {"type": "score", "score": 1}}}]'` | + +### Decision Input Messages + +`gen_ai.input.messages` holds a one-element array: `[{"type": "evaluation", "state", "questions"}]`. Copy `state` and the questions as the caller passed them. If the provider uses a different layout, convert it to this layout (see [Mapping Provider Payloads](#mapping-provider-payloads)). `` is the question key. + +| Field | Type | Required | Description | Example | +| :------------------------------ | :--------------------- | :----------------------- | :--------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------ | +| `type` | string | Yes | Always `"evaluation"`. | `"evaluation"` | +| `state` | any | Yes | The content being evaluated: a string, an object, or a message array, as the caller passed it. | `"I was charged twice and want a refund."` | +| `questions` | object | Yes | The questions, keyed by question name. | `{"urgency": {...}}` | +| `questions..type` | string | Yes | `"boolean"`, `"predicate"`, `"noul"`, `"choice"`, or `"score"`. Copy other values as they are. | `"choice"` | +| `questions..instructions` | string | Yes | The question text. | `"Which team should handle this?"` | +| `questions..criteria` | object or string array | For `choice` and `score` | For `choice`, an object that maps each option to its description. For `score`, an ordered array of labels. The index is the score. | `{"billing": "Charges and refunds"}`, `["low", "high"]` | +| `questions..descriptions` | object | No | Only for `score`. Maps a score index (as a string) to the description of that level. Omit the levels that have no description. | `{"1": "Blocks the customer"}` | + +### Decision Output Messages + +`gen_ai.output.messages` holds a one-element array: `[{"type": "evaluation", "answers"}]`. `answers` uses the same keys as `questions`, and each answer has a `type` that matches its question. + +| Field | Type | Required | Description | Example | +| :----------------------------- | :----- | :---------------------------- | :---------------------------------------------------------------------------------------------------- | :------------------------------------- | +| `type` | string | Yes | Always `"evaluation"`. | `"evaluation"` | +| `answers` | object | Yes | The answers, keyed by question name. | `{"urgency": {...}}` | +| `answers..type` | string | Yes | The type of the question. | `"score"` | +| `answers..probability` | number | For `boolean` and `predicate` | The probability that the answer is yes. Shown as Yes when it's 0.5 or higher. | `0.97` | +| `answers..noul` | number | For `noul` | The probability that the answer is yes. Jev's variant of `boolean`. | `0.98` | +| `answers..choice` | string | For `choice` | The selected option. | `"billing"` | +| `answers..score` | number | For `score` | The score. It can be a fraction between two levels. | `1.58` | +| `answers..probabilities` | object | No | Only for `choice` and `score`. Maps each option, or each score index as a string, to its probability. | `{"billing": 0.64, "technical": 0.36}` | +| `answers..legend` | object | No | Only for `score`. Maps each score index (as a string) to its label. | `{"0": "low", "1": "high"}` | +| `answers..confidence` | number | No | Only for `choice` and `score`. The confidence of the answer. | `0.28` | + +Omit optional fields when the provider doesn't report them. A field with the wrong type makes that question or answer unreadable, and Sentry shows it raw. + +### Mapping Provider Payloads + +TypeSafe Jev, Cloudflare Clef, and Mastra use the layout above, so copy their `state`, `questions`, and `answers` as they are. Other providers need these changes: + +- **OpenAI** (`client.decisions.create`): `input` becomes `state`. `questions` and `answers` are arrays, so key each item by its `name` and remove `name` (use the array index as the key when an item has no name). For `choice` questions, `choices: [{value, description}]` becomes `criteria: {: }`. For `score` questions, `levels: [{label, description}]` becomes `criteria: [