Skip to content

feat(openai-agents)!: emit invoke_agent, chat and execute_tool spans - #33

Draft
apucacao wants to merge 3 commits into
ag/py-telemetry-openai-messagesfrom
ag/py-telemetry-openai-agents
Draft

feat(openai-agents)!: emit invoke_agent, chat and execute_tool spans#33
apucacao wants to merge 3 commits into
ag/py-telemetry-openai-messagesfrom
ag/py-telemetry-openai-agents

Conversation

@apucacao

@apucacao apucacao commented Aug 11, 2026

Copy link
Copy Markdown

Replaces one flat span per call with the tree the TypeScript SDK emits, for openai-agents.

invoke_agent                     one per call, carries the LD identity and the run total
├── chat {model}                 one per model turn, with that turn's own tokens
└── execute_tool {name}          one per tool call, a sibling of chat

The per-turn data was already in hand: this handler walked the Runner's raw responses to sum usage, and simply never opened a span per turn.

Abandoning the stream was costing money

The old path iterated the Runner's event stream with no cleanup at all. Breaking out of that loop only stops us reading: the Runner's own background task keeps calling the model and spending tokens until told to stop.

Teardown now cancels the streamed run as well as closing the span tree.

Two things specific to this handler

  1. gen_ai.response.model stays the requested name, on both the root and the chat spans, which is deliberately different from openai-messages (feat(openai-messages)!: emit invoke_agent, chat and execute_tool spans #32). The TypeScript twin has never resolved the answering model here and no test pins it, so reporting one would invent behaviour rather than match it.

  2. Finish reasons are derived from the Responses API's status rather than mapped through the shared table, which does not apply: there is no finish_reason field to map. A function call in the output takes precedence over status, because a live capture put completed on every turn including the six that stopped to call a tool.

Other changes

Cached tokens are now reported, read from the cached-tokens detail and left out of the input total, because OpenAI already counts them inside it. Cache creation is always zero.

Breaking change

The span is renamed from openai.agent.run to invoke_agent. Queries selecting on the old name will not match. Prompt and completion content is no longer on spans unless the caller passes capture_content=True.

Where this sits

Needs the usage layer (#28) and the content layer (#29). Independent of the other five handler PRs; the stack orders them only because gh stack is linear.

Tests: 781 to 799.


Note

Overview
Replaces the flat openai.agent.run span with the TypeScript-aligned tree: an invoke_agent root, one chat {model} child per model turn, and sibling execute_tool {name} spans per tool call—driven via agents.RunHooks rather than a Model wrapper.

Breaking: span rename (openai.agent.runinvoke_agent), and prompt/completion/tool content is off by default unless capture_content=True.

Also cancels abandoned streamed runs (previously the Runner kept spending tokens after the consumer stopped), reports cached-token usage, and writes failed-run spend onto the root without double-counting the SDK aggregate.

Reviewed by Cursor Bugbot for commit 169ab0c. Bugbot is set up for automated code reviews on this repo. Configure here.

@apucacao

Copy link
Copy Markdown
Author

bugbot run

Comment thread packages/openai-agents/src/launchdarkly_ai_openai_agents/handler.py Outdated
@apucacao
apucacao force-pushed the ag/py-telemetry-openai-agents branch from efe4f71 to aec3684 Compare August 11, 2026 20:43
@apucacao

Copy link
Copy Markdown
Author

bugbot run

Comment thread packages/openai-agents/src/launchdarkly_ai_openai_agents/handler.py
@apucacao
apucacao force-pushed the ag/py-telemetry-openai-agents branch from aec3684 to 60c2410 Compare August 11, 2026 21:01
@apucacao

Copy link
Copy Markdown
Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 60c2410. Configure here.

One flat span named openai.agent.run becomes the tree the TypeScript SDK emits:
an invoke_agent root, one `chat {model}` child per model turn, one
`execute_tool {name}` child per tool call. The per-turn data was already in
hand: this handler walked the Runner's raw responses to sum usage, and simply
never opened a span per turn.

BREAKING CHANGE: the span this handler emits is renamed from `openai.agent.run`
and `openai.agent.run.stream` to `invoke_agent`. Queries selecting on the old
names will not match. Prompt and completion content is no longer on spans
unless the caller passes capture_content=True.

Cached tokens are now reported, read from the cached-tokens detail and left out
of the input total, because OpenAI already counts them inside it. Cache creation
is always zero.

gen_ai.response.model stays the requested name here, on both the root and the
chat spans, which is deliberately different from openai-messages. The
TypeScript twin has never resolved the answering model in this handler and no
test pins it, so reporting one would invent behaviour rather than match it.

Finish reasons are derived from the Responses API's status rather than mapped
through the shared table, which does not apply: there is no finish_reason field
to map. A function call in the output takes precedence over status, because a
live capture put `completed` on every turn including the six that stopped to
call a tool.

Abandoning the stream needed more than ending our spans. The old path iterated
the Runner's event stream with no cleanup at all, and breaking out of that loop
only stops us reading: the Runner's own background task keeps calling the model
and spending tokens until told to stop. Teardown now cancels the streamed run
as well as closing the span tree, so an abandoned stream stops costing money.

Tests: 53 to 71.
Two sources describe the same spend and they overlap. The run hooks add each
turn as it finishes, so by the time the run raises they already hold every
completed turn, and the exception carries the SDK's own aggregate over those
same turns. The error path added the aggregate to the accumulator, so any run
that failed after paid turns reported roughly twice what it cost.

MaxTurnsExceeded does that by definition, which makes this the common case
rather than an edge one. A three-turn run reporting 70 input tokens reported
140.

The aggregate is the authoritative figure, so it now replaces the accumulator
rather than adding to it, matching what the TypeScript handler does. When the
error carries no aggregate, which is what a tool handler's own error looks
like, the accumulator is all there is and is used instead. Neither having
anything still writes nothing, because all-zero attributes would assert the run
cost nothing.

Three tests, one per branch. The double-count one fails with 140 against 70
when the fix is reverted, which is how I checked it pins the bug rather than
the behaviour.

Found by Bugbot on #33, severity High.
The wrapper never passed capture_content to the factory, so it stayed in kwargs
and reached config(), which takes no such argument. A caller asking for content
on spans got a TypeError rather than content.

Lifted out alongside variables, which was already handled the same way and for
the same reason: one configures the handler, the other belongs to the
invocation, and config() accepts neither.

Two tests, one per branch, asserting the flag reaches the factory and does not
reach config().

Found by Bugbot on #33. It flagged this handler; five of the six wrappers have
it, and the other four are fixed in their own layers.
@apucacao
apucacao force-pushed the ag/py-telemetry-openai-agents branch from 60c2410 to 169ab0c Compare August 11, 2026 21:18
@apucacao

Copy link
Copy Markdown
Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 169ab0c. Configure here.

@apucacao

Copy link
Copy Markdown
Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 169ab0c. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant