Skip to content

Recognize native Codex and Claude Code telemetry in Agents - #33

Merged
VibeSWE merged 4 commits into
mainfrom
feat/agents-native-harness-fields
Oct 13, 2026
Merged

VibeSWE merged 4 commits into
mainfrom
feat/agents-native-harness-fields

Conversation

@VibeSWE

@VibeSWE VibeSWE commented Oct 13, 2026 •

Copy link
Copy Markdown

Summary

Codex and Claude Code are the coding agents people most often point at Parseable, but both export OTLP traces with their own attribute names:

  • Codex: the dataset never appeared in Agents. Its spans have no gen_ai.agent.*, gen_ai.operation.name, gen_ai.request.model or gen_ai.tool.name, so discovery skipped it.
  • Claude Code: runs appeared because model-call spans carry gen_ai.request.model. Tool calls had no name, token usage showed as not recorded, and the run was named after the raw span (claude_code.interaction).

This PR teaches the Agents view to recognize both harnesses and read their native fields.

What changes

  • Detection (lib/agentHarnesses.ts): a dataset is recognized from schema fields specific to each harness (codex.turn.token_usage.input_tokens / codex.turn.phase, interaction.sequence / llm_request.context), together with span_name. One dataset can hold both harnesses; each row is mapped by its own harness (Codex rows are those whose service.name starts with codex). Logs datasets are still not listed.
  • Mapping: derives the GenAI fields the view already reads.
    • Codex: session_task.turn becomes the agent span, try_run_sampling_request the model calls, and spans with tool_name the tool calls. Per-request usage is lifted from handle_responses to its model call.
    • Claude Code: claude_code.interaction becomes the agent span, claude_code.llm_request the model calls (input usage = input_tokens + cache_read_tokens + cache_creation_tokens, left unknown when input_tokens is missing or malformed), and claude_code.tool the tool calls (tool_name).
    • Recorded values are never replaced, and absent usage stays unknown.
  • Query: Codex traces a turn with about 475 internal spans (file reads, rollout writes…), which would use up the 1,000-record limit after about two runs. For Codex services the run query keeps only turn, model-call, stream, usage and tool spans, plus any span that already carries GenAI agent fields; rows from other services are untouched. The record-limit banner reflects the queried row count, not the rows left after mapping. Runs are seeded from Codex turns.
  • Tree: Codex's intermediate spans aren't loaded, so model and tool calls are attached to their nearest loaded ancestor, or to the turn whose time range contains their start. A call that no single loaded turn contains (for example one from a turn that is still running) is left alone and keeps the missing-parent warning. Whichever parent field a row uses (span_parent_span_id or parent_span_id) is rewritten.
  • Details panel: shows a "Mapped from native … telemetry" note on spans that had fields derived or their parent rewritten. Recorded fields and Parent span ID still show exactly what the harness stored.
  • Setup guide and agent brief: no longer tell users to map these two harnesses by hand.

Evidence

The fixtures are trimmed from real exports into a local Parseable: Codex CLI 0.162.1 (app server and codex exec) and Claude Code 2.1.296 with CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1. Running this branch's query and mapping against that data produced:

Dataset Before After
codex-traces not discovered 4 runs named Codex: turn → model calls + exec tool; tokens match Codex's own turn totals (e.g. 32,558 in / 38 out); 28 rows loaded instead of thousands
claude-code-traces runs without tools or usage Claude Code run: 2 model calls + Bash tool; 42,554 in / 154 out including cache tokens
codex-logs not discovered not discovered (logs aren't runs)

Testing

  • npm run format:check, tsc --noEmit, vite build
  • npm test: 2,350 passed, 14 skipped. agentHarnesses.test.ts covers detection (including mixed datasets), SQL (including that absent columns are never referenced and GenAI-field spans are kept), Codex usage lifting, time-range turn attachment across several turns, cross-service parents, both parent fields, Claude cache-inclusive and unknown usage, the mapping note, and recorded fields staying unchanged.
  • Playwright agents.spec.ts + agent-setup.spec.ts on a Blacksmith testbox: 26 passed. New specs: native Codex traces → discovered dataset, run row, tree, mapping note and recorded parent ID; record-limit banner when Codex mapping drops queried rows. The other Agents specs (agents-navigation, agents-review-*) also passed; one test failed once on cold-start Vite re-optimization and passed on rerun.

Limitations

  • Only traces are mapped. Codex/Claude Code logs (prompts, tool results) are not joined into the run conversation yet.
  • Claude Code's standalone session-title request has no interaction span, so it shows as its own small run named claude_code.llm_request.
  • Codex's nested code-mode tool executions are exported in a separate trace with no turn span. They don't appear as runs; the exec call inside the turn represents them.
  • Field names are tied to the versions above; a harness that renames its spans would fall back to the current generic behaviour.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Codex and Claude Code export OTLP traces with their own attribute names,
so the Agents view either skipped their datasets (Codex) or showed runs
without tool names or token usage (Claude Code).

Detect both harnesses from the dataset schema and derive the GenAI span
fields the view reads: the turn becomes the agent span, model calls and
tool calls get their operation, model, tool name and usage. Claude Code
input usage includes cache reads and writes. Recorded values are never
replaced, and the details panel still shows the fields as recorded.

Codex traces a turn with hundreds of internal spans, so the run query
keeps only turn, model-call and tool spans from Codex services, and the
mapping attaches model and tool calls to their turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@comment-clanker comment-clanker Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 0febb045707ee9be014ee8ab71cdf2f0850d1b51 with two GPT-6.1-Sol and two Claude Opus 5.5 reviewers, each covering a different area. Actionable findings are in the threads below.

What holds up: datasets with only generic OTel GenAI telemetry get byte-identical SQL and untouched rows. Codex usage lifting from handle_responses matches the turn totals, recorded values are never overwritten, "Recorded fields" and "Parent span ID" read the original rows, and the turn total isn't double-counted. agentHarnesses.test.ts passes. The edge cases were reproduced with scratch scripts against the PR's functions; Playwright and live ingestion were not rerun.

Comment thread frontend/src/lib/agentHarnesses.ts Outdated
Comment thread frontend/src/lib/agentHarnesses.ts Outdated
Comment thread frontend/src/lib/agentHarnesses.ts Outdated
Comment thread frontend/src/features/agents/AgentsPage.tsx
Comment thread frontend/src/lib/agentHarnesses.ts Outdated
Comment thread frontend/src/lib/agentHarnesses.ts Outdated
Comment thread frontend/src/lib/agentHarnesses.ts
VibeSWE and others added 3 commits October 12, 2026 23:22
…afely

Address review findings on native Codex and Claude Code telemetry:

- Detect every harness in a dataset. Claude Code mapping always runs, and
  Codex mapping, dropping and reparenting apply only to rows from Codex
  services (or, without a service name, to a Codex-only dataset).
- Keep Codex spans that already carry standard GenAI fields, such as tool
  spans whose tool_name a collector renamed to gen_ai.tool.name.
- Attach a detached Codex call to a turn only when it started during
  exactly one loaded turn, so calls of an unexported turn stay detached and
  multi-turn traces attach each call to its own turn.
- Rewrite whichever parent field a row uses (span_parent_span_id and/or
  parent_span_id).
- Leave Claude Code input usage unknown unless input_tokens parses.
- Show the "Mapped from native telemetry" note only on rows something was
  derived for, and drive the record-limit banner from the queried row count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Map Codex rows in place among all loaded rows so a turn whose parent
belongs to another service keeps that parent. Keep Codex stream and
response spans that carry GenAI agent fields themselves, as the run
query already does, and lift usage only from the ones that are dropped.
Treat rows without a service name as Codex unless they are Claude Code
spans, so mixed datasets without service names still map Codex turns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Codex response span kept for its own GenAI fields lost its dropped
stream parent and was attached to the turn beside its model call, so its
usage was counted twice. Walk dropped parents up to the nearest kept
ancestor first. In mixed datasets, take a row without a service name to
be Codex only if it has a Codex span name or a Codex tool call, so
generic spans keep their recorded parent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VibeSWE added a commit that referenced this pull request Oct 13, 2026
…ness-conversation-logs

Bring in the review fixes from #33. Harness detection now returns a list;
conversation logs stay joined for a single detected harness until the
page handles mixed datasets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@VibeSWE
VibeSWE merged commit 20120c1 into main Oct 13, 2026
7 of 9 checks passed
@VibeSWE
VibeSWE deleted the feat/agents-native-harness-fields branch October 13, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant