Skip to content

Join Codex and Claude Code logs into Agents run conversations - #34

Merged
VibeSWE merged 6 commits into
mainfrom
feat/agents-harness-conversation-logs
Oct 13, 2026
Merged

VibeSWE merged 6 commits into
mainfrom
feat/agents-harness-conversation-logs

Conversation

@VibeSWE

@VibeSWE VibeSWE commented Oct 13, 2026 •

Copy link
Copy Markdown

Stacked on #33. Review that first; this PR's diff is against its branch.

Summary

#33 builds Codex and Claude Code runs from their traces. The prompts, responses and tool results are exported separately, as log records, so the Conversation tab stayed empty. Both harnesses stamp each log record with the trace and span it happened in, so the join can be automatic.

What changes

  • Logs dataset pairing (lib/agentHarnessLogs.ts): when the selected traces dataset holds Codex or Claude Code traces, Agents scans the workspace once for logs datasets of either harness (Rescan datasets repeats the scan; Refresh doesn't). Schemas that return 403/404 are skipped; other failures are shown with Retry instead of reporting that no logs dataset exists. It recognizes Codex logs by conversation.id + originator, and Claude Code logs by prompt.id + event.sequence, each with trace_id and event.name. It picks the one named like the traces dataset (codex-traces → codex-logs). A Conversation logs selector shows the choice and can turn the join off (logs=none in the URL). The choice is cleared when switching traces datasets, and a logs= value that isn't a candidate for the current dataset is ignored. A traces dataset with both harnesses offers logs of either and joins the selected one with its own harness.

  • Query: one extra query fetches only conversation events (event.name IN …) for the loaded runs' trace IDs, within the same time bounds and record limit, newest first, so the most recent runs keep their events. A note says when the limit was reached.

  • Attachment, in the order things happened:

    Codex Claude Code
    User prompt turn span: the turn whose recorded parent is the codex.turn_input span the prompt is logged on, else the turn containing it in time; ambiguous prompts stay unattached interaction span
    Assistant response not exported by Codex its model call, by request_id
    Tool input arguments on the tool span, by call_id within the trace tool_input on the tool span, by tool_use_id within the trace
    Tool output output on the tool span the tool span's tool.output trace event (shown even without a logs dataset)
    Approval decision tool span, by call_id tool span, by tool_use_id
  • Redacted content keeps its recorded value and gets a "(redacted)" label, so it's clear the exporter withheld it rather than the dashboard.

  • Unloaded runs: records for runs outside the current view are ignored, never guessed. Codex's code-mode child tool calls run in a separate trace with no turn, so they're skipped; the outer exec result already includes their output.

  • Conversation tab and search: the tab lists spans in time order, and run search now matches conversation text.

  • Setup guide: says to export logs to their own dataset as well.

Evidence

Captured with content flags on for a probe run: codex exec -c otel.log_user_prompt=true, and Claude Code with OTEL_LOG_USER_PROMPTS, OTEL_LOG_ASSISTANT_RESPONSES, OTEL_LOG_TOOL_DETAILS and OTEL_LOG_TOOL_CONTENT. Running this branch's discovery, queries and attachment against the local codex-traces/codex-logs and claude-code-traces/claude-code-logs produced:

Codex
  [session_task.turn] User prompt: "Run: cat sample.txt  then reply with just the file contents."
  [exec] Tool arguments · exec: "const r = await tools.exec_command({cmd:\"cat sample.txt\", …
  [exec] Tool output · exec: "Script completed\nWall time 0.2 seconds\nOutput:\n\nhello\n"
Codex (earlier turn, prompt logging off)
  [session_task.turn] User prompt (redacted): "[REDACTED]"
Claude Code
  [claude_code.interaction] User prompt: "Run cat sample.txt with the Bash tool, then reply …"
  [claude_code.tool] Tool decision · Bash: "accept (config)"
  [claude_code.tool] Tool input · Bash: "{\"command\":\"cat sample.txt\", …}"
  [claude_code.tool] Tool output · Bash: "hello"
  [claude_code.llm_request] Assistant response: "hello"

Testing

  • npm run format:check, tsc --noEmit, vite build
  • npm test: 2,389 passed, 14 skipped. New agentHarnessLogs.test.ts, with fixtures trimmed from the real logs, covers detection, discovery (skip, unchecked and rethrow paths), pairing, quoted newest-first SQL, Codex/Claude attachment and ordering, per-turn prompt attribution, trace-scoped tool calls, repeated events, redaction labels, trace-only Claude tool output, and ignoring unloaded runs.
  • Playwright on a Blacksmith testbox, agents.spec.ts + agent-setup.spec.ts: 32 passed. New specs: the paired logs dataset (selector default, Conversation tab, log query, search, turning logs off), switching traces datasets with logs set, Refresh without rediscovery and Rescan picking up a new logs dataset, the logs record-limit note, a failed logs scan with Retry, and a mixed traces dataset.
    • In local runs of all five Agents spec files with one worker, a few different tests fail each run, from "Target crashed" or a LogsPage dynamic import failing while Vite re-optimizes dependencies. origin/main fails the same way on this machine (2 failed / 42 passed, including a "Target crashed"), and the failing tests pass on rerun. I've treated this as local dev-server instability, not a regression; CI should confirm.

Limitations

  • Codex doesn't export assistant reply text, so Codex conversations show the prompt and tool activity only.
  • What appears depends on each CLI's content settings. With defaults, prompts and responses are redacted. Codex exports tool arguments and output even with log_user_prompt = false.
  • Logs exported after the page loaded appear on refresh.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Both harnesses export prompts, responses and tool results as log records
stamped with the trace and span they happened in. For a harness traces
dataset, Agents now finds the matching logs dataset (preferring the one
named like the traces dataset), queries only conversation events for the
loaded runs' trace IDs, and attaches them to their spans in order:

- Codex: the user prompt on its turn; tool arguments and output on the
  tool span, matched by call ID.
- Claude Code: the user prompt, assistant responses on their model call
  (by request ID), tool input and approval decisions on the tool span
  (by tool use ID), and tool output from the tool span's trace event.

Redacted content is labelled rather than hidden, and records for runs that
are not loaded are ignored. A "Conversation logs" selector shows the paired
dataset and can turn the join off; search also matches conversation text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@comment-clanker comment-clanker Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 64925c231ffdc0df6334cf653b57a960a11cb027 (the diff against #33's 0febb045) with two GPT-6.1-Sol and two Claude Opus 5.5 reviewers, each covering a different area. Actionable findings are in the threads below.

What holds up: generic GenAI datasets and demo mode send no extra requests, the SQL quoting is sound, stale responses can't overwrite newer ones, and a failed logs query leaves runs rendering. agentHarnessLogs.test.ts passes (8/8). The edge cases were reproduced with scratch scripts against the PR's functions; Playwright and live ingestion were not rerun.

Comment thread frontend/src/lib/agentHarnessLogs.ts Outdated
Comment thread frontend/src/features/agents/AgentsPage.tsx
Comment thread frontend/src/features/agents/AgentsPage.tsx Outdated
Comment thread frontend/src/lib/agentHarnessLogs.ts Outdated
Comment thread frontend/src/lib/agentHarnessLogs.ts
Comment thread frontend/src/lib/agentHarnessLogs.ts Outdated
Comment thread frontend/src/lib/agentHarnessLogs.ts Outdated
VibeSWE and others added 5 commits October 12, 2026 23:39
…ness-conversation-logs

Bring in the review fixes from #33. Harness detection now returns a list;
conversation logs stay joined for a single detected harness until the
page handles mixed datasets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Address review findings on joining harness logs into run conversations:

- Query logs newest first so the most recent runs keep their events when
  the record limit is reached, and say when the limit was reached.
- Discover logs datasets once for all harnesses, keyed on a stable
  "needed" flag rather than the reloading run data, and reload it from
  Rescan datasets. Skip 403/404 schemas, report other failures with
  Retry, and only say no logs dataset exists when the scan succeeded.
- Drop the logs choice when switching traces datasets, and ignore a logs
  param that is not a candidate for the current dataset's harnesses.
  Mixed traces datasets offer logs of every detected harness and join
  the selected one with its own harness.
- Attach prompts to the turn whose time range contains them, scope tool
  call IDs to their trace and prefer the record's own tool span, and
  keep repeated log events while still collapsing flattened span events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Match a log event to the harness's turn spans, so an enclosing agent span
of another service no longer makes every prompt ambiguous. An event inside
exactly one turn goes to that turn; a Codex prompt otherwise goes to the
turn starting within 100 ms after it. An event after the only loaded turn
is left unattached, since it may belong to a turn not exported yet, and
untimed events fall back to the only turn rather than the trace root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex logs a prompt on the turn's input span, which the turn records as
its parent. Match prompts through that parent first, unless several turns
share it, and keep the timing rule only as a fallback. Only the harness's
own turns are candidates, and turns that start together leave a prompt
unattached instead of picking one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Other events logged on a span a turn records as its parent are placed by
their time, as before, instead of being taken to belong to that turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@VibeSWE
VibeSWE changed the base branch from feat/agents-native-harness-fields to main October 13, 2026 05:23
@VibeSWE
VibeSWE merged commit bd20b04 into main Oct 13, 2026
7 of 8 checks passed
@VibeSWE
VibeSWE deleted the feat/agents-harness-conversation-logs branch October 13, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant