feat(sleep): harvest Codex CLI rollout sessions (sessions/YYYY/MM/DD)

[loegaire]

Summary

SkillOpt-Sleep's codex transcript source only supports the Codex Desktop layout: a flat ~/.codex/archived_sessions/*.jsonl directory. Machines running the Codex CLI (which is most CLI-only setups) never archive sessions there — the CLI writes nested rollout files under ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl — so harvest_codex() returns zero sessions and the auto source silently falls back to Claude transcripts. This PR makes the harvester support both layouts.

Changes

• harvest_codex() now collects candidates from the flat Desktop archive and recursively walks the sibling sessions/ tree (derived from codex_home, so the existing --codex-home / codex_home config keeps working).
• digest_codex_archived_session() deduplicates adjacent (role, text) pairs. CLI rollouts record every turn twice — once as response_item/message and once as event_msg/user_message / event_msg/agent_message (adjacent in either order) — which previously double-counted n_user_turns / n_assistant_turns and duplicated prompts. Non-adjacent repeats (a user legitimately sending the same message twice) are preserved.
• The existing record parser already handles CLI rollout payload types (user_message, agent_message, message with role, exec_command_end, patch_apply_end, ...) and strips <environment_context> / <user_instructions> wrappers, so no parsing changes were needed beyond the dedup.

Testing

• New tests/test_harvest_codex_cli.py: nested-layout discovery, desktop+CLI combination, turn dedup, <environment_context> dropping, tool-name extraction, and the no-messages → None case.
• Full suite: python -m pytest -q → 1499 passed, 11 skipped, 353 subtests passed.

Verification on a real Codex CLI install

Before: skillopt-sleep harvest --source codex → 0 sessions (217 rollout files, 1.5 GB on disk). After: full scan digests 120 sessions in ~14 s and mines 39 tasks with the mock backend; a nightly run completes end-to-end against the real Codex backend.

[Copilot]

🟡 Changes recommended

Reset deduplication across skipped records and strengthen assistant-turn assertions.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds Codex CLI rollout harvesting alongside existing Desktop archive support.

Changes:

• Recursively discovers nested CLI sessions.
• Deduplicates mirrored user and assistant messages.
• Adds coverage for discovery, parsing, tools, and empty sessions.

File summaries

File	Summary
tests/test_harvest_codex_cli.py	Tests CLI rollout harvesting behavior.
skillopt_sleep/harvest_codex.py	Supports both layouts and message deduplication.

Review details

Suppressed comments (1)

tests/test_harvest_codex_cli.py:77

• The fixture includes an event_msg/response_item assistant duplicate, but the assertions only verify user-turn deduplication. A regression that double-counts or duplicates assistant turns would still pass; please assert the assistant count and final text as well.

    assert d.n_user_turns == 2
    assert d.user_prompts == ["fix the failing tests", "wrong flag, try again"]

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Yifan Yang (Yif-Yang)]

Thank you for extending discovery to the current Codex CLI rollout layout.

Reviewed 2f14366a06e2.

skillopt_sleep/harvest_codex.py:230: The newly discovered CLI tree also contains SkillOpt's own codex exec rollouts. With the actual prompt captured from CodexCliBackend.attempt() and written into a synthetic sessions/2026/09/30/rollout-replay.jsonl, this change harvests it as user evidence. This still reproduces after locally combining #287: its markers miss the current normal attempt prompt. Please coordinate with #286/#287 to exclude engine-generated sessions and add a nested-layout provenance regression before enabling this source. The existing dedup review is separate and still applies.

[loegaire]

loegaire please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree