Skip to content

feat(compaction): Summarize with the agent model and cached request prefix - #2602

Merged
thomasjiangcy merged 2 commits into
mainfrom
compaction-summary-main-model
Nov 3, 2026
Merged

thomasjiangcy merged 2 commits into
mainfrom
compaction-summary-main-model

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Capacity compaction now gets its summary from the active agent model. Before, it sent the whole history to openai/gpt-6-luna with cacheRetention: "none". Big compactions (~170k input tokens) took ~12s p50 and read nothing from the cache.

Request

buildAgentModelRequest (new chat/agent/context-summary.ts) builds the request that the Pi agent loop would send for the same messages: the agent's transformContext → convertToLlm, state.model, systemPrompt, tools, thinkingLevel, sessionId, thinkingBudgets, transport, and API key. The summary call adds one user message with the instruction. Pi does not export its request builder, so this code repeats streamAssistantResponse and createLoopConfig. A unit test runs a real Pi Agent and checks that the built request matches what the loop passed to streamFn. If Pi changes these steps, the test fails.

  • tool_choice, thinking, and effort stay the same, because Anthropic drops cached messages when they change. The prompt forbids tool calls instead.
  • Only max_tokens changes (8k). For budget models, Pi adds the thinking budget on top, so the budget does not change. For adaptive models (Opus 5.5), thinking counts against the 8k.
  • The call goes through the agent's traced stream with gen_ai.prompt.name: junior.context_compaction. Its gen_ai.usage.* attributes include cache read and cache creation.

Prompt

The prompt says: do not continue the task, do not call tools, and return only <summary>…</summary>. It asks for fixed sections: Goal / Constraints & preferences / Progress (Done, In progress, Blocked) / Key decisions / Open asks (in the user's exact words) / References (≤20) / Next steps. Each item is a one-line bullet, and the whole summary stays under 1,000 words. It leaves out content that is added again anyway. In update mode, it merges an earlier summary, and newer history wins.

Validation

These count as failures: a tool call, a length stop, empty output, or a missing tag. The first failure gets one nudge (instruction + nudge, so the cached prefix still applies). A second failure throws, and the history stays untouched. As before, it throws only if the history is already at the input limit. This path has no slice(0, 6000) anymore, so Junior never stores a truncated summary.

Turn start

Slack's pre-agent maybeCompact is removed. It had no agent request to reuse. The agent already compacts before its first model request on every surface (web already worked this way). A new Turn's compaction keeps the old replacement shape: up to 20k tokens of recent user messages from committed history, with their original provenance and source events, then the summary, then the new Turn's context and instruction. The new instruction is left out of the selection if the Turn committed it before compaction. Mid-turn replacement is unchanged: the instruction, then the summary.

Handoff compaction (fast model, old prompt) and thresholds are unchanged.

Telemetry

Rebased on current main (#2599, #2600, #2601). The agent's compaction before the first model request (applyActiveContextCompaction → compactActiveContextIfNeeded with mode: "blocking") now carries #2599's instrumentation:

  • One agent.context_compaction span per compaction, with app.compaction.mode = blocking (before the first model request), active (between model calls), or handoff. The summary gen_ai.chat span (gen_ai.prompt.name: junior.context_compaction) is its child, so you can group cache reads by mode.
  • app.turn.compaction_wait_ms on chat.turn still counts this wait, because the agent runs inside the Slack chat.turn span.
  • TELEMETRY.md now says that blocking runs in the agent, and app.compaction.retained_messages counts all history messages kept with the summary.

mode replaces the earlier retainUserMessages flag. A blocking compaction with a new instruction keeps recent user messages. This is the same behavior as before.

#2601's read-back eval and the new probe evals are both in continuity.eval.ts.

Tests / evals

  • Unit: tests/unit/agent/context-summary.test.ts covers request equality with the real Pi loop, the single appended message, and the nudge-then-fail path.
  • Component: a new case checks that earlier user asks stay before the summary at Turn start (provenance kept, the committed new instruction not duplicated). The retained-message selection tests are kept. Other compaction tests are updated.
  • A full-wiring integration test that compares provider bytes passed locally. I dropped it because check-test-architecture forbids new agent-running tests in packages/junior/tests.
  • New probe evals in evals/conversation/continuity.eval.ts ask "what's still open", "which PR", and "what failed" after a long thread is compacted. Not run: the sandbox has no model keys. They need a run and recordings.

via @thomasjiangcy.

--

View Session [Sentry]

@vercel

vercel Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
junior-docs Ready Ready Preview Nov 3, 2026 12:33pm UTC

Request Review

@github-actions github-actions Bot added the risk: high PR risk score: high label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

5 screenshot changes — 5 changed · 0 added · 0 removed

Review screenshots in Frameshift

Conversation Attachment · Desktop
Conversation Attachment · Desktop
Changed
Conversation Attachment · Mobile
Conversation Attachment · Mobile
Changed
Conversation Attachment Modal · Desktop
Conversation Attachment Modal · Desktop
Changed
Conversation Subagents · Desktop
Conversation Subagents · Desktop
Changed
Settings Mcp · Desktop
Settings Mcp · Desktop
Changed

…refix

Capacity compaction asks the active agent model for the summary. The
request is the agent request for the same messages plus one user message
with the summary instruction, so the provider reads the cached prompt
prefix. The output must be a complete <summary> block; one nudge, then the
full history stays. The fast model and the 6,000 character cut are gone
for this path.

Slack no longer compacts before the agent starts. The agent compacts
before its first model request and keeps recent user messages, the
summary, then the new instruction. Handoff compaction is unchanged.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e71a23c. Configure here.

The Turn commits its context before its instruction, and the selection
checked one message at a time. A context message then looked like a user
ask, took the retained budget, and repeated after the summary.
@thomasjiangcy
thomasjiangcy merged commit 436e67c into main Nov 3, 2026
50 of 51 checks passed
@thomasjiangcy
thomasjiangcy deleted the compaction-summary-main-model branch November 3, 2026 13:25

This branch was successfully deployed

1 active deployment
Preview – junior-docs — bc4463a4 Deployed Nov 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: high PR risk score: high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant