@file mentions are injected at the top of the conversation and invalidate the prompt cache

[brdloush]

@file mentions are injected at the top of the conversation and invalidate the prompt cache

Human summary 👨

I discovered yet another source of cache busting (precious snapshots being thrown out) 🎉

1. I have a conversation of multiple turns, so far so good
2. then I say something like what do you think about @file1.clj

• this places the with its inlined file content to the chat, positioned basically just after a system prompt. ie: very early
• if you already have like 100k of context, this results in very early (close to the tip) edit

3. that's the culprit -> similar issue as with cache busting the cursor position: rewriting the "chat history" too close to the tip

IMO this problem is very relatable to the issue #464

More info in this part of slack thread:

https://clojurians.slack.com/archives/C093426FPUG/p1783317730264019?thread_ts=1783281642.707959&cid=C093426FPUG

The rest of this ticket is a summary of my investigation together with claude Fable + Opus.

I would consider the Proposed fixes part as just a possible inspiration. I didn't fact-checked those proposals, as I don't have knowledge of the two referenced code-bases. Just wanted to have some more framing regarding what seems to be happening for my case.

The rest should be solid (I've read it and it's correctly based on multiple inputs I fed the agent & observations / hypotheses we proved)

Summary 🤖 + 👨

When a file is @-mentioned in a chat, eca embeds its content as a <file> block
inside the cached static system-prompt prefix, at the very top of the
conversation — immediately after the system prompt. Adding (or refreshing) an
@-mention mid-session therefore mutates the prompt prefix and shifts every
token after it, invalidating the entire downstream KV/prefix cache.

This is the same class of bug as the cursor-position issue fixed in
#464 ("Deliver cursor
context per-turn", commit 1c8e011d9): volatile/late-arriving content must not
be placed in the cached prefix. File contexts were left in the static prefix and
have the same effect.

Per one of ECA's coauthors: rewriting the history — appending just after the
system prompt — looks like a bug and should not happen even for the @ variant. Late-arriving file
content belongs at the conversation tip, not the prefix.

Impact

• Local inference (llama.cpp / recurrent + hybrid SSM models): a full prompt
  re-prefill on every mid-session @-mention. On the reporter's setup
  (Qwen3.6-27B hybrid SSM+attention, RX 7900 XTX) this is ~60–70 s per mention at
  40k+ tokens. Recurrent SSM state cannot be partially rewound, so any early-
  prefix divergence rolls back to the nearest context checkpoint and re-prefills
  everything after it.
• Cloud providers: prefix caching (discounted cached-input pricing, e.g.
  Anthropic/OpenAI) is invalidated by the same top-of-conversation mutation, so
  mid-session @-mentions silently cost API users money.
• Resume: structured contexts are per-request and are not persisted into
  the stored turn, so on /resume the reconstructed static prefix no longer
  contains the <file> — the referenced content silently disappears from
  history.

Background: the static/dynamic prompt split

eca splits the system prompt into a cacheable static prefix and a per-turn
dynamic section, and (since #464) delivers volatile editor state (cursor) in
the user message rather than the prefix. File contexts, however, are still
classified as static:

• src/eca/features/prompt.clj:202-209 — static-prompt-context-types #{:file :agents-file :repoMap} and static-prompt-context?.
• src/eca/features/prompt.clj:257-318 — build-static-instructions renders all
  static-prompt-context? contexts into a ## Context block at the end of the
  cached static prefix (lines 313-318), i.e. the top of the conversation.
• src/eca/features/chat.clj:70-90 — static-prompt-cache-signature hashes the
  static contexts (:static-contexts, line 85). Adding a new @file changes the
  signature, so the whole static prefix is rebuilt (chat.clj:1485-1499) with the
  new <file> embedded up front → the cached prefix is invalidated.

A cache-safe path already exists for the same content:

• src/eca/features/chat.clj:1502-1505 — expanded-prompt-contexts are appended
  to the end of the user message (Route B below).
• eca's file-read tool results also land at the conversation tip. Session logs
  show dozens of tool-driven reads at sim_best ≈ 0.99, ~1.5 s turns — no busts.

The @-mention path takes equivalent content and places it in the one location
guaranteed to invalidate everything downstream.

Root cause: two ingestion routes, only one is cache-safe

The eca server accepts a file two ways on chat/prompt:

• Route A — structured contexts[] (ChatPromptParams.contexts):
  context.clj:129 raw-contexts->refined → :file is static-prompt-context?
  → static prefix (build-static-instructions). Cache-busting; not persisted;
  lost on resume.
• Route B — inline @/abs/path text in message: context.clj:164-179 contexts-str-from-prompt (regex #"@([/~\.][^\s:]+)…") → appended to the end
  of the user message. Cache-safe; persisted in the turn.

eca-emacs: @ vs #

The eca-emacs client exposes two prefixes (eca-chat-context.el:55-63):

• @ = eca-chat-context-prefix ("context"). Completions insert a string
  carrying the eca-chat-context-item text property
  (eca-chat-context.el:196-199, exit fn :451-460). At send time
  eca-chat--extract-contexts-from-prompt (eca-chat.el:1896-1908) collects those
  into the structured :contexts[] array (eca-chat.el:1910-1935). → Route A →
  static prefix. Inline @ also leaves an @/abs/path token in the message
  (normalize-prompt, eca-chat.el:1881-1883), which the server additionally
  expands via Route B — so an inline @ can embed the file twice (prefix +
  user message).
• # = eca-chat-filepath-prefix ("filepath"). Completions insert a string
  with eca-chat-item-type 'filepath but no eca-chat-context-item
  (eca-chat-context.el:201-214, exit fn :462-470), so it never enters
  :contexts[]. normalize-prompt strips the # and inserts the bare path
  only (eca-chat.el:1878-1880, literal comment "Removes # from #files"). No
  file content is embedded server-side; the agent must read the path with a tool.

So in eca-emacs, @ is the cache-busting variant and # is a plain path
reference. (Path style is irrelevant here — the client always sends the resolved
absolute path via eca--path-local-to-remote.)

Reproduction

Prerequisite: a session with enough prior context to have a populated cache
(e.g. ~30k+ tokens after several turns).

1. Have a healthy, cache-hitting session going (tool calls, edits, etc.).
2. Send a message that @-mentions a file, e.g.
   I changed @/abs/path/to/file.clj, take a look.
3. Observe: the next request re-prefills from an early divergence point instead
   of appending at the tip.

Payload-level confirmation (promptoscope)

Captured with promptoscope: the
@-mentioned file is embedded as

<file path="/abs/path/to/file.clj">;; file contents…</file>

immediately after the system prompt, at the very top of the conversation.
Every additional @-mention mutates that same top block, so every mention
re-busts.

llama.cpp evidence

Healthy turn (append at tip):

slot get_availabl: selected slot by LCP similarity, sim_best = 0.994 (> 0.100 thold), f_keep = 0.997
prompt eval time = 946.95 ms / 209 tokens

Turn immediately after an @-mention (prefix divergence + full re-prefill):

slot get_availabl: selected slot by LCP similarity, sim_best = 0.296 (> 0.100 thold), f_keep = 0.395
prompt processing, n_tokens = 2048, progress = 0.05 ...
... (full re-prefill of ~44k tokens, ~60+ s)

Reading the fingerprint: the prompt grew (~33k → ~44k) while sharing only its
first ~13k with the cached state — not an append (divergence would be at the tip)
and not a compaction (prompt would shrink). ~11k tokens were inserted starting
at position ~13k, which is exactly where the static preamble ends and the file
block begins. Two separate @-mentions in the same session produced the identical
signature; zero busts occurred on any other turn type (tool calls, edits, even a
user-interrupted generation all recovered at sim_best ≈ 0.99).

Proposed fixes (ranked)

Mirror the #464 cursor fix: deliver file content per-turn at the conversation
tip, never in the cached prefix.

1. Render @-mentions as a synthetic tool-call/result pair at the tip — reuse
   the existing read-tool formatting (already proven cache-safe in the logs).
   Minimal new serialization; content also gets persisted with the turn, so it
   survives resume.
2. Append the <file> block to the user message that mentioned it — reuse the
   existing expanded-prompt-contexts path (chat.clj:1502-1505) for Route-A file
   contexts too; drop :file from static-prompt-context-types. Optionally dedup
   with a per-chat "already delivered" set (like :last-editor-state in Move @cursor from system prompt to the user prompt #464) so a
   pinned file that the client re-sends each turn isn't re-appended.
3. Non-fix to avoid: keeping the top block but appending new files to its end.
   Any growth of that block still shifts every token after it. Top placement is
   only safe for content fixed before turn one.

Note the coauthor's guidance: option 3 is explicitly not acceptable — even the
@ variant should not rewrite history. The content should land at the tip.

Workarounds until fixed

• Mid-session, prefer #file (path-only reference) or plain words ("look at
  src/…/common.clj, I changed the CSS loading") and let the model read the file
  via its tool — lands tip-side, cache survives. # does not auto-embed content.
• @-mentions in the first message of a session are harmless (nothing cached
  yet).
• Server-side damage control (bounds the bust, cannot prevent it): deeper
  checkpoint coverage, e.g. llama.cpp --checkpoint-min-step 1024 --ctx-checkpoints 40 (~40k-token recovery window) so a checkpoint survives at
  ≤ the divergence point and the rebuild is incremental rather than from zero.

Code references

eca (server)

• src/eca/features/prompt.clj:202-209 — static-prompt-context-types /
  static-prompt-context? (:file classified static).
• src/eca/features/prompt.clj:257-318 — build-static-instructions, ## Context
  block at 313-318.
• src/eca/features/prompt.clj:336-343 — build-editor-state-context (the Move @cursor from system prompt to the user prompt #464
  per-turn delivery pattern to mirror).
• src/eca/features/chat.clj:70-90 — static-prompt-cache-signature
  (:static-contexts).
• src/eca/features/chat.clj:1466-1505 — refined-contexts, static/dynamic
  instructions, and the cache-safe expanded-prompt-contexts.
• src/eca/features/context.clj:129-179 — raw-contexts->refined (Route A) and
  contexts-str-from-prompt (Route B).
• Prior art: commit 1c8e011d9 "Deliver cursor context per-turn" (Move @cursor from system prompt to the user prompt #464).

eca-emacs (client)

• eca-chat-context.el:55-63 — @/# prefix defcustoms.
• eca-chat-context.el:196-199, :451-460 — @ → eca-chat-context-item →
  structured context.
• eca-chat-context.el:201-214, :462-470 — # → filepath only, no context.
• eca-chat.el:1866-1887 — normalize-prompt (strips #, re-prepends @).
• eca-chat.el:1896-1908 — extract-contexts-from-prompt.
• eca-chat.el:1910-1935 — send-prompt assembling :message + :contexts.

Appendix: reporter environment

• llama.cpp mainline (router mode). Flags (excerpt): -c 262144 -ub 512 -ctk q4_0 -ctv q4_0 --spec-type draft-mtp --checkpoint-min-step 256 --ctx-checkpoints 32 --cache-ram 8000 --no-mmproj --reasoning off --jinja --kv-unified --parallel 1.
• Model: nilayparikh/Qwen3.6-27B-Text-NVFP4-MTP-GGUF (Qwen3.6-27B hybrid
  SSM+attention, NVFP4, 18.3 GiB, MTP speculative-decoding head).
• GPU: AMD RX 7900 XTX 24 GB (gfx1100), Vulkan/RADV, KHR_coopmat. Ubuntu 26.04.
• Note: the recurrent SSM state cannot be partially rewound, which is what turns
  an early-prefix divergence into a full re-prefill; the checkpoint pool
  (min-step 256 × 32) only covered the last ~8–11k tokens, so no checkpoint
  survived at the ~13k divergence point.