[REQUEST] Make the special-tag parser code-span aware, and delete the backtick sanitizer

[j15z]

Is your feature request related to a problem? Please describe.

sanitizeChatDisplayContent (chat-content/chat-sanitize.ts) mutates the backticks around a <workspace_resource> tag before the parser runs. It exists because the parser is blind to backticks: it extracts a tag wherever it finds one, including inside an inline code span, which leaves the span's delimiters wrapped around a hole.

Use `<workspace_resource>{"type":"file","path":"a.md","title":"a.md"}</workspace_resource>` like so.
  ->  Use ` [chip] ` like so.        two orphaned backticks

So the sanitizer is not really about backticks. It is cleaning up after a decision the parser already made, using a weaker signal than the parser had. To do that it has to re-derive "is this a real tag" from backtick adjacency, and that guess is where the bugs come from.

This produced four separate corruption bugs on #5952, each one a fix for the last:

Shape	What broke
A prose mention — The `<workspace_resource>` tag	Opening backtick stripped, closing one left unpaired, opening a code span that inverted every span in the rest of the message
A message writing the opener and closer as two backticked spans	Read as one wrapped tag, outer pair stripped
A neighbouring span two words away	Pattern started at one span's delimiter and finished at another's
A span sitting flush against the tag	Delimiter consumed as if it were the tag's wrapping

Plus a quadratic (154ms per call at 168KB, on the main thread, per streamed chunk) introduced while fixing the first one.

All are fixed as of #5952, and the three patterns are now one left-to-right pass. But the recurrence is the point: four arrangements, each correct for the cases it was written for, each breaking a different one. Two independent reviewers on that PR flagged it as an altitude problem rather than a run of bad luck.

Describe the solution you'd like

Move the decision into the parser and delete the sanitizer's tag handling.

If a tag sits inside an inline code span, the model is showing syntax, not referencing a resource. Honour that: do not extract the tag at all — render the span as code, tag text and all.

	Today	Proposed
Sanitizer	mutates backticks so the parser's output does not strand them	tag patterns deleted; keeps only the unrelated hidden-reference rule
Parser	blind to backticks	skips a tag inside a code span

Nothing then touches backticks, so there are none to unbalance. Every shape in the table above stops being fixed and becomes impossible. It also removes the second, weaker notion of "what a tag looks like" — the parser already knows exactly where each tag begins and ends.

It fixes an asymmetry too: the sanitizer only knows workspace_resource, while the parser knows seven tags. A backticked <question> or <credential> has the same problem today and no handling at all.

The hard part, so it is not a surprise: mid-stream, "am I inside a code span" is undecidable in exactly the way "will this tag close" is.

Use `<workspace_resource>{"type":"file","path":"a.md"}</workspace_resource>
                                                                    ^ stream is here

That opening backtick may close after the tag, or may be an unpaired stray. Either guess retracts on the other case:

• Guess "not in a span" -> render the chip -> the closing backtick arrives -> it becomes code. A rendered card un-renders, which is the defect class fix(copilot): stop the special-tag parser discarding text it cannot resolve #5952 spent several rounds removing.
• Guess "in a span" -> hold it -> no closing backtick ever arrives -> release late.

The second is the safer guess and matches what the parser already does with its pending outcome: hold when the evidence is not in yet. But it means an open backtick becomes another kind of pending state inside the streaming state machine, which is the part of this code with the subtlest invariants.

Accepted behaviour change: a real tag that the model wrapped in backticks stops rendering a chip and renders as code. That is the intended semantics of this change rather than a regression — the model asked for code — but it is a visible difference worth agreeing on before starting.

Describe alternatives you've considered

• Leave it as one pass (status quo). Correct across thirteen verified shapes today. The objection is not that it is broken now, it is that the next edit has no structural reason to stay correct — four previous arrangements each passed their own tests.
• Widen the sanitizer to all seven tag names. A one-line change that fixes the asymmetry but keeps the duplicate notion of a tag, so it inherits the same failure mode on six more tags.
• Strip backticks inside the parser rather than skipping the tag. Removes the second implementation but keeps overriding the model's explicit "render this as code" signal, and still has to decide what a stray backtick means.

Additional context

Depends on a prerequisite that is worth doing on its own: special-tags.tsx is 'use client' end to end, so its parser cannot be imported by server code — which is why lib/mothership/inbox/executor.ts carries a separate, weaker <thinking>-stripping regex rather than reusing the parser. Extracting the parser into a plain .ts module is a mechanical move with no logic change, and it unblocks both this and that duplication.

Suggested order:

1. Extract the parser out of the 'use client' module (no logic change)
2. Teach it to skip tags inside code spans; delete the sanitizer's tag patterns
3. Migrate inbox/executor.ts onto the shared parser

The groundwork for step 2 is already in place. #5952 added four property invariants over generated messages — no character lost, every valid tag renders whatever surrounds it, no card un-renders and no text retracts across streamed frames, and the settled parse never shows less than the last streaming frame. Those are exactly the invariants a code-span-aware streaming parser must not violate, and each was validated by reverting a real bug and confirming the property catches it. Characterize first, restructure second — the characterization is done.

Relevant code: apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/chat-content/chat-sanitize.ts, .../special-tags/special-tags.tsx (parseSpecialTags, resolveTagAt, classifyBody), apps/sim/lib/mothership/inbox/executor.ts.