fix(copilot): stop the special-tag parser discarding text it cannot resolve

[j15z]

Summary

The chat renderer parses seven inline special tags out of the assistant's streamed text and turns them into interactive cards and chips: <workspace_resource> (a clickable workflow/table/file chip), <question> and <options> (selection cards), <credential> (a secret input or connect control), <usage_upgrade> and <mothership-error> (notices), and <thinking> (reasoning, suppressed from the rendered output). Anything it could not turn into a tag, it used to throw away.

Two shapes cost the user real text:

• An opening tag with no close suppressed everything after it for the rest of the stream. Simply mentioning a tag name in prose — "the <workspace_resource> chip needs a real path" — blanked the remainder of the message until the stream ended.
• A matched pair whose body failed to parse was dropped silently, along with everything between the tags. When the model misspelled a closing tag, the opener reached forward and matched a later tag's close, so several paragraphs became "the body" and vanished. The message resumed mid-sentence.

resolveTagAt now resolves every opening tag to one of four named outcomes. Each section below gives a failing message, when the parser detects it, how, and what the reader ends up seeing.

1. segment — the body parsed; render the card

The interesting case is a tag that looks broken mid-stream but is not. A closing marker arrives one character at a time, so a valid tag spends several frames looking like a payload with junk after it.

mid-stream   Ready when you are. <options>[{"title":"Ship it","description":"Open the PR"}]</opt
complete     Ready when you are. <options>[{"title":"Ship it","description":"Open the PR"}]</options>

• Where it's caught: nowhere. No rule fires on any frame — this is the outcome that must not trigger a false alarm.
• What keeps it from being caught: the JSON value already closed at }], so the trailing </opt would otherwise look like stray content and trip the viability rule below. dropArrivingClose strips a trailing fragment that is a prefix of this tag's own closing marker, so </opt reads as a close in transit. It only strips at the very end of the buffer, so a genuinely wrong close such as </workflow_resource> is not a prefix of </workspace_resource> and still trips the rule immediately.
• What the reader sees: Ready when you are. and a shimmer while </options> finishes arriving, then the options card. The payload JSON is never on screen.

2. literal — provably not a tag; render the span verbatim

The `<workspace_resource>` chip only renders when the path points to a real file.

• Where it's caught: on the first frame containing the complete opener — one character into the body, without waiting for the stream to end.
• What catches it: the JSON-viability rule. Every tag body except <thinking> must be JSON, and this one begins with a backtick, which cannot start a JSON value. The same rule fires later in a body the moment its top-level value closes and non-whitespace follows — which is how a misspelled close, a truncated close, and a complete payload trailed by prose with no close are all caught by one check rather than by pattern-matching each mistake.
• What the reader sees: the whole sentence as ordinary markdown, with no delay. Previously everything from the backtick onward stayed blank until the stream finished, then appeared at once.

<thinking> is the exception, since its body is prose. There the test is that tags never nest: a marker for another known tag in the body proves the opener was text. Tag names specifically, not anything tag-shaped — reasoning that mentions <div> or Promise<void> is still reasoning. The rule applies whether or not the close arrives, so a late </thinking> cannot retroactively swallow content the stream had already put on screen.

Rejection resumes inside the message rather than abandoning it, so a genuine tag later in the same reply still renders as a card.

3. discard — a payload that failed its shape guard; drop it

Here are the options. <question>{"type":"single_select"}</question> Let me know.

• Where it's caught: at the closing tag. This outcome is unreachable until one arrives — with no close, a broken body can only end up as outcome 2 or 4.
• What catches it: the <question> type guard, running after JSON.parse already succeeded. The body is valid JSON, but a question needs a prompt and at least one option.
• What is deliberately NOT caught here: a body that will not parse at all. Dropping text is only defensible for a payload the agent actually formed, and bracket depth cannot tell {the Q4 report} — prose someone wrapped in braces — from a real payload. A body that fails JSON.parse falls through to outcome 2 and is shown, which also covers the two commonest slips a model makes, unquoted keys and single quotes.
• What the reader sees: Here are the options. Let me know. The payload is dropped rather than printed, because showing a reader raw JSON is worse than showing nothing. This is the only outcome that intentionally removes text, and a message consisting solely of one renders nothing rather than falling back to the raw content.

4. pending — still streaming and a close is plausible; hold the remainder

Before I start, one question: <question>[{"type":"single_sel

• Where it's caught: nowhere yet, deliberately. This is the only outcome that is not terminal — it always resolves into one of the other three.
• What catches it: nothing does, and that is the correct answer. The body is an open JSON array, so viability still holds, and the parser genuinely cannot tell a broken payload from a half-arrived one: [{"type":"single_sel is one keystroke from a valid card. Releasing it early would print raw JSON that snaps into a card a frame later.
• What the reader sees: Before I start, one question: and a shimmer. If the payload completes it becomes a card; if the stream ends first it falls to outcome 2 and the text is shown in full. Nothing is lost either way — the cost of being held is a delay, never deleted text.

How the decision is structured

Deciding this well turned out to need three separate questions, and answering them in one function is what made the first several rounds of this PR regress each other. They are now split:

	Answers	Knows nothing about
inspectWithin	how much of a body may be read	what it is, where to resume
classifyBody	what the body is — one of five closed cases	positions, outcomes, resume
resumeForClass	where scanning continues	what the outcome is

The closed set is the point: resolveMatchedPair and resumeForClass each switch over it exhaustively, so adding a case fails to compile until both questions are answered for it. Verified by adding a sixth case — tsc reports exactly two errors, one per switch.

That refactor is behaviour-identical, not merely believed to be: diffed against the previous implementation over 7,500 comparisons (1,500 generated messages, each parsed streaming, settled, and at three mid-stream cut points), asserting deep equality of the whole result.

What these checks cost

These checks are not free. Deciding whether a body is still viable JSON means reading it — twice: once to blank out the contents of JSON strings, once to walk it counting bracket depth.

It gets worse for two reasons. The parser re-runs on every streamed chunk, over the whole message received so far. And it runs on the main thread, so the time lands directly in a frozen chat window.

With no limit, one check is O(body length) — and a misspelled closing tag makes a single body stretch across most of the message. So a reply of length n that mentions tags k times costs O(k·n) per parse, paid again on every chunk. That is what froze the tab.

Capping how much of a body gets inspected makes each check constant work instead. The limit is 4096 characters, comfortably above the largest payload these tags emit — a <question> card runs about 1,500 characters, a <workspace_resource> about 100 — so it never changes the verdict for real content. With it, the worst shapes we could construct (a 58 KB reply with a borrowed close, and a 68 KB reply mentioning tag names 360 times) parse in under 35 ms.

Two cheaper guards do the rest: opener and closing-tag lookups are memoized per parse rather than rescanning to the end of the buffer for every opener, and both the viability check and the string-blanker return early when a single character already settles the answer.

The limit has one blind spot, stated in its docstring and pinned by a test: a JSON body whose value closes beyond 4096 characters, followed by prose and no closing tag, still looks like a viable prefix inside the window, so it waits for the stream to end instead of settling mid-stream. It stays lossless, and it needs a payload several times larger than any tag emits. A tag name mentioned in prose — the case that actually happens — settles at its first character no matter how long the message is.

One more fix, outside the parser

Preserving this text exposed a second bug in the display sanitizer that runs before the parser. It strips a backtick next to a <workspace_resource> tag so a code span cannot stop a chip rendering — but it could not tell a real tag from prose mentioning the tag name, and stripping a mention's opening backtick left the closing one unpaired. That opens a markdown code span running to the next backtick, inverting every code span in the rest of the message.

It now pairs backticks the way markdown does — a backtick to the next one on the same line — and unwraps a span only when it genuinely contains a complete tag. Pairing is what makes two further cases correct structurally rather than as separate patches: an adjacent unrelated code span, and a fenced block containing a tag. A negative lookahead bounds the scan, which also removed a quadratic in the previous version.

This was only reachable because of the parser fix: until now a prose mention blanked the rest of the message, so the mangled formatting was never on screen.

Two deliberate trades

• An unclosed <thinking> is hidden while streaming and shown once the stream ends. Hiding is the right mid-stream default — a close is still plausible, and the body renders as nothing anyway. Showing it at the end does surface reasoning when the close never arrives; that is accepted, because the alternative swallows the answer whenever the model opens the tag and then writes its reply without closing it.
• A resource whose title or path contains a backtick renders as text rather than a chip. The sanitizer tells a real tag from a mention by requiring no backtick inside the payload, so a payload that has one is not matched. That costs one chip and is rare; the failure it replaces corrupts a whole message and is common.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• Other: ___________

Testing

162 tests across the touched area, plus tsc and Biome clean.

Coverage is built around failure shapes rather than line count:

• Four malformed messages taken verbatim from production replies are pinned as regression tests: a matched pair whose prose was swallowed by a later backticked example, a truncated closing tag, a matched pair wrapping prose instead of a payload, and no closing tag at all. The last is asserted lossless — every character survives mid-stream, so nothing waits for the stream to end.
• Four property invariants over generated messages, composed from fragments and seeded so a failure reproduces: no character is lost from a message containing nothing droppable; every valid tag renders as a card whatever surrounds it; across streamed frames a card never un-renders and text never retracts; the settled parse never shows less than the last streaming frame. Each was validated by reverting a real fix and confirming the property catches it.
• Complexity is pinned, not just correctness. Two tests assert that pathological inputs stay linear. Every fixture was under 1 KB before, which is exactly why two separate quadratics went unnoticed.
• Each behavioural fix is mutation-checked: reverting it fails its test, and only its test. Where a fix could not be pinned that way — one only shifts a text-segment boundary, and adjacent text segments are concatenated before rendering — the test was deleted rather than kept as decoration, and the invariant documented in the code.

What reviewers should focus on:

• Where a rejected tag resumes scanning. The rejection reasons deliberately resume at different offsets, and collapsing them passes every test while silently deleting text, so both are pinned.
• Rejected spans are emitted as several text segments rather than one. Display-neutral, since the renderer concatenates them; tests assert the joined text, which is what a reader sees.
• The sanitizer runs before the parser and is the one piece here that duplicates the parser's knowledge of what a tag looks like.

Known follow-ups, deliberately not in this PR: extracting the parser out of this 'use client' module (it cannot be imported by server code today, which is why the inbox executor carries its own weaker <thinking> regex), and moving backtick handling into the parser so the sanitizer's separate notion of "what is a tag" goes away.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Screenshots/Videos

No visual change to any component — the difference is which text survives parsing and renders correctly. The before/after for each failure shape is given as rendered output in the Summary above.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Jul 28, 2026 5:35am

[cursor]

PR Summary

Medium Risk
Touches hot-path streaming parsing and pre-render sanitization for all assistant messages; behavior is heavily tested but edge cases (scan window, <thinking> tradeoffs) remain documented.

Overview
Fixes silent text loss in copilot chat when assistant messages mention special tags (<workspace_resource>, <question>, etc.) or contain malformed tag markup.

Special-tag parser (parseSpecialTags) is refactored so each opener resolves to one of four outcomes: render a card (segment), show the span as plain text (literal), drop only valid JSON that fails shape guards (discard), or hold while streaming (pending). Unclosed openers no longer blank the rest of the message; prose mentions and non-JSON bodies are shown instead of discarded; borrowed/wrong closes resume scanning at safe offsets; JSON viability, arriving-close tolerance, and a 4096-character inspection window plus memoizedIndexOf keep streaming parses bounded on the main thread.

Display sanitizer (sanitizeChatDisplayContent) replaces multi-pass backtick stripping with a single left-to-right pass that unwraps only real <workspace_resource> tags (or stray flank backticks) without unpairing backticks when the model only mentions the tag name in prose or when code spans/fences sit nearby.

Tests add production-shaped regressions, frame-replay streaming invariants, property tests over generated messages, and performance ceilings for pathological inputs.

^{Reviewed by Cursor Bugbot for commit 7dc9148. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

This PR makes streamed special-tag parsing lossless for unresolved content.

• Adds explicit segment, literal, discard, and pending resolution paths for malformed and incomplete tags.
• Prevents quoted tag markers inside JSON strings from bypassing payload discard behavior.
• Reworks workspace-resource backtick sanitization to preserve ordinary code spans and prose mentions.
• Adds regression, streaming-invariant, property, and complexity coverage for parser and sanitizer behavior.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure related to the previous review thread remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/chat-content/chat-content.test.ts	Adds sanitizer regressions for prose mentions, adjacent code spans, fenced blocks, stray backticks, and pathological repeated openers.
apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/chat-content/chat-sanitize.ts	Replaces independent backtick-stripping expressions with a bounded left-to-right matcher that distinguishes code spans from complete workspace-resource tags.
apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/special-tags/special-tags.test.ts	Adds extensive malformed-tag, streaming-stability, payload-classification, scan-bound, memoization, and property-based regression coverage.
apps/sim/app/workspace/[workspaceId]/home/components/message-content/components/special-tags/special-tags.tsx	Refactors special-tag resolution into explicit outcomes, preserves unresolved literal text, blanks JSON strings before marker detection, and bounds body inspection work.

_{Reviews (8): Last reviewed commit: "fix(copilot): sanitize backticks in one ..." | Re-trigger Greptile}

[j15z]

@greptile

[j15z]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c61dd9d. Configure here.}

[j15z]

@greptile

[j15z]

@cursor review

[j15z]

@greptile

[j15z]

@cursor review

[j15z]

@greptile

[j15z]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 7dc9148. Configure here.}

[j15z]

Follow-up filed as #6001 — make the parser code-span aware and delete the sanitizer's tag handling, with the 'use client' parser extraction as its prerequisite.

Both are named in the description; the issue carries the reasoning, the four corruption shapes that motivated it, and the mid-stream complication (whether a tag sits inside a code span is undecidable while the span is still open, the same way a tag's close is).