Show streamed Expert tool calls, reasoning and progress messages by andypalmi · Pull Request #8895 · FlowFuse/flowfuse

andypalmi · 2026-10-13T13:03:03Z

Closes FlowFuse/engineering#547
Closes FlowFuse/engineering#549
Closes FlowFuse/engineering#551
Closes FlowFuse/engineering#553
Closes FlowFuse/engineering#554
Closes FlowFuse/engineering#556

Part of FlowFuse/engineering#545.

The Expert chat shows a turn's tool calls, reasoning and progress messages as they happen, inside the reply bubble, instead of only a tool-call count with the final answer. Streamed events arrive on the existing expert:status-message inflight topic with a kind field (tool-record, reasoning or message), so no new MQTT topics or ACL changes are needed.

Adds

supportsStreamedActivity in the chat context. Expert only streams to clients that send it; older Expert versions ignore it and keep today's behaviour.
Store handling for streamed events (product-expert.js): each event is placed in the turn's reply bubble in seq order, an event with a known id updates its item in place, and the final answer merges into the same bubble by id, so nothing is duplicated.
A guard that drops a late update older than the one shown, or one that would reopen a finished item.
TimelineItem.vue: one collapsible card per tool call. The header is the reason for the call on one line, and expanding it shows the full reason, the tool, its input and output.
ReasoningBlock.vue: a collapsed "Thinking" / "Thought" block with one line of the reasoning under it, the latest sentence while streaming and the first once done.
AgentMessage.vue: progress messages from the Expert as assistant bubbles between the cards.
streamedActivity.js: the shared list of streamed and activity kinds.

Changes

AiMessage.vue renders tool records, reasoning and messages in order above the answer. Insights answers and saved conversations use the same tool cards; Insights does not stream.
The loading indicator keeps its three dots while a turn streams but drops the status lines under them. Turns that do not stream keep both.
CollapsibleSection.vue gains a summary slot under the header.

Removes

ToolCalls.vue and ToolCallItem.vue, the old "N tool calls" strip, replaced by the cards.

Tests

Store: streamed items are inserted by seq, updated by id, merged with the final answer, and late updates are ignored.
Components: tool cards render for streamed, Insights and saved answers; the reasoning preview shows one line; the loading indicator hides its status lines when asked.

codecov · 2026-10-13T13:18:37Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.29%. Comparing base (111f500) to head (e8883b7).
⚠️ Report is 13 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8895      +/-   ##
==========================================
+ Coverage   79.08%   79.29%   +0.20%     
==========================================
  Files         476      476              
  Lines       25890    25912      +22     
  Branches     6886     6896      +10     
==========================================
+ Hits        20475    20546      +71     
+ Misses       5415     5366      -49

Flag	Coverage Δ
backend	`79.29% <ø> (+0.20%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…expanded

n-lark

Hey just a couple things good work. Two things that look deliberate but don't match the issues this closes:

#549 asks for the tool name in the header with the reason inside
#551 asks for the reasoning block to be open while text arrives and collapse when it ends.

Fine to keep the implementation as is. Just worth a note or updating the issues.

An update replaced the item with its new seq but left it in place, so the bubble was no longer sorted and a later insert could land above it. The item now keeps its first seq, the newest update's seq is tracked separately for the late-update check, and fields an update leaves out are kept.

The header is now a button with aria-expanded, so it can be focused and toggled from the keyboard. A collapsible prop renders a static header with no chevron or toggle when there is nothing to expand.

Tool cards show input and output in the shared JsonViewer instead of their own highlighter and toggles. A card with no reason, error or data has nothing to expand, so it shows no chevron.

andypalmi · 2026-10-14T20:20:13Z

Thanks! Updated FlowFuse/engineering#549 and FlowFuse/engineering#551 to match what shipped: the reason heads the tool card, and the reasoning block stays collapsed with a one-line preview so the chat does not jump while it streams.

Adds an align prop and a plain kind to ff-button so it can render a left-aligned header with no background or padding.

Steve-Mcl

when a turn leaves _inFlightRequests without a final answer (Stop, handleMqttError, Start over), set done on its bubble's open items, or add a stopped marker.
Alternatively, have ReasoningBlock only show "Thinking" while the turn is live. It's a small store change plus one test. Below is a screenshot of what happens if convo is stopped during thinking (note the partial streamed character "I" at the end:

A few smaller things, none blocking:

_streamed now means two things. On streamed items it means "arrived over MQTT", and updateAnswerStreamedState uses it to mean "typewriter finished". hasStreamedActivity relies on the first meaning. It works today, but a separate flag (_live?) would be harder to break.
The bubble is placed at its first event. A system or event message pushed mid-turn lands below it, and later cards keep growing the bubble above that notice, so the order on screen stops matching the order things happened.
A tool card with no result yet looks the same as a finished one, since ok undefined isn't treated as failed. Does the agent send a start event before the result? If so, a running state on the card would help.

A stopped turn marks its open items done, so a reasoning block no longer shows Thinking forever. Streamed items carry their own _live flag instead of sharing _streamed with the typewriter. Activity that follows a message pushed mid-turn starts a new bubble below it, and final-answer tool records still upgrade their card wherever it is.

andypalmi · 2026-10-15T10:29:48Z

Thanks! Done in d7a1553:

When a turn ends without a final answer (Stop or a connection error), its open items are marked done, so a stopped reasoning block shows "Thought" instead of "Thinking". Start over clears the chat, so it needs nothing.
Streamed items now carry their own _live flag, and hasStreamedActivity checks that. _streamed is back to meaning only that the typewriter has finished.
Activity that arrives after a system or event message landed mid-turn now starts a new bubble below that message, so the chat keeps the order things happened in. Updates and final-answer tool records still find their card in whichever bubble it is in.
There is no start event: the Expert sends the tool record once the call returns, so every card already has its result. A running state would need the Expert to send an event when a call starts. That could be a follow-up if we want it.

Steve-Mcl

local testing now good. Approving from an operational perspective - so I am asking that we wait for another tick from one the FE folk before you merge pls.

cstns · 2026-10-15T12:35:43Z

Reviewing now

cstns

Ran this on the branch itself against the dev Expert through the FlowFuse central broker. Streamed reasoning, tool records and progress messages render in order, the final answer merges in by id, Stop, reload, keyboard access and Insights cards all behave, and the full frontend suite is green. One thing worth fixing inline (the hover on the Input/Output toggles), the rest are small notes.

cstns · 2026-10-15T13:00:43Z

+            </div>
+            <!-- Input/output are dev-only today, but render whenever present so a future
+                 flag flip needs no UI change. -->
+            <json-viewer v-if="hasInput" :value="parseJson(item.input)" label="Input" collapsible default-collapsed />


The Input/Output toggles come out as full tertiary buttons here, so hovering them fills the row with the accent colour and fades it in over 0.3s. Inside a card that reads like a bug. Might be worth giving the viewer a plain variant for this spot, or scoping the hover away in this component.

Done in e8883b7. JsonViewer takes an opt-in toggleKind, and the tool card passes plain, so the toggles no longer fill on hover. Other viewers are unchanged.

cstns · 2026-10-15T13:00:43Z

+    if (typeof value !== 'string') return false
+    const trimmed = value.trim()
+    if (trimmed === '' || trimmed === AUTOMATION_PLACEHOLDER) return false
+    return !trimmed.startsWith('invoke_')


On Insights the tool line under the reason comes out as mcp_tool__flowfuse-mcp-gateway-0__invoke_read_tool, since toolTitle is empty and toolName equals toolId. The guard only catches names that start with invoke_. Not a regression, but it defeats the idea of hiding wrappers. Maybe match invoke_ anywhere, or fall back to input.tool_name for gateway calls.

Done in e8883b7. invoke_ is now rejected after a server prefix too, and a gateway call falls back to input.tool_name.

cstns · 2026-10-15T13:00:43Z

+                this._insertBySeq(target, { ...item, _uuid: uuidv4(), _streamed: false })
+            }
+            const { answer: _answer, ...fields } = response
+            Object.assign(target, fields)


Small robustness one: if a reasoning block is still done: false when the final answer lands, it stays "Thinking" for good and survives a reload. The Expert does send done before the answer, so I couldn't hit it for real, but calling _closeTurnItems(transactionId) here too would make an Expert error path harmless.

Done in e8883b7. The final answer now marks any open items done as well.

cstns · 2026-10-15T13:00:44Z

+            return typeof this.item.error === 'string' ? this.item.error.trim() : ''
+        },
+        // First usable name: friendly title, else the tool's own name, else the raw key.
+        title () {


The old strip showed per-call and total ms and the records still carry durationMs, but nothing displays it now. Fine if that's on purpose, just checking it wasn't dropped by accident.

On purpose. In support the time per call isn't useful to the user. Insights is where it might help, so we can add it there if people want it.

cstns · 2026-10-15T13:00:44Z

-            }
-
-            return null
+        activity () {


Small UX note, not a bug: a paginated turn gave me ten identical "List Hosted Instances" cards with nothing to expand, ten rows where the old strip was one collapsed line. Could be worth folding consecutive identical cards without a reason into one, or leaving it as a follow-up.

Every tool call now has to carry a reason, in support and Insights alike, so each paged call should get its own card with its reason and something to expand. Cards with nothing to expand would come from a turn on an Expert from before that change, or from an older saved conversation. Could you check which it was? If it still happens on a fresh turn, I'll look into it.

Try with Expert on Cloud, it should be there @cstns

cstns · 2026-10-15T13:00:44Z

            </li>
            <li v-if="isWaitingForResponse">
-                <expert-loading-indicator />
+                <expert-loading-indicator :show-messages="!hasStreamedActivity" />


Nit: after Stop, hasStreamedActivity goes false as soon as the in-flight map clears, so the loader comes back with the generic rotating lines under a bubble that was just streaming. Cosmetic, and the stopping indicator predates this PR.

Done in e8883b7. A turn that is still stopping keeps the loader to its dots.

n-lark

LGTM for my fixes but please address Serban's feedback as well.

JsonViewer takes an opt-in toggleKind, and the tool card uses the plain kind so its Input and Output toggles no longer fill with the accent colour on hover. A gateway wrapper call is named after the tool it invoked. The final answer finishes any item still open, and a stopping turn keeps the loader to its dots.

andypalmi added 7 commits October 13, 2026 12:38

Route streamed activity on the Expert status topic into the reply bubble

9c1302d

Render Expert tool calls as collapsible cards

5e21fe3

Show streamed Expert reasoning as a collapsible block

76b80b8

Show intermediate Expert messages as assistant bubbles in order

16759a1

Store streamed tool records under their own record kind

4691884

Cover tool record cards for Insights and saved answers

679afd5

Remove the unused Expert tool-call strip components

a95e217

andypalmi requested review from Steve-Mcl and cstns October 13, 2026 13:03

andypalmi self-assigned this Oct 13, 2026

andypalmi deployed to staging October 13, 2026 13:06 — with GitHub Actions Active

andypalmi added 8 commits October 13, 2026 15:49

Hide the Expert loading indicator once the turn streams activity

23655d4

Advertise streamed Expert activity with one capability flag

d4c632b

Lead Expert tool cards with the reason for the call

fe2b006

Keep streaming Expert reasoning on one line instead of opening it

fee3cbc

Keep Expert tool card reasons on one line and show them in full when …

2b1551f

…expanded

Show one line of Expert reasoning under its header

61bcac2

Ignore late streamed updates that would roll back Expert activity

55ef515

Keep the Expert loading dots while a turn streams, without status lines

9133790

andypalmi deployed to staging October 14, 2026 07:48 — with GitHub Actions Active

andypalmi marked this pull request as ready for review October 14, 2026 07:52

Merge branch 'main' into feat/expert-streamed-activity

68bd792

andypalmi deployed to staging October 14, 2026 07:59 — with GitHub Actions Active

n-lark self-requested a review October 14, 2026 14:53

n-lark requested changes Oct 14, 2026

View reviewed changes

andypalmi added 3 commits October 14, 2026 21:55

Make the collapsible header a button

45ef699

The header is now a button with aria-expanded, so it can be focused and toggled from the keyboard. A collapsible prop renders a static header with no chevron or toggle when there is nothing to expand.

Use JsonViewer for tool card input and output

e823c2b

Tool cards show input and output in the shared JsonViewer instead of their own highlighter and toggles. A card with no reason, error or data has nothing to expand, so it shows no chevron.

andypalmi deployed to staging October 14, 2026 20:02 — with GitHub Actions Active

andypalmi requested a review from n-lark October 14, 2026 21:42

Use ff-button for the collapsible header

5bd0a12

Adds an align prop and a plain kind to ff-button so it can render a left-aligned header with no background or padding.

andypalmi deployed to staging October 14, 2026 21:50 — with GitHub Actions Active

Keep default ff-button markup unchanged

b3aa0d0

andypalmi deployed to staging October 14, 2026 22:04 — with GitHub Actions Active

Steve-Mcl reviewed Oct 15, 2026

View reviewed changes

andypalmi requested a review from Steve-Mcl October 15, 2026 10:29

andypalmi deployed to staging October 15, 2026 10:33 — with GitHub Actions Active

Steve-Mcl approved these changes Oct 15, 2026

View reviewed changes

cstns reviewed Oct 15, 2026

View reviewed changes

n-lark approved these changes Oct 15, 2026

View reviewed changes

andypalmi deployed to staging October 15, 2026 16:13 — with GitHub Actions Active

andypalmi requested a review from cstns October 15, 2026 16:22

cstns approved these changes Oct 16, 2026

View reviewed changes

andypalmi merged commit 87d6371 into main Oct 16, 2026
29 checks passed

andypalmi deleted the feat/expert-streamed-activity branch October 16, 2026 09:24

andypalmi deployed to staging October 16, 2026 09:24 — with GitHub Actions Active

Conversation

andypalmi commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

n-lark left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andypalmi commented Oct 14, 2026

Uh oh!

Steve-Mcl left a comment

Choose a reason for hiding this comment

Uh oh!

andypalmi commented Oct 15, 2026

Uh oh!

Steve-Mcl left a comment

Choose a reason for hiding this comment

Uh oh!

cstns commented Oct 15, 2026

Uh oh!

cstns left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andypalmi Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n-lark left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andypalmi commented Oct 13, 2026 •

edited

Loading

codecov Bot commented Oct 13, 2026 •

edited

Loading

andypalmi Oct 15, 2026 •

edited

Loading