Skip to content

feat: enable UI actions from AI chat - #9881

Open
djbarnwal wants to merge 2 commits into
mainfrom
feat/ui-action-ai
Open

djbarnwal wants to merge 2 commits into
mainfrom
feat/ui-action-ai

Conversation

@djbarnwal

Copy link
Copy Markdown
Member

Adds an opt-in mechanism that allows Rill’s AI agents to trigger approved UI actions in the browser.

Adding capability to support ejection of Rill templates to vega lite spec. (#9850)

UI elements can expose themselves to the agent using a stable data-ai-action identifier. Available actions are sent as transient context with each chat request, and the agent can invoke the new click_ui tool to request an action.

Implementation

  • Added the click_ui backend tool for developer and analyst agents.
  • Added UIContext and UIAction to the completion API.
  • Discover visible, enabled, uniquely identified [data-ai-action] elements when sending a chat message.
  • Revalidate actions immediately before clicking them in the browser.
  • Added initial actions for:
    • Editing a canvas
    • Starting a dashboard pivot
  • Added frontend and backend tests for action discovery, validation, and execution.

Safety

UI actions are explicitly opt-in. The agent cannot use arbitrary CSS selectors or click arbitrary page elements.

An action is available only when it:

  • Has a non-empty data-ai-action identifier.
  • Is visible and enabled.
  • Has a unique identifier on the current page.
  • Is outside a data-ai-ignore subtree.
  • Is included in the current request’s UI context.

The action is checked again in the browser before the click is performed.

image

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

@djbarnwal
djbarnwal requested review from AdityaHegde and begelundmuller and removed request for begelundmuller September 11, 2026 14:13
@nishantmonu51 nishantmonu51 added Type:Feature New feature request Size:XL Very large change: 2,000+ lines labels Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Size:XL Very large change: 2,000+ lines Type:Feature New feature request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants