Skip to content

[Feat] Integration tool approvals are no longer experimental; Auto becomes its own experiment - #3169

Merged
daniel-lxs merged 4 commits into
developfrom
feat/tool-approvals-out-of-experimental
Sep 23, 2026
Merged

daniel-lxs merged 4 commits into
developfrom
feat/tool-approvals-out-of-experimental

Conversation

@daniel-lxs

@daniel-lxs daniel-lxs commented Sep 23, 2026

Copy link
Copy Markdown
Member

Related issue

No linked issue; internal Roomote work.

Why this PR exists

  • A maintainer explicitly invited this PR in the linked issue or discussion
  • I am a maintainer / this is internal Roomote work

What changed

Per-tool integration approvals come out of Experimental. Auto stays experimental, behind its own switch.

Per-tool approvals are always on. Always allow, Always ask, Disable, "ask me for this tool this session", task asks, and the chat approval actions no longer depend on integrationToolApprovals. The Fast session rules, the integration proxy, the native MCP guard, the task path, the approval routes, and the Manage tools controls all run for every deployment. Personal policies for private servers are available to every member.

Auto is its own experiment, integrationToolAutoApprovals. Settings → Experimental now shows "Auto tool approvals" instead of "Integration tool approvals".

  • Off (the default): nothing is assessed, not even in the background. A tool nobody has made a choice about simply runs. In Manage tools it shows as Always allow, and the group row offers no Auto button.
  • On: the "Auto-approval decisions" card appears in Settings → Agent Guidance, and Auto behaves exactly as it does today, including presence-aware asks.

The old kill switch is gone. Turning the approvals experiment off used to cancel every open approval and make claims fail under a share lock on the settings row. With no experiment to turn off, the cancel sweep, the share-locked read, and the experiment re-checks in the relay loop are removed.

Legacy availability UI is removed. Both Manage tools dialogs lose the old enable checkboxes and their Save footer. Disable replaces them and still keeps the legacy disabled_tools columns in sync.

Docs. The integrations docs describe the per-tool choices, and the custom MCP servers page notes that Always ask is enforced at the proxy for tasks.

Rollback. No schema change. The old integration_tool_approvals_experiment_enabled metadata key stays in the database and now classifies as a legacy flag. Deployments that had Auto On keep their saved setting, but Auto stays inactive until an admin turns on the new experiment.

How it was tested

Unit suites updated for the new gates. Kill-switch tests are removed, and experiment-off cases are replaced with "no experiment needed" assertions and an Auto-off rendering case. Typecheck, format, and oxlint pass locally. CI runs the full suites. Live smoke at head ab6b0710d on a running sandbox stack: steps 1–6 passed on the observed UI and runtime checks. Full smoke session.

  • 1 — Pass: With experiments untouched, Auto tool approvals was off; the old Integration tool approvals experiment and Agent Guidance Auto card were absent.
  • 2 — Pass: DeepWiki tools showed Always allow selected by default; the group offered Always allow / Always ask / Disable without Auto, checkboxes, or Save footer. read_wiki_structure changed to Always ask.
  • 3 — Pass: A web session displayed an approval card without an Auto line. The audit row moved pending → approved → consumed after Allow once, and DeepWiki returned “Roomote Overview.” An initial approval POST failed with 403 due to sandbox origin configuration; retry after correcting the origin succeeded.
  • 4 — Pass: Disable reduced DeepWiki ListTools from 3 to 2, and the session reported read_wiki_structure unavailable without running it. Resetting Always allow and clicking it again left it selected with Auto off. The earlier report of a repeat-click failure came from testing under Auto-on semantics and was incorrect for this case.
  • 5 — Pass: Enabling Auto exposed the Agent Guidance card, and the requested instruction was saved. Untouched tool rows showed nothing selected while the group offered Auto. A fresh unset DeepWiki server called read_wiki_structure without a card and recorded auto_approved with NULL human decider. Turning the experiment off allowed another call without a card and left integration_tool_auto_evaluations at zero.
  • 6 — Pass (UI): A separately signed-in non-admin member created a personal DeepWiki server and changed a tool among Always ask, Disable, and Always allow.

The sandbox initially misrouted model calls to an inherited production inference gateway, yielding Provider error (openrouter/openai/gpt-5.6-terra): HTTP 401: Unauthorized: authentication_required. Isolating XDG_CONFIG_HOME for the local web process restored live execution; the successful checks above were observed after that correction. No product code changed or PR merge occurred.

Checklist

  • The PR title follows the repo convention
  • This PR is small and scoped to one change
  • pnpm lint and pnpm check-types pass locally
  • I added tests or included a clear manual validation note above
  • I removed secrets, tokens, private keys, and customer data from code, logs, and screenshots
  • If this change should appear in the changelog, I ran pnpm changeset

Screenshots

Experimental page with Auto tool approvals off
Experimental page: Auto tool approvals is off; the old integration approvals switch is absent.

Manage tools with Auto off
Manage tools, Auto off: Always allow is pressed and the group row has no Auto.

Manage tools with Auto on
Manage tools, Auto on: untouched tool rows have nothing pressed and the group row offers Auto.

Approval card: displayed without an Auto line during step 3; screenshot pending capture.

…eriment

Per-tool approvals (Always allow, Always ask, Disable, session overrides,
task asks, chat actions) are now a normal feature: no experiment gates the
rules, the proxy, the task path, the approval routes, or the Manage tools
controls. The old experiment also acted as a kill switch that cancelled
open approvals and blocked claims; that path goes with it.

Auto gets its own experiment, integrationToolAutoApprovals. With it off,
nothing is assessed, not even in the background, and a tool nobody has
made a choice about simply runs; the Manage tools rows show it as Always
allow and the group row offers no Auto. With it on, the Auto-approval
decisions card appears in Agent Guidance and Auto behaves as before.

The legacy enable checkboxes and Save footer in both Manage tools dialogs
are removed; Disable replaces them and still syncs the legacy columns.
The old experiment's metadata key stays in the database as a legacy flag.
@roomote-community

roomote-community Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

No new code issues found. See task

  • Remove the now-unused ToggleLeft and ToggleRight exports so Knip passes.
  • Update the Vercel MCP test mock for the now-always-on approval-state lookup; the affected test now returns 200 (apps/api/src/handlers/mcp/__tests__/vercel-auth.test.ts:15-34,390).

Reviewed c127533

Comment thread apps/web/src/components/settings/McpToolManagementDialog.tsx
@roomote-roomote

roomote-roomote Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Smoke test at exact head ab6b0710db1b944e5114de6a7cb6c19dd15b2028 on the local running stack (API/web health 200, clean tracked worktree). All six scenarios exercised; no code changed, pushed, or merged. Added unauthenticated remote DeepWiki via Settings → Integrations → Add integration.

  1. PASS: Untouched Experimental showed Auto tool approvals off, no legacy Integration tool approvals switch; Agent Guidance had no Auto-approval decisions card. Screenshot: Experimental, Auto off.
  2. PASS: Manage tools listed all three DeepWiki tools; all-tools Always allow and each row were pressed. The group offered Always allow / Always ask / Disable with no Auto, enable checkboxes, or Save footer. Selecting read_wiki_structure Always ask pressed it immediately. Screenshot: Manage tools, Auto off.
  3. PASS: A web session called read_wiki_structure for RooCodeInc/Roomote. The Always ask card had no Auto line. SQL observed the integration_tool_approval_requests row pending; after Allow once, that same row reached consumed with decided_at populated and DeepWiki returned “Roomote Overview.” Another successful call returned “Core Architecture.” An earlier turn interrupted by sandbox restarts separately showed the approved intermediate status; the successful row was sampled at pending and consumed, not at the brief approved interval.
  4. PASS: Disable hid read_wiki_structure: the live MCP ListTools log changed from toolCount=3 tools=ask_wiki_question,read_wiki_contents,read_wiki_structure to toolCount=2 tools=ask_wiki_question,read_wiki_contents; the web prompt reported the requested tool unavailable and did not run it. Resetting to Always allow and pressing it again with the experiment off left its pressed state true both times. An earlier re-click under Auto-on cleared the explicit choice back to the unpressed Auto default; that was not this off-state criterion.
  5. PASS: Enabling the experiment exposed the Auto-approval decisions card. It was turned on with the exact saved instruction Reading a repository's wiki structure or contents is routine. Using ask_wiki_question is high risk. A fresh untouched shared DeepWiki server showed group Auto pressed with every individual row unpressed. Its web read_wiki_structure call returned real topics without an approval card; the audit row recorded auto_approved, NULL decided_by_user_id, and auto_evaluation.recommendation=approve. After turning the experiment off, a second call returned “Core Architecture” without a card; integration_tool_auto_evaluations remained at 0 and no additional approval-request row was created. Screenshot: Agent Guidance, Auto on · Screenshot: shared Manage tools, Auto on.
  6. PASS (UI): A separately signed-in non-admin member created a personal remote DeepWiki MCP and saw the same group choices. Switching its read_wiki_structure among Always ask / Disable / Always allow updated the pressed row immediately. Personal-server tool execution was not part of this UI check.

Sandbox setup: The first approval POST got 403 due to mismatched bundled/preview origins; aligning the web origin fixed it. Subsequent model calls initially got 401 because the local web process inherited this task's global OpenCode config pointing to a different inference gateway. Isolating its XDG config restored local DeepWiki execution. The final runtime claims above come from the successful runs after those environmental interruptions.

…icies

The native MCP handlers' auth tests used to switch the approvals experiment
off to keep the approval guard inert. With no experiment to switch, they
now mock the approval enforcement to 'no policies' instead. Also drops the
ToggleLeft/ToggleRight icon exports the removed legacy footers used.
@daniel-lxs
daniel-lxs merged commit 7160780 into develop Sep 23, 2026
18 checks passed
@daniel-lxs
daniel-lxs deleted the feat/tool-approvals-out-of-experimental branch September 23, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant