Skip to content

feat(browser): add opt-in headless browser automation tools - #415

Draft
tanvi-red wants to merge 6 commits into
Observal:mainfrom
tanvi-red:browser-tools
Draft

tanvi-red wants to merge 6 commits into
Observal:mainfrom
tanvi-red:browser-tools

Conversation

@tanvi-red

Copy link
Copy Markdown

Purpose

Add browser automation as an opt-in first-party extension that the model can use when no programmatic interface exists, while keeping browser state daemon-owned and every client a consistent projection:

  • register eleven browser tools only after explicit enablement, contributing no prompt text, schema, UI, or background work while disabled
  • drive a real headless Chromium through Playwright for navigation, interaction, content extraction, and screenshots
  • give the model prompt guidance to prefer web_fetch for static pages and reach for the browser only when a page needs JavaScript or interaction
  • keep the browser process lifecycle bound to the session: closed on toggle-off, runtime rebuild, and daemon shutdown

Fixes

Advances #72 (axl-backlog:sandbox.browser-tool). Satisfies acceptance criteria 1 (opt-in, nothing while disabled) and 6 (daemon-owned state). Criterion 2 (sandbox confinement), criterion 5 (per-site authenticated browsing), and full typed-RPC exposure remain follow-ups.

Approach

  • Keep tool registration, browser lifecycle, and prompt construction in the daemon runtime; clients only render and submit intent.
  • Add a new @axl/extension-browser package exposing browser_navigate, browser_screenshot, browser_click, browser_type, browser_scroll, browser_read, browser_back, browser_forward, browser_wait, browser_eval, and browser_select.
  • Validate every navigation target before it reaches the browser: reject non-http(s) schemes, credential-bearing URLs, and malformed input, at both the tool layer and the Playwright session layer.
  • Gate the tools behind a browser selection flag wired through the protocol (config.tools, session selection, and the session.configure result), daemon, runtime, SDK projector, CLI (--browser/--no-browser), and the TUI settings toggle.
  • Save screenshots and downloads only to workspace-scoped directories (.axl-screenshots/, .axl-downloads/) and open screenshots in the host viewer on macOS, Linux, and Windows.
  • Add a client-local TUI renderer through the public extension API that shows readable tool labels instead of raw output.
  • Track the live browser session per session ID so a runtime rebuild closes the prior process before starting a new one, and toggling the tool off closes it with no replacement.

How was this tested?

  • pnpm typecheck, pnpm lint, pnpm format:check, pnpm check:boundaries, pnpm check:generated, and pnpm build: all passed.
  • @axl/extension-browser unit tests: 42 passed, covering policy flag generation, URL safety rejection, every tool's input validation, screenshot-to-disk, and the eleven-tool roster.
  • uvx reuse lint: passed with 589 of 589 files carrying copyright and license information.
  • pnpm audit --audit-level high: no known vulnerabilities.
  • git diff --check: passed.
  • Live Anthropic testing through the real daemon: browser_navigate, browser_eval (returned document.title), browser_wait, browser_back, and browser_forward all executed correctly; the settings toggle removed the tools when disabled and the model fell back to web_fetch.
  • Security checks verified live: file://, credential URLs, ftp://, javascript:, and chrome:// are all rejected before the browser is touched.
  • Pre-existing environment-dependent failures on macOS (kernel path-canonicalization under /private/var, OCI tests needing /proc/self/mountinfo, one flaky daemon queue-drain test) are unrelated to this change; this branch touches no kernel, sandbox, or queue code.

Learning

  • Reviewed Playwright's launchPersistentContext and CDP model as the automation reference; chose persistent context so the user-data directory stays workspace-scoped.
  • A headless screenshot returned as inline base64 overflows the model context and caused a provider 400; returning a saved path plus a short description keeps the context small while the artifact stays on disk.
  • Browser tools are most reliable when the model is told to prefer web_fetch first, since browser automation is slower and only needed for JavaScript-rendered or interactive pages.
  • The browser selection flag had to be threaded through every RPC handler explicitly; a missing field in the daemon's session.create and session.configure destructuring silently dropped it and defeated the toggle.

Note for reviewers

The wire protocol gained a required browser field on config.tools, session selection, and the session.configure result, but the wire version was left at 16. Please confirm whether this warrants a version bump.

Checklist

  • I reviewed the complete diff.
  • I added or updated the smallest relevant test for behavior changes.
  • I ran the relevant formatting, lint, type-check, test, boundary, and license checks.
  • Every new file has SPDX metadata, directly or through REUSE.toml.
  • Every commit has a matching DCO Signed-off-by trailer.
  • UI changes include screenshots attached to the pull request, not committed to the repository.

AI assistance

  • Generative AI materially assisted this change. Tool and model/version: pi coding agent; the model/version was not exposed to the session.
  • I manually reviewed, understood, and tested the generated work.

New @axl/extension-browser package providing six Playwright-backed tools:
browser_navigate, browser_screenshot, browser_click, browser_type,
browser_scroll, and browser_read. The browser runs headless and is opt-in
through the --browser flag or the TUI settings toggle.

- Screenshots save to disk and auto-open in the OS image viewer
- Prompt guidance so the model prefers the browser for JS-heavy pages
- TUI renderer shows clean tool labels instead of raw output
- Browser process closes on toggle-off, runtime rebuild, and daemon shutdown
- Full wiring through protocol, daemon, runtime, SDK, CLI, TUI, and web
- URL safety checks reject file://, credentials, and non-http schemes
- 31 tests covering policy, tool behavior, validation, and screenshot save

Signed-off-by: Tanvi Reddy <reddyplayer22@gmail.com>
The session.create and session.configure RPC handlers in daemon.ts
destructured request params by name but omitted browser, so the flag
was silently dropped before reaching the session manager. This left the
browser tool stuck on its initial state: toggling it off in settings had
no effect because update.browser arrived as undefined.

Both handlers now forward browser alongside webFetch and webSearch.

Signed-off-by: Tanvi Reddy <reddyplayer22@gmail.com>
Adds five tools to the browser extension:
- browser_back / browser_forward: navigate session history
- browser_wait: wait for a selector to become visible (bounded timeout)
- browser_eval: evaluate a JS expression and return a JSON-safe result
- browser_select: choose an option in a select dropdown

Each has a TUI renderer and unit tests. The five tools bring the
registered browser roster to eleven. Verified live against Chromium.

Signed-off-by: Tanvi Reddy <reddyplayer22@gmail.com>
Signed-off-by: Tanvi Reddy <reddyplayer22@gmail.com>
The new browser extension package files were authored by me. Correct the
SPDX copyright headers and REUSE.toml entries from the template default to
my name.

Signed-off-by: Tanvi Reddy <reddyplayer22@gmail.com>
The release package bundles the CLI with esbuild. Because the browser
extension depends on Playwright, esbuild tried to bundle playwright-core,
which fails on its dynamic chromium-bidi requires.

Mark playwright and playwright-core external so they are not bundled, and
declare playwright as an optional dependency of the published package so it
installs for browser use without being forced on every install. The browser
tools load it through a runtime dynamic import and fail loudly when absent.

Signed-off-by: Tanvi Reddy <reddyplayer22@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant