Skip to content

Show the running test's browser as view-only - #179

Closed
Jonksar wants to merge 13 commits into
browser-use:mainfrom
iter8-ai:hermes/test-run-live-view
Closed

Jonksar wants to merge 13 commits into
browser-use:mainfrom
iter8-ai:hermes/test-run-live-view

Conversation

@Jonksar

@Jonksar Jonksar commented Sep 30, 2026 •

Copy link
Copy Markdown

While a setup test runs, the Test step now shows the run's Browserbase browser as view-only; the user cannot click, type, scroll or focus it.

Chain: FIRE publishes the Browserbase fullscreen live-view URL when a run's browser opens → GET /agent/{id}/run/{run}/live-view (null once the run has a result) → ICE proxy → CFE setup controller returns liveViewUrl from getTestRun only while running → workflow-use renders it inert behind a pointer shield.

Host: . New e2e asserts the frame is inert, tabindex=-1, and covered by a shield (red confirmed without the UI change); 17/17 e2e pass.


Summary by cubic

Adds a demonstration-based setup flow for computer-use agents: describe a task, demonstrate it in an ephemeral browser, review the captured steps, test, and schedule. The previous workflow canvas is replaced by the new setup UI, and a separate recording service captures bounded semantic events from a Browserbase session. While a test runs, the Test step shows the run's browser as a watch-only live view behind an inert pointer shield.

New Features

  • Setup UI with host message bridge: drafts, credential placeholders, compiled computer-use configs, and step review.
  • Recording service: public-HTTP URL validation, 15-minute TTL, 200-step cap, and credential-field classification that never retains typed values.

Infrastructure

  • CI runs UI unit and e2e tests plus recording service tests; production images for ui and recording are built on main.
  • Docker images, Playwright setup, and a Playwright e2e suite covering the full journey were added.

Written for commit b8de132. Summary will update on new commits.

Review in cubic

Joonatan Samuel and others added 13 commits September 11, 2026 12:24
Add demonstration-based setup for computer-use agents
Deploy the demonstration setup UI and recorder images
Generate the UI client in clean builds
* fix(setup): handle null recorder step fields

* fix(setup): collect downloaded files

* fix(setup): block credential URL queries

* fix(setup): reject URLs with query data

* fix(recording): reject URLs with query data

* fix(setup): validate navigation URL fallbacks

* fix(setup): never retain demonstrated values

* fix(setup): reject credential intent

* fix(setup): guard all credential boundaries

* fix(setup): disable runtime form arguments

* docs(setup): describe form-entry restriction

* docs(setup): define empty argument contract

* fix(setup): type empty host arguments

---------

Co-authored-by: Joonatan Samuel <jonksar@github.com>
* fix(setup): match credential intent by term

* fix(setup): block MFA and hyphenated sign-in intent

* fix(setup): normalize credential intent matching

* fix(setup): normalize credential intent paths

* fix(setup): reject conventional PIN values

* fix(setup): normalize separated credential terms

* fix(setup): normalize credential intent tokens

* fix(setup): refine credential intent grammar

* fix(setup): block concatenated sign-in intent

* fix(setup): normalize credential separators

* fix(setup): scope credential intent matching

* fix(setup): preserve credential guard boundaries

* fix(setup): normalize credential path separators

* fix(setup): tokenize credential intent consistently

* fix(setup): validate raw credential intent

* fix(setup): constrain credential exceptions

* fix(setup): scope recorder hostname exception

* fix(setup): reject PIN and backup codes

* fix(setup): distinguish PIN codes from pin actions

* fix(setup): narrow PIN action exemption

* fix(setup): constrain pin action destinations

* fix(setup): require exact pin action destination

* fix(setup): scope pin action exception

* fix(setup): close remaining pin intent gaps

* fix(setup): separate pin credentials from content

* fix(setup): scope safe pin identifiers

* fix(setup): reject bare PIN credential actions

* fix(setup): narrow PIN context exceptions

* fix(setup): close credential boundary bypasses

* test(setup): cover credential boundary regressions

* fix(setup): reject numeric credential suffixes

* test(setup): cover prefixed credential values

* fix(setup): reject prefixed credential values

* test(setup): cover concatenated credential prefixes

* fix(setup): cover concatenated credential values

* test(setup): cover credential prefix heuristics

* fix(setup): refine credential token detection

* test(setup): cover credential token boundaries

* fix(setup): bound concatenated credentials

* test(ui): cover structural credential boundaries

* fix(ui): classify structural credential intent

* test(ui): cover mixed credential structures

* fix(ui): classify mixed credential structures

---------

Co-authored-by: Joonatan Samuel <jonksar@github.com>
* Record sign-in fields as credential placeholders

Sign-in fields become credential steps that carry only their kind
(username, password, otp). The compiler turns them into exact $placeholders,
and the host collects and stores the values, so the setup page, recorder and
model never receive them.

* fix(recording): keep sign-in submit clicks; let the host decide re-prompts

Review: a submit button inside a password form was classified as a
username field, so the Sign in click was dropped. Only typed fields are
credential fields now. The setup page always asks the host, which binds
saved sign-in details to the demonstrated website.

* fix(setup): reject literal sign-in values; ignore checkbox input events

Review: removing the word heuristic let a goal like 'password
example-secret-123' reach the prompt. A narrow check now rejects a
credential word followed by an assigned or digit-bearing value, while
sign-in wording and $placeholders stay allowed. Checkbox and radio
input events no longer become form-entry steps.

* fix(setup): classify secret fields before the username fallback; keep reviewed descriptions

Review: a 'Secret key' field in a password form became $username, and an
edited credential step description was dropped from the prompt.

* fix(setup): reject symbol-bearing sign-in values and credential URL paths

Review: 'password correct-horse-battery-staple', 'verification code
482913' and /token/<value> paths compiled into prompts.

* fix(setup): check raw and repeatedly decoded URL paths; labelled codes

Review: /token/abc123/../reports and %2574oken paths, and 'code sent to
me: 482913', bypassed the disclosure check.

* fix(setup): treat backslashes as path separators when checking URLs

Review: https://host\token\abc123\..\reports resolved to a clean path
while the saved URL kept the token segment.

* fix(setup): classify password and username fields by type first; nearby codes

Review: a password field labelled Passcode became an OTP step, and an
autocomplete=username field with 'auth' in its id became a password.
Credential steps can now be reclassified in review. Codes a few words
after their label are rejected.

---------

Co-authored-by: Joonatan Samuel <jonksar@github.com>
Each step is one row: number, instruction, optional expected outcome and
an icon remove button. A ten-step demonstration now fits on a 1440x900
screen with the actions visible; narrow screens stack the outcome under
its instruction.

Co-authored-by: Joonatan Samuel <jonksar@github.com>
@Jonksar

Jonksar commented Sep 30, 2026

Copy link
Copy Markdown
Author

Opened against the wrong repository by mistake; sorry for the noise.

@Jonksar Jonksar closed this Sep 30, 2026
@gitguardian

gitguardian Bot commented Sep 30, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
- - Generic Password e04e326 recording/tests/test_api.py View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@Jonksar
Jonksar deleted the hermes/test-run-live-view branch September 30, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant