Conversation
Add demonstration-based setup for computer-use agents
Deploy the demonstration setup UI and recorder images
Generate the UI client in clean builds
* fix(setup): handle null recorder step fields * fix(setup): collect downloaded files * fix(setup): block credential URL queries * fix(setup): reject URLs with query data * fix(recording): reject URLs with query data * fix(setup): validate navigation URL fallbacks * fix(setup): never retain demonstrated values * fix(setup): reject credential intent * fix(setup): guard all credential boundaries * fix(setup): disable runtime form arguments * docs(setup): describe form-entry restriction * docs(setup): define empty argument contract * fix(setup): type empty host arguments --------- Co-authored-by: Joonatan Samuel <jonksar@github.com>
* fix(setup): match credential intent by term * fix(setup): block MFA and hyphenated sign-in intent * fix(setup): normalize credential intent matching * fix(setup): normalize credential intent paths * fix(setup): reject conventional PIN values * fix(setup): normalize separated credential terms * fix(setup): normalize credential intent tokens * fix(setup): refine credential intent grammar * fix(setup): block concatenated sign-in intent * fix(setup): normalize credential separators * fix(setup): scope credential intent matching * fix(setup): preserve credential guard boundaries * fix(setup): normalize credential path separators * fix(setup): tokenize credential intent consistently * fix(setup): validate raw credential intent * fix(setup): constrain credential exceptions * fix(setup): scope recorder hostname exception * fix(setup): reject PIN and backup codes * fix(setup): distinguish PIN codes from pin actions * fix(setup): narrow PIN action exemption * fix(setup): constrain pin action destinations * fix(setup): require exact pin action destination * fix(setup): scope pin action exception * fix(setup): close remaining pin intent gaps * fix(setup): separate pin credentials from content * fix(setup): scope safe pin identifiers * fix(setup): reject bare PIN credential actions * fix(setup): narrow PIN context exceptions * fix(setup): close credential boundary bypasses * test(setup): cover credential boundary regressions * fix(setup): reject numeric credential suffixes * test(setup): cover prefixed credential values * fix(setup): reject prefixed credential values * test(setup): cover concatenated credential prefixes * fix(setup): cover concatenated credential values * test(setup): cover credential prefix heuristics * fix(setup): refine credential token detection * test(setup): cover credential token boundaries * fix(setup): bound concatenated credentials * test(ui): cover structural credential boundaries * fix(ui): classify structural credential intent * test(ui): cover mixed credential structures * fix(ui): classify mixed credential structures --------- Co-authored-by: Joonatan Samuel <jonksar@github.com>
* Record sign-in fields as credential placeholders Sign-in fields become credential steps that carry only their kind (username, password, otp). The compiler turns them into exact $placeholders, and the host collects and stores the values, so the setup page, recorder and model never receive them. * fix(recording): keep sign-in submit clicks; let the host decide re-prompts Review: a submit button inside a password form was classified as a username field, so the Sign in click was dropped. Only typed fields are credential fields now. The setup page always asks the host, which binds saved sign-in details to the demonstrated website. * fix(setup): reject literal sign-in values; ignore checkbox input events Review: removing the word heuristic let a goal like 'password example-secret-123' reach the prompt. A narrow check now rejects a credential word followed by an assigned or digit-bearing value, while sign-in wording and $placeholders stay allowed. Checkbox and radio input events no longer become form-entry steps. * fix(setup): classify secret fields before the username fallback; keep reviewed descriptions Review: a 'Secret key' field in a password form became $username, and an edited credential step description was dropped from the prompt. * fix(setup): reject symbol-bearing sign-in values and credential URL paths Review: 'password correct-horse-battery-staple', 'verification code 482913' and /token/<value> paths compiled into prompts. * fix(setup): check raw and repeatedly decoded URL paths; labelled codes Review: /token/abc123/../reports and %2574oken paths, and 'code sent to me: 482913', bypassed the disclosure check. * fix(setup): treat backslashes as path separators when checking URLs Review: https://host\token\abc123\..\reports resolved to a clean path while the saved URL kept the token segment. * fix(setup): classify password and username fields by type first; nearby codes Review: a password field labelled Passcode became an OTP step, and an autocomplete=username field with 'auth' in its id became a password. Credential steps can now be reclassified in review. Codes a few words after their label are rejected. --------- Co-authored-by: Joonatan Samuel <jonksar@github.com>
Each step is one row: number, instruction, optional expected outcome and an icon remove button. A ten-step demonstration now fits on a 1440x900 screen with the actions visible; narrow screens stack the outcome under its instruction. Co-authored-by: Joonatan Samuel <jonksar@github.com>
|
Opened against the wrong repository by mistake; sorry for the noise. |
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| - | - | Generic Password | e04e326 | recording/tests/test_api.py | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
While a setup test runs, the Test step now shows the run's Browserbase browser as view-only; the user cannot click, type, scroll or focus it.
Chain: FIRE publishes the Browserbase fullscreen live-view URL when a run's browser opens →
GET /agent/{id}/run/{run}/live-view(null once the run has a result) → ICE proxy → CFE setup controller returnsliveViewUrlfromgetTestRunonly while running → workflow-use renders it inert behind a pointer shield.Host: . New e2e asserts the frame is
inert,tabindex=-1, and covered by a shield (red confirmed without the UI change); 17/17 e2e pass.Summary by cubic
Adds a demonstration-based setup flow for computer-use agents: describe a task, demonstrate it in an ephemeral browser, review the captured steps, test, and schedule. The previous workflow canvas is replaced by the new setup UI, and a separate
recordingservice captures bounded semantic events from a Browserbase session. While a test runs, the Test step shows the run's browser as a watch-only live view behind an inert pointer shield.New Features
Infrastructure
uiandrecordingare built on main.Written for commit b8de132. Summary will update on new commits.