Run the per-scenario emulators locally, and let a paused execution outlive a human - #1712
Open
RhysSullivan wants to merge 4 commits into
Open
Run the per-scenario emulators locally, and let a paused execution outlive a human#1712RhysSullivan wants to merge 4 commits into
RhysSullivan wants to merge 4 commits into
Conversation
The hosted control plane put the public internet on the critical path of
scenarios that are otherwise entirely local: connect ETIMEDOUT and 502s
reaching the edge failed 17 shards in two weeks, and by Aug 20
graphql-introspection-health failed every run on a 403 rate limit the
instance answered ahead of the fault it had armed - raising the budget from
10 to 100 changed nothing, so it was never ours to stay under.
@executor-js/emulate is built for this ("local drop-in replacement services
for CI and no-network sandboxes") and the suite already boots WorkOS and
Autumn that way. Same package, same wire behaviour, same per-run isolation,
minus the network. Scoped, so the emulator dies with the scenario.
Contributor
Cloudflare preview
Sign-in is Cloudflare Access (one-time PIN to an allowed email). The preview has its own database and encryption key; it is destroyed when this PR closes. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-marketing | 2e1fdac | Commit Preview URL Branch Preview URL |
Aug 20 2026, 06:22 PM |
The browser approval scenario measures what that costs: 8.7s from page load to the Approve click on a loaded runner, against a 6s ceiling, so the POST came back 404 and the page said the execution was no longer available. 12 shards in two weeks. Production allows 9 minutes; 6s was a test-speed knob borrowed from the expiry scenario, which now sleeps correspondingly longer - the one place the raise is paid for.
@executor-js/cli
@executor-js/config
@executor-js/execution
@executor-js/sdk
@executor-js/codemode-core
@executor-js/runtime-quickjs
@executor-js/plugin-file-secrets
@executor-js/plugin-graphql
@executor-js/plugin-keychain
@executor-js/plugin-mcp
@executor-js/plugin-onepassword
@executor-js/plugin-openapi
executor
commit: |
RhysSullivan
changed the title
RhysSullivan
marked this pull request as ready for review
August 20, 2026 18:18
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-cloud | 2e1fdac | Aug 20 2026, 06:24 PM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1707, which left two failures. This fixes both; the matrix is now green three runs in a row (36/36 each).
The hosted emulator was never ours to keep green
graphql-introspection-healthhad started failing on every CI run with403 API rate limit exceededfrom the hosted GitHub emulator instead of the 401 it had armed a fault for. Raising the fault budget from 10 to 100 changed nothing, so whatever produced that 403 sits outside the instance's own fault accounting — and from a shared CI egress IP there is no version of the scenario that can stay under it. Same dependency, different symptom, cost 17 more shards in the two weeks to Aug 20:connect ETIMEDOUTand bare 502s reaching<service>.emulators.dev.@executor-js/emulatedescribes itself as "local drop-in replacement services for CI and no-network sandboxes", and the suite already boots WorkOS and Autumn in-process insetup/cloud.boot.ts.createEmulatorInstancenow does the same per scenario: OS-assigned port,createEmulator, a control-plane readiness probe, andEffect.acquireReleaseso the emulator dies with the scenario's scope. Same package, same wire behaviour, same per-run isolation — a fresh process-local instance keeps ledger assertions as clean as a fresh hosted one — minus the network.Five scenarios move:
graphql-introspection-health(github),oauth-client-handoff(microsoft),connect-handoffandconnect-handoff-session(resend),google-health-checks(google). Four needed their bodies wrapped inEffect.scoped.selfhost/mcp-oauth-scope-discovery-emulatedeliberately stays on the deployed instance — proving executor works against a real deployment is the point of that scenario, andmcp-oauth-scope-discoverycovers the same contract in-process.A paused execution has to outlive a human
The browser approval scenario measures what that costs. From the failing trace:
GET /api/mcp-sessions/<id>200 at 17:05:11.898,POSTthe approval at 17:05:20.629 → 404. 8.7 s from page load to the Approve click on a loaded runner, against a 6 s ceiling, with the page showing "This paused execution is no longer available". 12 shards in two weeks.Production allows 9 minutes. The 6 s was a test-speed knob, and
cloud/mcp-client-sessionsis the only thing that wants it small — it derives its teardown wait from the value, so raising it to 30 s is paid for there and nowhere else. That is the trade: seconds on one scenario against a 30-second timeout plus a re-run of the whole matrix.Note on AGENTS.md
AGENTS.mdsaid to "create per-run hosted instances through the service's/_emulate/instancescontrol route". This PR rewrites that line to prefer in-process instances and to say when a hosted one is still right. Flagging it because it is a documented-policy change, not just a code change.Verified
Three consecutive full-matrix CI runs, 36/36 green. All five moved scenarios green on selfhost locally, with no emulator process or listener surviving the run.
typecheck,lint,format:checkclean.Still open after this
The foreign-org-slug not-found bug behind the biggest flake is filed as #1713 — a product fix in the shell rendering path, not CI hardening, and it needs a cloud stack to verify.