Bound the e2e waits that were failing CI on nothing - #1707
Merged
RhysSullivan merged 1 commit intoAug 20, 2026
Conversation
Contributor
Cloudflare previewTorn down — the PR is closed. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-cloud | b806f5c | Aug 20 2026, 05:34 PM |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-marketing | b806f5c | Commit Preview URL Branch Preview URL |
Aug 20 2026, 05:33 PM |
@executor-js/cli
@executor-js/config
@executor-js/execution
@executor-js/sdk
@executor-js/codemode-core
@executor-js/runtime-quickjs
@executor-js/plugin-file-secrets
@executor-js/plugin-graphql
@executor-js/plugin-keychain
@executor-js/plugin-mcp
@executor-js/plugin-onepassword
@executor-js/plugin-openapi
executor
commit: |
Three causes accounted for 31 of the 40 flake-signature E2E shard failures on the current 16-shard layout (145 runs, Aug 18-20). networkidle cannot be a gate in this suite, because the suite keeps the network busy: every browser session exports OTel spans to the run's motel on a one-second interval, and Playwright wants 500ms of silence. A page that keeps producing spans never opens that window, so the navigation burns its full 30s on a page that is visibly loaded. Route the 104 goto sites and 9 waitForLoadState sites through visit()/settle(): wait for load, then for React to own the document (__reactContainer$, the interactivity networkidle was standing in for), then bounded quiet. Readiness is the assertion that follows. createEmulatorInstance had no timeout and no retry on the one request a scenario makes to the public internet, so connect ETIMEDOUT and the occasional 502 failed scenarios over the network. ledger.list(n) is the last n entries, so checking both Google rows at the end let the first fall out of the window behind the second account's traffic, and read as "Calendar never reached the emulator" when its probe had already come back healthy. Also arm the GraphQL introspection fault with a budget rather than a count: when the armed 401s run out the emulator answers as GitHub would, 403 rate limited, and the assertion fails on a message about nothing.
RhysSullivan
force-pushed
the
ci/bound-networkidle-and-emulator-network-waits
branch
from
August 20, 2026 17:31
3a43003 to
b806f5c
Compare
RhysSullivan
marked this pull request as ready for review
August 20, 2026 17:36
RhysSullivan
deleted the
ci/bound-networkidle-and-emulator-network-waits
branch
August 20, 2026 17:36
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two weeks of
ci.yml(350 runs, 7,636 jobs, Aug 6-20), classified from job logs. On the current 16-shard layout (145 runs since Aug 18): 1.61% of E2E shard jobs failed with a flake signature, which put a false red on 23% of full runs. Four causes account for 31 of the 40.networkidle is not a gate in this suite (17 of 40)
Every browser session exports OTel spans to the run's motel through
packages/react'sOtlpTraceron a one-second interval (VITE_PUBLIC_OTLP_TRACES_URL, wired bysetup/motel.ts). Playwright'snetworkidlewants 500 ms with zero requests in flight. A page that keeps producing spans never opens that window, so the navigation sits out its full 30 s on a page that is visibly loaded.103
gotosites and 9waitForLoadStatesites now go throughvisit()/settle().visitwaits forload, then for React to own the document, then — bounded — for quiet:hydratedis the readinessnetworkidlewas standing in for, asked directly. React DOM stamps__reactContainer$on its root whenhydrateRootruns; before that line a click or afillgoes to markup nobody is listening to and is lost without a trace. Locators cannot see the difference, which is why a swallowed interaction surfaces 20 s later as "the thing it should have opened never appeared".settlekeeps the quiet when it comes (usually milliseconds) and gives up at 5 s. Readiness is the assertion that follows, which is a real page-specific signal. Nothing that was asserted stops being asserted.The one request that leaves the runner (8 of 40)
createEmulatorInstancePOSTed to<service>.emulators.devwith no timeout and no retry, soconnect ETIMEDOUTand the occasional 502 failed a scenario over the network rather than the product. Now 10 s per attempt, jittered exponential backoff, 3 retries. The inline duplicate ingoogle-health-checksshares it.A ledger window, not eventual consistency (6 of 40)
ledger.list(n)is the last n entries, and connecting the second Google account is easily a hundred emulator requests — so asserting both rows at the end let the first fall out of the window and read as "Calendar never reached the emulator" when its probe had already come back healthy. Each row is now checked as it happens. Confirmed fixed: shard 4of16 failed this on two runs before the change and passes after.A swallowed click before hydration (4, on the earlier 8-shard layout)
All four popup timeouts were
selfhost/posthog-mcp-oauthwaiting for the OAuth window.clickForPopupretries the click until the window is there — the idempotency argumentclickToRevealalready makes for a reveal click.Verified
Full selfhost suite (all 10 CI shards, 92 files) green locally, twice. Repo-wide
typecheck,lint,format:checkclean. Cloud shards can only be verified in CI — see the last note.Two real bugs this surfaces, both out of scope here
1. A foreign org slug renders a workspace, not a 404. This was the single largest flake (36 of 61 across the full two weeks):
page.goto("/zz-no-such-org/policies", { waitUntil: "networkidle" }). With the wait bounded, the underlying failure is legible — the console renders the full Policies shell under the foreign URL, with the user's own org in the switcher and a retrying "Failed to load policies" card. That retry loop is what kept the network busy andnetworkidleunreachable.OrgSlugGate's header says the shell "404s upstream" when the server returns no organization for a slug the caller cannot see; when the query resolves after the shell renders, nothing 404s. The scenario's stated contract — "a wrong address, never a silent redirect into a workspace the URL didn't name" — is not held. Intermittent, soexpectedFailurewould be wrong; it needs a fix in the shell's foreign-slug path.2.
mcp-browser-resume-pageraces a 6 s ceiling. The failure screenshot says "The paused execution could not be resumed. It may have already completed or expired." The click lands ~14 s after the pause, againstMCP_PAUSED_SESSION_IDLE_TIMEOUT_MS=6000; it only passes while the concurrentresumelong-poll keeps the session warm. Raising the ceiling costs wall clock inmcp-client-sessions, which derives its teardown wait from it — a trade worth making deliberately, not in this PR.Still failing and not addressed here:
graphql-introspection-healthgets403 API rate limit exceededfrom the hosted GitHub emulator instead of the armed 401. Arming a larger budget did not help, so the limiter appears to sit ahead of the fault system and to key on the runner's egress IP. Together with the eight network failures above, that is the case for running these five scenarios against locally spawned emulators (createEmulator, as the suite already does for WorkOS and Autumn) rather than the hosted control plane — which would contradict the line inAGENTS.mdprescribing per-run hosted instances, so it is a call to make explicitly.Control
mainre-run at the same hour as this branch, same runner pool, failed three shards — and all three are the causes above:page.goto("/zz-no-such-org/policies")30 s timeoutvisit/settleTypeError: fetch failedreaching emulators.devNone of the three has appeared on this branch. What remains on this branch is
Approve sentand the GraphQL 403, both described above.