Skip to content

Make the companion check deterministic on pull requests - #417

Merged
BunsDev merged 5 commits into
mainfrom
fix/companion-ui-flow-flake
Oct 9, 2026
Merged

BunsDev merged 5 commits into
mainfrom
fix/companion-ui-flow-flake

Conversation

@BunsDev

@BunsDev BunsDev commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Closes #414

What was happening

The failing assertion was never the pairing field: it was the wait for the familiar picker after Connect. In every failed run the app showed "Could not reach your Mac securely", which is what the app says for any non-protocol error, and the recording shows "Connecting…" for about 12 s with no system prompt. The pairing request from the simulator to the fixture at the runner VM's 192.168.64.x address timed out (NSURLError -1001, the client's 12 s request timeout; visible in the transport test's own log on the main-push run 37948926187, where that test went first and failed the same way after 25 s).

The pattern across all runs: the first connection from the simulator to the fixture is the slow one, whichever test makes it. When the transport unit test went first it took 17 s, 5 s and 0.6 s across three green runs; when the UI flow went first it failed twice. The host reaches the fixture instantly (a curl probe answered in 0 s on every run), so the fixture is not the slow side.

What changed

  1. Host-side probe (test-e2e.sh): the fixture is asked for /v1/snapshot from the host before xcodebuild, bounded to about a minute; it fails loudly with firewall state and interfaces if the fixture never answers, and prints how long the first answer took.
  2. Simulator-side warm-up: the script now boots the simulator itself, waits for bootstatus, and opens the fixture's address from inside it once before any test runs. That first in-simulator connection took 18 to 83 s on the runs below, which is the stall, now spent before the tests. With it, the transport unit test is a steady 0.2 to 0.7 s on six consecutive runs instead of 0.6 to 17 s.
  3. The UI flow still stalled once in three runs after that (its own first connection from the UI-launched app, 110 s, same assertion), so per the issue's fallback it now runs on pushes to main, on pull requests labelled ci:full, and on a manual run with the new ui_flow input. Every other pull request runs the native unit tests and the Release device build (CHAT_IOS_UI_FLOW=0, -only-testing:ChatCompanionTests). When the flow does run, the simulator's network log is captured and its error lines are printed on failure, since the app's own copy deliberately does not name the error.
  4. docs/iphone-companion.md documents the switch.

Not changed: what the test asserts, what the companion pairs with or sends, Rust, TypeScript, the desktop app, ci.yml.

Evidence (all on this branch)

Run Mode Result Transport test UI flow In-simulator warm-up
37985643572 host probe only success 17.2 s 85.6 s n/a
37987013753 host probe only success 5.2 s 236.7 s n/a
37988516921 host probe only success 0.6 s 70.3 s n/a
37990036658 + simulator warm-up failure (UI flow, same assertion) 0.6 s 110.6 s failed 60 s
37991597112 + simulator warm-up success 0.6 s 83.3 s 26 s
37992529057 + simulator warm-up success 0.7 s 65.2 s 33 s
37994176526 PR path (gated) success 0.4 s not run 35 s
37995252373 PR path (gated) success 0.2 s not run 29 s
37996062557 PR path (gated) success 0.5 s not run 70 s
37997263575 full (ui_flow on) false green: no tests ran (bash 3.2 empty-array bug, fixed in the last commit; the job now refuses to pass without an .xcresult) not run not run 83 s
37998311195 PR path, with the result-bundle guard success, bundle retained 0.25 s not run 55 s
37999284660 PR path success, bundle retained 0.31 s not run 23 s
38000133677 PR path failure before any test: simctl openurl timed out (POSIX 60) after two minutes, then xcodebuild could not find the device not run not run 32 s
38000865422 PR path success, bundle retained 0.32 s not run 89 s
38002091000 full (ui_flow on) success, bundle retained 0.29 s 65.1 s 51 s

Where this leaves #414

Run 38000133677 names the mechanism: a TCP connect from inside the simulator to the fixture's 192.168.64.x address times out (POSIX error 60 from simctl openurl), while the host reaches the same address in 0 s on every run. Everything above, the 5 to 17 s unit-test durations, the 12 s client timeouts, the stalled UI pairings, is that same dropped connect seen through different timeouts. It is below the test script: the simulator's path to the runner VM's NAT interface drops SYNs now and then.

What this PR does about it: the warm-up spends that first connect before the tests and makes the unit tests fast when it succeeds (0.2 to 0.7 s on seven runs in a row, against 0.6 to 17 s before); the UI flow, which makes a fresh first connection from its own app process, is kept off the pull-request path; the job can no longer go green without a result bundle; and after the last commit the warm-up itself is bounded to 45 s and re-boots the device if it dropped out, so a dropped connect costs seconds, not the device. What it cannot do is stop the drop. Pull-request runs with the final script: 6 of 7 green, the one failure being the dropped connect above, before the bound was added.

The real fix is to take the fixture off that interface: bind it to a private-range alias on the loopback interface (for example 10.255.255.1 on lo0, which Pairing.parse accepts and 127.0.0.1 would not) via an explicit address override in companion/fixture.rs. That is a small Rust change outside this PR's scope, so it is left on #414 as the follow-up.

🤖 Generated with Claude Code

BunsDev and others added 5 commits October 9, 2026 15:14
On hosted macOS runners the first connection to the fixture's LAN
address timed out after 12 s while every later one answered within a
second, whichever test made it: the UI flow when it ran first (#414),
and the transport unit test when that ran first. The fixture binds to
the runner VM's 192.168.64.x interface, so that first connection is
the expensive one. The test script now asks the fixture for /v1/snapshot
from the host, bounded to about a minute, right before xcodebuild, and
prints how long the first answer took; if it never answers, the script
fails with the firewall state and interfaces instead of leaving the
simulator to time out.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tests

The host-side probe answers at once, so the fixture is not the slow
side: the simulator's own first connection to the fixture is, taking
5 to 17 s across three green runs and more than the client's 12 s
request timeout on the runs that failed (#414). xcodebuild boots the
simulator and starts the first network test within seconds of that.
The script now boots the simulator itself, waits for bootstatus, and
opens the fixture's address from inside it once before xcodebuild
runs, so the first connection is already behind whichever test makes
the first request.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two warm-ups made the native unit tests deterministic on hosted
runners (the transport test went from 0.6 to 17 s down to a steady
0.6 s), but the simulator UI flow still stalled on its own first
connection once in three runs, past the client's 12 s request timeout
(#414). The flow therefore runs on pushes to main, on pull requests
labelled ci:full, and on a manual run with ui_flow on; every other
pull request runs the native unit tests and the Release device build.
When the flow does run, the simulator's network log is kept and its
error lines are printed on failure, since the app's own copy does not
name the error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The previous commit's full-flow run went green without running a single
test: the runner's bash 3.2 treats an empty array expansion as unbound
under set -u, the script ended at the xcodebuild line, the status was
lost inside the if, and the upload step only warned about the missing
.xcresult. The expansion now uses the bash 3.2-safe form, the script
refuses to finish without a result bundle, and the upload step fails
when there is nothing to retain.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On one run simctl openurl hung for two minutes on a dropped connection
to the fixture (POSIX error 60) and xcodebuild then could not find the
device at all. The warm-up is now limited to 45 s, reports when it
could not reach the fixture, and boots the simulator again if it is
no longer listed as booted before the tests start.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@BunsDev
BunsDev merged commit 73ef8bf into main Oct 9, 2026
11 checks passed
@BunsDev
BunsDev deleted the fix/companion-ui-flow-flake branch October 9, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

companion UI flow test is intermittent on hosted macOS runners and blocks unrelated merges

1 participant