Repository navigation
Make the companion check deterministic on pull requests - #417
Merged
Merged
Conversation
On hosted macOS runners the first connection to the fixture's LAN address timed out after 12 s while every later one answered within a second, whichever test made it: the UI flow when it ran first (#414), and the transport unit test when that ran first. The fixture binds to the runner VM's 192.168.64.x interface, so that first connection is the expensive one. The test script now asks the fixture for /v1/snapshot from the host, bounded to about a minute, right before xcodebuild, and prints how long the first answer took; if it never answers, the script fails with the firewall state and interfaces instead of leaving the simulator to time out. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tests The host-side probe answers at once, so the fixture is not the slow side: the simulator's own first connection to the fixture is, taking 5 to 17 s across three green runs and more than the client's 12 s request timeout on the runs that failed (#414). xcodebuild boots the simulator and starts the first network test within seconds of that. The script now boots the simulator itself, waits for bootstatus, and opens the fixture's address from inside it once before xcodebuild runs, so the first connection is already behind whichever test makes the first request. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two warm-ups made the native unit tests deterministic on hosted runners (the transport test went from 0.6 to 17 s down to a steady 0.6 s), but the simulator UI flow still stalled on its own first connection once in three runs, past the client's 12 s request timeout (#414). The flow therefore runs on pushes to main, on pull requests labelled ci:full, and on a manual run with ui_flow on; every other pull request runs the native unit tests and the Release device build. When the flow does run, the simulator's network log is kept and its error lines are printed on failure, since the app's own copy does not name the error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The previous commit's full-flow run went green without running a single test: the runner's bash 3.2 treats an empty array expansion as unbound under set -u, the script ended at the xcodebuild line, the status was lost inside the if, and the upload step only warned about the missing .xcresult. The expansion now uses the bash 3.2-safe form, the script refuses to finish without a result bundle, and the upload step fails when there is nothing to retain. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On one run simctl openurl hung for two minutes on a dropped connection to the fixture (POSIX error 60) and xcodebuild then could not find the device at all. The warm-up is now limited to 45 s, reports when it could not reach the fixture, and boots the simulator again if it is no longer listed as booted before the tests start. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #414
What was happening
The failing assertion was never the pairing field: it was the wait for the familiar picker after Connect. In every failed run the app showed "Could not reach your Mac securely", which is what the app says for any non-protocol error, and the recording shows "Connecting…" for about 12 s with no system prompt. The pairing request from the simulator to the fixture at the runner VM's
192.168.64.xaddress timed out (NSURLError -1001, the client's 12 s request timeout; visible in the transport test's own log on the main-push run 37948926187, where that test went first and failed the same way after 25 s).The pattern across all runs: the first connection from the simulator to the fixture is the slow one, whichever test makes it. When the transport unit test went first it took 17 s, 5 s and 0.6 s across three green runs; when the UI flow went first it failed twice. The host reaches the fixture instantly (a curl probe answered in 0 s on every run), so the fixture is not the slow side.
What changed
test-e2e.sh): the fixture is asked for/v1/snapshotfrom the host before xcodebuild, bounded to about a minute; it fails loudly with firewall state and interfaces if the fixture never answers, and prints how long the first answer took.bootstatus, and opens the fixture's address from inside it once before any test runs. That first in-simulator connection took 18 to 83 s on the runs below, which is the stall, now spent before the tests. With it, the transport unit test is a steady 0.2 to 0.7 s on six consecutive runs instead of 0.6 to 17 s.ci:full, and on a manual run with the newui_flowinput. Every other pull request runs the native unit tests and the Release device build (CHAT_IOS_UI_FLOW=0,-only-testing:ChatCompanionTests). When the flow does run, the simulator's network log is captured and its error lines are printed on failure, since the app's own copy deliberately does not name the error.docs/iphone-companion.mddocuments the switch.Not changed: what the test asserts, what the companion pairs with or sends, Rust, TypeScript, the desktop app,
ci.yml.Evidence (all on this branch)
ui_flowon)simctl openurltimed out (POSIX 60) after two minutes, then xcodebuild could not find the deviceui_flowon)Where this leaves #414
Run 38000133677 names the mechanism: a TCP connect from inside the simulator to the fixture's
192.168.64.xaddress times out (POSIX error 60 fromsimctl openurl), while the host reaches the same address in 0 s on every run. Everything above, the 5 to 17 s unit-test durations, the 12 s client timeouts, the stalled UI pairings, is that same dropped connect seen through different timeouts. It is below the test script: the simulator's path to the runner VM's NAT interface drops SYNs now and then.What this PR does about it: the warm-up spends that first connect before the tests and makes the unit tests fast when it succeeds (0.2 to 0.7 s on seven runs in a row, against 0.6 to 17 s before); the UI flow, which makes a fresh first connection from its own app process, is kept off the pull-request path; the job can no longer go green without a result bundle; and after the last commit the warm-up itself is bounded to 45 s and re-boots the device if it dropped out, so a dropped connect costs seconds, not the device. What it cannot do is stop the drop. Pull-request runs with the final script: 6 of 7 green, the one failure being the dropped connect above, before the bound was added.
The real fix is to take the fixture off that interface: bind it to a private-range alias on the loopback interface (for example
10.255.255.1onlo0, whichPairing.parseaccepts and127.0.0.1would not) via an explicit address override incompanion/fixture.rs. That is a small Rust change outside this PR's scope, so it is left on #414 as the follow-up.🤖 Generated with Claude Code