fix: keep iOS synthesized drags off AX

[thymikee]

Summary

Make iOS scroll and synthesized coordinate drags avoid XCTest AX/window/keyboard lookup so they keep working when the app accessibility tree is unhealthy.

Default iOS scroll to the synthesized drag lane, add finite screenshot-size frame fallback/validation, and reuse the AX-free path for synthesized sequence drags while keeping macOS/tvOS behavior unchanged.

Validation

• pnpm build:xcuitest:ios
• pnpm build:xcuitest:macos
• pnpm check:quick
• pnpm exec vitest run src/platforms/apple/core/tests/index.test.ts
• pnpm build
• Live iPhone 17 Pro simulator C25DBB5B-9254-4293-A8D5-2785C78DE03A: opened Bluesky, confirmed sparse/AX-rejected snapshot, then verified scroll down --pixels 400, gesture pan, gesture fling, press, double-tap, swipe, and screenshot capture on the same session.

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	1.6 MB	1.6 MB	+280 B
JS gzip	502.9 kB	503.0 kB	+66 B
npm tarball	601.5 kB	602.5 kB	+1.0 kB
npm unpacked	2.1 MB	2.1 MB	+4.0 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	26.7 ms	27.0 ms	+0.3 ms
CLI --help	54.5 ms	54.0 ms	-0.5 ms

Top changed chunks: no changes in the largest emitted chunks.

[thymikee]

Swift Runner Unit Compile is failing with a real compile error:

• apple-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift:1455:50: ambiguous use of .infinity
• The failing line is fallbackScreenshotSize: { CGSize(width: .infinity, height: 932) }

Please make the type explicit there, for example CGFloat.infinity or another finite/typed value that matches the test intent, then rerun pnpm build:xcuitest:ios / the Swift runner unit compile gate.

[thymikee]

Review — keep iOS synthesized drags off AX

Solid direction: routing iOS scroll and synthesized coordinate drags through an AX-free lane (no window/keyboard/element lookup) is the right way to keep interaction working when the accessibility tree is unhealthy, and the frame-resolution ladder is well unit-tested. A few consequences of going AX-free are worth a conscious look before merge — the biggest is a hot-path perf regression, plus a couple of behaviors that silently changed.

Findings (most-severe first)

1. Every synthesized gesture now takes redundant full-screen screenshots (hot-path regression).
RunnerTests+Interaction.swift:1064 — synthesizedGestureReferenceFrame(app _:) ignores the app and calls finiteSynthesizedReferenceFrame(appFrame: .zero, fallbackBounds: .zero, …). isUsableReferenceFrame(.zero) is always false, so production always falls through to XCUIScreen.main.screenshot().image.size (:1069) — the well-tested appFrame/fallbackBounds branches are dead in production. Worse, each layer re-resolves the frame independently: a single scroll captures 3 screenshots (scrollReferenceFrame → axFreeSynthesizedDragPlan → synthesizedDragAt), a synthesized drag 2, a synthesized tap 1 — where the old path used the free app.frame. On the hottest gesture (tap) this replaces a cheap property read with a synchronous main-thread screen capture (tens of ms), which cuts against the perf goal and compounds under settle/record loops.
Fix: resolve the frame once and thread plan.referenceFrame into synthesizedDragAt instead of recomputing; and/or wire a cheap AX-free source (UIScreen.main.bounds, points) into fallbackBounds so the screenshot is only a last resort.

2. iOS synthesized drag/scroll dropped keyboard avoidance.
RunnerTests+CommandExecution.swift:1230 / :1338 — the iOS lane bypasses both keyboardAvoidingDragPoints and resolvedTouchReferenceFrame→frameAvoidingKeyboard. With a text field focused (healthy tree), a downward scroll now starts at ~85% of full screen height — inside the keyboard region — so the synthesized swipe lands on the keys instead of scrolling the list underneath. This is an intentional cost of staying AX-free, but it silently regresses the common focused-textfield case; worth confirming it's acceptable.

3. scroll/drag lost the XCUICoordinate fallback; scroll is now unconditionally synthesized.
RunnerTests+CommandExecution.swift:857 / :1291 — executeSynthesizedDragGesture returns non-nil for every iOS outcome, and scroll passes synthesized: shouldUseSynthesizedScrollPath() (always true), where it previously synthesized only when durationMs != nil. If the private XCTest event-synthesis symbol ever goes unavailable (Xcode drift / API removal), every iOS scroll and synthesized drag hard-fails UNSUPPORTED_OPERATION where the old default path degraded to a coordinate dragAt. Reasonable tradeoff given the new hint text, but it removes the safety net for a whole feature.

4. Preflight-skip now bypasses the foreground guard + interaction stabilization for scroll/drag/sequence.
RunnerTests+CommandExecution.swift:1388 + :541 — extending shouldSkipAppActivationPreflight to scroll/drag/sequence takes the early resolveAppWithoutActivation branch, skipping interaction_foreground_guard re-activation (:582) and applyInteractionStabilizationIfNeeded() (:599). A scroll issued after the target app was backgrounded injects the swipe into whatever is foreground (SpringBoard/another app) instead of re-activating it, and gestures can fire mid-transition-animation. This was already true for tap, so it's consistent — but scroll/drag previously always ran the activation path, so it's a behavior change worth an explicit sign-off.

5. Landscape coordinate mapping — low confidence, worth a device check.
RunnerTests+Interaction.swift:1064 — nativeSynthesizedPoint assumes frame is interface-oriented (as app.frame was). XCUIScreen.main.screenshot().image.size orientation/points-vs-pixels semantics differ from app.frame; portrait was validated live, but in landscape the height - localY rotation could place touches at the wrong native coordinate. Recommend a quick landscape drag/tap check on device.

Minor cleanups

• The min(max(… ?? 250, 16), 10000) duration clamp is duplicated across executeDragGesture, executeSynthesizedDragGesture, and performSequenceStep — a shared helper avoids drift.
• The finite-coordinate guard runs up to 3× per drag (executeDragGesture → axFreeSynthesizedDragPlan → synthesizedDragAt) with three different error messages; the inner two are unreachable once the outer passes.

Automated review — please sanity-check findings 2–5 against intended behavior; several are deliberate AX-free tradeoffs rather than outright bugs.

[thymikee]

Addressed the review/CI blockers in ab81d9d.

Changes:

• Fixed the Swift unit compile failure by using a typed CGFloat.infinity in the finite-frame test.
• Removed repeated frame resolution on synthesized drags: scroll computes the screenshot frame once and passes it through to synthesized drag; sequence drag does the same.
• Tried a cheap AX-free UIKit bounds source, but live verification showed it reports 320x480 on the iPhone 17 Pro runner while the actual interaction viewport is 402x874, so I removed it rather than ship bad coordinates. Screenshot sizing remains the fallback source, but now it is resolved once per gesture path instead of redundantly.
• Restored foreground guard/stabilization for drag, scroll, and sequence by limiting activation-preflight skip back to synthesized coordinate taps only.
• Added best-effort keyboard avoidance for synthesized drag/scroll when the last snapshot did not report AX rejection; after AX-rejected snapshots, the runner avoids keyboard AX probes to preserve the unhealthy-tree coordinate lane.
• Restored XCUICoordinate fallback for explicit synthesized drag when private synthesis is unavailable. Default iOS scroll intentionally still returns unsupported instead of falling back to AX-backed coordinate drag after private synthesis failure, because that fallback is the failing path on AX-broken screens.

Validation:

• pnpm build:xcuitest:ios
• pnpm build:xcuitest:macos
• pnpm check:quick
• pnpm exec vitest run src/platforms/apple/core/tests/index.test.ts
• pnpm build
• Live iPhone 17 Pro simulator C25DBB5B-9254-4293-A8D5-2785C78DE03A: Bluesky portrait scroll down --pixels 400 succeeded with referenceWidth 402/referenceHeight 874; landscape-left pan succeeded with referenceWidth 874/referenceHeight 402; screenshot artifact captured at /private/tmp/ios-ax-drag-rereview-landscape.png; session closed and verified no sessions remain.

Residual note:

• A full interactive snapshot on the current Bluesky screen still hit the severe AX runner recycle budget path during smoke. That is the pre-existing unhealthy-tree condition this PR routes coordinate scroll/drag around; coordinate scroll still succeeded afterward.

[thymikee]

Re-review after 47f0063: no actionable blockers found. The prior Swift compile failure is fixed, synthesized scroll/drag now reuses the resolved reference frame instead of recomputing screenshots, drag/scroll/sequence stay on the foreground guard/stabilization path, keyboard avoidance is gated away from AX-rejected states, and explicit synthesized drag keeps a coordinate fallback. CI is green across 21 checks, and the PR includes live iPhone 17 Pro simulator evidence for portrait scroll and landscape pan with session cleanup noted. Labeling ready-for-human for maintainer judgment.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-08 09:07 UTC