feat: e2e tests by MrMuzyk · Pull Request #1501 · callstack/repack

MrMuzyk · 2026-10-16T12:25:49Z

Summary

Adds an e2e lane that runs the MF2 tester app on a real emulator/simulator on CI, so Re.Pack bundling/runtime regressions are caught before merge instead of in a
consumer app.

New workflow .github/workflows/e2e.yml — e2e (Android) on an API 35 emulator, e2e (iOS) on an iPhone 16 simulator. Each job starts both Re.Pack dev servers (host app:8081, MiniApp :8082), warms both platform compilers, builds the Debug app, then runs the suite against it.
Suite: apps/tester-federation-v2/e2e/smoke.e2e.ts — 3 deterministic tests (app boots, Home → Detail → back, Home → Gallery loads the MiniApp over the MF remote), via tester-army/e2e + @e2e-dev/mobile. No model credential needed, which is what lets fork PRs run it at all.
Fork safety: pull_request_target with an explicit head.sha checkout, permissions: contents: read, no secrets to fork code. Same-repo PRs always run; external contributors run when a maintainer adds the e2e label.
E2E define gates LogBox.ignoreAllLogs(true) in the host app (active only with E2E=true servers) — a warning toast was observed absorbing a tap on CI. Errors still show; only warnings are silenced.

Test plan

CI is the evidence — current head passes 3/3 on both lanes. On failure, the e2e-android / e2e-ios artifacts carry junit + screenshots + logcat, and the job tail carries both dev server logs.

Last successful run

Locally (needs Node ≥ 24.8; CI pins 24.18):

Boot any local emulator/simulator and install the Debug app (pnpm android / pnpm ios work with the app's own scripts).
cd apps/tester-federation-v2 and start both servers with the flag: E2E=true pnpm start:hostapp and E2E=true pnpm start:miniapp.
pnpm exec e2e run --target android (or --target ios).

Before merging: create the e2e label in repo settings — until it exists, no fork PR can ever match the gate.

vercel · 2026-10-16T12:25:53Z

@MrMuzyk is attempting to deploy a commit to the Callstack Team on Vercel.

A member of the Team first needs to authorize it.

changeset-bot · 2026-10-16T12:25:55Z

⚠️ No Changeset found

Latest commit: c33c7bb

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

# Conflicts: # apps/tester-federation-v2/package.json # pnpm-lock.yaml

dannyhw

Thanks for your work on this. I ran the suite locally at 4feac606 on a dedicated API 35 emulator and an iOS 26.4 simulator: 3/3 on Android and 3/3 on iOS, with --workers 1 --retries 0, in a separate run per platform. So the tests themselves work.

Inline comments cover the details. The main points:

Trigger model: I'd switch from pull_request_target to pull_request. See the comment on on:.
The root cause from #1499 is still there. LogBox.ignoreAllLogs(true) hides the toast, but the MF2 shared versions still resolve to catalog:testers. Without E2E=true, the Gallery test still fails locally.
Determinism: pin the device, set retries: 0 explicitly, and turn telemetry off.

Also:

e2e label: the repo doesn't have it yet (gh label list returns nothing). Without it, labelled fork runs are skipped silently. The description already lists this as a step before merge, so I'm noting it here so it doesn't get missed.
No CI run at this head. The green run linked in the description is a workflow_dispatch on c33c7bb3. The only difference from 4feac606 is that src/host/index.js loses the comment explaining the LogBox gating. If the gating stays, please keep that comment.
Running locally with Xcode 27: the iOS build needs IPHONEOS_DEPLOYMENT_TARGET=15.1 passed to xcodebuild, because AsyncStorage's pod targets 13.0. This doesn't affect CI on macos-15/Xcode 16.4.

dannyhw · 2026-10-17T13:46:32Z

+name: e2e
+
+on:
+  pull_request_target:


I'd use pull_request here instead of pull_request_target. The job uses no secrets and only needs contents: read, so the target trigger adds risk without enabling anything:

The label stays on the PR. Once a maintainer adds e2e to a fork PR, every later synchronize push runs fork code with no fresh review. With pull_request, GitHub's fork-workflow approval does this gating.

The fork picks the pnpm version. setup-pnpm-node passes no version, so pnpm/action-setup installs whatever packageManager says in the checked-out, fork-controlled package.json, against the fork's lockfile.

Workflow changes can't be tested before merge. With pull_request_target the workflow file always comes from main, so a PR's edits to e2e.yml never run on that PR. That includes this PR.

It tests the PR head, not the merge. The checkout uses head.sha. The default pull_request checkout uses the merge ref, which is the right thing for a merge gate.

On cache poisoning, the concern from #1499: GitHub now gives untrusted triggers a read-only cache token for the default-branch scope, so this is mitigated by the platform. The points above still apply.

With pull_request, I'd also:

drop labeled/unlabeled from types, since each label change currently starts a new run

add paths: (e.g. apps/tester-federation-v2/**, packages/**, pnpm-lock.yaml and this workflow file) so docs-only PRs don't spend emulator minutes

dannyhw · 2026-10-17T13:46:32Z

+# one run per PR; a new push cancels the older one
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true


This also applies to push on main, so the next merge cancels the run for the previous one. That run is the signal that main is healthy.

Suggested change

cancel-in-progress: true

cancel-in-progress: ${{ github.event_name != 'push' }}

dannyhw · 2026-10-17T13:46:32Z

+            adb reverse --list
+            adb reverse --list | grep -q 'tcp:8081' || { echo '::error::adb reverse missing for tcp:8081'; exit 1; }
+            adb reverse --list | grep -q 'tcp:8082' || { echo '::error::adb reverse missing for tcp:8082'; exit 1; }
+            adb install -r -g android/app/build/outputs/apk/debug/app-debug.apk


android-emulator-runner runs script: with sh -c and no -e. The two adb reverse --list checks are guarded, but adb install isn't. If the install fails (e.g. INSTALL_FAILED_UPDATE_INCOMPATIBLE), the suite still runs and fails about 60 s later with "home-details-button not found".

set -e would break the ec=$? / logcat pattern below, so guarding this line is simpler:

Suggested change

adb install -r -g android/app/build/outputs/apk/debug/app-debug.apk

adb install -r -g android/app/build/outputs/apk/debug/app-debug.apk || { echo '::error::adb install failed'; exit 1; }

dannyhw · 2026-10-17T13:46:32Z

+            adb reverse --list | grep -q 'tcp:8081' || { echo '::error::adb reverse missing for tcp:8081'; exit 1; }
+            adb reverse --list | grep -q 'tcp:8082' || { echo '::error::adb reverse missing for tcp:8082'; exit 1; }
+            adb install -r -g android/app/build/outputs/apk/debug/app-debug.apk
+            pnpm exec e2e run --target android --reporter list,junit; ec=$?; adb logcat -d -v time > .e2e/logcat.txt 2>&1; exit $ec


e2e sends anonymous usage telemetry to PostHog by default. Neither job opts out, and the e2e docs recommend setting the opt-out in CI. A workflow-level env: covers both lanes:

env: E2E_TELEMETRY_DISABLED: '1'

dannyhw · 2026-10-17T13:46:32Z

+          name: e2e-android
+          path: apps/tester-federation-v2/.e2e
+          include-hidden-files: true
+          if-no-files-found: error


If the job fails before the e2e step (Gradle, pod install, dev server timeout), .e2e doesn't exist. This step then fails too, so the run shows a second error next to the real one. warn keeps the upload without that extra failure. The same applies to the iOS job.

Related: Show dev server logs runs after the upload, so the host and miniapp server logs never end up in the artifact. iOS artifacts also contain only the report and junit (no device log). Copying /tmp/hostapp.log and /tmp/miniapp.log into .e2e/ before uploading would make failures debuggable from the artifact.

dannyhw · 2026-10-17T13:46:32Z

+  targets: [
+    {
+      name: 'android',
+      engine: mobile({ platform: 'android' }),


Without device, @e2e-dev/mobile uses every booted device of the platform. Locally, e2e defaults to workers = cores / 2. When I ran pnpm exec e2e run --target android on a machine with several booted emulators and simulators, it would have spread across all of them, including ones other work was using. CI behaves only because each job has exactly one device and CI mode defaults workers to 1.

Pinning it in the config, with an env override for local runs, solved this for me:

engine: mobile({ platform: 'android', device: process.env.E2E_DEVICE }),

On iOS the job already names its simulator e2e, so CI can pass that in.

I'd also set retries: 0 and workers: 1 explicitly at the top level. In CI mode e2e defaults to retries: 1, and a test that fails once and then passes is counted as flaky, not failed, so the check stays green. For a 3-test smoke lane that targets cold start and MF remote loading, an intermittent first-attempt failure is exactly the regression we want to see.

dannyhw · 2026-10-17T13:46:32Z

 import App from './App';

+if (__E2E__) {
+  LogBox.ignoreAllLogs(true);


This hides the symptom from #1499, but the cause is still there. The green run's logcat.txt still logs [ Federation Runtime ] Version catalog:testers from HostApp of shared singleton module react-native-safe-area-context does not satisfy the requirement… (and the same for react-native-screens) on every launch. Locally, at this head, the Gallery test fails without E2E=true because the toast covers the tile.

Cause: the four MF configs pass pkg.dependencies['react-native-safe-area-context'] and ['react-native-screens'] as the shared version/requiredVersion. In package.json those values are the literal string catalog:testers, which MF2's semver check can't parse. Importing the real versions fixes it, the same way the configs already handle react and react-native:

import safeAreaPkg from 'react-native-safe-area-context/package.json' with { type: 'json' }; import screensPkg from 'react-native-screens/package.json' with { type: 'json' }; // ... version: safeAreaPkg.version, requiredVersion: safeAreaPkg.version,

With that in place, this line can go entirely, or be narrowed to LogBox.ignoreLogs([...]) for anything specific that's left.

Also, ignoreAllLogs(true) hides console.error toasts too, not just warnings (see LogBoxNotificationContainer, which returns nothing when isDisabled). Only fatal errors and red boxes still render. The description says errors are still shown, which only holds for fatal ones. A global mute also means future MF or React Native warnings won't show up in this lane, and catching those is what it's for.

dannyhw · 2026-10-17T13:46:32Z

        __WITH_PRELOAD__:
          process.env.WITH_PRELOAD === 'true' ||
          process.env.WITH_PRELOAD === '1',
+        __E2E__: process.env.E2E === 'true',


Minor: __E2E__ (like the existing __WITH_PRELOAD__) isn't declared anywhere for TypeScript, and this app has no typecheck script, so turbo run typecheck skips it. That means e2e.config.ts and e2e/*.e2e.ts are also never type-checked. A small src/globals.d.ts with declare const __E2E__: boolean; plus "typecheck": "tsc --noEmit" (as tester-expo has) would cover both.

dannyhw · 2026-10-17T13:46:32Z

  },
  "devDependencies": {
    "@babel/core": "^7.29.7",
+    "@e2e-dev/mobile": "0.11.0",


Two things about the new deps:

Catalog: every other tester dependency is catalog-managed, and @e2e-dev/mobile has a peer dependency on e2e >=0.15 <1, so the two need to move together. Adding them to catalog:testers (or adding a "pinned together" comment) would make that explicit. They also bring @vitest/*@5 into the lockfile while the catalog pins vitest ^4. That's harmless, but worth knowing.

Node version: e2e@0.19.0 and @e2e-dev/mobile@0.11.0 declare engines.node: ^22.22.3 || >=24.8.0, but the root engines allows >=24. There's no engine-strict, so on 24.0–24.7 a contributor only sees a warning and then gets a confusing failure.

dannyhw · 2026-10-17T13:46:32Z

    if (this.state.hasError) {
      return (
-        <View style={styles.container}>
+        <View testID="miniapp-error" style={styles.container}>


Nit: nothing uses miniapp-error yet. When the remote fails to load, the Gallery test keeps polling miniapp-gallery for 45 s and then reports a timeout, so a broken remote looks the same as a slow device. Checking miniapp-error alongside the gallery check would make it fail fast with a clear message. Otherwise this testID can be dropped.

MrMuzyk added 2 commits October 13, 2026 15:34

ci: temp e2e workflow

6ff87a3

Merge branch 'callstack:main' into main

c5fa84d

feat: repack e2e tests

4feac60

# Conflicts: # apps/tester-federation-v2/package.json # pnpm-lock.yaml

MrMuzyk force-pushed the e2e-ci branch from c33c7bb to 4feac60 Compare October 16, 2026 13:58

MrMuzyk marked this pull request as ready for review October 16, 2026 13:59

dannyhw reviewed Oct 17, 2026

View reviewed changes

	cancel-in-progress: true
	cancel-in-progress: ${{ github.event_name != 'push' }}

	adb install -r -g android/app/build/outputs/apk/debug/app-debug.apk
	adb install -r -g android/app/build/outputs/apk/debug/app-debug.apk \|\| { echo '::error::adb install failed'; exit 1; }

Conversation

MrMuzyk commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

vercel Bot commented Oct 16, 2026

Uh oh!

changeset-bot Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

dannyhw left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MrMuzyk commented Oct 16, 2026 •

edited

Loading

changeset-bot Bot commented Oct 16, 2026 •

edited

Loading

dannyhw left a comment •

edited

Loading