Skip to content

ci: add E2E inventory and retry reporting - #291

Draft
piotr-iohk wants to merge 18 commits into
mainfrom
codex/264-inventory-shadow
Draft

piotr-iohk wants to merge 18 commits into
mainfrom
codex/264-inventory-shadow

Conversation

@piotr-iohk

@piotr-iohk piotr-iohk commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #264.

Companion: synonymdev/bitkit-ios#1199

This PR supplies the inventory and attempt reporting consumed by the smaller iOS gate. The iOS repository owns path policy and automatic selection. Android's workflow selection and legacy retry behavior stay unchanged.

Description

  • Supplies stable case IDs and backend/disposition metadata to the iOS-owned selector. No iOS source paths or app selection rules live in this repository.
  • Inventories all 69 existing declarations, including fixtures, manual cases and explicit gaps. The executable iOS nightly set is 42 local, six staging and six migration cases.
  • Restores the established ciIt() pass locks: cases passed earlier in this run are skipped, so only failed or uncompleted cases repeat. The iOS workflows expose three numbered attempts, stopping on success. The final report combines current-run passes, retains the first failure and fails after three unsuccessful attempts. App/E2E/artifact identities remain pinned.
  • Loads selected spec files as one sequential WDIO worker per simulator and adds an optional immutable source pin for migration fixture preparation.
  • Removes the earlier replacement smoke implementation. The final change uses existing test assertions and has no shadow rollout.

Merge this producer before its iOS consumer. The consumer pins the reusable fixture workflow so development branches can be validated before merge.

Validation

  • 105 shared unit/contract tests and 33 iOS policy tests passed, covering inventory, actual app paths and policy drift, retries/recovery, missing or stale reports, installed WDIO scheduling and fixture-source pinning.
  • Inventory validation and TypeScript checking passed.
  • Cross-repository workflow contract checks and actionlint passed, including 19 aggregate-status scenarios.
  • Independent implementation review completed with source findings corrected.

At E2E b691df2 / iOS 7380b949, controlled Mocha executions through the real attempt CLI verify that passed bodies run once, only failures repeat, recovery works on attempts two and three, persistent failures fail, and startup/timeouts/failed hooks/stale locks retain correct outcomes. The workflow check verifies three visible attempts and a final report that fails the job.

The runner contract tests use controlled fixtures; they are not hosted Appium evidence. The initial hosted five-case gate passed first attempt at E2E 39f8964 / iOS f0a4aa3 (run); that is historical runtime evidence before policy relocation. The current hosted policy check, full lanes and optional cost comparison are recorded separately. Existing iOS mainnet/skip gaps are documented, not counted as passes.

Current full-suite validation

Keep one automatic local shard. All 42 local cases passed on both runs at iOS 5ba352f7 / E2E ee7a429. Two shards took 37.6% longer and used 71.0% more estimated Namespace compute on this pair.

Local measurement One shard Two shards
Elapsed, including setup and cleanup 96.57 min 132.87 min
Total Namespace Mac time 96.57 min 165.17 min
Total backend lifetime 95.93 min 163.87 min
First-attempt cases 42/42 41/42
Estimated prepaid-equivalent compute $6.56 $11.22
Estimated overage compute $9.84 $16.83

The 24-case shard passed first try in 30m44s of execution. In the 18-case shard, boost_1 timed out waiting for BoostButton. The retained screenshot and recording show iOS Settings in the foreground. Its other 17 cases passed, but the then-current runner repeated all 18 and then passed. The retry correction restores failed-only retries; this historical pair does not measure that behavior. This is one observation with a recovered failure. Estimates use measured runner/backend lifetimes and published rates. They exclude storage, rounding and account discounts. One-shard baseline · Two-shard experiment · Namespace rates

The baseline completed 54/54 cases under the preceding workflow wiring. The regular two-shard run completed 48/48, covering 42 local and six staging cases. Its selection contained no migration. Only local elapsed time and paid Namespace compute are compared. An earlier two-shard dispatch mistakenly included migration and was cancelled. Its estimated $1.55 prepaid-equivalent / $2.32 overage is recorded separately. Cancelled dispatch

In the draft at iOS 94088e98, e2e-tests.yml contains no migration job, call, output or status dependency. e2e_migration.yml owns a separate build, nightly schedule, release-push trigger and result. All six case assertions passed in the historical run. The independent build and artifact handoff will be verified on its next nightly or release run. Both PRs remain drafts.

The preceding run restored the correct 449,502 sats in RN, but its upgrade and passphrase tests could not read the nested amount on iOS. The corrected test reads the balance from its parent accessibility label. A local iOS simulator using the hosted target build and funded RN fixture passed the passphrase upgrade, including balances, history and tags after relaunch. Preceding run

The previous full run passed all six staging cases on the first attempt and five of six migration cases within three attempts. All 12 RN source wallets were prepared. The RN passphrase case failed. One recording showed the correct 449,502 sats behind a USDT introduction. Another showed a truncated passphrase and an empty wallet before the upgrade. The harness now checks the exact passphrase and funded RN balance before installing the native app, then dismisses the introduction before verifying balances and history. Migration job

The previous local job reached its 90-minute limit three times, with 37, 38 and 39 of 42 cases passing on the respective attempts. Security and transfer each passed on a subsequent attempt. The first PIN reset showed a regtest log-deletion error and returned to home instead of onboarding. The transfer recording showed QuickPay covering the activity assertions. The security assertion remains in the suite. The transfer case handles the one-time introduction and still verifies the balance and both activity rows. Local job

That unsuccessful run consumed 271.72 Namespace Mac job-minutes and 271.00 observed backend minutes. Published rates imply $18.47 prepaid-equivalent or $27.71 overage for compute. The estimate excludes storage, credits, rounding and account discounts. Run · Namespace pricing

The initial diagnostic also exposed a redundant Android Back after RN send success that backgrounded the source wallet. Its correction was exercised by all three sweep wallets in the next run. Initial diagnostic

Models used

  • Planning/scoping: gpt-6-astra (reasoning: high), gpt-6-astra (reasoning: xhigh), gpt-6.1-sol (reasoning: high)
  • Implementation: gpt-6-astra (reasoning: high), gpt-6-astra (reasoning: xhigh), gpt-6.1-sol (reasoning: high)
  • Review: gpt-6-astra (reasoning: high), gpt-6.1-sol (reasoning: high)

@piotr-iohk piotr-iohk changed the title ci: add iOS risk selection and retry reporting ci: add E2E inventory and retry reporting Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant