[danceratopz]

Description

In consume enginex and consume wirex, each xdist worker starts one client per pre-allocation group, runs the group's tests, stops it, and starts the next one. On the hive-tests mainnet dashboard (8 vCPU runners, --sim.parallelism=4), workers spend most of their time between containers waiting for that start: in consume enginex, besu starts in 7.6 s and nethermind in 3.9 s per group, and workers spend 73 to 78% of their time between containers for those clients.

This PR overlaps the start with the current group's tests:

• Start ahead. Once a worker's current client is ready, it starts the client of the next group queued to it on a background thread. That group's first test claims it, waiting if the start has not finished. A worker holds at most one client started ahead, so -n 4 has at most 8 client containers alive.
• Lookahead scheduler. xdist's loadgroup hands a worker its next group only when it has two or fewer tests left, and most groups have one or two tests, so the next group is rarely known in time. GroupLookaheadScheduling (through pytest_xdist_make_scheduler) keeps one more group queued per worker, handed out once the current group has at most 100 tests left, so a long group cannot hold another back from idle workers. Groups still run whole on one worker.
• wirex. A client started ahead is a fresh group client, so the wirex reuse policy (replacement_reason) applies to it unchanged, and isolated multi-target clients never use it.
• Cleanup. Remaining clients are now stopped when the module ends, before the shared hive test they belong to, so a start still running when a run is interrupted cannot outlive it. This also stops leftover clients before hive ends their test, rather than after.

It is on by default for both simulators. --no-pipeline-client-start turns it off, for example to debug a client start or on a host short of memory. Since the dashboards run the published simulator images, they pick it up without a hive or hive-tests change.

Measurements

Think of it as a pit crew warming up the next car while the current one is still on the track. Local A/B on a host pinned to the CI runner layout (4 physical cores with SMT: hive, consume and every client container confined to the same 8 threads), -n 4, tests@v21.0.2, client images pinned by digest, runs interleaved off and on.

Latest, with a cleaned-up Docker store (like CI, which starts each job with an empty one) and the pending hive start fixes (output attach, anonymous volumes, client wrappers):

Simulator, selection	Client	Off (s)	On (s)	Change
enginex, full release (83,598 tests)	ethrex	1,421	1,341	−5.6%
enginex, fork_Cancun (12,875 tests)	ethrex	230	221	−4.1%
enginex, fork_Cancun	nethermind	966	895	−7.4%

Earlier, before this round of host and hive fixes, when a container start and stop cost about twice as much on this host:

Simulator, selection	Client	Off (s)	On (s)	Change
enginex, full release	ethrex	2,206	1,899	−13.9%
enginex, fork_Cancun	go-ethereum	402	336	−16.4%
enginex, fork_Cancun	nethermind	1,027	921	−10.3%
enginex, fork_Cancun	besu	1,896	1,811	−4.5%¹
wirex, tests/(cancun|prague) at fork_Prague	besu	745	752	+1.0%¹

(Means of one to three runs per arm.) No test changed verdict between off and on in any of the more than 50 runs, no client container was left running, and no client stop failed. The cheaper the start, the less there is to hide, so the latest table is the better guide to CI: expect a few percent for most clients, more where the client's own boot is slow.

¹ Besu's start is CPU-bound (about 16 s here, 7.6 s on CI) and longer than most groups, so only about 30% of it is hidden, and the boot running in the background competes with the tests next to it for the same 8 threads: workers wait less for starts but run tests more slowly by almost as much. Running more besu clients at once does not help either: at -n 8 with pipelining off, besu enginex takes about the same time as at -n 4 with it on.

Notes for review

• The per-group builders in multi_test_client.py (pre-alloc group, client genesis, environment) and the hive start are moved out of their fixtures into plain functions, so the start ahead can build the next group's inputs outside pytest's fixture graph. Fixture behaviour is unchanged.
• The next group's environment is the current test's environment with the group-specific part swapped, which keeps what a simulator adds to the environment fixture (wirex's sync loop interval).
• A worker is sent whole groups in order, so next_group_item reads the last queued item first and only scans the queue once another group has arrived. Without that, every test of a long group would scan the group's remaining tests while it waits for the lookahead (quadratic; it added about 2 minutes to a wirex run with one 11,000-test group).
• xdist runs its worker module as __channelexec__, so the worker's test queue is found by its attributes rather than by isinstance(…, WorkerInteractor).
• With pipelining off, the "Start client" timing total now includes the genesis serialization (a few milliseconds).
• pyproject.toml gains a mypy override for xdist.*, which ships no type information.

Related Issues or PRs

N/A.

Checklist

• Ran fast static checks to avoid CI fails, see Code Standards & Verifying Changes: just static
• PR title has the form <type>(<area>): <description>, where <type> and <area> come from an appropriate C-<type>, respectively A-<area>, label. The title should match the target squash commit message.

Cute Animal Picture

No picture, but here is a snail that learned to start its engine before the light turns green: 🐌💨

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.10%. Comparing base (fd92984) to head (3220155).

Additional details and impacted files

@@              Coverage Diff              @@
##           forks/bogota    #3952   +/-   ##
=============================================
  Coverage         95.10%   95.10%           
=============================================
  Files               673      673           
  Lines             40904    40904           
  Branches           3778     3778           
=============================================
  Hits              38903    38903           
  Misses             1393     1393           
  Partials            608      608           

Flag	Coverage Δ
unittests	95.10% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bt-recneps]

Lookahead scheduling works on real xdist workers, but the wirex env carry-over is untested and the docs' container cap misses wirex :)

_{Reviewed in 10m 15s}

[danceratopz]

Needs more better analysis on more data, but it seems to help a wee bit. Could improve much more for managed runtime clients if we make more cores available (soon).