Skip to content

ci: run the windows python tests on two xdist workers - #313

Open
lstein wants to merge 1 commit into
mainfrom
fix/windows-pytest-worker-count
Open

lstein wants to merge 1 commit into
mainfrom
fix/windows-pytest-worker-count

Conversation

@lstein

@lstein lstein commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

The Windows python-test jobs flake non-deterministically. They ran pytest -n logical — one xdist worker per vCPU, four on a windows-2022 standard runner. OMP_NUM_THREADS/MKL_NUM_THREADS cap torch's pools at one thread per worker, but they do not cap the helper processes some tests spawn (ffmpeg decoders in tests/app/util/test_video_thumbnails.py, installer subprocesses), so a full-width run leaves the runner with no spare core and the deadline-sensitive tests miss their budgets at random.

Observed on recent runs:

  • tests/app/util/test_video_thumbnails.py::TestStreamedDecoderIsBounded::test_capacity_wait_and_first_frame_share_one_deadlinefirst frame got a fresh timeout after the capacity wait (2.89s / 3.58s)
  • tests/app/services/shared/test_for_scheduler_performance.py::test_loop_scheduler_overhead_is_linear[iterate]

Windows now runs two workers. The worker count and the job timeout moved into the matrix; linux and macos keep -n logical and the 30-minute timeout. Windows gets 45 minutes, because the pytest step was already ~13.5 min of a ~15 min job at four workers and halving the workers does not leave room under 30.

Not fixed here, and still open: the second Windows failure mode on these runs is a hard Windows fatal exception: code 0xc000001d (illegal instruction) inside F.linear on bfloat16 CPU tensors — invokeai/backend/quantization/dequantizing_linear.py:49, reached from the NVFP4 loader tests (test_krea2_nvfp4_loader.py, test_z_image_loader_boundaries.py). It is a CPU-ISA dispatch crash, not contention, and it is non-deterministic because GitHub's Windows pool mixes CPU generations. Fewer workers does not address it.

QA Instructions

  • python -c "import yaml; yaml.safe_load(open('.github/workflows/python-tests.yml'))" — parses.
  • Effect is observable only on the hosted Windows runners; this PR's own windows-cpu jobs are the check. Confirm both Windows jobs run -n 2 and finish inside 45 minutes.
  • Baseline measured from run 35675232701 (py3.11: windows-cpu, passing): run pytest 809s, whole job ~14 min.

Review

Mechanical CI configuration change; no independent review subagents were run. Self-checks: YAML parses and the matrix resolves to the intended per-platform values; release.yml calls this workflow via workflow_call and does not touch the matrix, so it is unaffected.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Meaningful regression coverage added / updated where needed; obsolete tests/code removed
  • Persisted-state and API changes include required migrations / compatibility validation
  • Relevant performance/efficiency opportunities considered; material claims have evidence
  • Material review findings resolved and relevant checks rerun
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

🤖 Generated with Claude Code

Windows ran one worker per vCPU, leaving no headroom for the helper
processes some tests spawn, and the deadline-sensitive tests missed their
budgets non-deterministically. The worker count and the job timeout are now
per-platform; linux and macos are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant