Conversation
Windows ran one worker per vCPU, leaving no headroom for the helper processes some tests spawn, and the deadline-sensitive tests missed their budgets non-deterministically. The worker count and the job timeout are now per-platform; linux and macos are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Windows python-test jobs flake non-deterministically. They ran
pytest -n logical— one xdist worker per vCPU, four on awindows-2022standard runner.OMP_NUM_THREADS/MKL_NUM_THREADScap torch's pools at one thread per worker, but they do not cap the helper processes some tests spawn (ffmpeg decoders intests/app/util/test_video_thumbnails.py, installer subprocesses), so a full-width run leaves the runner with no spare core and the deadline-sensitive tests miss their budgets at random.Observed on recent runs:
tests/app/util/test_video_thumbnails.py::TestStreamedDecoderIsBounded::test_capacity_wait_and_first_frame_share_one_deadline—first frame got a fresh timeout after the capacity wait (2.89s / 3.58s)tests/app/services/shared/test_for_scheduler_performance.py::test_loop_scheduler_overhead_is_linear[iterate]Windows now runs two workers. The worker count and the job timeout moved into the matrix; linux and macos keep
-n logicaland the 30-minute timeout. Windows gets 45 minutes, because the pytest step was already ~13.5 min of a ~15 min job at four workers and halving the workers does not leave room under 30.Not fixed here, and still open: the second Windows failure mode on these runs is a hard
Windows fatal exception: code 0xc000001d(illegal instruction) insideF.linearon bfloat16 CPU tensors —invokeai/backend/quantization/dequantizing_linear.py:49, reached from the NVFP4 loader tests (test_krea2_nvfp4_loader.py,test_z_image_loader_boundaries.py). It is a CPU-ISA dispatch crash, not contention, and it is non-deterministic because GitHub's Windows pool mixes CPU generations. Fewer workers does not address it.QA Instructions
python -c "import yaml; yaml.safe_load(open('.github/workflows/python-tests.yml'))"— parses.windows-cpujobs are the check. Confirm both Windows jobs run-n 2and finish inside 45 minutes.py3.11: windows-cpu, passing):run pytest809s, whole job ~14 min.Review
Mechanical CI configuration change; no independent review subagents were run. Self-checks: YAML parses and the matrix resolves to the intended per-platform values;
release.ymlcalls this workflow viaworkflow_calland does not touch the matrix, so it is unaffected.Checklist
What's Newcopy (if doing a release after this PR)🤖 Generated with Claude Code