Conversation
The macOS runner has 7GB for its three vCPUs. One worker per vCPU peaked at 6.70GB summed, leaving too little wired memory for the MPS allocator, which then failed a 256-byte buffer. Worker count and job timeout are now per-platform; Linux and Windows keep one worker per vCPU. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The macOS test job ran out of memory rather than out of CPU.
macos-14-arm64has 3 vCPUs but only 7GB of RAM, and-n logicalstarted one worker per vCPU: the suite's own reporter measured 6.70GB summed across 3 workers (6.77GB on py3.12), against 10.04GB/4 workers on the 16GB Linux runner and 7.82GB/4 on Windows. macOS was the only platform running at the wall, and it is also the only runner whoseautodevice is a GPU, so MPS buffers come out of that same 7GB. The failure mode was an allocator that could not satisfy a 256-byte Metal buffer.Worker count and job timeout are now per-platform:
-n logical, 30 min-n logical, 30 min-n logical, 30 min-n 2, 40 minmacOS is the only behavioral change in the workflow; the Linux and Windows command lines are byte-identical to before.
This PR reduces the memory pressure. The matching root cause — a dtype query that answered by allocating on the device, and treated an allocation failure as fatal — is fixed separately in #310, which is what makes the affected tests pass on a loaded runner.
QA Instructions
This workflow is in its own change filter, so this PR runs its own six pytest jobs, including both macOS jobs at
-n 2. What to look for in the macOS logs:2 workers [N items]at startup.peak RSS: …GB worst worker, …GB summed over 2 workersline, expected near 4.5GB.OMP_NUM_THREADS: 1leaves each of two workers a full core of the runner's three. If the real number lands well under, the timeout can come back down in a follow-up.Verified locally: the matrix expands to the same six jobs with both new keys present on every one;
timeout-minutesaccepts thematrixcontext at job level (the job already usesmatrix.osinruns-on); theworkflow_callpath fromrelease.ymlis unaffected.Review
Two independent read-only reviews, both adversarial, on the fixed diff:
tests/AGENTS.mdand rootAGENTS.mdboth stated-n logicalas CI's invocation, one per vCPU. Both corrected here. No defect found in the matrix expansion, expression legality, shell quoting, or the reusable-workflow path.RUSAGE_SELF, so it never counts the Python children that ~11 test files spawn. I measured one at 1.53GB — a worker runningtest_module_compatibility.pyholds a child that large while resident. The comment no longer claims a specific clearance and names this blind spot instead. Follow-up worth taking: teach the reporter to report children's peak, so the number CI decisions are sized from covers them. Not done here becauseRUSAGE_CHILDRENreports the max over reaped children, which cannot simply be added toRUSAGE_SELFto give a concurrent peak; doing it honestly is its own change.A reviewer also proposed
INVOKEAI_device: cpuon the macOS job as a cheaper fix for the immediate symptom. Not taken: it would drop the incidental MPS coverage this job uniquely provides, and #310 removes the symptom without that cost.Checklist
What's Newcopy (if doing a release after this PR)🤖 Generated with Claude Code