Skip to content

ci: size pytest workers per platform - #309

Open
lstein wants to merge 1 commit into
mainfrom
ci/cap-macos-pytest-workers
Open

lstein wants to merge 1 commit into
mainfrom
ci/cap-macos-pytest-workers

Conversation

@lstein

@lstein lstein commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

The macOS test job ran out of memory rather than out of CPU. macos-14-arm64 has 3 vCPUs but only 7GB of RAM, and -n logical started one worker per vCPU: the suite's own reporter measured 6.70GB summed across 3 workers (6.77GB on py3.12), against 10.04GB/4 workers on the 16GB Linux runner and 7.82GB/4 on Windows. macOS was the only platform running at the wall, and it is also the only runner whose auto device is a GPU, so MPS buffers come out of that same 7GB. The failure mode was an allocator that could not satisfy a 256-byte Metal buffer.

Worker count and job timeout are now per-platform:

platform before after
linux-cpu (4 vCPU / 16GB) -n logical, 30 min unchanged
windows-cpu (4 vCPU / 16GB) -n logical, 30 min unchanged
macos-default (3 vCPU / 7GB) -n logical, 30 min -n 2, 40 min

macOS is the only behavioral change in the workflow; the Linux and Windows command lines are byte-identical to before.

This PR reduces the memory pressure. The matching root cause — a dtype query that answered by allocating on the device, and treated an allocation failure as fatal — is fixed separately in #310, which is what makes the affected tests pass on a loaded runner.

QA Instructions

This workflow is in its own change filter, so this PR runs its own six pytest jobs, including both macOS jobs at -n 2. What to look for in the macOS logs:

  • 2 workers [N items] at startup.
  • The peak RSS: …GB worst worker, …GB summed over 2 workers line, expected near 4.5GB.
  • Actual wall time, which the 40-minute budget is sized against. That budget is extrapolated, not measured: 16 min at three workers, and OMP_NUM_THREADS: 1 leaves each of two workers a full core of the runner's three. If the real number lands well under, the timeout can come back down in a follow-up.

Verified locally: the matrix expands to the same six jobs with both new keys present on every one; timeout-minutes accepts the matrix context at job level (the job already uses matrix.os in runs-on); the workflow_call path from release.yml is unaffected.

Review

Two independent read-only reviews, both adversarial, on the fixed diff:

  • GitHub Actions mechanics. Found two stale documentation claims — tests/AGENTS.md and root AGENTS.md both stated -n logical as CI's invocation, one per vCPU. Both corrected here. No defect found in the matrix expansion, expression legality, shell quoting, or the reusable-workflow path.
  • Operational safety and lever choice. Three material findings, all addressed in the diff or disclosed below:
    1. The peak-RSS reporter uses RUSAGE_SELF, so it never counts the Python children that ~11 test files spawn. I measured one at 1.53GB — a worker running test_module_compatibility.py holds a child that large while resident. The comment no longer claims a specific clearance and names this blind spot instead. Follow-up worth taking: teach the reporter to report children's peak, so the number CI decisions are sized from covers them. Not done here because RUSAGE_CHILDREN reports the max over reaped children, which cannot simply be added to RUSAGE_SELF to give a concurrent peak; doing it honestly is its own change.
    2. The timeout's "~23 min" was extrapolated and did not say so. The comment now states its provenance and reasoning.
    3. Running macOS ~0.3GB from the ceiling made the reporter self-enforcing: a memory regression turned that job red at the PR that caused it. At two workers a +0.5GB/worker regression can land green. Follow-up worth taking: a budget assertion on the summed figure, in the style of the frontend's maintained performance budgets. Deliberately not added here — it is a new gate, not a worker cap.

A reviewer also proposed INVOKEAI_device: cpu on the macOS job as a cheaper fix for the immediate symptom. Not taken: it would drop the incidental MPS coverage this job uniquely provides, and #310 removes the symptom without that cost.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Meaningful regression coverage added / updated where needed; obsolete tests/code removed
  • Persisted-state and API changes include required migrations / compatibility validation
  • Relevant performance/efficiency opportunities considered; material claims have evidence
  • Material review findings resolved and relevant checks rerun
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

🤖 Generated with Claude Code

The macOS runner has 7GB for its three vCPUs. One worker per vCPU peaked
at 6.70GB summed, leaving too little wired memory for the MPS allocator,
which then failed a 256-byte buffer. Worker count and job timeout are now
per-platform; Linux and Windows keep one worker per vCPU.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant