Skip to content

fix(test runner): do not count statically skipped tests when sharding - #42916

Merged
Yury Semikhatsky (yury-s) merged 1 commit into
microsoft:mainfrom
yury-s:fix-42875
Sep 24, 2026
Merged

Yury Semikhatsky (yury-s) merged 1 commit into
microsoft:mainfrom
yury-s:fix-42875

Conversation

@yury-s

Copy link
Copy Markdown
Member

Summary

  • Statically skipped tests (test.skip, test.fixme, describe.skip) no longer count toward shard sizes, so shards with many skipped tests are not starved of real work.
  • Fully skipped groups follow the preceding group; if all tests are skipped, they are balanced as before.

Fixes #42875

@github-actions

Copy link
Copy Markdown
Contributor

Test results for "MCP"

1 failed
❌ [chromium] › mcp/http.spec.ts:105 › http transport browser lifecycle (isolated) @mcp-ubuntu-latest-chromium

8711 passed, 1480 skipped


Merge workflow run.

@github-actions

Copy link
Copy Markdown
Contributor

Test results for "tests 1"

8 flaky ⚠️ [installation tests] › playwright-cli-install-should-work.spec.ts:144 › install command should work with HTTPS proxy for HTTP downloads `@package-installations-ubuntu-latest`
⚠️ [installation tests] › playwright-test-package-managers.spec.ts:19 › npm: @playwright/test should work `@package-installations-ubuntu-latest`
⚠️ [chromium-library] › library/beforeunload.spec.ts:130 › should support dismissing the dialog multiple times `@chromium-ubuntu-22.04-arm-node20`
⚠️ [chromium-library] › library/chromium/chromium.spec.ts:301 › should report intercepted service worker requests in HAR `@chromium-ubuntu-22.04-node20`
⚠️ [chromium-library] › library/video.spec.ts:762 › screencast › should work with video+trace `@chromium-ubuntu-22.04-node22`
⚠️ [chromium-page] › page/workers.spec.ts:191 › should attribute network activity for worker inside iframe to the iframe `@chromium-ubuntu-22.04-node22`
⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:257 › third party 'Partitioned;' cookies `@firefox-ubuntu-22.04-node20`
⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:470 › top level 'Partitioned;' cookie and same origin iframe `@firefox-ubuntu-22.04-node20`

52282 passed, 1241 skipped


Merge workflow run.

@github-actions

Copy link
Copy Markdown
Contributor

Hi, I'm the Playwright bot and I took a first look at the CI failures.

🟢 The failure looks like a pre-existing flake, not caused by this PR

The only failing test, mcp/http.spec.ts:105, is a known intermittent timeout on main and on unrelated PRs. This PR only changes how the test runner builds shards.

Details

This PR changes packages/playwright/src/runner/testGroups.ts (shard grouping for statically skipped tests), plus a docs line and new tests in tests/playwright-test/shard.spec.ts. None of that runs inside the MCP HTTP transport or its browser lifecycle, and the MCP suite doesn't use sharding logic that could starve or drop this test.

Pre-existing flake / infra

Triaged by the Playwright bot - agent run

@yury-s
Yury Semikhatsky (yury-s) merged commit 5ea3ac5 into microsoft:main Sep 24, 2026
44 of 45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Opt-in flag to exclude statically skipped tests when calculating CI shard distribution

2 participants