Repository navigation
Add a Running Tests in Parallel guide and link it from the reference pages - #1300
Conversation
…pages Closes #272. Adds docs/articles/nunit/writing-tests/parallel-tests.md, a task-oriented topic page in the same style as Flaky and Slow Tests, covering the questions that come up most often in nunit/nunit discussions: turning parallel execution on, tests that only fail in parallel (shared state), fixtures in parallel with tests in order, parallel test cases, isolating a group of fixtures, throttling one test, and why a parallel run can be slower. All examples are executable snippets in ParallelTestsGuideExamples.cs, plus a verification fixture that builds test methods the way NUnit does to prove which scopes are valid. The Parallelizable, NonParallelizable and LevelOfParallelism reference pages now point to the guide and to the Framework Parallel Test Execution note, and the home page links the guide instead of the technical note. Corrections found while verifying the claims against NUnit 5.0.0: - ParallelScope.All and Children are invalid on a non-parameterized test method but valid on a parameterized one; the reference table said All was valid on methods. - Framework Parallel Test Execution case 4 claimed a Parallelizable SetUpFixture with NonParallelizable fixtures runs as a group in parallel with other groups. A timing experiment shows every non-parallel fixture queues on the single non-parallel worker regardless of namespace, so the case now describes what actually happens and points to case 3. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
📖 Docs preview removed because this PR was closed. |
The NUnit console runner runs separate assemblies in parallel by default, so the unqualified claim only holds for the framework within one assembly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The guide needs corrections to runner-specific concurrency advice, parallel opt-in wording, and the distinction between parameterless and non-parameterized tests.
3 open issues
What changed in this PR
Adds a practical parallel-testing guide to NUnit’s documentation, addressing shared state, execution scopes, fixture grouping, and concurrency limits.
Changes:
- Adds six executable examples and scope-validation tests.
- Links the guide from the home page, Writing Tests navigation, and attribute references.
- Updates scope-validity documentation and SetUpFixture scheduling explanations.
| File | Description |
|---|---|
docs/snippets/Snippets.NUnit/ParallelTestsGuideExamples.cs |
Adds guide examples and scope-validation tests. |
docs/index.md |
Links the new guide from popular topics and the Writing Tests card. |
docs/articles/nunit/writing-tests/toc.yml |
Adds the guide to navigation. |
docs/articles/nunit/writing-tests/parallel-tests.md |
Introduces practical parallel-testing guidance. |
docs/articles/nunit/writing-tests/attributes/parallelizable.md |
Updates valid scopes and adds guide links. |
docs/articles/nunit/writing-tests/attributes/nonparallelizable.md |
Links practical and technical parallelism guidance. |
docs/articles/nunit/writing-tests/attributes/levelofparallelism.md |
Adds per-test throttling guidance and related links. |
docs/articles/nunit/technical-notes/usage/Framework-Parallel-Test-Execution.md |
Revises scope and fixture-group scheduling explanations. |
Review effort: Balanced
- Say that [Parallelizable] is enough to get parallel scheduling, and separate that from making the test thread-safe. - Describe the Self-only restriction in terms of non-parameterized test methods rather than argument count. - Distinguish runner defaults: console runner parallel by default, VSTest mode needs MaxCpuCount, Microsoft.Testing.Platform mode of dotnet test runs modules in parallel by default (--max-parallel-test-modules). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
manfred-brands
left a comment
There was a problem hiding this comment.
Nice document.
I had one question that could use clarification.
| ## Running the cases of one test in parallel | ||
|
|
||
| A test method with `[TestCase]`, `[TestCaseSource]` or `[Values]` arguments is a small suite of test cases. Mark it with | ||
| `[Parallelizable(ParallelScope.All)]` and the cases run in parallel with one another, even when the fixture runs its |
There was a problem hiding this comment.
Would ParallelScope.Children work to allow running the test cases in parallel but not with other tests concurrently.
There was a problem hiding this comment.
@manfred-brands I think it should, barring bugs. Might be good to try both as parameterized test suites have been a source of many of them in the past.
There was a problem hiding this comment.
@manfred-brands Yes, and it turns out Children and All are identical on a test method. I took @CharliePoole's advice and tried both, with a [TestCase] method next to two plain tests in the same fixture, under two configurations:
- Fixture not parallel (no assembly setting): with either scope, the three cases overlapped each other and nothing else, not even the sibling tests in the same fixture.
- Fixture parallel (
[assembly: Parallelizable(ParallelScope.Fixtures)]): with either scope, the cases overlapped each other, the sibling tests, and other fixtures.
The scheduler explains it: GetExecutionStrategy returns Direct for any TestSuite that isn't a fixture, which includes a ParameterizedMethodSuite, before it ever checks Self. So the suite always runs on the fixture's thread, and the only thing the attribute contributes is Children in the context for the cases, which sends them to the parallel queue. Whether they overlap anything else is decided by the fixture, not by the method's scope.
So the pattern you describe is a non-parallel fixture with Children (or All) on the method. I've updated the section to say that (40a3ee2), switched the snippet to Children since it says what it does, and added a sentence to the technical note's ParallelScope list so the why is recorded in the "how it works" place.
- Turning parallel execution on now presents the two approaches, per fixture first and assembly-wide second, states the trade-off of the assembly-wide setting, and notes that teams commonly start with the first. - Running the cases of one test in parallel uses ParallelScope.Children, explains that All is equivalent on a test method, and describes how the fixture decides whether the cases overlap with anything else. Verified by timing runs: in a non-parallel fixture the cases overlapped only each other; under an assembly-level Fixtures setting they overlapped sibling tests and other fixtures, identically for Children and All. - Technical note records why: a parameterized method's suite always runs directly on its fixture's thread, so only Children has an effect there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
manfred-brands
left a comment
There was a problem hiding this comment.
Thanks @SeanKilleen for investigating and clarifying my question.
…pages (#1300) * Add a Running Tests in Parallel guide and link it from the reference pages Closes #272. Adds docs/articles/nunit/writing-tests/parallel-tests.md, a task-oriented topic page in the same style as Flaky and Slow Tests, covering the questions that come up most often in nunit/nunit discussions: turning parallel execution on, tests that only fail in parallel (shared state), fixtures in parallel with tests in order, parallel test cases, isolating a group of fixtures, throttling one test, and why a parallel run can be slower. All examples are executable snippets in ParallelTestsGuideExamples.cs, plus a verification fixture that builds test methods the way NUnit does to prove which scopes are valid. The Parallelizable, NonParallelizable and LevelOfParallelism reference pages now point to the guide and to the Framework Parallel Test Execution note, and the home page links the guide instead of the technical note. Corrections found while verifying the claims against NUnit 5.0.0: - ParallelScope.All and Children are invalid on a non-parameterized test method but valid on a parameterized one; the reference table said All was valid on methods. - Framework Parallel Test Execution case 4 claimed a Parallelizable SetUpFixture with NonParallelizable fixtures runs as a group in parallel with other groups. A timing experiment shows every non-parallel fixture queues on the single non-parallel worker regardless of namespace, so the case now describes what actually happens and points to case 3. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Scope the opening sentence of the parallel guide to a single assembly The NUnit console runner runs separate assemblies in parallel by default, so the unqualified claim only holds for the framework within one assembly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Address Copilot review on the parallel guide - Say that [Parallelizable] is enough to get parallel scheduling, and separate that from making the test thread-safe. - Describe the Self-only restriction in terms of non-parameterized test methods rather than argument count. - Distinguish runner defaults: console runner parallel by default, VSTest mode needs MaxCpuCount, Microsoft.Testing.Platform mode of dotnet test runs modules in parallel by default (--max-parallel-test-modules). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Address review from manfred-brands and CharliePoole - Turning parallel execution on now presents the two approaches, per fixture first and assembly-wide second, states the trade-off of the assembly-wide setting, and notes that teams commonly start with the first. - Running the cases of one test in parallel uses ParallelScope.Children, explains that All is equivalent on a test method, and describes how the fixture decides whether the cases overlap with anything else. Verified by timing runs: in a non-parallel fixture the cases overlapped only each other; under an assembly-level Fixtures setting they overlapped sibling tests and other fixtures, identically for Children and All. - Technical note records why: a parameterized method's suite always runs directly on its fixture's thread, so only Children has an effect there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> 6ac4954

Closes #272.
What this adds
A Running Tests in Parallel topic page under Writing Tests, in the same style as Flaky and Slow Tests: question-shaped sections, a few sentences each, executable snippets. It answers the questions that came up repeatedly in nunit/nunit discussions (listed in #272):
FixturesvsChildrenmean at assembly level (described, not recommended, per @CharliePoole's note in the issue)InstancePerTestCasepattern (the Selenium static-driver case)SemaphoreSlimMaxCpuCount), with a pointer toAssemblySelectLimitThe page is linked from the Writing Tests TOC, the home page (the "Running tests in parallel" popular link now goes here instead of to the technical note, plus a card entry), and from the Parallelizable, NonParallelizable and LevelOfParallelism reference pages. Those reference pages also now link to the Framework Parallel Test Execution note, which nothing outside Technical Notes linked before.
Corrections to existing pages
Verified against NUnit 5.0.0 source and a timing experiment:
ParallelScopetable listedAllas valid on methods.AllisSelf | Children, andChildrenis rejected on a non-parameterized test method (May not specify ParallelScope.Children on a non-parameterized test method). It is valid on a parameterized one, where it runs the cases in parallel. Table and notes updated.[Parallelizable]SetUpFixture with[NonParallelizable]fixtures runs as a group in parallel with other groups. It does not.IsolateQueuesis only called from the non-parallel worker, so every non-parallel fixture in the assembly queues on that single worker and runs one at a time after the parallel shift. In the experiment (two namespaces, each with a parallel SetUpFixture and two non-parallel fixtures) all four fixtures ran sequentially on one thread. Case 3 (non-parallel SetUpFixture, parallel fixtures inside) works as described. This is what @OsirisTerje saw in Tests in parallel fixtures not running in parallel per namespace nunit#4679 when the console runner also did not run the namespaces in parallel. The case text now describes the real behaviour and points to case 3.Snippets
ParallelTestsGuideExamples.csholds the six page examples plusParallelScopeVerification, which is not referenced from the docs. It builds a test method withNUnitTestCaseBuilderand a parameterized one withDefaultTestCaseBuilderand asserts the run states and skip reason above, so the claims stay checked.Checks
dotnet teston Snippets.NUnit: 631 passed, 0 faileddocfx docs/docfx.json: builds, all xrefs resolve (only the usual missing-API warnings in a checkout withoutcode-output)🤖 Generated with Claude Code