Skip to content

Add a Running Tests in Parallel guide and link it from the reference pages - #1300

Merged
SeanKilleen merged 4 commits into
masterfrom
272_parallel-tests-guide
Oct 24, 2026
Merged

SeanKilleen merged 4 commits into
masterfrom
272_parallel-tests-guide

Conversation

@SeanKilleen

Copy link
Copy Markdown
Member

Closes #272.

What this adds

A Running Tests in Parallel topic page under Writing Tests, in the same style as Flaky and Slow Tests: question-shaped sections, a few sentences each, executable snippets. It answers the questions that came up repeatedly in nunit/nunit discussions (listed in #272):

  • Turning parallel execution on, and what Fixtures vs Children mean at assembly level (described, not recommended, per @CharliePoole's note in the issue)
  • Tests that only fail when run in parallel: fixture instance state, static state, external resources, and the InstancePerTestCase pattern (the Selenium static-driver case)
  • Running fixtures in parallel while each fixture's tests run in order
  • Running the cases of one parameterized test in parallel
  • Keeping a group of fixtures apart from the rest, and what NUnit cannot do (namespace groups in parallel with serial fixtures inside)
  • Limiting parallelism for one test with a SemaphoreSlim
  • Why a parallel run can be slower, and that assembly-level parallelism is the runner's job (MaxCpuCount), with a pointer to AssemblySelectLimit

The page is linked from the Writing Tests TOC, the home page (the "Running tests in parallel" popular link now goes here instead of to the technical note, plus a card entry), and from the Parallelizable, NonParallelizable and LevelOfParallelism reference pages. Those reference pages also now link to the Framework Parallel Test Execution note, which nothing outside Technical Notes linked before.

Corrections to existing pages

Verified against NUnit 5.0.0 source and a timing experiment:

  • parallelizable.md: the ParallelScope table listed All as valid on methods. All is Self | Children, and Children is rejected on a non-parameterized test method (May not specify ParallelScope.Children on a non-parameterized test method). It is valid on a parameterized one, where it runs the cases in parallel. Table and notes updated.
  • Framework-Parallel-Test-Execution.md, case 4: claimed a [Parallelizable] SetUpFixture with [NonParallelizable] fixtures runs as a group in parallel with other groups. It does not. IsolateQueues is only called from the non-parallel worker, so every non-parallel fixture in the assembly queues on that single worker and runs one at a time after the parallel shift. In the experiment (two namespaces, each with a parallel SetUpFixture and two non-parallel fixtures) all four fixtures ran sequentially on one thread. Case 3 (non-parallel SetUpFixture, parallel fixtures inside) works as described. This is what @OsirisTerje saw in Tests in parallel fixtures not running in parallel per namespace nunit#4679 when the console runner also did not run the namespaces in parallel. The case text now describes the real behaviour and points to case 3.

Snippets

ParallelTestsGuideExamples.cs holds the six page examples plus ParallelScopeVerification, which is not referenced from the docs. It builds a test method with NUnitTestCaseBuilder and a parameterized one with DefaultTestCaseBuilder and asserts the run states and skip reason above, so the claims stay checked.

Checks

  • dotnet test on Snippets.NUnit: 631 passed, 0 failed
  • markdownlint-cli2 and cspell: clean
  • docfx docs/docfx.json: builds, all xrefs resolve (only the usual missing-API warnings in a checkout without code-output)

🤖 Generated with Claude Code

…pages

Closes #272.

Adds docs/articles/nunit/writing-tests/parallel-tests.md, a task-oriented
topic page in the same style as Flaky and Slow Tests, covering the questions
that come up most often in nunit/nunit discussions: turning parallel execution
on, tests that only fail in parallel (shared state), fixtures in parallel with
tests in order, parallel test cases, isolating a group of fixtures, throttling
one test, and why a parallel run can be slower. All examples are executable
snippets in ParallelTestsGuideExamples.cs, plus a verification fixture that
builds test methods the way NUnit does to prove which scopes are valid.

The Parallelizable, NonParallelizable and LevelOfParallelism reference pages
now point to the guide and to the Framework Parallel Test Execution note, and
the home page links the guide instead of the technical note.

Corrections found while verifying the claims against NUnit 5.0.0:

- ParallelScope.All and Children are invalid on a non-parameterized test
  method but valid on a parameterized one; the reference table said All was
  valid on methods.
- Framework Parallel Test Execution case 4 claimed a Parallelizable
  SetUpFixture with NonParallelizable fixtures runs as a group in parallel
  with other groups. A timing experiment shows every non-parallel fixture
  queues on the single non-parallel worker regardless of namespace, so the
  case now describes what actually happens and points to case 3.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

📖 Docs preview removed because this PR was closed.

@SeanKilleen
SeanKilleen requested review from a team and OsirisTerje October 23, 2026 20:23
The NUnit console runner runs separate assemblies in parallel by default,
so the unqualified claim only holds for the framework within one assembly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The guide needs corrections to runner-specific concurrency advice, parallel opt-in wording, and the distinction between parameterless and non-parameterized tests.

3 open issues
What changed in this PR

Adds a practical parallel-testing guide to NUnit’s documentation, addressing shared state, execution scopes, fixture grouping, and concurrency limits.

Changes:

  • Adds six executable examples and scope-validation tests.
  • Links the guide from the home page, Writing Tests navigation, and attribute references.
  • Updates scope-validity documentation and SetUpFixture scheduling explanations.
File Description
docs/​snippets/​Snippets.NUnit/​ParallelTestsGuideExamples.cs Adds guide examples and scope-validation tests.
docs/​index.md Links the new guide from popular topics and the Writing Tests card.
docs/​articles/​nunit/​writing-tests/​toc.yml Adds the guide to navigation.
docs/​articles/​nunit/​writing-tests/​parallel-tests.md Introduces practical parallel-testing guidance.
docs/​articles/​nunit/​writing-tests/​attributes/​parallelizable.md Updates valid scopes and adds guide links.
docs/​articles/​nunit/​writing-tests/​attributes/​nonparallelizable.md Links practical and technical parallelism guidance.
docs/​articles/​nunit/​writing-tests/​attributes/​levelofparallelism.md Adds per-test throttling guidance and related links.
docs/​articles/​nunit/​technical-notes/​usage/​Framework-Parallel-Test-Execution.md Revises scope and fixture-group scheduling explanations.

Review effort: Balanced

Comment thread docs/articles/nunit/writing-tests/parallel-tests.md Outdated
Comment thread docs/articles/nunit/writing-tests/parallel-tests.md Outdated
Comment thread docs/articles/nunit/writing-tests/parallel-tests.md Outdated
- Say that [Parallelizable] is enough to get parallel scheduling, and
  separate that from making the test thread-safe.
- Describe the Self-only restriction in terms of non-parameterized test
  methods rather than argument count.
- Distinguish runner defaults: console runner parallel by default, VSTest
  mode needs MaxCpuCount, Microsoft.Testing.Platform mode of dotnet test
  runs modules in parallel by default (--max-parallel-test-modules).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@manfred-brands manfred-brands left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice document.
I had one question that could use clarification.

## Running the cases of one test in parallel

A test method with `[TestCase]`, `[TestCaseSource]` or `[Values]` arguments is a small suite of test cases. Mark it with
`[Parallelizable(ParallelScope.All)]` and the cases run in parallel with one another, even when the fixture runs its

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would ParallelScope.Children work to allow running the test cases in parallel but not with other tests concurrently.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@manfred-brands I think it should, barring bugs. Might be good to try both as parameterized test suites have been a source of many of them in the past.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@manfred-brands Yes, and it turns out Children and All are identical on a test method. I took @CharliePoole's advice and tried both, with a [TestCase] method next to two plain tests in the same fixture, under two configurations:

  • Fixture not parallel (no assembly setting): with either scope, the three cases overlapped each other and nothing else, not even the sibling tests in the same fixture.
  • Fixture parallel ([assembly: Parallelizable(ParallelScope.Fixtures)]): with either scope, the cases overlapped each other, the sibling tests, and other fixtures.

The scheduler explains it: GetExecutionStrategy returns Direct for any TestSuite that isn't a fixture, which includes a ParameterizedMethodSuite, before it ever checks Self. So the suite always runs on the fixture's thread, and the only thing the attribute contributes is Children in the context for the cases, which sends them to the parallel queue. Whether they overlap anything else is decided by the fixture, not by the method's scope.

So the pattern you describe is a non-parallel fixture with Children (or All) on the method. I've updated the section to say that (40a3ee2), switched the snippet to Children since it says what it does, and added a sentence to the technical note's ParallelScope list so the why is recorded in the "how it works" place.

Comment thread docs/articles/nunit/writing-tests/parallel-tests.md Outdated
- Turning parallel execution on now presents the two approaches, per
  fixture first and assembly-wide second, states the trade-off of the
  assembly-wide setting, and notes that teams commonly start with the
  first.
- Running the cases of one test in parallel uses ParallelScope.Children,
  explains that All is equivalent on a test method, and describes how the
  fixture decides whether the cases overlap with anything else. Verified
  by timing runs: in a non-parallel fixture the cases overlapped only each
  other; under an assembly-level Fixtures setting they overlapped sibling
  tests and other fixtures, identically for Children and All.
- Technical note records why: a parameterized method's suite always runs
  directly on its fixture's thread, so only Children has an effect there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@manfred-brands manfred-brands left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @SeanKilleen for investigating and clarifying my question.

@SeanKilleen
SeanKilleen merged commit 6ac4954 into master Oct 24, 2026
6 checks passed
@SeanKilleen
SeanKilleen deleted the 272_parallel-tests-guide branch October 24, 2026 15:35
github-actions Bot pushed a commit that referenced this pull request Oct 24, 2026
…pages (#1300)

* Add a Running Tests in Parallel guide and link it from the reference pages

Closes #272.

Adds docs/articles/nunit/writing-tests/parallel-tests.md, a task-oriented
topic page in the same style as Flaky and Slow Tests, covering the questions
that come up most often in nunit/nunit discussions: turning parallel execution
on, tests that only fail in parallel (shared state), fixtures in parallel with
tests in order, parallel test cases, isolating a group of fixtures, throttling
one test, and why a parallel run can be slower. All examples are executable
snippets in ParallelTestsGuideExamples.cs, plus a verification fixture that
builds test methods the way NUnit does to prove which scopes are valid.

The Parallelizable, NonParallelizable and LevelOfParallelism reference pages
now point to the guide and to the Framework Parallel Test Execution note, and
the home page links the guide instead of the technical note.

Corrections found while verifying the claims against NUnit 5.0.0:

- ParallelScope.All and Children are invalid on a non-parameterized test
  method but valid on a parameterized one; the reference table said All was
  valid on methods.
- Framework Parallel Test Execution case 4 claimed a Parallelizable
  SetUpFixture with NonParallelizable fixtures runs as a group in parallel
  with other groups. A timing experiment shows every non-parallel fixture
  queues on the single non-parallel worker regardless of namespace, so the
  case now describes what actually happens and points to case 3.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Scope the opening sentence of the parallel guide to a single assembly

The NUnit console runner runs separate assemblies in parallel by default,
so the unqualified claim only holds for the framework within one assembly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Address Copilot review on the parallel guide

- Say that [Parallelizable] is enough to get parallel scheduling, and
  separate that from making the test thread-safe.
- Describe the Self-only restriction in terms of non-parameterized test
  methods rather than argument count.
- Distinguish runner defaults: console runner parallel by default, VSTest
  mode needs MaxCpuCount, Microsoft.Testing.Platform mode of dotnet test
  runs modules in parallel by default (--max-parallel-test-modules).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Address review from manfred-brands and CharliePoole

- Turning parallel execution on now presents the two approaches, per
  fixture first and assembly-wide second, states the trade-off of the
  assembly-wide setting, and notes that teams commonly start with the
  first.
- Running the cases of one test in parallel uses ParallelScope.Children,
  explains that All is equivalent on a test method, and describes how the
  fixture decides whether the cases overlap with anything else. Verified
  by timing runs: in a non-parallel fixture the cases overlapped only each
  other; under an assembly-level Fixtures setting they overlapped sibling
  tests and other fixtures, identically for Children and All.
- Technical note records why: a parameterized method's suite always runs
  directly on its fixture's thread, so only Children has an effect there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> 6ac4954
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parallelism guide/FAQ

4 participants