diff --git a/docs/articles/nunit/technical-notes/usage/Framework-Parallel-Test-Execution.md b/docs/articles/nunit/technical-notes/usage/Framework-Parallel-Test-Execution.md index 073573dd3..a99e1c8cf 100644 --- a/docs/articles/nunit/technical-notes/usage/Framework-Parallel-Test-Execution.md +++ b/docs/articles/nunit/technical-notes/usage/Framework-Parallel-Test-Execution.md @@ -7,6 +7,11 @@ test run. By default, no parallel execution takes place. Attributes are used to indicate which tests may run in parallel and how they relate to other tests. +> [!TIP] +> This note describes how NUnit schedules parallel work. For practical questions, such as how to turn parallel execution +> on and what to do about state shared between tests, see +> [Running Tests in Parallel](../../writing-tests/parallel-tests.md). + ## Platform Support Parallel execution is supported by the NUnit framework on desktop .NET runtimes and .NET Standard 2.0. It is not @@ -36,9 +41,11 @@ This is a `[Flags]` type enumeration used to specify which tests may run in para it appears and any subordinate tests. The following values are available for use: * `ParallelScope.Self` indicates that the test itself may be run in parallel with other tests. This is the default for - the `ParallelizableAttribute` and is the only value permitted on a test method. + the `ParallelizableAttribute` and is the only value permitted on a test method that is not parameterized. * `ParallelScope.Children` indicates that the descendants of the test may be run in parallel with respect to one - another. + another. On a parameterized test method this is the only part of the scope that has an effect: the suite of test + cases always executes directly on the thread of its fixture, so `Self` is ignored there and `Children` and `All` are + equivalent. The individual cases are then dispatched to the parallel queue. * `ParallelScope.Fixtures` indicates that test fixtures that are the descendants of the test may be run in parallel with one another.` @@ -67,8 +74,11 @@ to other tests at the same level in the tree. The following are a few examples o 3. __Non-parallel SetUpFixture with parallel test fixtures:__ The entire group of fixtures runs separately from any fixtures outside the group. Within the group, multiple fixtures run in parallel. -4. __Parallel SetUpFixture with non-parallel test fixtures:__ The group runs in parallel with other fixtures and groups. - Within the group, only one fixture at a time may execute. +4. __Parallel SetUpFixture with non-parallel test fixtures:__ The SetUpFixture's own one-time setup and teardown may run + in parallel with other work, but the non-parallel fixtures inside it are queued together with every other + non-parallel fixture in the assembly and run one at a time, after all parallel work has finished. This does not + produce a group that runs in parallel with other groups while its own fixtures run in sequence; see (3) for the + way to isolate a group. 5. __Parallel SetUpFixture with non-parallel test fixtures containing parallel test cases:__ This is just one example of a more complex setup. The fixtures themselves run as described in (4) but the cases within each fixture run in diff --git a/docs/articles/nunit/writing-tests/attributes/levelofparallelism.md b/docs/articles/nunit/writing-tests/attributes/levelofparallelism.md index 66eed190d..80bbf361d 100644 --- a/docs/articles/nunit/writing-tests/attributes/levelofparallelism.md +++ b/docs/articles/nunit/writing-tests/attributes/levelofparallelism.md @@ -45,9 +45,12 @@ using NUnit.Framework; 3. Setting the level to 1 effectively disables parallelism (only one test runs at a time). 4. This value can be overridden using the `--workers` command-line option in the NUnit console runner. 5. Consider your test dependencies and shared resources when setting this value. Higher parallelism can cause issues with tests that share state. +6. The attribute cannot be applied to a single fixture or test. To limit how many cases of one test run at the same time, throttle inside the test itself, for example with a `SemaphoreSlim`. See [Running Tests in Parallel](xref:paralleltests#limiting-parallelism-for-one-test). ## See Also +* [Running Tests in Parallel](xref:paralleltests) +* [Framework Parallel Test Execution](../../technical-notes/usage/Framework-Parallel-Test-Execution.md) * [Parallelizable Attribute](xref:attribute-parallelizable) * [NonParallelizable Attribute](xref:attribute-nonparallelizable) * [Console Command Line](xref:consolecommandline) diff --git a/docs/articles/nunit/writing-tests/attributes/nonparallelizable.md b/docs/articles/nunit/writing-tests/attributes/nonparallelizable.md index 7ffd5ab23..6d1b53692 100644 --- a/docs/articles/nunit/writing-tests/attributes/nonparallelizable.md +++ b/docs/articles/nunit/writing-tests/attributes/nonparallelizable.md @@ -30,5 +30,7 @@ does not support parallelization. ## See Also +* [Running Tests in Parallel](xref:paralleltests) +* [Framework Parallel Test Execution](../../technical-notes/usage/Framework-Parallel-Test-Execution.md) * [Parallelizable Attribute](xref:attribute-parallelizable) * [LevelOfParallelism Attribute](xref:attribute-levelofparallelism) diff --git a/docs/articles/nunit/writing-tests/attributes/parallelizable.md b/docs/articles/nunit/writing-tests/attributes/parallelizable.md index 64556d9bf..0284f308d 100644 --- a/docs/articles/nunit/writing-tests/attributes/parallelizable.md +++ b/docs/articles/nunit/writing-tests/attributes/parallelizable.md @@ -10,6 +10,12 @@ The `ParallelizableAttribute` marks an assembly, fixture, or method (and optiona When used without an argument, scope defaults to `ParallelScope.Self`. +> [!TIP] +> This page describes the attribute. For the common questions, such as how to turn parallel execution on, what to +> do about state shared between tests, and how to run fixtures in parallel while keeping each fixture's tests in +> order, see [Running Tests in Parallel](xref:paralleltests). How NUnit schedules parallel work is described in +> [Framework Parallel Test Execution](../../technical-notes/usage/Framework-Parallel-Test-Execution.md). + ## Constructors ```csharp @@ -53,14 +59,16 @@ Named argument form: | Value | Meaning | Valid On | | ------------------------ | --------------------------------------------------------------------------------- | ----------------- | | `ParallelScope.Self` | the test itself may be run in parallel with other tests | Classes, Methods | -| `ParallelScope.Children` | child tests may be run in parallel with one another | Assembly, Classes | +| `ParallelScope.Children` | child tests may be run in parallel with one another | Assembly, Classes, parameterized test methods | | `ParallelScope.Fixtures` | fixtures may be run in parallel with one another | Assembly, Classes | -| `ParallelScope.All` | the test and its descendants may be run in parallel with others at the same level | Classes, Methods | +| `ParallelScope.All` | the test and its descendants may be run in parallel with others at the same level | Classes, parameterized test methods | ## Notes * Some values are invalid on certain elements, although they will compile. NUnit will report any tests so marked as - invalid and will produce an error message. + invalid and will produce an error message. On a test method that is not parameterized, only `ParallelScope.Self` + is valid. On a parameterized test method, such as one with `[TestCase]` or `[TestCaseSource]`, `Children` or `All` + lets the individual test cases run in parallel with one another. * The `ParallelScope` enum has additional values, which are used internally but are not visible to users through Intellisense. @@ -70,6 +78,8 @@ Named argument form: ## See Also +* [Running Tests in Parallel](xref:paralleltests) +* [Framework Parallel Test Execution](../../technical-notes/usage/Framework-Parallel-Test-Execution.md) * [FixtureLifeCycle Attribute](xref:attribute-fixturelifecycle) * [NonParallelizable Attribute](xref:attribute-nonparallelizable) * [LevelOfParallelism Attribute](xref:attribute-levelofparallelism) diff --git a/docs/articles/nunit/writing-tests/parallel-tests.md b/docs/articles/nunit/writing-tests/parallel-tests.md new file mode 100644 index 000000000..156b52249 --- /dev/null +++ b/docs/articles/nunit/writing-tests/parallel-tests.md @@ -0,0 +1,162 @@ +--- +uid: paralleltests +--- + +# Running Tests in Parallel + +Within a test assembly, NUnit runs tests one at a time unless you tell it otherwise. Marking a test +[`[Parallelizable]`](xref:attribute-parallelizable) is enough to have NUnit schedule it on a pool of worker threads, +alongside any other test marked the same way. The attribute is a statement that the test is *able* to run at the same +time as other tests. Making that true is your job, and most of this page is about that. It answers the questions that +come up most often. The attribute reference pages describe each attribute, and +[Framework Parallel Test Execution](../technical-notes/usage/Framework-Parallel-Test-Execution.md) describes how NUnit +schedules the work. + +> [!TIP] +> Parallel execution only helps when the tests are independent. Most problems reported with parallel tests turn out to +> be state shared between tests, not a problem with NUnit. See +> [Tests that only fail when run in parallel](#tests-that-only-fail-when-run-in-parallel) before you reach for +> `[NonParallelizable]`. + +## Turning parallel execution on + +There are two ways to opt in, and choosing between them is the first decision to make. + +**Mark fixtures and tests one at a time.** `[Parallelizable]` on a class means the fixture runs alongside other +fixtures while its own tests run in order. `[Parallelizable(ParallelScope.All)]` lets its tests run in parallel with +one another as well. Everything you have not marked keeps running one at a time, so each fixture you mark is one you +have checked. + +[!code-csharp[ParallelFixtureSelf](~/snippets/Snippets.NUnit/ParallelTestsGuideExamples.cs#ParallelFixtureSelf)] + +**Turn it on for the whole assembly** and mark the exceptions `[NonParallelizable]`. The attribute usually goes in a +file such as `AssemblyInfo.cs` or `GlobalUsings.cs`: + +```csharp +using NUnit.Framework; + +[assembly: Parallelizable(ParallelScope.Fixtures)] +``` + +With `ParallelScope.Fixtures`, test fixtures run in parallel with one another, but the tests inside each fixture still +run one at a time. With `ParallelScope.Children`, every fixture and every test runs in parallel with everything else, +unless a fixture or test overrides that with its own attribute. This is the quickest way to get a parallel run, but +not necessarily the safest. Every fixture in the assembly becomes a candidate, including ones written before the +setting existed and ones added later, so a test that shares state shows up later as an intermittent failure rather +than at the moment you turned parallelism on. + +Many teams start with the first approach and switch to the second once their fixtures are known to be independent. + +NUnit uses as many worker threads as the machine has processors, with a minimum of two. The +[`[LevelOfParallelism]`](xref:attribute-levelofparallelism) assembly attribute changes that number. + +## Tests that only fail when run in parallel + +A test that passes on its own and fails when run in parallel is almost always sharing something mutable with another +test. The usual suspects are: + +- **Fixture instance fields.** By default NUnit creates one instance of your fixture class and runs every test in it + on that instance. A `[SetUp]` method that assigns a field is overwriting the same field that another test, running on + another thread, is still using. +- **Static fields and singletons.** A static `HttpClient`, database connection, or browser driver is shared by every + test in the process, whatever fixture they are in. +- **External resources.** Two tests writing to the same file, the same database table, or the same port. + +For fixture instance state, [`[FixtureLifeCycle(LifeCycle.InstancePerTestCase)]`](xref:attribute-fixturelifecycle) +gives each test its own instance of the fixture, so `[SetUp]` and `[TearDown]` work on a resource that no other test +can see. This example creates a session per test, which is also the pattern for a Selenium web driver or any other +object that must not be driven from two threads at once: + +[!code-csharp[ParallelInstancePerTestCase](~/snippets/Snippets.NUnit/ParallelTestsGuideExamples.cs#ParallelInstancePerTestCase)] + +For static state and external resources, NUnit cannot isolate the tests for you. Either give each test its own copy of +the resource, or mark the tests that need it [`[NonParallelizable]`](xref:attribute-nonparallelizable) so that they run +on their own after all parallel work has finished: + +[!code-csharp[ParallelNonParallelizable](~/snippets/Snippets.NUnit/ParallelTestsGuideExamples.cs#ParallelNonParallelizable)] + +> [!NOTE] +> `[OneTimeSetUp]` and `[OneTimeTearDown]` run once per fixture even with `InstancePerTestCase`, and must then be +> static. Anything they create is shared by every test in the fixture. + +## Running fixtures in parallel but each fixture's tests in order + +A fixture is the unit NUnit uses to keep tests in order. `[Parallelizable]` without an argument, which is +`ParallelScope.Self`, means the fixture may run at the same time as other fixtures while its own tests run one after +another. The same applies when the assembly uses `ParallelScope.Fixtures`. If a set of tests shares a resource that +must not be used from two threads at once, keeping those tests in one fixture marked `[Parallelizable]` is the simplest +way to serialize them without slowing down the rest of the run. + +## Running the cases of one test in parallel + +A test method with `[TestCase]`, `[TestCaseSource]` or `[Values]` arguments is a small suite of test cases. Mark it with +`[Parallelizable(ParallelScope.Children)]` and the cases run in parallel with one another. `ParallelScope.All` does +the same thing on a test method, because the suite itself always runs on its fixture's thread and only the `Children` +part has an effect. Whether the cases also overlap with other tests depends on the fixture. In a fixture that is not +itself parallel, the cases run in parallel with one another and nothing else runs at the same time, not even the +fixture's other tests. In a parallel fixture, the cases share the worker pool with everything else that is eligible. + +[!code-csharp[ParallelTestCases](~/snippets/Snippets.NUnit/ParallelTestsGuideExamples.cs#ParallelTestCases)] + +On a test method that is not parameterized, that is a plain `[Test]` method with no `[TestCase]`, `[TestCaseSource]` or +`[Values]`, `ParallelScope.Self` is the only valid scope. NUnit reports the test as invalid if you use `Children` or +`All` there, and the NUnit Analyzers flag it at build time as +[NUnit1009](../../nunit-analyzers/NUnit1009.md). + +## Keeping a group of fixtures apart from the rest + +There is no attribute for a namespace, but a [`[SetUpFixture]`](xref:attribute-setupfixture) applies to every fixture +in its namespace, and its parallel settings apply to them as well. A `[NonParallelizable]` SetUpFixture with +`[Parallelizable]` fixtures inside it runs as a group. Nothing outside the group runs while the group runs, and within +the group the fixtures run in parallel with one another: + +[!code-csharp[ParallelGroupIsolated](~/snippets/Snippets.NUnit/ParallelTestsGuideExamples.cs#ParallelGroupIsolated)] + +The opposite arrangement, a group that runs in parallel with other groups while its own fixtures run one at a time, is +not something NUnit supports. A `[Parallelizable]` SetUpFixture with `[NonParallelizable]` fixtures inside it does not +do that: every non-parallel fixture in the assembly waits in the same queue and runs one at a time, whatever namespace +it is in. If you need that shape, put the tests that share the resource in one fixture, as described above, or move +each group to its own test project and let the test runner run the projects in parallel. Whether a runner does that by +default is covered in [Parallel execution did not make the run faster](#parallel-execution-did-not-make-the-run-faster). + +## Limiting parallelism for one test + +[`[LevelOfParallelism]`](xref:attribute-levelofparallelism) sets the number of worker threads for the whole assembly and +cannot be applied to a single fixture or test. To let a parameterized test run its cases in parallel while limiting how +many run at once, for example because each case calls a rate-limited service, throttle inside the test with a static +`SemaphoreSlim`: + +[!code-csharp[ParallelThrottle](~/snippets/Snippets.NUnit/ParallelTestsGuideExamples.cs#ParallelThrottle)] + +## Parallel execution did not make the run faster + +Parallel execution has a cost. For a suite of very fast tests, starting and switching between worker threads can take +longer than the tests themselves, so adding `[Parallelizable]` may make such a run slightly slower rather than faster. +It pays off when the tests spend their time waiting, for a database, a web request, or a browser. + +If your solution has several test projects, note that running them in parallel with one another is the job of the test +runner, not of NUnit, and each runner has its own default: + +- The NUnit console runner runs each assembly in its own process, in parallel, by default. See + [Engine Parallel Test Execution](../technical-notes/usage/Engine-Parallel-Test-Execution.md). +- `dotnet test` in its default VSTest mode, and Visual Studio Test Explorer, run one project at a time unless + `MaxCpuCount` is set in a runsettings file. +- `dotnet test` in Microsoft.Testing.Platform mode, which the .NET 10 SDK uses when `global.json` selects that runner, + runs test projects in parallel by default, up to the number of processors. `--max-parallel-test-modules` changes the + limit. See [NUnit and Microsoft.Testing.Platform](../../vs-test-adapter/NUnit-And-Microsoft-Test-Platform.md). + +When a project with more than a couple of thousand tests runs much more slowly as a whole than the same tests run in +smaller batches, check the adapter's [`AssemblySelectLimit`](xref:tipsandtricks#assemblyselectlimit) setting before +looking at parallelism. + +## See also + +- The [Parallelizable](xref:attribute-parallelizable), [NonParallelizable](xref:attribute-nonparallelizable), + [LevelOfParallelism](xref:attribute-levelofparallelism) and [FixtureLifeCycle](xref:attribute-fixturelifecycle) + reference pages +- [Framework Parallel Test Execution](../technical-notes/usage/Framework-Parallel-Test-Execution.md), for how NUnit + schedules parallel and non-parallel work +- [Engine Parallel Test Execution](../technical-notes/usage/Engine-Parallel-Test-Execution.md), for running assemblies + in parallel with the console runner +- [Preparing and Cleaning Up Tests](xref:setupandteardownguide), for which setup methods run per test and which run + once diff --git a/docs/articles/nunit/writing-tests/toc.yml b/docs/articles/nunit/writing-tests/toc.yml index ecb896df6..852a685ef 100644 --- a/docs/articles/nunit/writing-tests/toc.yml +++ b/docs/articles/nunit/writing-tests/toc.yml @@ -10,6 +10,8 @@ href: dependent-tests.md - name: Flaky and Slow Tests href: flaky-and-slow-tests.md +- name: Running Tests in Parallel + href: parallel-tests.md - name: Organizing and Selecting Tests href: organizing-tests.md - name: Attributes diff --git a/docs/index.md b/docs/index.md index 7cede9921..c485717e9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -29,7 +29,7 @@ _description: Documentation for NUnit, the open-source unit-testing framework fo Preparing and cleaning up Tests that depend on each other Flaky tests -Running tests in parallel +Running tests in parallel Microsoft.Testing.Platform Upgrading to NUnit 5 Analyzer warnings @@ -64,6 +64,7 @@ public class CalculatorTests
  • Checking several things at once
  • Tests that depend on other tests
  • Flaky and slow tests
    • +
  • Running tests in parallel
  • Organizing and selecting tests

    • Reference

    diff --git a/docs/snippets/Snippets.NUnit/ParallelTestsGuideExamples.cs b/docs/snippets/Snippets.NUnit/ParallelTestsGuideExamples.cs new file mode 100644 index 000000000..5b702ecfa --- /dev/null +++ b/docs/snippets/Snippets.NUnit/ParallelTestsGuideExamples.cs @@ -0,0 +1,246 @@ +using System.Collections.Concurrent; +using NUnit.Framework; +using NUnit.Framework.Interfaces; +using NUnit.Framework.Internal.Builders; +using MethodWrapper = NUnit.Framework.Internal.MethodWrapper; +using ParameterizedMethodSuite = NUnit.Framework.Internal.ParameterizedMethodSuite; +using PropertyNames = NUnit.Framework.Internal.PropertyNames; + +#pragma warning disable CA1822 + +namespace Snippets.NUnit.ParallelGuide +{ + /// + /// Stands in for a resource that must not be used from two threads at once, + /// such as a Selenium web driver or a database connection. + /// + public sealed class Session : IDisposable + { + public int CallCount { get; private set; } + + public string Call(string input) + { + CallCount++; + return input.ToUpperInvariant(); + } + + public void Dispose() + { + } + } + + #region ParallelFixtureSelf + // Runs alongside other fixtures; its two tests run one after the other. + [Parallelizable] + public class OrderCalculatorTests + { + [Test] + public void Total_AddsLineItems() + { + Assert.That(2 + 3, Is.EqualTo(5)); + } + + [Test] + public void Total_IsZeroForEmptyOrder() + { + Assert.That(0, Is.EqualTo(0)); + } + } + #endregion + + #region ParallelInstancePerTestCase + [Parallelizable(ParallelScope.All)] + [FixtureLifeCycle(LifeCycle.InstancePerTestCase)] + public class SessionTests + { + private Session _session = null!; + + [SetUp] + public void OpenSession() + { + // Each test gets its own fixture instance, so each test gets its own session. + _session = new Session(); + } + + [TearDown] + public void CloseSession() + { + _session.Dispose(); + } + + [Test] + public void Call_ReturnsUpperCase() + { + Assert.That(_session.Call("abc"), Is.EqualTo("ABC")); + } + + [Test] + public void Call_CountsCalls() + { + _session.Call("x"); + _session.Call("y"); + + // Would be unreliable if another test on another thread shared this session. + Assert.That(_session.CallCount, Is.EqualTo(2)); + } + } + #endregion + + #region ParallelNonParallelizable + public static class SharedCache + { + public static readonly ConcurrentDictionary Entries = new(); + } + + // Uses a static resource, so it must not overlap with anything else that touches it. + [NonParallelizable] + public class SharedCacheTests + { + [SetUp] + public void ClearCache() + { + SharedCache.Entries.Clear(); + } + + [Test] + public void Cache_StartsEmpty() + { + Assert.That(SharedCache.Entries, Is.Empty); + } + + [Test] + public void Cache_StoresEntry() + { + SharedCache.Entries["key"] = "value"; + + Assert.That(SharedCache.Entries, Has.Count.EqualTo(1)); + } + } + #endregion + + #region ParallelTestCases + public class SlugTests + { + // The three cases of this test run in parallel with one another. + [TestCase("Hello World", "hello-world")] + [TestCase(" NUnit ", "nunit")] + [TestCase("A/B", "a-b")] + [Parallelizable(ParallelScope.Children)] + public void ToSlug_NormalizesInput(string input, string expected) + { + var slug = input.Trim().ToLowerInvariant().Replace(' ', '-').Replace('/', '-'); + + Assert.That(slug, Is.EqualTo(expected)); + } + } + #endregion + + #region ParallelThrottle + public class RateLimitedServiceTests + { + // At most two cases call the service at the same time, whatever the worker count. + private static readonly SemaphoreSlim Throttle = new(2); + + [TestCase(1)] + [TestCase(2)] + [TestCase(3)] + [TestCase(4)] + [Parallelizable(ParallelScope.Children)] + public async Task Service_HandlesRequest(int request) + { + await Throttle.WaitAsync(); + try + { + await Task.Delay(10); + Assert.That(request, Is.Positive); + } + finally + { + Throttle.Release(); + } + } + } + #endregion +} + +#region ParallelGroupIsolated +namespace Snippets.NUnit.ParallelGuide.Reporting +{ + // Nothing outside this namespace runs while the fixtures in it run. + [SetUpFixture] + [NonParallelizable] + public class ReportingSetUp + { + [OneTimeSetUp] + public void StartReportServer() + { + } + + [OneTimeTearDown] + public void StopReportServer() + { + } + } + + // Within the group, these two fixtures run in parallel with each other. + [Parallelizable] + public class DailyReportTests + { + [Test] + public void Report_HasHeader() => Assert.Pass(); + } + + [Parallelizable] + public class MonthlyReportTests + { + [Test] + public void Report_HasTotals() => Assert.Pass(); + } +} +#endregion + +namespace Snippets.NUnit.ParallelGuide +{ + /// + /// Not referenced from the docs. Checks the claims the guide makes about which scopes + /// are valid on a test method, by building the tests the way NUnit does. + /// + public class ParallelScopeVerification + { + private static class Holder + { + // Not a test: no [Test] attribute, so NUnit never discovers it. The analyzer + // still flags the invalid scope, which is the point being verified here. +#pragma warning disable NUnit1009 + [Parallelizable(ParallelScope.All)] + public static void PlainMethodWithAll() + { + } +#pragma warning restore NUnit1009 + } + + [Test] + public void AllOnPlainTestMethod_IsNotRunnable() + { + var method = new MethodWrapper(typeof(Holder), nameof(Holder.PlainMethodWithAll)); + + var test = new NUnitTestCaseBuilder().BuildTestMethod(method, null, null); + + Assert.That(test.RunState, Is.EqualTo(RunState.NotRunnable)); + Assert.That(test.Properties.Get(PropertyNames.SkipReason), + Is.EqualTo("May not specify ParallelScope.Children on a non-parameterized test method")); + } + + [Test] + public void ChildrenOnParameterizedTestMethod_IsRunnable() + { + var method = new MethodWrapper(typeof(SlugTests), nameof(SlugTests.ToSlug_NormalizesInput)); + + var suite = new DefaultTestCaseBuilder().BuildFrom(method, null); + + Assert.That(suite, Is.TypeOf()); + Assert.That(suite.RunState, Is.EqualTo(RunState.Runnable)); + Assert.That(suite.Tests, Has.Count.EqualTo(3)); + } + } +}