test(integration): survive intermittent SQL Server 2025 startup crashes

[samtrion]

Summary

On main, the integration tests failed with ContainerNotRunningException: Container <id> exited with code 1. The container is the SQL Server fixture, and SQL Server crashed while starting up. This is an intermittent upstream bug, and changing the image did not remove it. The fixture now replaces a container that crashes during startup, with at most 3 attempts. Every retry is reported. The PR also moves the image off a stale preview tag and fixes an unrelated flaky RabbitMQ test.

Root cause

• Run 36242350353 (main, 1160c28): 125 failed tests, all from SqlServerContainerFixture.InitializeAsync on net10.0. All 250 exited with code 1 lines name the same container id; net8.0 and net9.0 passed.
• Crash signature. Testcontainers already includes the container's stdout and stderr in the exception:

  This program has encountered a fatal error and cannot continue running
           Reason: 0x00000002
      Stack Trace: ... sqlpal.dll ... lsasrv.dll ... lsass.exe ...
     Distribution: Ubuntu 24.04.3 LTS
       Last errno: 11 (Resource temporarily unavailable)
  

  This is failure signature B in microsoft/mssql-docker#974. SQL Server 2025 crashes intermittently at startup, and more often when several containers start at once. Our run starts one SQL Server container per target framework (net8/9/10) at the same moment, so it hits the concurrent case.
• The ubuntu-22.04 base does not avoid it. The first commit of this PR moved to 2025-CU9-ubuntu-22.04, based on the issue's "0 crashes in ~250 starts" on 22.04. The issue itself says that only bounds the rate and does not make it zero. After the rebase, run 36255943687 crashed the same way on the new image: 133 SQL Server failures on net10.0, one container, Ubuntu 22.04.5 LTS, Reason: 0x00000002, sqlpal and lsasrv in the stack. The 22.04 claim from the first version of this PR was wrong.
• Frequency. In the 80 CI runs I scanned before this PR (49 failed), the crash appears once. So it is rare per run, but when it hits, it takes out every SQL Server test for that TFM.
• Stale image. The fixture had used 2025-RTM-ubuntu-24.04-preview since chore(docker): update mcr.microsoft.com/mssql/server docker tag to v2025 #534. No GA tag has the -preview suffix, so loose versioning never proposed a cumulative update.

Decision: stay on SQL Server 2025 and retry

• A 2022 image would work technically. The provider scripts in src/NetEvolve.Pulse.SqlServer/Scripts use nothing specific to 2025, and EF Core's UseSqlServer defaults to compatibility level 150.
• But I found no evidence that 2022 is unaffected. The same LSASS Reason: 0x00000002 signature appears in older reports such as This program has encountered a fatal error and cannot continue running. The following diagnostic information is available: Reason: 0x00000002 Status: 0xd000000d Parameters: 0x280bafe00 [0] 0x10010 Stacktrace: 000000006a4ea978 000000006a43f6fd 000000006a514015 000000006a514044 0000000280f70aa5 Process: 7 - sqlservr Thread: 51 (application thread 0x10a0) Instance Id: 1dd11547-c8f1-4f55-ba66-e00daa0a5908 Crash Id: 5cf4d7ff-06b5-4135-9269-ef0938cd46e7 Build stamp: 7d599fe53e35b5a1b0c8a5e4185d8b7334e01a8c5fa77540415502a85f37ef27 Capturing core dump and information... dmesg: read kernel buffer failed: Operation not permitted No journal files were found. No journal files were found. Attempting to capture a dump with paldumper microsoft/mssql-docker#524, which predates 2025. Changing the engine major would reduce test coverage of the 2025 engine without a proven benefit.
• #974 has no replies and no documented mitigation beyond changing the base image and reducing concurrent starts. The three TFM test processes run separately, so their starts can't be serialized cheaply from inside one process.
• A bounded restart of a container that crashed before it was ready is a legitimate mitigation for a documented intermittent upstream crash. It does not hide a bug in Pulse: the tests only run against a container that started successfully.

Changes

• SqlServerContainerFixture
  • Uses mcr.microsoft.com/mssql/server:2025-CU9-ubuntu-22.04: same engine major, current CU, and the lower-rate base.
  • On ContainerNotRunningException during StartAsync, it disposes the container and builds a fresh one, up to 3 attempts. A crash on the final attempt still fails the tests with the full container output.
  • Each retry is logged to stderr with the crash output. On GitHub Actions it is also written to the job summary as a warning, because dotnet test does not print output from passing runs.
• renovate.json: regex versioning for mcr.microsoft.com/mssql/server (YYYY-CU<n>[-GDR<n>]-ubuntu-<ver>). It stays on the same Ubuntu base and never matches preview, RC or CTP tags. The file is byte-identical to the template after chore(renovate): track SQL Server CU/GDR tags on a fixed Ubuntu base template-dotnet#371, so the next template sync changes nothing.
• RabbitMqMessageTransportIntegrationTests: the test exchange is now declared with autoDelete: false.
  • TUnit creates a new test class instance per test, so each test opens its own connection and exclusive queue.
  • When the previous test's connection closed, its queue binding was removed and the broker deleted the auto-delete exchange asynchronously.
  • Sometimes that happened after the next test's no-op ExchangeDeclare, and the following QueueBind failed with 404 NOT_FOUND - no exchange 'pulse.integration.test'.
  • Seen in run 36229149003 (PR chore(deps): update dependency microsoft.aspnetcore.signalr.client to v10.0.12 #776) and in run 36247301567 on this PR.

Test evidence

• dotnet build Pulse.slnx -c Release: 0 warnings, 0 errors, including the CSharpier.MSBuild check.
• CI on the final commit: results are added below once the Run Tests job has run 3 times.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.77%. Comparing base (7e0ac93) to head (b9456a2).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #796      +/-   ##
==========================================
+ Coverage   95.70%   95.77%   +0.07%     
==========================================
  Files         270      270              
  Lines       10466    10466              
  Branches      953      953              
==========================================
+ Hits        10016    10024       +8     
+ Misses        243      235       -8     
  Partials      207      207              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.