Skip to content

Test: remove flaky markers from stabilised tests - #4483

Open
cheenamalhotra wants to merge 3 commits into
mainfrom
dev/cheena/dev-automation-remove-stabilized-markers
Open

Test: remove flaky markers from stabilised tests#4483
cheenamalhotra wants to merge 3 commits into
mainfrom
dev/cheena/dev-automation-remove-stabilized-markers

Conversation

@cheenamalhotra

@cheenamalhotra cheenamalhotra commented Jul 29, 2026

Copy link
Copy Markdown
Member

Summary

  • These tests have not failed in the last 90 days in CI-SqlClient and CI-SqlClient-Package pipelines, so can be marked as non-flaky.
  • Remove the class-wide flaky quarantine from DateTimeVariantTests so the nine stabilised test methods run in regular CI.
  • retain the flaky trait on TestSqlDataReaderParameterToTVP_Variant, which was not in the stabilisation set and remains environment-sensitive.
  • skip xsql because it is a private helper and has no flaky marker of its own.

Validation

  • dotnet build src/Microsoft.Data.SqlClient/tests/ManualTests/Microsoft.Data.SqlClient.ManualTests.csproj --no-restore --configuration Release --framework net8.0 --verbosity minimal --maxcpucount:1

Checklist

  • Tests added or updated
  • Public API changes documented (not applicable)
  • Verified against customer repro (not applicable)
  • Ensure no breaking changes introduced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 64a6d241-8263-4451-b207-65459e29a980
Copilot AI review requested due to automatic review settings July 29, 2026 02:41
@cheenamalhotra
cheenamalhotra requested a review from a team as a code owner July 29, 2026 02:41
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Jul 29, 2026
@cheenamalhotra cheenamalhotra added this to the 7.1.0-preview3 milestone Jul 29, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Removes the xUnit flaky quarantine at the DateTimeVariantTests class level so stabilized DateTime/sql_variant parameter tests are eligible to run in the standard CI test set, while keeping the one known environment-sensitive test quarantined.

Changes:

  • Removed [Trait("Category", "flaky")] from DateTimeVariantTests so other tests in the class are no longer globally quarantined.
  • Added [Trait("Category", "flaky")] specifically to TestSqlDataReaderParameterToTVP_Variant to keep only that test excluded from the default CI filter.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 64a6d241-8263-4451-b207-65459e29a980
Copilot AI review requested due to automatic review settings July 29, 2026 02:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e3e5c96e-86fb-4c23-a009-ebfcc9898f04
@cheenamalhotra cheenamalhotra moved this from To triage to In review in SqlClient Board Jul 29, 2026
@cheenamalhotra cheenamalhotra added the Area\Tests Issues that are targeted to tests or test projects label Jul 29, 2026
Copilot AI review requested due to automatic review settings July 29, 2026 03:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (9)

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/DateTimeVariantTests.cs:166

  • This test method is still marked [Trait("Category", "flaky")], which conflicts with the PR description stating the stabilized DateTimeVariant tests should run in regular CI (only TestSqlDataReaderParameterToTVP_Variant should remain flaky). Remove the flaky trait here so it is no longer quarantined.
        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [MemberData(nameof(GetParameterCombinations), DisableDiscoveryEnumeration = true)]

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/DateTimeVariantTests.cs:203

  • This test method is still marked [Trait("Category", "flaky")], which conflicts with the PR description stating the stabilized DateTimeVariant tests should run in regular CI (only TestSqlDataReaderParameterToTVP_Variant should remain flaky). Remove the flaky trait here so it is no longer quarantined.
        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [MemberData(nameof(GetParameterCombinations), DisableDiscoveryEnumeration = true)]

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/DateTimeVariantTests.cs:247

  • This test method is still marked [Trait("Category", "flaky")], which conflicts with the PR description stating the stabilized DateTimeVariant tests should run in regular CI (only TestSqlDataReaderParameterToTVP_Variant should remain flaky). Remove the flaky trait here so it is no longer quarantined.
        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [MemberData(nameof(GetParameterCombinations), DisableDiscoveryEnumeration = true)]

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/DateTimeVariantTests.cs:291

  • This test method is still marked [Trait("Category", "flaky")], which conflicts with the PR description stating the stabilized DateTimeVariant tests should run in regular CI (only TestSqlDataReaderParameterToTVP_Variant should remain flaky). Remove the flaky trait here so it is no longer quarantined.
        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [MemberData(nameof(GetParameterCombinations), DisableDiscoveryEnumeration = true)]

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/DateTimeVariantTests.cs:387

  • This test method is still marked [Trait("Category", "flaky")], which conflicts with the PR description stating the stabilized DateTimeVariant tests should run in regular CI (only TestSqlDataReaderParameterToTVP_Variant should remain flaky). Remove the flaky trait here so it is no longer quarantined.
        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [MemberData(nameof(GetParameterCombinations), DisableDiscoveryEnumeration = true)]

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/DateTimeVariantTests.cs:463

  • This test method is still marked [Trait("Category", "flaky")], which conflicts with the PR description stating the stabilized DateTimeVariant tests should run in regular CI (only TestSqlDataReaderParameterToTVP_Variant should remain flaky). Remove the flaky trait here so it is no longer quarantined.
        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [MemberData(nameof(GetParameterCombinations), DisableDiscoveryEnumeration = true)]

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/DateTimeVariantTests.cs:539

  • This test method is still marked [Trait("Category", "flaky")], which conflicts with the PR description stating the stabilized DateTimeVariant tests should run in regular CI (only TestSqlDataReaderParameterToTVP_Variant should remain flaky). Remove the flaky trait here so it is no longer quarantined.
        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [MemberData(nameof(GetParameterCombinations), DisableDiscoveryEnumeration = true)]

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/DateTimeVariantTests.cs:597

  • This test method is still marked [Trait("Category", "flaky")], which conflicts with the PR description stating the stabilized DateTimeVariant tests should run in regular CI (only TestSqlDataReaderParameterToTVP_Variant should remain flaky). Remove the flaky trait here so it is no longer quarantined.
        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [MemberData(nameof(GetParameterCombinations), DisableDiscoveryEnumeration = true)]

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/DateTimeVariantTests.cs:129

  • This test method is still marked [Trait("Category", "flaky")], which conflicts with the PR description stating the stabilized DateTimeVariant tests should run in regular CI (only TestSqlDataReaderParameterToTVP_Variant should remain flaky). Remove the flaky trait here so it is no longer quarantined.

This issue also appears in the following locations of the same file:

  • line 164
  • line 201
  • line 245
  • line 289
  • line 385
  • ...and 3 more
        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [MemberData(nameof(GetParameterCombinations), DisableDiscoveryEnumeration = true)]

@priyankatiwari08

priyankatiwari08 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

CI Failure: DateTimeVariantTests.SqlBulkCopyDataRow_Type on macOS (Set 3)

The macOS Set 3 jobs are failing on both net8.0 and net9.0. The failing test is SqlBulkCopyDataRow_Type — one of the methods that lost its flaky marker when the class-level [Trait(Category, flaky)] was removed without method-level re-addition.

Error:

The given value '1/1/0001 12:00:00 AM' of type DateTime from the data source cannot be converted to type time for Column 0 [f1] Row 1.

This seems to fail deterministically on macOS with DateTime → time boundary values (0001-01-01, 9999-12-31), which looks more like a platform-specific issue than intermittent flakiness. We might need to consider restoring the flaky marker on this method (and potentially the other SqlBulkCopy* variants), or alternatively adding a macOS-specific skip condition if this is a known platform limitation.

@priyankatiwari08 priyankatiwari08 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI is failing on macOS Set 3 (net8.0 + net9.0) due to SqlBulkCopyDataRow_Type — please address the comment above before merging.

@github-project-automation github-project-automation Bot moved this from In review to Waiting for customer in SqlClient Board Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area\Tests Issues that are targeted to tests or test projects

Projects

Status: Waiting for customer

Development

Successfully merging this pull request may close these issues.

4 participants