Quarantine flaky pipeline tests; add triage prompt#4476
Conversation
Two intermittently-failing tests observed in public CI at/after bf4cd14 are quarantined, and a reusable, access-agnostic (MCP/CLI/REST) prompt for triaging pipeline test failures is added. - SimulatedServerTests.ConnectionTests.TransientFault_RetryEnabled_ShouldSucceed_Async: transient-fault retry timing under CI load (SqlException surfaces on the retry login). Marked [Trait("Category","flaky")]. - ConnectionPoolTest.TransactionPoolTest.TransactionCleanupTest: process-global pool count race (Expected 2, Actual 3) from cross-test contamination inside a TransactionScope. Marked flaky with an accurate failure-signature comment. - Add .github/prompts/triage-pipeline-failures.prompt.md. Signed-off-by: Paul Medynski <31868385+paulmedynski@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Quarantines two intermittently failing tests in the SqlClient CI by marking them with the existing Category=flaky trait, and adds a reusable Copilot prompt to triage pipeline test failures at/after a specific commit SHA.
Changes:
- Mark
TransientFault_RetryEnabled_ShouldSucceed_Async(simulated server unit test) as quarantined via[Trait("Category", "flaky")]with an embedded failure signature comment. - Mark
TransactionCleanupTest(manual pooling test) as quarantined via[Trait("Category", "flaky")]with an embedded failure signature comment. - Add
.github/prompts/triage-pipeline-failures.prompt.mdto guide ADO/GitHub pipeline failure investigation and classification workflows.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/Microsoft.Data.SqlClient/tests/UnitTests/SimulatedServerTests/ConnectionTests.cs | Quarantines an intermittently failing transient-retry simulated-server test. |
| src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ConnectionPoolTest/TransactionPoolTest.cs | Quarantines an intermittently failing transaction-pool cleanup test. |
| .github/prompts/triage-pipeline-failures.prompt.md | Adds a triage workflow prompt for classifying pipeline test failures and deciding fix vs quarantine. |
Comments suppressed due to low confidence (1)
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ConnectionPoolTest/TransactionPoolTest.cs:93
- Quarantining this test as
[Trait("Category", "flaky")]reduces regular CI coverage, but the described failure mode (poolConnectionCountoff by 1) can be made deterministic by isolating the pool key per invocation. Since pools are process-global and keyed by connection string, appending a uniqueApplicationName(viaSqlConnectionStringBuilder) prevents cross-test contamination without quarantining the test.
[Trait("Category", "flaky")]
[ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
[ClassData(typeof(ConnectionPoolConnectionStringProvider))]
public static void TransactionCleanupTest(string connectionString)
{
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## main #4476 +/- ##
==========================================
- Coverage 70.88% 62.70% -8.19%
==========================================
Files 288 283 -5
Lines 43935 66821 +22886
==========================================
+ Hits 31142 41898 +10756
- Misses 12793 24923 +12130
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- Fix prompt tool identifier 'codebase/search' -> 'search/codebase' in triage-pipeline-failures.prompt.md, matching the documented identifier in generate-prompt.prompt.md.
- Quarantine flaky ConnectionFailoverTests.NetworkError_WithUserProvidedPartner_RetryDisabled_ShouldConnectToFailoverPartner with [Trait("Category", "flaky")]. It fails under CI load only: the 5s ConnectTimeout can be exhausted by the failover connection itself on slow agents. Captured the xUnit output and stack trace in a comment for future triage.
Description
Quarantines two intermittently-failing tests observed in the public CI pipelines on
mainat/after commit bf4cd14, and adds a reusable prompt for triaging pipeline test failures.Test changes (engineering processes / test coverage):
SimulatedServerTests.ConnectionTests.TransientFault_RetryEnabled_ShouldSucceed_Async— under CI load the simulated transient error occasionally resurfaces on the retry login, so the async open propagates theSqlExceptioninstead of succeeding. Same transient-fault retry-timing family as its already-quarantined siblings. Marked[Trait("Category", "flaky")]with the observed failure signature.ConnectionPoolTest.TransactionPoolTest.TransactionCleanupTest— the process-global pool (keyed by connection string) intermittently reports one extra connection (Assert.EqualExpected 2, Actual 3) due to cross-test contamination while running inside aTransactionScope. It is a test-isolation / pool-count race, not a product defect. Marked[Trait("Category", "flaky")]with an accurate root-cause comment.Tooling:
.github/prompts/triage-pipeline-failures.prompt.md— an access-agnostic (ADO MCP /azCLI / REST) workflow to find and classify failing tests in the SqlClient pipelines at/after a given commit, capture full xUnit output/stack traces, distinguish regression vs pre-existing, and present findings for approval before fixing or quarantining.No product code changes; no public API changes.