Repository navigation
feat(testing): flag C# typeof-only, construct-only and BDD cannot-fail tests in the audit #6040
Description
Part of #6038
Problem
The testing:audit C# adapter misses the worst cannot-fail shapes in .NET suites, does not scan step-definition or UnitTest1.cs files, and its generous check assertion token suppresses print-only diagnostic tests.
On main (testing 0.21.3), plugins/testing/skills/audit/adapters/cs-xunit.yaml:
:9files: ['*Test.cs', '*Tests.cs'], so*StepDefinitions.cs(Reqnroll[Given]/[When]/[Then]) andUnitTest1.csare never enumerated.:11-12test_startrecognizesFact,Theory,Test,TestMethod,TestCase,DataTestMethod, not Reqnroll step attributes.:18-33assertion tokens include[Cc]heck,[Ee]nsure,[Vv]alidate, which match method and type names (for example*_CheckConnectionStrings, a...Checkertype), so a test that only prints counts as asserting.:42-46weakcoversAssert.NotNull, soAssert.NotNull(typeof(X))is a report-onlyweak-oracle, not a cannot-fail finding.
Evidence
Verified this pass by running main's plugins/testing/skills/audit/scripts/cant-fail-scan.sh --findings (extracted from origin/main into a scratch directory) over a synthetic xUnit fixture:
| Shape | Result on main |
|---|---|
Assert.NotNull(typeof(Widget)); |
weak-oracle (report-only) |
Assert.True(nameof(Widget) == typeof(Widget).Name); |
not flagged |
Assert.True(typeof(ComponentBase).IsAssignableFrom(typeof(Widget))); |
not flagged |
var x = new Widget(); Assert.NotNull(x); Assert.IsType<Widget>(x); |
not flagged |
var x = new Widget(); Assert.NotNull(x); |
weak-oracle (report-only) |
print-only Diagnostics_CheckConnectionStrings (Console.WriteLine only) |
not flagged |
if (cfg is null) return; before the only assertion |
not flagged |
UnitTest1.cs and LoginStepDefinitions.cs with [Then] ... Assert.True(true); |
not enumerated (1 of 1 files examined was ShapeTests.cs) |
The item reported Assert.NotNull(typeof(X)) as "not flagged at all"; on main it is flagged, but only as report-only weak-oracle.
Reported by the producing session from agent reading on a real repo, not rechecked here: logic re-implemented in the test body over literals with no production call; a discarded Result as the only oracle (including through a shared helper); identical-body duplicate tests in one class under different names; inherited architecture-test methods that return before building a rule. False positives it saw (4 of 218 findings, so precision is good): an assertion two hops away in a static helper in another file; an assertion in a fixture or base-class helper inside a DependsOn chain; await task.WaitAsync(timeout) as an implicit no-throw oracle; null-returning or throwing lookups (FindProperty, GetCustomAttribute, smart-enum FromName) read as weak oracles.
Proposed approach
- Add cannot-fail rules for C#: typeof-only assertions (
typeof(...)andnameof(...)with no other operand, andIsAssignableFrombetween twotypeofexpressions), and construct-then-check-construction (new T()followed only byNotNull/IsType<T>on that variable). Note the construct shape can fail if the constructor throws; classify it as "throw is the only oracle" (below) unless the constructor is known trivial. - Claim
*StepDefinitions.cswith Reqnroll step attributes as test starts, and widen globs toUnitTest*.cs(or detect test files by attribute presence). - Narrow
check/ensure/validateto call shapes that are assertions (for example a call on an assertion receiver or anAssert*/Should*-named extension), not any identifier containing the word. - Add a third class beside zero-assertion: "throw is the only oracle" (a smoke or operational test that passes if nothing throws; weak, can fail), so users stop deleting tests that guard config and DI wiring.
- Report an early
return;that precedes every assertion as a finding (the test reports Passed instead of Skipped).
Files: plugins/testing/skills/audit/adapters/cs-xunit.yaml (and cs-nunit.yaml, cs-mstest.yaml which extend it), plugins/testing/skills/audit/scripts/cant-fail-scan.awk, plugins/testing/skills/audit/SKILL.md (rule table), plugins/testing/skills/test-value/SKILL.md (taxonomy keyed to rule ids), plugins/testing/skills/audit/scripts/cant-fail-scan.test.sh.
Acceptance criteria
- Each "not flagged" or "not enumerated" row in the table above produces a finding on a fixture, and each new rule has a fixture case that fails before the change.
-
Assert.True(true)in a Reqnroll[Then]step in*StepDefinitions.csis flagged as cannot-fail. - A test calling a production method named
CheckX()and asserting nothing else is still treated as asserting only if the call is an assertion shape; a print-only test withCheckin its name is flagged. - Zero-assertion and "throw is the only oracle" are reported under different rule ids.
- Existing
cant-fail-scan.test.shcases pass; precision on the existing corpus does not drop (check-corpus-grid.sh).
Constraints and gotchas
- A trial-license guard (
catch (Exception ex) when (IsTrialFailure(ex))) is legitimate and must not be flagged. - xUnit v2 has no
Assert.Skip; the early-return finding's suggested action must not assume one (offer xUnit v3'sAssert.Skipor a skip package as options). - Hook false-positive fixes need a stay-quiet test that fails before the fix (
docs/conventions/hook-precision/README.md); the PostToolUse scan hook runs the same scanner. - Version bump and CHANGELOG entry for testing.
Context
Source: local handoff item 20261003-024929-testing-cleanup-and-audit-full-dotnet-support.md, gap 3.
Activity
kyle-sexton commentedon Oct 4, 2026
This was generated by AI during work-loop.
Approve this for autonomous dispatch, or keep it for a human?
The work-loop admission gate human-gates feature-shaped C3 work (
work-class: scoped, feature-shaped title). The lane removedagent-readyand appliedneeds-human. To dispatch it, an operator flips it back through/work-items:attend-queue.Lane: work-loop, instance melo-worker-1, cycle 1.
kyle-sexton commentedon Oct 4, 2026
kyle-sexton commentedon Oct 4, 2026
This was generated by AI during attend-queue.
Operator direction (2026-10-04): "go with the recommendations." This records that decision.
Operator decision: C3 (scoped), feature-shaped, priority high; approved for autonomous dispatch with the decisions below. Every row of this issue's table still reproduces on testing 0.22.6.
Land as four small PRs, in order:
cant-fail-scan.awk:1868-1869feeding:1694).:1707,:1710skip them today).Assert.True(true)and typeof/nameof constant oracles to the cs-xunitinertlist, as cs-mstest already does forIsTrue(true). No new rule id.typeof(A).IsAssignableFrom(typeof(B))andIsType<Base>can fail, so they stay out ofinert.[Then]step methods only, never[Given]or[When], plusUnitTest*.csglobs. Regenerate the hook filters (gen-hook-filters.sh).--checkgates. Zero-assertion keeps gating bodies with no call. Updatetier_of, thecant-fail-scan.shcase lists, the corpus GRID and the detector-findings README row.Decisions:
check/ensure/validatetoken. That choice is documented in the audit SKILL.md Gotchas. Fix the name leak (A) instead.cant-fail-scan.test.sh".check-corpus-grid.shchecks grid structure, not precision.Research: attend-queue slice
intake-6040.kyle-sexton commentedon Oct 4, 2026
kyle-sexton commentedon Oct 4, 2026
This was generated by AI during work-loop.
Resume point (work-loop lane stopping, 2026-10-04). Merged: PR A #6213 (testing 0.22.9) and PR B #6216 (testing 0.22.10). Remaining: PR C (Reqnroll
[Then]step methods,UnitTest*.csglobs, hook-filter regeneration) and PR D (a throw-is-the-only-oracle rule id, the early-return finding report-only, tier_of, the case lists, GRID and the README row), as split in the operator decision above. The lane released its lease.Open review notes carried from B, not yet decided:
IsNotNull(typeof/nameof)and the Assert.Equal/NotEqual/Same typeof forms were left out of B. MSTest's own inert list replaces the shared one, so MSTest still reports that shape as weak-oracle. Decide: follow-up or drop.inert-assertion:csremedy text now names typeof/nameof constants on MSTest and NUnit findings too.Related follow-ups filed: #6211 (the same name-as-assertion leak in Python and Go), #6218 (per-statement oracle judging). For C and D, plan first (medium scope): read B's diff on main and the cant-fail-scan corpus layout before dispatch.
kyle-sexton commentedon Oct 4, 2026
kyle-sexton commentedon Oct 11, 2026
Slice C (the last remaining slice) is up as draft PR #7049 on branch
feat/6040-reqnroll-then-test-start.[Then]step is now a test start in cs-xunit, cs-nunit and cs-mstest;[Given]and[When]are not. TheUnitTest*.cs/*StepDefinitions.csglobs and hook filters had already landed (gen-hook-filters.sh --checkis clean).[Then]under each adapter, anAssert.True(true)[Then], and a silent step file with[Given]/[When]. The four new bad cases fail against main's adapters and pass with the change;cant-fail-scan.test.sh(1292 checks) andaffected-tests.sh --base origin/main --runpass.[Then]slice.Left for a human: the carried review note from B (MSTest
IsNotNull(typeof/nameof)and theAssert.Equal/NotEqual/Sametypeof forms): follow-up or drop.Generated by Claude Code
kyle-sexton commentedon Oct 11, 2026
Handoff note for whoever picks this up next. The overnight Claude run has stopped, and Kyle is moving issue work to Cursor.
feat/6040-reqnroll-then-test-startWhere a branch is listed it is pushed, so it can be taken over as is. A draft PR shows a red
ci-statusby design until it is marked ready.Generated by Claude Code