feat(testing): tag audit findings with test project kind and whether CI runs it #6041

Activity

1. kyle-sexton

   added

   status: needs-decisionAwaiting a human or maintainer judgment call.Awaiting a human or maintainer judgment call.

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.Refactors, migrations, contract changes; cross-cutting and hard to reverse.

   on Oct 3, 2026
2. kyle-sexton commented on Oct 13, 2026

   

   kyle-sexton

   on Oct 13, 2026

   MemberAuthor

   More actions

   Decision recorded. The owner approved applying independently verified recommendations on 2026-10-12.

   • How should the audit learn each test project's kind and CI membership? → Heuristic detection (nearest .csproj, workflow dotnet test targets and .sln membership, name and package tokens) with an optional config override, reporting unknown when unsure
     Basis: plugins/testing/skills/audit/scripts/ (git grep for csproj/project kind on origin/main: no hits, so nothing exists yet); issue feat(testing): tag audit findings with test project kind and whether CI runs it #6041 Proposed approach; judgment

   Wants testing:audit findings to carry the owning test project, whether CI runs it, and a project kind (unit, integration, operational) so live or CI-skipped projects rank differently and warn. Nothing has shipped; the findings table on main still has no project or CI field.

   Verified by an independent fresh-context check (confirmed, risk low): gh issue view 6041 (OPEN, no comments); git grep csproj|project_kind|ci_run in plugins/testing/skills/audit/scripts and plugins/testing/scripts: no hits (hits only in eval fixtures); git log since 2026-10-03 on testing audit (#7049, #6840, #6471: none adds project kind); #6038 OPEN; gh pr list search 6041: none

   Admission to an agent lane stays with the work-class rules and owner ratification.
3. kyle-sexton

   removed

   status: needs-decisionAwaiting a human or maintainer judgment call.Awaiting a human or maintainer judgment call.

   on Oct 13, 2026
4. kyle-sexton commented on Oct 13, 2026

   

   kyle-sexton

   on Oct 13, 2026

   MemberAuthor

   More actions

   Owner batch-ratified this admission in an attended session on 2026-10-12: work-class: scoped. Rule: class C1-C3, no trust surface, the independent check confirmed the recorded decision, and no owner step remains.
5. kyle-sexton

   added

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   and removed

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   on Oct 13, 2026
6. kyle-sexton commented on Oct 13, 2026

   

   kyle-sexton

   on Oct 13, 2026

   MemberAuthor

   More actions

   Correction: this issue already carried work-class: structural, which batch ratification never admits. The admission above is reverted; it stays escalated for an attended decision.
7. kyle-sexton

   added the

   Feature

   issue type on Oct 15, 2026
8. kyle-sexton

   removed

   work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.Refactors, migrations, contract changes; cross-cutting and hard to reverse.

   on Oct 17, 2026
9. kyle-sexton commented on Oct 17, 2026

   

   kyle-sexton

   on Oct 17, 2026

   MemberAuthor

   More actions

   Owner decision 2026-10-17: work classes are being retired (Q505); the recorded decision stands, so the structural hold is removed and this issue joins a later admission wave.
10. kyle-sexton commented on Oct 18, 2026

    

    kyle-sexton

    on Oct 18, 2026

    MemberAuthor

    More actions

    Agent Brief

    Type: Feature
    Summary: testing:audit findings carry the owning test project, whether CI runs it, and the project's kind, and warn on operational or live projects.

    Current behavior:
    The findings table written by the cannot-fail scanner carries rank, tier, confidence, location, surface, finding and action. Nothing says which test project a finding lives in, whether CI builds that project, or whether the project touches live environments.

    Desired behavior:

    • For C#, each finding names its owning project (the nearest .csproj above the test file).
    • Each finding carries ci: yes|no|unknown: yes when the project belongs to a solution or project a repository workflow's dotnet test or dotnet build targets, no when workflows were read and none targets it, unknown when no workflow can be read or parsing fails.
    • Each finding carries a kind from cheap explicit evidence: project and folder name tokens (Integration, Smoke, Functional, Architecture, Provisioning) and package references (Reqnroll, Playwright, ArchUnitNET, Testcontainers). An optional repo config override maps a project to a kind. Anything else is unknown, never a guess.
    • Findings in CI-run unit projects rank first. A finding in a project of kind operational or live carries a warning not to run its tests.
    • The audit stays read-only and never executes a test project to learn its kind.
    • Non-C# findings show the new fields as unknown and are otherwise unchanged.

    Key interfaces:

    • The scanner's findings record: new project, ci and kind fields.
    • testing config resolution: an optional per-project kind override key.
    • The audit skill's description of the findings table.

    Acceptance criteria:

    • On a fixture with two test projects, one in a workflow's dotnet test solution and one not, findings carry ci: yes and ci: no.
    • A project named *.Provisioning.Tests gets kind operational and its findings carry the warning.
    • A config override changes a project's kind.
    • An unclassifiable project reports unknown.
    • A malformed workflow file yields ci: unknown, not an error exit.
    • The scanner's test suite passes under scripts/run-test-file.sh with these cases.
    • scripts/check-changelog-fragments.sh passes with a testing fragment.

    Out of scope:

    • Repo-wide cleanup planning (feat(testing): let cleanup plan a whole repo as per-project batches #6043), which consumes these fields.
    • Kind detection for non-.NET ecosystems.
    • Running any test project.

    ---

    This was generated by AI during backlog shaping. The owner approved this in an attended backlog-shaping session on 2026-10-17.
11. kyle-sexton

    added

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

    and removed

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

    on Oct 18, 2026
12. kyle-sexton

    closed this as completedin #7945on Oct 19, 2026