[Mgmt] Harden SDK review contracts and trusted evidence preflight

[Yuchao Yan (msyyc)]

Replaces #49223 with the same implementation commit on an Azure-owned source branch: Azure:mgmt-review-reliability. No code changes are introduced by this PR migration.

Summary

Implement the management SDK review reliability improvements in one draft PR, rather than adding another prompt-only repair.

• Introduce a strict version-2 draft/publication contract: source IDs and checked ranges instead of model-built URLs and citation flags; snapshot-owned release state, entry identities and required check keys; separate specification evidence and permitted SDK context.
• Collect bounded immutable SDK evidence, including synchronous/asynchronous clients and version metadata, without importing or executing package code. Derive version consistency, preview/beta compatibility, stability flags and the more-than-21-day date reminder in shared trusted code. A rule fingerprint prevents silently using stale deterministic policy after authoritative rules change.
• Add a read-only, host-side evidence/preflight service using the existing gh-aw v0.88.8 Python MCP-script integration. It registers specification files by fetching pinned GitHub content itself, checks complete drafts, reports independent errors with codes/paths, and allows an initial attempt plus at most two corrections. Only successful preflight returns a fixed submission envelope.
• Preserve the independent publisher gate, immutable producer artifact ID, event/tooling binding, fixed resolved-PR target and max-one comment. Re-fetch registered specification evidence independently, verify content hashes, recompute routine checks, and render only validated output before the built-in publication/hiding handler.
• Accept and remove a matching redundant item_number; reject mismatches and unsupported publication fields. Partial reviews explicitly require human review, even when no findings are proven.
• Update workflow maintenance/rollout documentation, retain sanitized historical failure shapes with run provenance, expand regression/runtime coverage, and regenerate only this workflow's lockfile.

Historical failures addressed

Preserves the protections and safeguards from #49097, #49147, #49149 and #49208.

The corpus covers stale base-revision tooling, restrictive Markdown grammar, the literal - false-success shape, matching transport targets, SDK evidence mixed into specification-only attribution, missing anchors (including the five-reference shape reported in #49209 / PR #49163), contradictory unavailable/anchored citations, stability-version evidence, and high-confidence attribution without verified specification lines.

Legacy fragments are explicit test migrations, not an accepted production fallback and not proof that a historical model's semantic claims were correct. Production accepts only schema version 2. Initial-release negatives for incomplete discovery, moves/renames, inaccessible evidence, historical releases, malformed dates and breaking entries remain covered.

Trust boundaries and remaining limitations

The agent, host service and publisher use separate copies of the pre-agent snapshot from the producer's immutable artifact ID. Executable tooling stays pinned to github.workflow_sha; no PR-controlled checkout or package execution is added. The service runs outside the sandbox and exposes only describe/read/register/preflight operations. The agent is not granted general Python, curl, arbitrary host execution or repository write permissions.

SDK collection retains existing request/per-file limits and adds an 8-MiB source-catalog text cap. Specification registration is limited to 20 files and up to 1 MiB of text per package, stopping conservatively when the remaining budget cannot cover another bounded file request. Missing/truncated/access-error evidence remains explicit.

The service mechanically caps validation calls within its process and closes after acceptance. Calling it before the built-in safe-output tool remains an agent instruction; the preflight digest is an edit detector, not a cryptographic attestation. A bypass/restart cannot bypass independent publisher validation. Line existence and structural/semantic-contract validation do not prove a citation supports a natural-language claim. Semantic README review and breaking-change causality remain model responsibilities.

Unsupported packaging layouts or a changed authoritative rule fingerprint produce unverified checks, not invented defaults. GitHub publication and older-comment hiding remain nontransactional; a later API failure is still possible. Preflight logs distinguish attempts/corrections and review completeness; publisher validation logs distinguish a validated artifact from pending built-in publication.

Test-only manual reviews (disabled by default)

Per the latest review decision, workflow_dispatch is commented out before merge. The label trigger remains enabled. Both the canonical Markdown source and lockfile include re-enable guidance; the lockfile reminder is comment-only and is not preserved by the compiler. To test again, uncomment the source block on a trusted Azure-owned branch, compile with gh-aw v0.88.8, commit/push the source and lockfile, and dispatch that branch. Comment the source block again and recompile before merging. Merely selecting --ref cannot enable a disabled trigger.

gh workflow run mgmt-sdk-pr-review.lock.yml --repo Azure/azure-sdk-for-python --ref mgmt-review-reliability -f pr_number=48997

This command requires an enabled revision to have been pushed first. Omitting --ref selects the default branch, where dispatch is unavailable while disabled. GitHub's manual-run UI also requires the dispatch entry point on the default branch. No live workflow was dispatched during this change.

Keep workflow/job concurrency active: label events use their PR number, and temporarily enabled manual runs use github.event.inputs.pr_number. Disabling only the group leaves an empty concurrency mapping, not a disabled trigger.

When enabled, manual dispatch uses exactly the label-triggered collector, agent, preflight and real publisher, including older-review hiding after validation. Both original and rerun actors must be msyyc, the execution repository remains Azure/azure-sdk-for-python, and the target SDK PR source must be Azure-organization-owned. The fixed target is resolved through the GitHub API and bound to its immutable head. Closed target PRs remain supported; no general-purpose write capability or additional credentials are introduced.

The previously deferred graceful-skip guards are now included: unauthorized manual actors skip the review jobs before setup. A permissionless notice logs the policy and a clear summary without failing; its explicit always() handles the compiler-added dependency on skipped activation. The overall run can be successful because this notice job succeeds, while review jobs are skipped. Each job also guards partial reruns, and the existing collector/service/publisher authorization checks remain fail-closed backstops. Label eligibility is unchanged.

Validation covered 146 tests in default-disabled mode, and 145 applicable tests in temporarily re-enabled mode, both with the pinned runtime and no skips (the disabled-state assertion is deliberately excluded only from the enabled-mode run). Both modes compiled strictly and passed targeted actionlint; the compiled notice Bash step was executed locally. Final lockfile executable content matches the disabled compiler output; only explanatory comments were added afterward. No authorization boundary was relaxed.

Initial-release client naming

Confirmed first releases now require synchronous and asynchronous public client names ending in the exact suffix MgmtClient. Trusted code parses collected _client.py declarations as data only, and the independent publisher derives a Blocking finding even if the model submits no findings. The remediation asks the author to add or update the client-name customization in client.tsp, then regenerate the SDK.

This does not force renames in existing releases. Unknown first-release status, unreadable declarations and unsupported/ambiguous client layouts remain explicitly unverified. The original initial-release discovery safeguards and generated-source exclusions remain intact.

The authoritative rule text and its implementation are now pinned to the same trusted workflow commit. This lets manual branch runs test the new requirement before merge without taking rules from the SDK PR or silently mixing candidate tooling with older default-branch policy. The public model-authored schema is unchanged; the new check is snapshot-owned.

New regressions cover exact/case-sensitive suffix matching, sync/async blockers, existing-release exemptions, uncertain/missing/stale evidence, data-only parsing, immutable policy binding and publication-derived findings. The existing collection-budget boundary tests account for removing two unnecessary default-branch discovery calls; request limits are unchanged.

Unified API-version metadata

Both API-version drift and preview/beta compatibility now use the _metadata.json apiVersions service-to-version map through one shared validator. The nullable or contradictory singular apiVersion is not used for judgments. A missing, empty or malformed map remains unverified, with no fallback to the singular field.

Drift compares complete mappings (including changed service associations, additions and removals) while ignoring key order. The rendered finding preserves both service-to-version maps. Any map value containing preview requires a beta SDK, including mixed stable/preview maps; a service name containing that word alone is not a preview API.

Regression provenance: #48997 (comment) . A read-only replay of the two cited revisions confirms ComputeGallery changed from 2025-12-03 to 2026-03-03. The new implementation reports that real drift instead of becoming unverified because apiVersion is null. No live review was published during this replay.

Follow-up review fixes

The collector now fetches pinned pyproject.toml before deriving version/client paths, including cold-cache packages handed off by the earlier per-package budget check. Both reads use the same bounded cache helper. The early project fetch counts toward the 16-file limit, and the final consistency-check request remains reserved. Missing/inaccessible/malformed project evidence is not converted into successful checks.

Multi-package comments share exact evidence URLs through labeled references across completed checks, findings, unverified checks, API-version drift and breaking-change attribution. Different revisions and line ranges remain distinct; unavailable-line explanations remain at each use. The seven-package regression now contains 35 links instead of 77, and passes actual pinned-runtime ingestion and publication with mocked GitHub writes. The 48-link/60,000-byte gates remain enforced independently by the publisher, including genuinely oversized reviews.

Single-package output is unchanged: complete, partial and finding-bearing examples were byte-compared with the previous commit. Added regressions cover cache handoff, near-exhausted budgets, retained cached evidence, file caps, multi-package findings and partial checks, distinct ranges/revisions, shared specification evidence and special-character URL preservation through the pinned publisher. No authorization, review-policy or publication-permission changes are included in these fixes.

Validation

• 146 tests passed, no skips, using python -m unittest discover -s .github/workflows/tests -p "test_mgmt_sdk_review*.py" with GH_AW_RUNTIME set to the v0.88.8 runtime.
• Pinned runtime coverage includes the compiled Python MCP handler, actual CLI JSON-stdin parsing and documented jq expressions, failed preflight followed by successful correction, safe-output capacity enforcement, ingestion, and the built-in comment handler with mocked GitHub writes. Exhaustion leaves existing comments untouched; positive controls verify hiding after valid publication.
• mgmt-sdk-pr-review compiled with gh-aw v0.88.8, strict mode. The existing pull_request_target warning remains; no compiler upgrade or unrelated workflow regeneration.
• Black formatting passed for implementation/contract tests, with changed-line formatting checked for the collector and existing collector tests.
• Repository CSpell: zero issues across the changed source, tests, workflow documentation and corpus.
• Repository-configured actionlint passed. The repository excludes generated locks; an additional targeted parser check passed with only the same three baseline incompatibilities suppressed (copilot-requests twice and generated concurrency.queue). No new action/container pins changed except the compiler-required Python setup action; no additional secrets were introduced.
• git diff --check passed.
• Read-only local smoke against PR [AutoPR azure-mgmt-computeworkloadmanager]-generated-from-SDK Generation - Python-6876016 #49163: the working-tree collector discovered the package with no source-discovery issues and completed all four deterministic checks. A deliberately partial semantic draft passed local preflight and the independent publisher gate, correctly retaining partial status. No GitHub review was published by that smoke test.

Post-merge rollout gate — not performed in this PR

Label runs execute default-branch workflow tooling; manual runs execute the selected workflow revision. Local tests, PR CI and feature-branch manual runs do not establish deployment on the production default branch. After merge, use fresh triggers across initial releases, ordinary updates, breaking changes and incomplete evidence, including #49209's scenario.

Require ten consecutive representative canaries with a valid published review or explicit expected incomplete outcome, no unexplained contract rejection and no false clean-review status. Verify actual toolingRevision, safe_outputs result and the comment. Track first-pass acceptance, correction success and publication success separately.

Related to #49209; do not consider the rollout verified until these canaries complete. Do not merge automatically.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Copilot review overview

🟡 Changes recommended

Client and version evidence can be omitted when pyproject.toml was not already cached.

Review effort: Balanced
Findings: 1

Open (1)

• Version discovery runs before pyproject.toml is cached · New

What changed in this PR

Hardens management SDK review evidence collection, validation, and publication.

Changes:

• Adds trusted evidence collection and deterministic checks.
• Introduces schema-v2 preflight and publication validation.
• Adds manual dispatch support and regression coverage.

File	Description
eng/​ignore-links.txt	Excludes the local review endpoint.
.github/​workflows/​mgmt-sdk-pr-review.md	Defines the hardened workflow.
.github/​workflows/​mgmt-sdk-pr-review.lock.yml	Compiles the updated workflow.
.github/​workflows/​scripts/​mgmt_sdk_review_context.py	Collects and pins review context.
.github/​workflows/​scripts/​mgmt_sdk_review_contract.py	Validates and renders reviews.
.github/​workflows/​scripts/​mgmt_sdk_review_evidence.py	Implements evidence policy and checks.
.github/​workflows/​scripts/​mgmt_sdk_review_service.py	Provides host-side preflight operations.
.github/​workflows/​tests/​test_mgmt_sdk_review_trigger.py	Tests trigger authorization.
.github/​workflows/​tests/​test_mgmt_sdk_review_reliability.py	Tests reliability boundaries.
.github/​workflows/​tests/​test_mgmt_sdk_review_context.py	Updates collector expectations.
.github/​workflows/​tests/​test_mgmt_sdk_review_comment.py	Updates publication contract tests.
.github/​workflows/​tests/​mgmt_review_runtime.cjs	Extends runtime integration coverage.
.github/​workflows/​tests/​fixtures/​mgmt-review/​historical/​contract-failures.json	Captures historical failure shapes.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Repeated automatic evidence links make valid reviews with several affected packages exceed the fixed publication limit.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Per-check URL duplication incorrectly triggers the fixed link cap · New
• Version discovery runs before pyproject.toml is cached

[Copilot]

Copilot review overview

🔵 Needs a closer look

The security-sensitive workflow and publication changes require human review and the documented post-merge canary rollout.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Per-check URL duplication incorrectly triggers the fixed link cap
• Version discovery runs before pyproject.toml is cached

[Copilot]

Copilot review overview

🟡 Changes recommended

The repository-scoped token cannot reliably fetch and independently revalidate cross-repository specification evidence.

Review effort: Balanced
Findings: 1

Open (1)

• Use credentials with read access to the specification repository · New

Resolved since last review (2)

• Per-check URL duplication incorrectly triggers the fixed link cap
• Version discovery runs before pyproject.toml is cached

[Copilot]

Copilot review overview

🟡 Changes recommended

The publisher lacks Contents permission for its independent specification re-fetch, blocking direct TypeSpec attribution publication.

Review effort: Balanced
Findings: 2

Open (2)

• safe_outputs job lacks Contents read permission for specification files · New
• Use credentials with read access to the specification repository