[Mgmt] Harden SDK review contracts and trusted evidence preflight

[Yuchao Yan (msyyc)]

Summary

Implement the management SDK review reliability improvements in one draft PR, rather than adding another prompt-only repair.

• Introduce a strict version-2 draft/publication contract: source IDs and checked ranges instead of model-built URLs and citation flags; snapshot-owned release state, entry identities and required check keys; separate specification evidence and permitted SDK context.
• Collect bounded immutable SDK evidence, including synchronous/asynchronous clients and version metadata, without importing or executing package code. Derive version consistency, preview/beta compatibility, stability flags and the more-than-21-day date reminder in shared trusted code. A rule fingerprint prevents silently using stale deterministic policy after authoritative rules change.
• Add a read-only, host-side evidence/preflight service using the existing gh-aw v0.88.8 Python MCP-script integration. It registers specification files by fetching pinned GitHub content itself, checks complete drafts, reports independent errors with codes/paths, and allows an initial attempt plus at most two corrections. Only successful preflight returns a fixed submission envelope.
• Preserve the independent publisher gate, immutable producer artifact ID, event/tooling binding, fixed resolved-PR target and max-one comment. Re-fetch registered specification evidence independently, verify content hashes, recompute routine checks, and render only validated output before the built-in publication/hiding handler.
• Accept and remove a matching redundant item_number; reject mismatches and unsupported publication fields. Partial reviews explicitly require human review, even when no findings are proven.
• Update workflow maintenance/rollout documentation, retain sanitized historical failure shapes with run provenance, expand regression/runtime coverage, and regenerate only this workflow's lockfile.

Historical failures addressed

Preserves the protections and safeguards from #49097, #49147, #49149 and #49208.

The corpus covers stale base-revision tooling, restrictive Markdown grammar, the literal - false-success shape, matching transport targets, SDK evidence mixed into specification-only attribution, missing anchors (including the five-reference shape reported in #49209 / PR #49163), contradictory unavailable/anchored citations, stability-version evidence, and high-confidence attribution without verified specification lines.

Legacy fragments are explicit test migrations, not an accepted production fallback and not proof that a historical model's semantic claims were correct. Production accepts only schema version 2. Initial-release negatives for incomplete discovery, moves/renames, inaccessible evidence, historical releases, malformed dates and breaking entries remain covered.

Trust boundaries and remaining limitations

The agent, host service and publisher use separate copies of the pre-agent snapshot from the producer's immutable artifact ID. Executable tooling stays pinned to github.workflow_sha; no PR-controlled checkout or package execution is added. The service runs outside the sandbox and exposes only describe/read/register/preflight operations. The agent is not granted general Python, curl, arbitrary host execution or repository write permissions.

SDK collection retains existing request/per-file limits and adds an 8-MiB source-catalog text cap. Specification registration is limited to 20 files and up to 1 MiB of text per package, stopping conservatively when the remaining budget cannot cover another bounded file request. Missing/truncated/access-error evidence remains explicit.

The service mechanically caps validation calls within its process and closes after acceptance. Calling it before the built-in safe-output tool remains an agent instruction; the preflight digest is an edit detector, not a cryptographic attestation. A bypass/restart cannot bypass independent publisher validation. Line existence and structural/semantic-contract validation do not prove a citation supports a natural-language claim. Semantic README review and breaking-change causality remain model responsibilities.

Unsupported packaging layouts or a changed authoritative rule fingerprint produce unverified checks, not invented defaults. GitHub publication and older-comment hiding remain nontransactional; a later API failure is still possible. Preflight logs distinguish attempts/corrections and review completeness; publisher validation logs distinguish a validated artifact from pending built-in publication.

Authorized manual reviews

Manual dispatch uses exactly the label-triggered production pipeline, including real comment publication and older-review hiding after validation. There is no dry-run publisher or separate test implementation.

• Run in Azure/azure-sdk-for-python, supplying only pr_number. Omitting --ref uses the default branch (main); selecting another reviewed branch in that repository tests that workflow revision. Tooling remains pinned to github.workflow_sha.
• Both the original actor and rerun actor must be msyyc. Target resolution, agent-service startup and publication independently check authorization so failed-job-only reruns do not reuse a previous actor's authorization.
• The SDK PR source must be owned by the Azure organization. Personal forks and deleted source repositories are rejected for manual runs; prior label-trigger eligibility is unchanged. Closed PRs are supported for historical reproduction.
• The producer resolves the PR and immutable head through the API. Its trusted outputs bind all consumers and the fixed max-one publisher. Invalid target/source/actor inputs fail before the agent reviews evidence.
• This actor allowlist is an operational guard on trusted maintainer branches, not isolation from someone who can rewrite the executing workflow itself. No additional credentials or write permissions are added.

gh workflow run mgmt-sdk-pr-review.lock.yml --repo Azure/azure-sdk-for-python -f pr_number=49163

Add --ref <maintainer-test-branch> to test a different workflow branch. The manual-dispatch entry point must be established on the default branch for GitHub's Run workflow UI; this unmerged PR does not establish deployment. No live workflow was dispatched during local validation.

New regressions cover actor and partial-rerun denials, malformed PR numbers, personal/deleted/lookalike sources, stale event heads, trusted job-output binding and identical manual/label publication plus older-comment hiding through the actual pinned runtime with mocked GitHub writes. A read-only API smoke resolved PR #49163 successfully.

Validation

• 122 tests passed, no skips, using python -m unittest discover -s .github/workflows/tests -p "test_mgmt_sdk_review*.py" with GH_AW_RUNTIME set to the v0.88.8 runtime.
• Pinned runtime coverage includes the compiled Python MCP handler, actual CLI JSON-stdin parsing and documented jq expressions, failed preflight followed by successful correction, safe-output capacity enforcement, ingestion, and the built-in comment handler with mocked GitHub writes. Exhaustion leaves existing comments untouched; positive controls verify hiding after valid publication.
• mgmt-sdk-pr-review compiled with gh-aw v0.88.8, strict mode. The existing pull_request_target warning remains; no compiler upgrade or unrelated workflow regeneration.
• Black formatting passed for implementation/contract tests, with changed-line formatting checked for the collector and existing collector tests.
• Repository CSpell: zero issues across the changed source, tests, workflow documentation and corpus.
• Repository-configured actionlint passed. The repository excludes generated locks; an additional targeted parser check passed with only the same three baseline incompatibilities suppressed (copilot-requests twice and generated concurrency.queue). No new action/container pins changed except the compiler-required Python setup action; no additional secrets were introduced.
• git diff --check passed.
• Read-only local smoke against PR [AutoPR azure-mgmt-computeworkloadmanager]-generated-from-SDK Generation - Python-6876016 #49163: the working-tree collector discovered the package with no source-discovery issues and completed all four deterministic checks. A deliberately partial semantic draft passed local preflight and the independent publisher gate, correctly retaining partial status. No GitHub review was published by that smoke test.

Post-merge rollout gate — not performed in this PR

Label runs execute default-branch workflow tooling; manual runs execute the selected workflow revision. Local tests, PR CI and feature-branch manual runs do not establish deployment on the production default branch. After merge, use fresh triggers across initial releases, ordinary updates, breaking changes and incomplete evidence, including #49209's scenario.

Require ten consecutive representative canaries with a valid published review or explicit expected incomplete outcome, no unexplained contract rejection and no false clean-review status. Verify actual toolingRevision, safe_outputs result and the comment. Track first-pass acceptance, correction success and publication success separately.

Related to #49209; do not consider the rollout verified until these canaries complete. Do not merge automatically.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Copilot review overview

🟡 Changes recommended

Semantic checks can accept stale SDK evidence, and source-collection failures can be omitted from completeness reporting.

Review effort: Balanced
Findings: 2

Open (2)

• Source collection issues incorrectly allow complete reviews · New
• Semantic checks can use stale historical SDK records · New

What changed in this PR

Hardens management SDK review evidence collection, preflight validation, and publication boundaries.

Changes:

• Adds version-2 evidence contracts and deterministic checks.
• Introduces a bounded read-only preflight service.
• Expands regression and runtime boundary coverage.

File	Description
mgmt-sdk-pr-review.lock.yml	Regenerates the compiled workflow.
mgmt-sdk-pr-review.md	Integrates preflight service and v2 contract.
mgmt_sdk_review_context.py	Collects bounded SDK evidence.
mgmt_sdk_review_contract.py	Validates and renders review submissions.
mgmt_sdk_review_evidence.py	Implements deterministic evidence policy.
mgmt_sdk_review_service.py	Hosts read-only evidence and preflight operations.
contract-failures.json	Records sanitized historical failure shapes.
mgmt_review_runtime.cjs	Extends runtime integration harness.
test_mgmt_sdk_review_comment.py	Migrates publication contract tests.
test_mgmt_sdk_review_context.py	Updates collector expectations.
test_mgmt_sdk_review_reliability.py	Adds v2 reliability and boundary tests.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The trusted version parser can accept a literal that is subsequently overwritten, producing incorrect deterministic results.

Review effort: Balanced
Findings: None

Resolved since last review (2)

• Semantic checks can use stale historical SDK records
• Source collection issues incorrectly allow complete reviews

Previously missed (1)

In code that hasn't changed since last review

Count all VERSION assignments before validating literal value

.github/​workflows/​scripts/​mgmt_sdk_review_evidence.py:106

This filters out non-literal assignments before counting, so an untrusted file such as VERSION = "1.0.0" followed by VERSION = compute_version() is accepted as a single literal assignment even though its effective value is different. That can make version consistency, preview, and stability checks report trusted results from a value that package code would not use. Count every top-level assignment to VERSION, then require that the sole assignment has a string constant value.

[Yuchao Yan (msyyc)]

Superseded by #49231, which contains the same implementation commit (00bcb5b) on Azure:mgmt-review-reliability rather than the personal fork. Continue review and branch-based workflow testing on #49231. Neither PR has been merged.

[Copilot]

Copilot review overview

🟡 Changes recommended

The publisher re-fetches evidence without the required contents permission.

Review effort: Balanced
Findings: 1

Open (1)

• Grant publisher job contents read permission for evidence re-fetch · New