ci: make SynapseML releases approved and resumable

[Rana Singh (ranadeepsingh)]

What changed

Make repeated OSS releases and independent Internal releases plan-driven and
resumable, with reviewed sources, explicit publication approval, durable
recovery records and immutable package coordinates.

• Derive and seal release plans; reject duplicate JSON members and overwritten plan files.
• Preview work before publication, record intent/build IDs, and resume only authorized missing work.
• Bind Maven and private-feed producers to exact source tags and commits, then require artifact receipts.
• Recover missing derivative tags at recorded reviewed merge commits without moving existing tags.
• Isolate ESRP staging, reject linked output paths, and reject incomplete publication requests explicitly.
• Document preview, recovery and rollout limits, with current links to release guide v3.

September 22 refresh

Head: 3f7d9fcf376212b721fad69c3b8291e2c5cf142a

Integrated master: 681bd96990c421de3b91d2b1bf8f8f470764199d

Rebased all 24 original commits and preserved their complete messages. The
conflict resolutions retain master's current review-artifact guidance and
two-commit checkout for E2E impact detection, alongside the release PR's full
history/tag checkout and publication admission guards.

The incoming E2E pipeline contract initially failed with KeyError: 'fetchDepth'.
A test-only follow-up now checks both checkout modes and the unique named impact
step. No release implementation or release workflow changed during this refresh.

Fresh local evidence:

• Release and CI-helper contracts: 959 passed, 1 skipped, 63 subtests passed.
• Native Windows version-bump/history contracts: 228 passed.
• Black 22.3.0 and whitespace checks passed.
• The one skipped check explicitly requires SYNAPSEML_TEST_RELEASE_SBT=1;
  no fresh SBT runtime result is claimed.
• Source-tree and test-file hashes confirm the final integration is identical
  to the tested bytes. Incoming master changes outside the original PR paths
  were preserved.

The bounded direct integration review is appended to
the existing follow-up report.
Earlier multi-model reviews and Azure preview results remain historical evidence,
not current-head CI or review approval.

Remaining gates

Fresh current-head Azure merge validation and automated review are being requested
after this update. Their absence or an old-head result is not a pass. See the
current Checks and review threads for results.

These existing findings still require disposition:

• Reject explicitly mismatched artifact versions during ESRP staging.
• Coordinate the port pipeline invocation with the changed ESRP CLI.
• Make repository-specific target support explicit. Until that is enforced,
  use an OSS-only plan for master,spark4.0,spark4.1 and a separate Internal
  plan for master,spark4.1; the combined default still includes Internal Spark 4.0.

Human last-push approval and applicable companion-pipeline gates remain.
No release tags, production publication, approvals or rollout changes were
performed. This PR is not being declared merge-ready by the refresh.
Same-coordinate Maven retry remains unsupported.

Documentation and companion changes

• Release guide v3 is merged; historical v2 remains unchanged.
• Official publisher and Internal automation remain separate adoption gates for their tracks.
• Command reference
• Release procedure
• Review history

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR adds release-engineering automation to make SynapseML-to-Fabric releases more deterministic and reviewable, including a GitHub Actions “prepare → tag merged commit → publish notes” flow plus supporting Python tooling to derive/verify release identifiers and safely bump BBC-VHD pins.

Changes:

• Introduces a release matrix generator (release_matrix.py) and an end-to-end verifier (verify_release.py) with accompanying tests.
• Adds bump_bbcvhd.py (plus tests) to safely update BBC-VHD component pins while preserving line endings and rolling back on failure.
• Adds/updates GitHub workflows for release prepare/tagging and manual, artifact-gated release notes; enables workflow_dispatch for PR validation so release PRs created via GITHUB_TOKEN can be validated.

Show a summary per file

File	Description
scripts/test_bump_version.py	Extends bump-version tests for denylisted repo-relative paths; normalizes path keys to POSIX.
scripts/bump-version.py	Improves Windows console encoding robustness; adds path-based denylist support; normalizes path handling via as_posix().
scripts/release/release_matrix.py	Adds a single-source-of-truth release matrix for tags and artifact versions + CLI rendering.
scripts/release/verify_release.py	Adds live verification of GitHub/ADO tags and Maven/PyPI/UPack/Azure Artifacts presence.
scripts/release/bump_bbcvhd.py	Adds deterministic, rollback-safe BBC-VHD component pin updater with CRLF/LF preservation.
scripts/release/README.md	Documents how to use the new release tooling and where it fits in the guide.
scripts/release/test_release_matrix.py	Adds regression/contract tests for the release matrix derivations and CLI validation.
scripts/release/test_verify_release.py	Adds unit tests for verifier networking/error handling and run plan behavior.
scripts/release/test_bump_bbcvhd.py	Adds tests for BBC-VHD bump idempotency, rollback, and newline preservation.
scripts/release/test_release_workflows.py	Adds tests asserting key workflow contract properties (manual gating, dispatch behavior).
scripts/release/test_prev_tag.sh	Adds a repo-taglist regression script for “previous primary tag” selection logic.
.github/workflows/release-prepare.yml	Adds a reviewed release PR generator and a merged-commit tagger that dispatches downstream orchestration.
.github/workflows/release-notes.yml	Adds a manual, artifact-gated GitHub Release publisher with explicit previous-tag selection.
.github/workflows/pr-validation.yml	Enables workflow_dispatch so validations can be dispatched for bot-opened release PR branches.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (1)

scripts/release/verify_release.py:260

• --skip internal does not currently skip the internal PyPI feed checks (the synapseml_internal package); it only skips internal git tag checks. This makes --skip internal behave inconsistently compared with --skip public.

    def pip(self, package: str, version: str) -> str:
        if "pip" in self.skip or "ado" in self.skip:
            return SKIPPED
        # Azure Artifacts normalises pypi names: synapseml_internal -> synapseml-internal
        return (
            OK
            if version
            in self._feed_versions("Synapse-Conda", "pypi", package.replace("_", "-"))
            else MISSING
        )

• Files reviewed: 14/14 changed files
• Comments generated: 2
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/release/verify_release.py:75

• On Windows, subprocess.run(..., shell=True) expects a command string, but this code passes a list. That combination can fail (or behave differently) on win32, which would break the fallback ADO auth path even when az is installed. Build a platform-appropriate command (string for shell=True, list otherwise).

    out = subprocess.run(
        [
            "az",
            "account",
            "get-access-token",
            "--resource",
            ADO_RESOURCE,
            "--query",
            "accessToken",
            "-o",
            "tsv",
        ],
        capture_output=True,
        text=True,
        shell=(sys.platform == "win32"),
    )

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Addressed the current-head suppressed Windows auth finding in 6e80f27.

• Windows now gives subprocess.run(..., shell=True) a command string; non-Windows retains an argument list with shell=False.
• Parameterized regression coverage locks both command shapes.
• A token-less Windows fallback replay authenticated through �z and found all 12 selected v1.1.3 master artifacts; the complete release-tooling suite is 279 passed.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/release/verify_release.py:313

• The Maven publication gate here verifies only synapseml-core_{scala} (via public_maven()), but the workflow-generated release notes advertise com.microsoft.azure:synapseml_2.12:${VERSION} (see .github/workflows/release-notes.yml:137) and the docs use the synapseml_2.12 coordinate (website/doctest.py:25). This can let verify_release.py report COMPLETE even if the primary install coordinate is missing (or vice versa). Consider aligning the verification to check the same Maven artifact(s) that users are instructed to consume (e.g., verify both synapseml_{scala} and synapseml-core_{scala}, or switch to the canonical one).

            "maven",
            tp.key,
            f"synapseml-core_{tp.scala}",
            tp.oss_maven_version,
            c.public_maven(tp.scala, tp.oss_maven_version),

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Lite

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🟡 Changes recommended

Two critical issues remain in ESRP staging: stale explicitly versioned artifacts can be relabeled, and mandatory arguments break the supported Spark 4.1 pipeline.

Get a fresh assessment by requesting another Copilot review.

Review tier: Lite
Findings: 2

Open (2)

• Reject stale versioned files during ESRP staging · New
• Required staging arguments break the Spark 4.1 release pipeline · New

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved release publication scoping and dependency issues, plus workflow-test coverage gaps, block approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3 · 1 · 4

Open (8)

• Maven-only plans trigger unauthorized non-Maven publication tasks · New
• Required staging arguments break the Spark 4.1 release pipeline
• Reject stale versioned files during ESRP staging
• Optional skipped dependencies can prevent release publication · New
• Review artifact uses a session-workspace path · New
• Review artifact exposes an absolute local session path · New
• Review artifact contains an absolute machine-local path · New
• Review artifact contains checkout-specific commands · New