feat: DH-23748: Multipass Release Matrix Workflow

[stanbrub]

• Reworked Release workflow to use matrix configs
• Improved dealing with APT conflicts after auto-provision
• Get version label from actual deephaven.version platform property
• Changed to using two always-on servers instead of one
• For release, use same iteration count for all benchmarks

[Copilot]

Copilot review overview

🟡 Changes recommended

The release profile remains temporary, repeated artifact names fail, mutable edge images are unpinned, and APT services are still interrupted.

Review effort: Balanced
Findings: 2 · 2

Open (4)

• No-block stop can orphan active APT processes · New
• Release matrix artifact names collide across rows · New
• Release profile runs incomplete benchmark suite · New
• Mutable edge image tag makes release non-reproducible · New

What changed in this PR

Introduces matrix-driven release benchmarks, dedicated benchmark hosts, improved provisioning, and engine-derived release labels.

Changes:

• Adds release and existing-server matrix orchestration.
• Separates nightly and matrix hosts.
• Improves APT setup, result labeling, filtering, and log redaction.

File	Description
docs/​ForkSetup.md	Documents separate host secrets.
.github/​workflows/​remote-benchmarks.yml	Adds host routing and controlled clearing.
.github/​workflows/​release-remote-benchmarks.yml	Converts releases to a matrix workflow.
.github/​workflows/​nightly-remote-benchmarks.yml	Enables pre-run clearing.
.github/​workflows/​matrix-exist-remote-benchmarks.yml	Integrates clearing and matrix run types.
.github/​workflows/​compare-remote-benchmarks.yml	Uses the matrix host.
.github/​workflows/​adhoc-exist-remote-benchmarks.yml	Moves clearing into the reusable workflow.
.github/​workflows/​adhoc-auto-remote-benchmarks.yml	Clears before both comparison runs.
.github/​scripts/​setup-test-server-remote.sh	Revises automatic-update handling.
.github/​scripts/​setup-ssh-local.sh	Detects unresolved host aliases.
.github/​scripts/​run-ssh-local.sh	Redacts host addresses from logs.
.github/​scripts/​run-benchmarks-remote.sh	Applies class filters with tag filters.
.github/​scripts/​matrix_array.sh	Gives releases uniform iterations.
.github/​scripts/​fetch-results-local.sh	Derives labels from engine versions.
.github/​scripts/​build-matrix.sh	Adds run types and matrix validation.
.github/​scripts/​adhoc.sh	Adds cloud-init data to provisioning.
.github/​resources/​adhoc-server-cloud-init.yml	Disables automatic APT services at boot.
.github/​matrix/​release.sh	Defines repeated release passes.
.github/​matrix/​gc-train-heap-fail.sh	Adds matrix run-type output.
.github/​matrix/​gc-train-1000-100.sh	Adds matrix run-type output.
.github/​matrix/​example-regression-matrix.sh	Adds matrix run-type output.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Release rows reuse an artifact name, and mutable edge images can change between sequential passes.

Review effort: Balanced
Findings: 2 · 1

Open (3)

• Release matrix artifact names collide across rows
• No-block stop can orphan active APT processes
• Mutable edge image tag makes release non-reproducible

Resolved since last review (1)

• Release profile runs incomplete benchmark suite

[Copilot]

Copilot review overview

🔵 Needs a closer look

Repeated release artifact names will fail after the first matrix leg, and failed dpkg repairs are currently ignored.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Release matrix artifact names collide across rows
• Mutable edge image tag makes release non-reproducible

Resolved since last review (1)

• No-block stop can orphan active APT processes

Previously missed (1)

In code that hasn't changed since last review

Do not ignore dpkg repair failures

.github/​scripts/​setup-test-server-remote.sh:69

The repair command's failure is discarded even though the following package installs depend on a consistent dpkg state. If an interrupted transaction cannot be configured, continuing only defers the failure (or leaves the host partially configured); let errexit stop setup at the repair failure.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The coupled remote-host lifecycle, cloud provisioning, and release publication changes require end-to-end workflow validation.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Release matrix artifact names collide across rows
• Mutable edge image tag makes release non-reproducible

Previously missed (1)

In code that hasn't changed since last review

Use canonical capitalization for GitHub

.github/​scripts/​run-ssh-local.sh:44

Use the product name’s canonical capitalization: “GitHub.”

[Copilot]

Copilot review overview

🟡 Changes recommended

Server cleanup depends on matrix scheduling order, which can cause stale state or mid-run clearing.

Review effort: Balanced
Findings: 2

Open (2)

• Matrix order does not guarantee server initialization · New
• Matrix scheduling can mix image revisions across release jobs · New

Resolved since last review (2)

• Release matrix artifact names collide across rows
• Mutable edge image tag makes release non-reproducible

[Copilot]

Copilot review overview

🔵 Needs a closer look

The release matrix still uses temporary values that publish only two benchmark classes across three passes.

Review effort: Balanced
Findings: 2

Open (2)

• Matrix scheduling can mix image revisions across release jobs
• Matrix order does not guarantee server initialization

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 3 · 3 · 1

Open (7)

• RUN_ID is computed from ${DEST_DIR}/run-*, but this script moves results/ into… · New
• Matrix scheduling can mix image revisions across release jobs
• Matrix order does not guarantee server initialization
• run_type is validated against only adhoc|release|compare, but other workflows in this PR use… · New
• This check relies on ${HOST} being set in the environment, but setup-ssh-local.sh is invoked… · New
• Adding a new workflow_call input as required: true is a breaking change for any existing… · New
• This header comment is now misleading: the workflow is invoked for adhoc and compare as well… · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

Long-running release rows can pull different repository revisions into one result set.

Review effort: Balanced
Findings: None

Resolved since last review (7)

• RUN_ID is computed from ${DEST_DIR}/run-*, but this script moves results/ into…
• Matrix scheduling can mix image revisions across release jobs
• Matrix order does not guarantee server initialization
• Adding a new workflow_call input as required: true is a breaking change for any existing…
• This check relies on ${HOST} being set in the environment, but setup-ssh-local.sh is invoked…
• run_type is validated against only adhoc|release|compare, but other workflows in this PR use…
• This header comment is now misleading: the workflow is invoked for adhoc and compare as well…

Previously missed (1)

In code that hasn't changed since last review

Pin matrix setup checkout to immutable workflow commit

.github/​workflows/​release-remote-benchmarks.yml:50

Each matrix row invokes the reusable setup independently, and that setup checks out github.ref_name and runs git pull before restaging the remote compose file. Since this release can run for roughly 28 hours, a branch update during the run can make later passes use a different repository configuration while all results still accumulate in one set. Pin the remote checkout to the workflow's immutable github.sha for every row.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Per-row reporting causes repeated publication side effects and leaves the release summary representing only one pass.

Review effort: Balanced
Findings: None

Previously missed (3)

In code that hasn't changed since last review

Generate the summary from accumulated results after all passes

.github/​matrix/​release.sh:38

These independent one-iteration rows do not produce a seven-pass summary. Every reusable-workflow setup calls build-benchmark-artifact-remote.sh, which deletes run/results before reusing the cached JAR; consequently each row generates release-benchmark-summary.svg from only its own pass, and each upload overwrites the shared set/actor summary with that single-pass SVG. Generate and upload the summary once from the accumulated set after all passes, while retaining the per-row raw-result uploads.

Avoid repeating notifications and NFS sync for each matrix row

.github/​workflows/​release-remote-benchmarks.yml:51

Each of the seven matrix rows invokes the reusable workflow's complete report phase, which posts a Slack notification and performs a full bucket-to-NFS sync. A single release will therefore emit seven notifications and seven expensive syncs, including reports over intermediate 1–6-pass data. Keep the failure-tolerant per-row GCloud upload, but move notification and NFS synchronization to one post-matrix job (or gate those side effects with a separate input).

Use standard “GitHub” capitalization

.github/​scripts/​run-ssh-local.sh:44

Use the product's standard capitalization: “GitHub”.

[Copilot]

Copilot review overview

🟡 Changes recommended

The release matrix overwrites its aggregate summary with a single-pass SVG instead of publishing the intended seven-pass median.

Review effort: Balanced
Findings: 1

Open (1)

• Matrix runs overwrite benchmark summaries instead of aggregating results · New