[mz-deploy] support autoscaling strategies

[sjwiesman]

Adds support for autoscaling policies to mz-deploy under two commands:

• Apply can now create clusters with auto scaling policies and alter policies when they change
• Stage will copy the autoscaling policy of the cluster it is copying

Most of the changes are added test coverage.

Motivation

Why does this change exist? Link to a GitHub issue, design doc, Slack
thread, or explain the problem in a sentence or two. A reviewer who has
no context should understand why after reading this section.

If this implements or addresses an existing issue, it's enough to link to that:
Closes
Fixes
etc.

Description

What does this PR actually do? Focus on the approach and any non-obvious
decisions. The diff shows the code --- use this space to explain what the
diff can't tell a reviewer.

Verification

How do you know this change is correct? Describe new or existing automated
tests, or manual steps you took.

[def-]

The QA LLM review found a few issues:

Finding 1: MEDIUM: A size change can fail permanently when the same apply resets autoscaling

Location: src/mz-deploy/src/cli/commands/clusters.rs:118-133

The reconciler always emits every SET before every RESET. That ordering is invalid for options whose constraints are checked against the effective cluster configuration. Consider a live cluster with SIZE = '25cc' and an ON HYDRATION policy whose HYDRATION SIZE = '100cc'. A natural declarative change is to make 100cc the steady-state size and remove the now-unnecessary policy. The new diff produces these statements in this order:

ALTER CLUSTER c SET (SIZE = '100cc');
ALTER CLUSTER c RESET (AUTO SCALING STRATEGY);

The first statement never succeeds. plan_alter_cluster keeps the existing autoscaling strategy when the statement does not mention it, combines that strategy with the proposed new size, and calls validate_auto_scaling_strategy (src/sql/src/plan/statement/ddl.rs:6560-6580). The validator rejects a hydration size equal to the effective cluster size, so execution stops with HYDRATION SIZE must differ from the cluster SIZE before the reset is reached. Retrying produces the same plan and the same failure.

The added integration test resets the policy without changing SIZE, so it does not cover this transition. A focused unit test of diff_cluster_options would also miss the server-side cross-option validation.

Reorder an autoscaling reset before a size SET when both are present, or otherwise plan the two statements according to their option dependencies. Add an execution-level test that promotes the old hydration size to the steady-state size while removing the policy.

Finding 2: MEDIUM: Existing clusters silently ignore a requested policy on regions without the introspection view

Location: src/mz-deploy/src/cli/commands/clusters.rs:101-106, 186-220

When supports_auto_scaling_strategies() returns false, diff_cluster_options omits the autoscaling option entirely. It does not even inspect whether the project declares a policy. On an older region, adding AUTO SCALING STRATEGY to an already-existing cluster while leaving size and replication factor unchanged therefore yields an empty diff. mz-deploy apply reports the cluster as up to date and exits successfully, but no policy is installed.

This is not necessary for compatibility. If the older server truly does not support the DDL, the command should return an explicit unsupported-version error. If the server supports the DDL but predates the introspection view, setting a declared policy is still safe because SET does not require reading the current value. The current behavior is also inconsistent with the missing-cluster path, which sends the original CREATE CLUSTER statement and surfaces the server's unsupported-feature error.

When introspection is unavailable, parse the desired option anyway. Skip reconciliation only when the project also has no policy. For a declared policy, either emit the SET statement or fail clearly rather than reporting success.

Finding 3: MEDIUM: The capability probe can be spoofed by an unrelated schema-scoped object

Location: src/mz-deploy/src/client/connection.rs:205-217

The probe treats any row in mz_catalog.mz_objects named mz_cluster_auto_scaling_strategies as proof that mz_internal.mz_cluster_auto_scaling_strategies exists. Catalog item names are schema-scoped, and mz_objects exposes both schema_id and type, but neither is checked. On a region that predates the builtin view, an ordinary table, view, type, or other item with that name in any user schema makes the probe cache true.

Every affected cluster query then injects a qualified join to the nonexistent mz_internal.mz_cluster_auto_scaling_strategies relation (src/mz-deploy/src/client/introspection.rs:102-106). As a result, apply clusters and stage fail before doing useful work. The new get_cluster dependency also makes unrelated commands such as setup and debug fail. A role with permission to create a catalog item can deliberately hold these deployment operations in this failure state on an older region.

Join mz_catalog.mz_schemas and require the exact mz_internal schema, expected object type, and object name. An exact qualified-regclass lookup would also avoid the name collision if it is supported across the intended server versions.

Here's some extra tests for #1 and #2:

diff --git a/src/mz-deploy/src/cli/commands/clusters.rs b/src/mz-deploy/src/cli/commands/clusters.rs
index 809a7f391e..af40646c75 100644
--- a/src/mz-deploy/src/cli/commands/clusters.rs
+++ b/src/mz-deploy/src/cli/commands/clusters.rs
@@ -375,4 +375,34 @@ mod tests {
         let (to_set, to_reset) = diff_cluster_options(&def, &existing, false).unwrap();
         assert!(to_set.is_empty() && to_reset.is_empty());
     }
+
+    #[mz_ore::test]
+    fn test_diff_unsupported_region_still_applies_declared_policy() {
+        // A declared policy must not be silently dropped just because the region
+        // lacks the introspection view. `SET (AUTO SCALING STRATEGY = ...)` does
+        // not read live state, so applying a declared policy is always safe. A
+        // fix either emits the SET or fails clearly. Either way the diff must not
+        // report an existing cluster with no live policy as up-to-date.
+        let def = definition(
+            "CREATE CLUSTER scaled (SIZE = '25cc', REPLICATION FACTOR = 2, \
+             AUTO SCALING STRATEGY = (ON HYDRATION (HYDRATION SIZE = '100cc')))",
+        );
+        // Live cluster matches SIZE and REPLICATION FACTOR but has no policy.
+        let existing = cluster("25cc", 2);
+        match diff_cluster_options(&def, &existing, false) {
+            Ok((to_set, to_reset)) => {
+                let sets: Vec<String> = to_set
+                    .iter()
+                    .map(AstDisplay::to_ast_string_simple)
+                    .collect();
+                assert!(
+                    sets.iter().any(|s| s.contains("AUTO SCALING STRATEGY")),
+                    "declared policy silently dropped on a region without the \
+                     introspection view: to_set={sets:?}, to_reset={to_reset:?}"
+                );
+            }
+            // Failing clearly is the acceptable alternative to emitting the SET.
+            Err(_) => {}
+        }
+    }
 }
diff --git a/test/mz-deploy/mzcompose.py b/test/mz-deploy/mzcompose.py
index e5dfc78808..513038febc 100644
--- a/test/mz-deploy/mzcompose.py
+++ b/test/mz-deploy/mzcompose.py
@@ -2392,3 +2392,29 @@ def workflow_autoscaling(c: Composition, parser: WorkflowArgumentParser) -> None
         result = run_mz_deploy(c, "autoscaling/v3", "apply")
         assert result.returncode == 0, f"apply v3 failed: {result.stderr}"
         assert live_strategy("scaled") is None, "expected the policy to be reset"
+
+    with c.test_case("autoscaling-promote-size-drops-policy"):
+        # Promote the hydration size to the steady-state SIZE and drop the now
+        # unnecessary policy in a single apply. The live cluster is
+        # 'scale=1,workers=1' with a hydration burst to 'scale=1,workers=2';
+        # the new definition makes 'scale=1,workers=2' the steady-state SIZE and
+        # removes the policy.
+        result = run_mz_deploy(c, "autoscaling/promote-before", "apply")
+        assert result.returncode == 0, f"apply promote-before failed: {result.stderr}"
+        assert live_strategy("promo") == ("scale=1,workers=2", 60)
+
+        # The reconciler emits every SET before every RESET, so it runs
+        # `ALTER CLUSTER promo SET (SIZE = 'scale=1,workers=2')` while the policy
+        # still lingers. `plan_alter_cluster` validates the untouched policy
+        # against the new size and rejects it because HYDRATION SIZE now equals
+        # the cluster SIZE, so the SET never lands and the apply cannot make
+        # progress. Ordering the RESET before the SET (or planning by option
+        # dependencies) is what lets this transition converge.
+        result = run_mz_deploy(c, "autoscaling/promote-after", "apply", check=False)
+        assert result.returncode == 0, (
+            "promoting the hydration size to SIZE while dropping the policy must "
+            f"converge, but apply failed: {result.stderr}"
+        )
+        size = c.sql_query("SELECT size FROM mz_clusters WHERE name = 'promo'")[0][0]
+        assert size == "scale=1,workers=2", f"expected promoted size, got {size}"
+        assert live_strategy("promo") is None, "expected the policy to be reset"

#3 I can't repro with the tests we have.

[bobbyiliev]

Previously a file that omitted REPLICATION FACTOR preserved the live value, now it resets to the default. If the declarative behavior is intended (seems reasonable!), maybe worth a mention in the PR description and apply-clusters.md.

[sjwiesman]

That was a bug! Fixed this as I went.

[bobbyiliev]

Small note, maybe worth a comment: mz_cluster_auto_scaling_strategies is a builtin materialized view, so it reflects DDL with a small delay (unlike mz_clusters). A dry-run right after an ALTER could briefly report drift, and the autoscaling-idempotent test could in theory flake on this. I hit the same thing in #37800.

[sjwiesman]

This should not have a delay as long as you are running under strict serializability

[bobbyiliev]

If both sides are None this passes vacuously, comparing against ("scale=1,workers=2", 60) directly would guard that.

[bobbyiliev]

Was removing this intentional? The command still looks idempotent.

[sjwiesman]

it was intentional

[bobbyiliev]

nit: "strategy policy" doubles up, maybe just "auto scaling strategy"?

[sjwiesman]

Appreciate the fast reviews, hopefully have addressed all the issues.

[bobbyiliev]

Approving, thanks for the quick turnaround! Two small things, neither blocking:

• If a cluster file doesn't set REPLICATION FACTOR, apply now resets it to the default instead of leaving the live value alone. Makes sense, but since folks can hit this without touching their project, probably worth a line in the PR description and the help doc so it shows up in release notes.
• Also might be nice to have a small test for the RESET-before-SET ordering (bump SIZE to the old hydration size and drop the strategy in one edit), just so a refactor doesn't quietly bring the issue back.