storage/sql-server: catch incompatible constraint changes in the CDC poll

[peterdukelarsen]

Motivation

Materialize currently doesn't catch a dropped UNIQUE constraint and can instead just report incorrect data for some queries.

Description

Materialize will now validate the schema while reading CDC events from SQL Server so that if a UNIQUE constraint is dropped Materialize can error instead of serving incorrect data.

Verification

Testdrive tests demonstrating new behavior.

🤖 Generated with Claude Code

[peterdukelarsen]

note for reviewer: this hint won't make total sense until the next PR lands. We'll wait to merge until both are approved.

[def-]

QA LLM Review

1. HIGH -- Two capture instances on one table make the second export report a bogus dropped key

src/sql-server-util/src/cdc.rs:366

constraints.remove(...) drains the per-table entry on the first SqlServerTableRaw, so when the same table appears twice (once per capture instance) the second CdcEvent::Schema is emitted with an empty constraint list. check_constraint_compatibility then reports every recorded PRIMARY KEY / UNIQUE as KeyDropped, writing a permanent, un-retracted definite error into that export's shard even though nothing changed upstream.

Details

get_tables_for_capture_instance keys its grouping on (schema, name, capture_instance, create_date) (src/sql-server-util/src/inspect.rs:933), so a table with two capture instances yields two entries with the same (schema_name, name). Both instances are in self.capture_instances whenever a single source has two exports of the same table bound to different instances. That is exactly the documented source-versioning flow in doc/user/content/ingest-data/sql-server/source-versioning.md:117-170: enable a new capture instance, then CREATE TABLE v2.t1 FROM SOURCE my_source (REFERENCE dbo.t1) while v1.t1 keeps running on the old instance. The instance that sorts later by name loses. In the doc's example (dbo_t1 vs dbo_t1_v2) that is the newly created v2.t1, which breaks on the first upstream change after it finishes snapshotting. There is no config gate on the check, so it cannot be turned off.

Fix, look up instead of draining:

-                    for table in tables {
-                        let constraints = constraints
-                            .remove(&(Arc::clone(&table.schema_name), Arc::clone(&table.name)))
-                            .unwrap_or_default();
+                    for table in tables {
+                        let constraints = constraints
+                            .get(&(Arc::clone(&table.schema_name), Arc::clone(&table.name)))
+                            .cloned()
+                            .unwrap_or_default();

(let constraints can then be non-mut; SqlServerTableConstraintRaw is Clone.) A regression test would be two CREATE TABLE ... FROM SOURCE on the same reference straddling a sp_cdc_enable_table with a second @capture_instance.

2. HIGH -- Migrated composite primary keys compare unequal and report a phantom KeyAltered

src/sql-server-util/src/desc.rs:155

The check compares column_names as an ordered Vec, but for source exports created before the constraints field existed, the catalog migration rebuilds those names in table column order, not key ordinal order. Any pre-existing export whose composite PK is declared out of column order will fail the check on the first poll after upgrade and be permanently poisoned with a false "was renamed or recreated upstream".

Details

ast_rewrite_sql_server_constraints (src/adapter/src/catalog/migrate.rs:969-985) reconstructs the key by iterating table.columns and collecting each column carrying primary_key_constraint. The old representation stored only the constraint name per column (src/sql-server-util/src/desc.proto:39), so ordinal information was never recorded and cannot be recovered. Live state comes from get_constraints_for_tables, which is explicitly ORDER BY ... kcu.ordinal_position (src/sql-server-util/src/inspect.rs:520). For CREATE TABLE t (a INT NOT NULL, b INT NOT NULL, CONSTRAINT pk PRIMARY KEY (b, a)) the stored desc holds ["a","b"] and the poll reads ["b","a"]. Mismatch, KeyAltered, definite error, on an unchanged table.

Column order is not semantically meaningful for the key property this check protects (a PK/UNIQUE over {a,b} is a key regardless of index order), so the simplest fix is to compare column sets:

Some(other_constraint)
    if other_constraint.constraint_type != constraint.constraint_type
        || other_constraint.column_names.iter().collect::<BTreeSet<_>>()
            != constraint.column_names.iter().collect::<BTreeSet<_>>() =>

That also keeps KeyRef's rendered order (which comes from the stored desc) intact for the message.

3. MEDIUM -- The hint tells SQL Server users to use an option SQL Server sources reject

src/storage/src/source/sql_server.rs:110

SchemaChange::hint ends with "create the table with WITH (EXCLUDE CONSTRAINTS ('name')) before the upstream drop", but purify_create_table_from_source hard-rejects that option for anything other than Postgres and MySQL (src/sql/src/pure.rs:1923-1932, "EXCLUDE CONSTRAINTS is not supported for SQL Server sources"). A user following the hint gets a plan error, and the new tests bake the wrong advice in as expected output (test/sql-server-cdc/upstream-schema-changes.td:233, :255).

Details

Either add GenericSourceConnection::SqlServer to the allowed set in pure.rs (the SQL Server desc already carries named constraints, so the pruning has something to key on), or have the SQL Server wrapper drop that sentence and surface only the recreate(None) half until the option is supported.

4. MEDIUM -- Every poll re-reads all columns of every captured table and rebuilds the desc

src/sql-server-util/src/cdc.rs:354 and src/storage/src/source/sql_server/replication.rs:545

Details

The new block runs get_tables_for_capture_instance (one row per column of every captured table) plus get_constraints_for_tables on each poll where the DB LSN advanced, i.e. roughly once per timestamp_interval for any active database, and then rebuilds a full SqlServerTableDesc per table. Constraint changes are rare; the cost is paid continuously.

The most visible symptom is log volume: SqlServerTableDesc::new calls SqlServerColumnDesc::new, which emits tracing::warn!("found an unsupported data type when parsing raw data") for every column it cannot parse (src/sql-server-util/src/desc.rs:355-368). A table with an excluded text/image/xml column, the exact configuration users are told to adopt since purification errors otherwise, now produces one warning per such column per second, indefinitely. Previously this path ran only during purification and snapshot.

Worth either polling the schema on its own slower cadence, or comparing the raw constraint rows directly and only constructing the desc when they differ.