mysql: drop keys over excluded columns from the table description

[peterdukelarsen]

Motivation

EXCLUDE COLUMNS removed the column from a MySQL table description but left any unique index over it in keys. Planning then emitted a UNIQUE constraint on a column the table does not have and failed with "unknown column in constraint", so a table with a unique index on an excluded column could not be created. Where planning did admit it (an earlier nullable UNIQUE constraint short-circuits constraint validation), the retained key stalled the table once the excluded column or its index was dropped upstream, since schema verification could no longer find it.

Description

Keys over an excluded column are handled where Postgres and SQL Server handle them, and the table description itself is left unchanged.

• Constraint emission in purification skips any key that spans an excluded column, so the table can be created and the key is not recorded as a Materialize key.
• determine_compatibility ignores keys in the recorded description that span a column the description marks as excluded, so dropping the excluded column or its index upstream is a non-event. This applies to existing tables too, whose persisted descriptions may already hold such keys.

Pruning inside to_desc was considered and rejected: to_desc also produces the upstream side of the runtime comparison, so pruning there would have made every persisted key over an excluded column look dropped on the next upstream DDL.

User-visible effect: a MySQL table with a unique index over a column listed in EXCLUDE COLUMNS can now be created, and dropping that index or column upstream no longer puts the table into an error state.

Verification

test/mysql-cdc/35-exclude-columns.td creates such tables, checks the excluded key is not recorded, and drops the upstream indexes afterwards, including a key that also spans a nullable included column. keys_over_excluded_columns_are_not_verified unit tests the compatibility check directly.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. HIGH -- Pruning inside to_desc also changes runtime schema verification, halting pre-existing sources

src/mysql-util/src/schemas.rs:187

to_desc has two callers: purification and verify_schemas at runtime. Pruning there changes not only the descriptor written into new statements but also the descriptor that running sources are compared against, so a MySQL table whose persisted descriptor already contains a key over an excluded column will now fail determine_compatibility on the next upstream DDL and be permanently marked IncompatibleSchema, requiring the table to be dropped and recreated. The PR description's "persisted descriptors are unchanged" holds for the catalog but not for this comparison.

Details

At src/storage/src/source/mysql/schemas.rs:58 the freshly fetched schema goes through to_desc with the output's stored exclude_columns, and the result is the other side of determine_compatibility. The check at src/mysql-util/src/desc.rs:164 is self.keys.difference(&other.keys), i.e. persisted-minus-fresh. Pruning only the fresh side makes that difference non-empty for any persisted key that mentions an excluded column. verify_schemas runs on every ALTER TABLE / RENAME TABLE seen in the binlog for a tracked table (src/storage/src/source/mysql/replication/events.rs:143), so an unrelated upstream ALTER TABLE is enough to trip it.

Such persisted descriptors are reachable today, which is what makes this a live regression rather than a theoretical one. Purification always stores the unpruned MySqlTableDesc in the statement details (src/sql/src/pure/mysql.rs:173 filters nothing, and details carries the same table), while the emitted constraints are validated by plan_source_export_desc. That validator does break 'c (src/sql/src/plan/statement/ddl.rs:1557) — not continue — as soon as a non-primary unique constraint reaches a nullable column, abandoning all remaining constraints unchecked. So the sql_bail!("unknown column in constraint") the PR relies on is skipped whenever an earlier-processed constraint short-circuits. Two shapes that create the descriptor today:

• UNIQUE KEY uq (a, b) with a nullable and EXCLUDE COLUMNS (b): the loop breaks on a before reaching b.
• UNIQUE KEY k_a (a) with a nullable plus UNIQUE KEY k_secret (secret) and EXCLUDE COLUMNS (secret): constraints are ordered by index name after the primary key, so k_a breaks first and k_secret is never checked.

Both create successfully at HEAD and persist a key naming the excluded column. test/mysql-cdc/constraints.td:33 already shows the nullable-unique constraint being silently dropped rather than rejected.

Fix: prune at constraint-emission time instead, which is exactly what the other two sources do — src/sql/src/pure/postgres.rs:284 (all_key_cols_included) and src/sql/src/pure/sql_server.rs:425 (excluded_columns) both skip the constraint and leave the persisted descriptor's key set intact. In generate_source_export_statement_values (src/sql/src/pure/mysql.rs:168), skip any table.keys entry whose columns are not all present in the columns vector just built, and revert the to_desc change. That fixes the creation failure this PR targets with no effect on verify_schemas for either new or existing sources.

If pruning must stay in to_desc, then the comparison has to tolerate the old shape, e.g. by having determine_compatibility ignore persisted keys that reference a column the persisted descriptor itself marks excluded.