Skip to content

Allow configuring the collation of asset name columns - #71249

Open
1fanwang wants to merge 2 commits into
apache:mainfrom
1fanwang:tidb-asset-collation-config
Open

Allow configuring the collation of asset name columns#71249
1fanwang wants to merge 2 commits into
apache:mainfrom
1fanwang:tidb-asset-collation-config

Conversation

@1fanwang

@1fanwang 1fanwang commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

closes: #31373

The name, uri and group columns on the asset tables hard-code the
latin1_general_cs collation on MySQL. The reasoning is sound: the values are
ASCII, and a single-byte charset keeps the 1500-character unique indexes inside
the 3072-byte index limit that utf8mb4 would blow past. But it is baked into
the ORM column definitions, so it cannot be overridden. A MySQL-compatible engine
that does not ship that collation cannot create Airflow's schema at all:

(1273, "Unsupported collation when new collation is enabled: 'latin1_general_cs'")

sql_engine_collation_for_ids already exists for the same reason on the
StringID columns. This adds the equivalent knob for the asset columns,
defaulting to today's value so nothing changes unless it is set.

The collation was repeated inline at ten sites; they now share one constant,
which is where the net line reduction comes from.

Design suggested by @hussein-awala in #31373 ("Adding a new Airflow configuration
to configure the table collation and set its default value to
latin1_general_cs can do the job").

Testing Done

Against TiDB v8.5.1, which offers 13 collations and no latin1_general_cs. Its
latin1_bin is the equivalent substitute: single-byte, case-sensitive, and it indexes
VARCHAR(1500). I verified that by inserting 'Abc' and 'abc' under a unique key
and confirming both are accepted.

With the option unset, the emitted DDL is byte-identical to before.

Raw logs

Before, airflow db migrate cannot create the first asset table:

sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError)
(1273, "Unsupported collation when new collation is enabled: 'latin1_general_cs'")
[SQL:
CREATE TABLE asset_alias (
        id INTEGER NOT NULL AUTO_INCREMENT,
        name VARCHAR(1500) COLLATE latin1_general_cs NOT NULL,
        `group` VARCHAR(1500) COLLATE latin1_general_cs NOT NULL,
        CONSTRAINT asset_alias_pkey PRIMARY KEY (id)
)
]

After, with AIRFLOW__DATABASE__SQL_ENGINE_COLLATION_FOR_ASSET_NAMES=latin1_bin:

[info] Creating Airflow database tables from the ORM
[info] Running stamp_revision  -> 7a98f1b7dbd3
[info] Database migration done!

tables created: 71
foreign keys : 86
latin1 columns: asset.name/uri/group, asset_active.name/uri,
                asset_alias.name/group, asset_watcher.name  -> all latin1_bin

Default unchanged: with the option unset, MySQL 8.4 still gets
COLLATE latin1_general_cs in the generated DDL.

A real Dag through the resulting schema (dynamic task mapping, XCom, fan-in):

sum of squares = 55
DagRun Finished: dag_id=tidb_smoke, run_duration=4.42, state=success

Regressions: tests/unit/models/test_base.py, test_asset.py,
tests/unit/core/test_configuration.py, tests/unit/utils/test_sqlalchemy.py.
The regression tests listed above cover this.

The asset name, uri and group columns hard-code the latin1_general_cs
collation on MySQL. Several MySQL-compatible engines do not provide that
collation, so Airflow cannot create its own schema on them even though the
rest of the database works. There is no way to override it from outside,
because the collation is baked into the ORM column definitions.

closes: apache#31373
Signed-off-by: 1fanwang <1fannnw@gmail.com>
@1fanwang
1fanwang force-pushed the tidb-asset-collation-config branch from 2b1699c to ea56fea Compare August 6, 2026 19:05
@1fanwang
1fanwang marked this pull request as ready for review August 6, 2026 20:07
@1fanwang
1fanwang requested review from XD-DENG and ashb as code owners August 6, 2026 20:07
The asset name/uri/group columns carry an explicit MySQL collation so their
1500-character unique indexes stay inside the 3072-byte index limit. That
collation is configurable on the models, but the migrations that create and
alter those columns still hard-coded `latin1_general_cs`, so the setting only
took effect when the schema was created from the ORM.

A fresh install on a MySQL-compatible engine that lacks that collation replays
the migrations instead and fails on the first asset table, which leaves the
setting useless in exactly the case it was added for.

Resolve the collation at migration run time through the same configuration key
the models read.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@1fanwang
1fanwang requested a review from ephraimbuddy as a code owner August 7, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

allow configuring dataset model uri collation

1 participant