Allow configuring the collation of asset name columns - #71249
Open
1fanwang wants to merge 2 commits into
Open
Conversation
1fanwang
force-pushed
the
tidb-asset-collation-config
branch
from
August 6, 2026 17:38
2e8d220 to
9f818c2
Compare
1fanwang
force-pushed
the
tidb-asset-collation-config
branch
from
August 6, 2026 18:12
9f818c2 to
2b1699c
Compare
The asset name, uri and group columns hard-code the latin1_general_cs collation on MySQL. Several MySQL-compatible engines do not provide that collation, so Airflow cannot create its own schema on them even though the rest of the database works. There is no way to override it from outside, because the collation is baked into the ORM column definitions. closes: apache#31373 Signed-off-by: 1fanwang <1fannnw@gmail.com>
1fanwang
force-pushed
the
tidb-asset-collation-config
branch
from
August 6, 2026 19:05
2b1699c to
ea56fea
Compare
1 task
1fanwang
marked this pull request as ready for review
August 6, 2026 20:07
2 tasks
The asset name/uri/group columns carry an explicit MySQL collation so their 1500-character unique indexes stay inside the 3072-byte index limit. That collation is configurable on the models, but the migrations that create and alter those columns still hard-coded `latin1_general_cs`, so the setting only took effect when the schema was created from the ORM. A fresh install on a MySQL-compatible engine that lacks that collation replays the migrations instead and fails on the first asset table, which leaves the setting useless in exactly the case it was added for. Resolve the collation at migration run time through the same configuration key the models read. Signed-off-by: 1fanwang <1fannnw@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes: #31373
The
name,uriandgroupcolumns on the asset tables hard-code thelatin1_general_cscollation on MySQL. The reasoning is sound: the values areASCII, and a single-byte charset keeps the 1500-character unique indexes inside
the 3072-byte index limit that
utf8mb4would blow past. But it is baked intothe ORM column definitions, so it cannot be overridden. A MySQL-compatible engine
that does not ship that collation cannot create Airflow's schema at all:
sql_engine_collation_for_idsalready exists for the same reason on theStringIDcolumns. This adds the equivalent knob for the asset columns,defaulting to today's value so nothing changes unless it is set.
The collation was repeated inline at ten sites; they now share one constant,
which is where the net line reduction comes from.
Design suggested by @hussein-awala in #31373 ("Adding a new Airflow configuration
to configure the table collation and set its default value to
latin1_general_cscan do the job").Testing Done
Against TiDB v8.5.1, which offers 13 collations and no
latin1_general_cs. Itslatin1_binis the equivalent substitute: single-byte, case-sensitive, and it indexesVARCHAR(1500). I verified that by inserting'Abc'and'abc'under a unique keyand confirming both are accepted.
With the option unset, the emitted DDL is byte-identical to before.
Raw logs
Before,
airflow db migratecannot create the first asset table:After, with
AIRFLOW__DATABASE__SQL_ENGINE_COLLATION_FOR_ASSET_NAMES=latin1_bin:Default unchanged: with the option unset, MySQL 8.4 still gets
COLLATE latin1_general_csin the generated DDL.A real Dag through the resulting schema (dynamic task mapping, XCom, fan-in):
Regressions:
tests/unit/models/test_base.py,test_asset.py,tests/unit/core/test_configuration.py,tests/unit/utils/test_sqlalchemy.py.The regression tests listed above cover this.