MDEV-41026 ER_DUP_KEY on '(temporary)' converting a subquery cache

[arcivanov]

https://jira.mariadb.org/browse/MDEV-41026

Important

The MSAN failures on this pull request are MDEV-27562, not a defect in this change. They come from the construction of the test added here: its cached subquery parameter is a VARCHAR column of an Aria table, so every cached lookup reaches the comparison that ticket reports. MDEV-27562 is fixed by #5648 against 10.11; these builders go green here once that merges up into main.

The defect

A subquery expression cache whose parameter is NULL for many rows fills its in-memory table with rows that all carry a NULL key. TABLE::add_tmp_key() builds that key without HA_NULL_ARE_EQUAL, ha_heap::create() propagates the absence, and hp_write_key() then skips the duplicate check for any key value containing NULL, so such rows are legal there. They accumulate rather than being exceptional: ref.null_rejecting is set, so every NULL parameter is a cache miss and adds one more NULL-keyed row.

The on-disk creators disagreed. Both the key and the unique constraint forced HA_NULL_ARE_EQUAL on, making the destination of a conversion stricter about NULLs than the table being converted from. create_internal_tmp_table_from_heap() re-inserts every stored row and treats a duplicate among them as fatal, so the second NULL-keyed row was rejected.

The reported symptom is ER_DUP_KEY. The same defect surfaces as ER_DUP_ENTRY once the cache table is already on disk, and as ER_DUP_UNIQUE when the key is too wide for the on-disk engine to index and is built as a unique constraint over a row hash instead.

The fix

Make KEY::flags & HA_NULL_ARE_EQUAL the single source of truth and read it at all four creator sites - keydef and uniquedef, in each of the Aria and the MyISAM creator - rather than deciding it there.

For that flag to be truthful, the GROUP BY key has to carry it when it is built as a unique constraint as well. The assignment sat inside the branch that only the real-key case takes, which also lays the group buffer out around the NULL flag, so it is now set in the unique-constraint case too. This is behaviour-neutral on Aria, whose creator tests HA_UNIQUE_HASH explicitly and therefore always took the uniquedef branch for such a key, where null_are_equal was hardcoded to 1.

choose_engine() routes every m_using_unique_constraint table to the on-disk engine, so that change cannot reach ha_heap::create(), the only other place a KEY's HA_NULL_ARE_EQUAL is read.

Testing

mysql-test/main/subquery_cache_null_convert covers both on-disk shapes. Each block prints Created_tmp_disk_tables, so the test cannot pass by quietly ceasing to overflow, and neither cached query aborts on error, so a regression in either shape is reported rather than hidden behind the other one. Reverting the fix and running the test unchanged gives, in one run:

shape	error
narrow key, stays a key on disk	ERROR 23000: Can't write; duplicate key in table '(temporary)' (1022)
key too wide for Aria, becomes a unique constraint	ERROR 23000: Can't write, because of unique constraint, to table '(temporary)' (1169)

main, heap and maria suites: 1500 of 1500 pass.

Not covered

The MyISAM halves of both creators compile only with -DUSE_ARIA_FOR_TMP_TABLES=OFF, which is not the default. They are changed to match and verified to compile warning-free in that configuration, but no test run has executed them.

[gkodinov]

Thank you for your contribution. This is a preliminary review.

There's a test failing in msan that looks related to your change. Can you please rectify?

[arcivanov]

@gkodinov

Important

The MSAN failures on this pull request are MDEV-27562, not a defect in this change. They come from the construction of the test added here: its cached subquery parameter is a VARCHAR column of an Aria table, so every cached lookup reaches the comparison that ticket reports. MDEV-27562 is fixed by #5648 against 10.11; these builders go green here once that merges up into main.

[gkodinov]

LGTM. Please stand by for the final review.

[gkodinov]

FYI: According to our development cycle we work on bugs In the following periods 15 Mar-30 Apr, 15 Jun-30 Jul, 15 Sep-30 Oct and 15 Dec-31 Jan. So, please, expect to get a review somewhere between these two dates and the goal is to have your PR merged before the second date