MDEV-40287 AES_ENCRYPT() and KDF() return NULL with OpenSSL 4.0 - #5652
MDEV-40287 AES_ENCRYPT() and KDF() return NULL with OpenSSL 4.0#5652vaintroub wants to merge 2 commits into
Conversation
MyCTX used an EVP_CIPHER_CTX in a stack buffer instead of allocating it with EVP_CIPHER_CTX_new(). OpenSSL 4.0 rejects a context that was not created that way for ciphers that use an IV: EVP_CipherInit_ex() fails with "invalid iv length", so AES_ENCRYPT() and KDF() in CBC/CTR/GCM modes return NULL. ECB has no IV and still works, which is why only the non-ECB modes broke. Fix: Allocate the context with EVP_CIPHER_CTX_new()/EVP_CIPHER_CTX_free(). Also remove check_openssl_compatibility() and the EVP_CIPHER_CTX_SIZE and EVP_CIPHER_CTX_init macros. Verified against OpenSSL 4.0.1: the mysys aes-t test fails on the CBC/CTR/GCM cases with the stack buffer and passes with EVP_CIPHER_CTX_new(). No visible performance degradation: the extra allocation costs ~12 ns/call (WolfSSL) and ~30 ns (OpenSSL) on Windows at a 30-byte payload, nothing at 16 KB, and nothing on Linux/glibc; sysbench OLTP over encrypted tables and redo log is unchanged. Assisted-by: Claude:claude-opus-4-8
There was a problem hiding this comment.
🟡 Changes recommended
The new heap allocation of EVP_CIPHER_CTX in MyCTX introduces a verified leak on init() error paths because the placement-new object is not destructed when my_aes_crypt_init() returns early.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR fixes OpenSSL 4.0 compatibility for MariaDB’s AES/KDF crypto paths by switching EVP_CIPHER_CTX handling from a stack-buffer “fake context” to proper OpenSSL-managed allocation, and removing the now-unneeded OpenSSL compatibility probe.
Changes:
- Replace stack-buffer
EVP_CIPHER_CTXusage withEVP_CIPHER_CTX_new()/EVP_CIPHER_CTX_free()inmysys_ssl/my_crypt.cc. - Remove
check_openssl_compatibility()(and its implementation file) and the associated startup check insql/mysqld.cc. - Clean up related compat macros and build wiring (CMake +
ssl_compat.h).
File summaries
| File | Description |
|---|---|
| sql/mysqld.cc | Removes server startup OpenSSL compatibility check. |
| mysys_ssl/openssl.c | Deletes the compatibility-check implementation. |
| mysys_ssl/my_crypt.cc | Switches AES context management to OpenSSL-owned allocation; adjusts init return paths. |
| mysys_ssl/CMakeLists.txt | Stops building the removed openssl.c. |
| include/ssl_compat.h | Removes now-unused compatibility macros and check_openssl_compatibility() declaration. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
🟢 Approval recommended
The changes are narrowly scoped to OpenSSL EVP context lifecycle, remove now-obsolete compatibility scaffolding, and the updated error-path cleanup addresses the previously identified leak without introducing new call-site breakage.
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 0 new
- Review effort level: Lite
MyCTX used an EVP_CIPHER_CTX in a stack buffer instead of allocating it with EVP_CIPHER_CTX_new(). OpenSSL 4.0 rejects a context that was not created that way for ciphers that use an IV: EVP_CipherInit_ex() fails with "invalid iv length", so AES_ENCRYPT() and KDF() in CBC/CTR/GCM modes return NULL. ECB has no IV and still works, which is why only the non-ECB modes broke.
Fix:
Allocate the context with EVP_CIPHER_CTX_new()/EVP_CIPHER_CTX_free(). Also remove check_openssl_compatibility() and the EVP_CIPHER_CTX_SIZE and EVP_CIPHER_CTX_init macros.
Verified against OpenSSL 4.0.1: the mysys aes-t test fails on the CBC/CTR/GCM cases with the stack buffer and passes with EVP_CIPHER_CTX_new().
No visible performance degradation: the extra allocation costs ~12 ns/call (WolfSSL) and ~30 ns (OpenSSL) on Windows at a 30-byte payload, nothing at 16 KB, and nothing on Linux/glibc; sysbench OLTP over encrypted tables and redo log is unchanged.