Fix unchecked i2d_PrivateKey() return value causing size_t overflow and abort

[metsw24-max]

The return value of i2d_PrivateKey() in ssl_asn1_table_set() is not validated before being assigned to an apr_size_t.

i2d_PrivateKey() returns an int and may return <= 0 on failure. This value is directly cast to apr_size_t, which can result in a large unsigned value when negative. This leads to excessive memory allocation attempts via ap_malloc()/ap_realloc(), causing the server to abort.

This patch introduces proper validation of the return value before casting, preventing invalid allocations.

Additionally, the caller in ssl_engine_pphrase.c is updated to handle NULL return values from ssl_asn1_table_set() to avoid NULL dereference.

Reproduction:

• Force i2d_PrivateKey() to return -1
• Apache attempts to allocate a huge buffer and aborts

After fix:

• Failure is handled gracefully
• No crash occurs

[metsw24-max]

@notroj any updates?

[notroj]

How is it it possible to trigger the i2d_PrivateKey failure case here? I would assume this is some only possible in some pathological case, not a practical thing anybody will really it. So add an ap_assert() on the first invocation at most here.

[metsw24-max]

Done
replaced the first-call error handling with ap_assert(derlen > 0) and removed the check on the second invocation as suggested.

[notroj]

this appears to be a noop apart from removing the APLOGNO()

[metsw24-max]

Reverted, that hunk slipped in by accident. Also dropped the leftover NULL check on ssl_asn1_table_set() further down, since with the assert it can't return NULL any more, so ssl_engine_pphrase.c is now unchanged from trunk.

[notroj]

C90 violation. Build with --enable-maintainer-mode

[metsw24-max]

Fixed, moved the declarations back above the statements. Compiles cleanly with -Wdeclaration-after-statement now.