Webcrypto: fix RSA-OAEP digest being ignored for non-SHA-1 hashes

[tompsota]

Problem

njs_cipher_pkey() sets up RSA-OAEP like this:

EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_OAEP_PADDING);
EVP_PKEY_CTX_set_signature_md(ctx, md);
EVP_PKEY_CTX_set_rsa_mgf1_md(ctx, md);

EVP_PKEY_CTX_set_signature_md() is not a valid control for an encrypt or
decrypt operation, so OpenSSL rejects it with command not supported. The
return value is not checked, so the OAEP digest silently stays at its SHA-1
default while MGF1 is set to the requested hash.

The result is that RSA-OAEP with any hash other than SHA-1 uses the digest
pair OAEP/SHA-1 + MGF1/<requested>, which is not what the caller asked for and
not what any other implementation produces:

• ciphertext from any other implementation fails to decrypt, with
  error:02000079:rsa routines::oaep decoding error
• ciphertext produced by njs cannot be decrypted anywhere else

EVP_PKEY_CTX_set_rsa_oaep_md() is currently not called anywhere in the tree.
qjs_cipher_pkey() in external/qjs_webcrypto_module.c has the same three
lines and the same problem.

Reproduction

Encrypt with OpenSSL using OAEP/SHA-256 and decrypt in njs:

const key = await crypto.subtle.importKey(
    'pkcs8', pkcs8, { name: 'RSA-OAEP', hash: 'SHA-256' }, false, ['decrypt']);
await crypto.subtle.decrypt({ name: 'RSA-OAEP' }, key, ciphertext);

Error: EVP_PKEY_decrypt() failed (SSL: error:03000093:digital envelope
routines::command not supported error:02000079:rsa routines::oaep decoding
error)

Decrypting the same ciphertext with OAEP digest SHA-1 and MGF1 digest SHA-256
succeeds, which confirms the digest njs actually negotiated.

Reproduced on njs 0.8.4, 0.8.9, 0.8.10, 0.9.6 and current master, against
OpenSSL 3.1.4 through 3.5.6, in both the njs and QuickJS engines.

Why the tests did not catch it

test/webcrypto/rsa.t.mjs is a round trip: it encrypts and decrypts with
crypto.subtle. Both directions make the same substitution, so the mismatch
cancels out and the SHA-256 and SHA-384 cases pass while being incompatible
with every other implementation.

test/webcrypto/rsa_decoding.t.mjs is the only test that decrypts an
externally generated ciphertext, and it covers SHA-1 only, which happens to be
the digest njs silently falls back to.

Changes

• use EVP_PKEY_CTX_set_rsa_oaep_md() in njs_cipher_pkey() and
  qjs_cipher_pkey()
• add externally generated OAEP ciphertexts for SHA-256 and SHA-384 to
  rsa_decoding.t.mjs, encrypted with the existing rsa.pkcs8 test key

The new fixtures were produced outside njs, so they fail on an unpatched build
and cannot be masked by a round trip.

Test results

Before the fix:

   {"src":"text.base64.rsa-oaep-sha256.enc","hash":"SHA-256"}
    with reason: Error: EVP_PKEY_decrypt() failed (SSL: error:03000093 ...
   {"src":"text.base64.rsa-oaep-sha384.enc","hash":"SHA-384"}
    with reason: Error: EVP_PKEY_decrypt() failed (SSL: error:03000093 ...
RSA-OAEP decoding FAILED: [1/3]

After the fix:

$ ./test/test262 --binary=build/njs test/webcrypto
TOTAL: PASSED [16/16]

Compatibility note

This changes the ciphertext njs produces and accepts for RSA-OAEP with non-SHA-1
hashes. Anything that stored ciphertext generated by an affected njs build using
those hashes was already unreadable by other implementations; it will now need
the OAEP digest that was actually used (SHA-1) to be read back.

Made with Cursor

[github-actions]

✅ All required contributors have signed the F5 CLA for this PR. Thank you!
_{Posted by the CLA Assistant Lite bot.}

[tompsota]

I have hereby read the F5 CLA and agree to its terms