add macro guard on SHA512 hashType

[JacobBarthelmeh]

This was found when developing and integrating wolfHSM into a project. The autoconf ./configure used to compile wolfSSL that wolfHSM was being linked to did not define WOLFSSL_SHA512_HASHTYPE. The changes in this PR allow for linking against wolfSSL compiled without WOLFSSL_SHA512_HASHTYPE and add a CI test case.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #357

Scan targets checked: wolfhsm-crypto-bugs, wolfhsm-src

No new issues found in the changed files. ✅

[Copilot]

Pull request overview

This PR improves compatibility with wolfSSL builds where SHA-512 is enabled but wc_Sha512 does not include the hashType member (WOLFSSL_SHA512_HASHTYPE is not defined), by adding compile-time guards and extending CI to cover that configuration.

Changes:

• Guard SHA-512 async/DMA test helpers and their invocation behind WOLFSSL_SHA512_HASHTYPE.
• Guard the server-side DMA SHA-512 resume-state assignment to sha512->hashType behind WOLFSSL_SHA512_HASHTYPE.
• Add a NO_SHA512_HASHTYPE=1 test build mode and a CI job to build/run with that configuration.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/wh_test_crypto.c	Avoids compiling/calling SHA-512 async/DMA tests when WOLFSSL_SHA512_HASHTYPE is not available.
test/Makefile	Adds NO_SHA512_HASHTYPE=1 build option to disable WOLFSSL_SHA512_HASHTYPE via test user settings.
test/config/user_settings.h	Makes WOLFSSL_SHA512_HASHTYPE conditional on WHTEST_NO_SHA512_HASHTYPE.
src/wh_server_crypto.c	Avoids referencing wc_Sha512.hashType when the member is not present.
.github/workflows/build-and-test.yml	Adds CI coverage for DMA builds with SHA-512 enabled but WOLFSSL_SHA512_HASHTYPE disabled.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[padelsbach]

@JacobBarthelmeh, please rebase to pass CI

[bigbrett]

@JacobBarthelmeh @padelsbach unless I'm misunderstanding the intent here, this one is a little silly/overkill to expose as a test makefile toggle. I really dont want to add every single switch (especially a trivial one like this) as a configurable option, lest our CI test suite starts blowing up. No end user is going to care about toggling this. If wolfHSM requires hashtype on, then it should require it on across the board. It is perfectly fine for certain wolfCrypt flags to be mandatory. If it is optional or only tied to a specific wolfHSM feature (like SHA512 DMA in this case) then the source code should be able to handle it on and off based on the proper feature guards or a check in wh_settings.h telling the user to turn it on (so we dont need redundant gates everywhere in the code). Things like this are fine to test once. Adding regression/edge cases for every single permutation of a build will result in our PR CI times blowing up and wont really add any value.

Otherwise EVERY single wolfCrypt config option that is relevant to a subset of features will need something like this and our code and tests will turn into spaghetti.

The POSIX tests really should be testing with everything on. The only things toggleable should me MAJOR wolfHSM features that are truly optional in the sense that a user may want to disable an entire subsystem (SHE, DMA, etc). No reason we really would ever want to make this high-level toggleable for an end user. A compile error saying "hey if you have SHA512 DMA you need the hashtype turned on in wolfCrypt" is plenty.

[JacobBarthelmeh]

@JacobBarthelmeh @padelsbach unless I'm misunderstanding the intent here, this one is a little silly/overkill to expose as a test makefile toggle. I really dont want to add every single switch (especially a trivial one like this) as a configurable option, lest our CI test suite starts blowing up. No end user is going to care about toggling this. If wolfHSM requires hashtype on, then it should require it on across the board. It is perfectly fine for certain wolfCrypt flags to be mandatory. If it is optional or only tied to a specific wolfHSM feature (like SHA512 DMA in this case) then the source code should be able to handle it on and off based on the proper feature guards or a check in wh_settings.h telling the user to turn it on (so we dont need redundant gates everywhere in the code). Things like this are fine to test once. Adding regression/edge cases for every single permutation of a build will result in our PR CI times blowing up and wont really add any value.

Otherwise EVERY single wolfCrypt config option that is relevant to a subset of features will need something like this and our code and tests will turn into spaghetti.

The POSIX tests really should be testing with everything on. The only things toggleable should me MAJOR wolfHSM features that are truly optional in the sense that a user may want to disable an entire subsystem (SHE, DMA, etc). No reason we really would ever want to make this high-level toggleable for an end user. A compile error saying "hey if you have SHA512 DMA you need the hashtype turned on in wolfCrypt" is plenty.

I can add a PR to revert the CI test for it. I still think we should allow the option to build without SHA512 hash type defined though.