Basic128Rsa15 and Basic256 with OpenSSL are broken on RHEL 9

[basyskom-jvoe]

Description

Due to the decision to disable SHA-1 signatures in OpenSSL on RHEL 9 (see here), the two security policies Basic128Rsa15 and Basic256 no longer work on that platform.

Building and linking works without any problem but a a runtime error occurs:
Sending OPN message failed with error BadInternalError.

This error message is generated by EVP_DigestSignInit() in UA_Openssl_RSA_Private_Sign() failing with error:03000098:digital envelope routines::invalid digest when EVP_sha1() is passed.

If there should be no way to bring back the support by using different API from OpenSSL, it would be nice if a way could be found to detect this situation and exclude the two broken security policies from the build.

For the open62541 plugin in Qt OPC UA, I am currently implementing an approach where I attempt to call EVP_DigestSignInit() as open62541 does it and exclude the two policies from the list of supported policies if the call fails.
But for open62541, it would be better to have a solution where this is detected during configuration so the corresponding API can be exluded from the build via #ifdef.

Steps to reproduce

• Build open62541 1.3.x with UA_ENABLE_ENCRYPTION=OPENSSL and UA_BUILD_UNIT_TESTS=ON
• Run the unit tests

The following output will be shown:

      Start 37: encryption_basic128rsa15
37/41 Test #37: encryption_basic128rsa15 ..........***Failed    0.07 sec
      Start 38: encryption_basic256
38/41 Test #38: encryption_basic256 ...............***Failed    0.07 sec

Without access to RHEL 9, the problem can also be reproduced with a current Fedora by setting the environment variable OPENSSL_FORCE_FIPS_MODE while running the unit tests.

[jpfr]

Check if fixed via ##5937.

[jpfr]

@basyskom-jvoe can this be closed after #5937?

[basyskom-jvoe]

@jpfr I don't have access to RHEL 9 at the moment. I'll try to check this soon.

[basyskom-jvoe]

@jpfr The problem is still present with v1.4.1.
Do you see a way how we could get Basic128Rsa15 and Basic256 back for this platform without switching back to mbedTLS (I have not tried it, but I'd assume that would work)?