p11test/p11test_case_pss_oaep.c segfault with using OpenSSL-3.0.0

[dengert]

p11test segfaults when built with OpenSSL 3.0.0 alpha. This appears to be caused when md == NULL in call to SHA1 in p11test_case_pss_oaep.c:206

This appears to be an OpenSSL 3.0.0 bug as man page SHA256_INIT(3ossl) says:
"SHA1() computes the SHA-1 message digest of the n bytes at d and places
it in md (which must have space for SHA_DIGEST_LENGTH == 20 bytes of
output). If md is NULL, the digest is placed in a static array. Note:
setting md to NULL is not thread safe."

The bug most likely does not occur in other hashes. SHA1 is treated special in OpenSSL 3.0.0 ./providers/implementations/digests/sha2_prov.c

But all other calls to SHA1 in OpenSC pass a buffer for md and do not have this problem.

Testing with gdb --args ./p11test -p 123456 shows this backtrace:

Thread 1 "p11test" received signal SIGSEGV, Segmentation fault.
0x00007ffff7da9202 in SHA1_Final (md=0x1 <error: Cannot access memory at address 0x1>, c=0x55555573de70)
    at ../openssl/include/crypto/md32_common.h:222
222	    HASH_MAKE_STRING(c, md);
(gdb) where
#0  0x00007ffff7da9202 in SHA1_Final (md=0x1 <error: Cannot access memory at address 0x1>, c=0x55555573de70)
    at ../openssl/include/crypto/md32_common.h:222
#1  0x00007ffff7e40cb2 in sha1_internal_final (ctx=0x55555573de70, out=0x0, outl=0x7fffffffcfc8, outsz=20)
    at ../openssl/providers/implementations/digests/sha2_prov.c:61
#2  0x00007ffff7cdb7a9 in EVP_DigestFinal_ex (ctx=0x5555555f1e60, md=0x0, isize=0x0)
    at ../openssl/crypto/evp/digest.c:426
#3  0x00007ffff7cdc246 in EVP_Digest (data=0x55555556aaf0, count=124, md=0x0, size=0x0, type=0x5555555b9e90, impl=0x0)
    at ../openssl/crypto/evp/digest.c:626
#4  0x00007ffff7cdc2dc in EVP_Q_digest
    (libctx=0x0, name=0x55555556ab88 "SHA1", propq=0x0, data=0x55555556aaf0, count=124, md=0x0, size=0x0)
    at ../openssl/crypto/evp/digest.c:640
#5  0x0000555555563623 in hash_message
    (message=0x55555556aaf0 "Simple message for signing & verifying. It needs to be little bit longer to fit also longer keys and allow the truncation.\n", message_length=124, hash=544)
    at ../../../../OpenSC/src/tests/p11test/p11test_case_pss_oaep.c:206
#6  0x0000555555564d0b in pss_sign_verify_test (o=0x5555557eb450, info=0x5555555788c0, mech=0x5555557eb4c0)
    at ../../../../OpenSC/src/tests/p11test/p11test_case_pss_oaep.c:667
#7  0x00005555555653b1 in pss_oaep_test (state=0x555555578450)
    at ../../../../OpenSC/src/tests/p11test/p11test_case_pss_oaep.c:784
#8  0x00007ffff7aae4e1 in  () at /usr/lib/x86_64-linux-gnu/libcmocka.so.0
#9  0x00007ffff7aaeec5 in _cmocka_run_group_tests () at /usr/lib/x86_64-linux-gnu/libcmocka.so.0
#10 0x00005555555583e4 in main (argc=3, argv=0x7fffffffddd8) at ../../../../OpenSC/src/tests/p11test/p11test.c:145

Note that 'md' is set to "1" and not set to point at a static array as stated in the man page.

Proposed Resolution

Change p11test to always provide a SHA1 buffer for md as this is also thread safe, and does not have to wait for OpenSSL to fix the problem. (I intend to report this as an OpenSSL-3.0.0 bug.)

As a side note, that may be useful in OpenSC, is OpenSSL-3.0.0 man pages also has "EVP_Q_digest() is a quick one-shot digest function." include/openssl/sha.h uses this in a define for SHA1().

Steps to reproduce

./p11test -p 123456`

[...]
[ RUN      ] pss_oaep_test
[  ERROR   ] --- Test failed with exception: Segmentation fault(11)
[  FAILED  ] pss_oaep_test
[...]
[==========] 9 test(s) run.
[  PASSED  ] 7 test(s).
[  SKIPPED ] 1 test(s), listed below:
[  SKIPPED ] wait_test

 1 SKIPPED TEST(S)
[  FAILED  ] 1 test(s), listed below:
[  FAILED  ] pss_oaep_test

 1 FAILED TEST(S)

Logs

No logs, p11test appears to catch the segfault and continue.

[Jakuje]

@dengert I think this was resolved with the opnessl update. Can you check if it is still the issue or it already works?

[dengert]

It should be fixed based on comments to the OpenSSL issue starting here: openssl/openssl#15718 (comment)

And in the master branch as:
openssl/openssl@987d7da

This is not a good week for doing much testing.

[Jakuje]

Tested locally with beta2:

# rpm -q openssl
openssl-3.0.0-0.beta2.2.el9.x86_64
# dnf builddep -y  opensc
# dnf install gnutls-utils softhsm
# ./bootstrap
# CFLAGS="-Wno-deprecated-declarations" ./configure 
# make
# cd src/tests/p11test/
# ./runtest.sh softhsm
[==========] 9 test(s) run.
[  PASSED  ] 7 test(s).
[  SKIPPED ] 2 test(s), listed below:
[  SKIPPED ] wait_test
[  SKIPPED ] interface_test

Closing as this is no longer issue.