Problems using OpenSSL 1.1.1 "CRYPTO_secure_*" routines

[dengert]

Problem Description

pkcs11-tool --test --login fails the Decryption tests when it tried to encrypt:

Decryption (currently only for RSA)
  testing key 0 (PIV AUTH key) 
    RSA-X-509: Encryption failed, returning
    RSA-PKCS: Encryption failed, returning
  testing key 1 (SIGN key) 
    RSA-X-509: Encryption failed, returning
    RSA-PKCS: Encryption failed, returning
  testing key 2 (KEY MAN key) 
    RSA-X-509: Encryption failed, returning
    RSA-PKCS: Encryption failed, returning

Commit ea6f7cf added these lines to ctx.c:

 835 #ifdef ENABLE_OPENSSL
 836         if (!CRYPTO_secure_malloc_initialized()) {
 837                 /* XXX What's a reasonable amount of secure heap? */
 838                 CRYPTO_secure_malloc_init(4096, 32);
 839         }
 840 #endif

As best I can tell from the OpenSSL man pages, once the secure heap is created its size is fixed.
and is used by internally by OpenSSL. For example in RAND_DRBG

Debugging the problem by adding ERR_print_errors_fp(stdout) to pkcs11-tool.c shows:

140737353951232:error:24074041:random number generator:rand_pool_new:malloc failure:../openssl/crypto/rand/rand_lib.c:447:
140737353951232:error:2406C06E:random number generator:RAND_DRBG_instantiate:error retrieving entropy:../openssl/crypto/rand/drbg_lib.c:320:
140737353951232:error:24074041:random number generator:rand_pool_new:malloc failure:../openssl/crypto/rand/rand_lib.c:447:
140737353951232:error:2406C06E:random number generator:RAND_DRBG_instantiate:error retrieving entropy:../openssl/crypto/rand/drbg_lib.c:320:
140737353951232:error:24074041:random number generator:rand_pool_new:malloc failure:../openssl/crypto/rand/rand_lib.c:447:
140737353951232:error:24074041:random number generator:rand_pool_new:malloc failure:../openssl/crypto/rand/rand_lib.c:447:
140737353951232:error:2406C06E:random number generator:RAND_DRBG_instantiate:error retrieving entropy:../openssl/crypto/rand/drbg_lib.c:320:
140737353951232:error:2406B072:random number generator:RAND_DRBG_generate:in error state:../openssl/crypto/rand/drbg_lib.c:579:
    RSA-PKCS: Encryption failed, returning

gdb shows where it is about to return NULL:

#1  0x00007ffff7a566bf in CRYPTO_secure_malloc (num=4096, file=0x7ffff7b50748 "../openssl/crypto/rand/rand_lib.c", line=445)
    at ../openssl/crypto/mem_sec.c:128
#2  0x00007ffff7a5673e in CRYPTO_secure_zalloc (num=4096, file=0x7ffff7b50748 "../openssl/crypto/rand/rand_lib.c", line=445)
    at ../openssl/crypto/mem_sec.c:143
#3  0x00007ffff7a87957 in rand_pool_new (entropy=256, min_len=32, max_len=4096) at ../openssl/crypto/rand/rand_lib.c:445
#4  0x00007ffff7a871ad in rand_drbg_get_entropy (drbg=0x7ffff7ff4e00, pout=0x7fffffffd2a8, entropy=256, min_len=32, max_len=4096, 
    prediction_resistance=0) at ../openssl/crypto/rand/rand_lib.c:149
#5  0x00007ffff7a85c7d in RAND_DRBG_instantiate (drbg=0x7ffff7ff4e00, pers=0x7ffff7b501e0 <ossl_pers_string> "OpenSSL NIST SP 800-90A DRBG", 
    perslen=28) at ../openssl/crypto/rand/drbg_lib.c:316
#6  0x00007ffff7a86c59 in drbg_setup (parent=0x0) at ../openssl/crypto/rand/drbg_lib.c:870
#7  0x00007ffff7a86cde in do_rand_drbg_init () at ../openssl/crypto/rand/drbg_lib.c:899
#8  0x00007ffff7a86c7c in do_rand_drbg_init_ossl_ () at ../openssl/crypto/rand/drbg_lib.c:884
#9  0x00007ffff7019a99 in __pthread_once_slow (once_control=0x7ffff7dd5548 <rand_drbg_init>, init_routine=0x7ffff7a86c73 <do_rand_drbg_init_ossl_>)
    at pthread_once.c:116
#10 0x00007ffff7ad49f3 in CRYPTO_THREAD_run_once (once=0x7ffff7dd5548 <rand_drbg_init>, init=0x7ffff7a86c73 <do_rand_drbg_init_ossl_>)
    at ../openssl/crypto/threads_pthread.c:113
#11 0x00007ffff7a86fe1 in RAND_DRBG_get0_public () at ../openssl/crypto/rand/drbg_lib.c:1024
#12 0x00007ffff7a86dd9 in drbg_bytes (out=0x69ad32 "", count=243) at ../openssl/crypto/rand/drbg_lib.c:941
#13 0x00007ffff7a8811b in RAND_bytes (buf=0x69ad32 "", num=243) at ../openssl/crypto/rand/rand_lib.c:776
#14 0x00007ffff7a986ba in RSA_padding_add_PKCS1_type_2 (to=0x69ad30 "", tlen=256, from=0x7fffffffd5b0 "abcdefghi", flen=10)
    at ../openssl/crypto/rsa/rsa_pk1.c:137
#15 0x00007ffff7a95f5b in rsa_ossl_public_encrypt (flen=10, from=0x7fffffffd5b0 "abcdefghi", to=0x7fffffffd5c0 "", rsa=0x694880, padding=1)
    at ../openssl/crypto/rsa/rsa_ossl.c:107
#16 0x00007ffff7a9208c in RSA_public_encrypt (flen=10, from=0x7fffffffd5b0 "abcdefghi", to=0x7fffffffd5c0 "", rsa=0x694880, padding=1)
    at ../openssl/crypto/rsa/rsa_crpt.c:30
#17 0x00007ffff7a45c17 in EVP_PKEY_encrypt_old (ek=0x7fffffffd5c0 "", key=0x7fffffffd5b0 "abcdefghi", key_len=10, pubk=0x693770)
    at ../openssl/crypto/evp/p_enc.c:29
#18 0x00000000004155f9 in encrypt_decrypt (session=6917488, mech_type=3, privKeyObject=6688864) at ../../../src/src/tools/pkcs11-tool.c:5252
#19 0x0000000000415ac8 in test_decrypt (sess=6917488) at ../../../src/src/tools/pkcs11-tool.c:5354
#20 0x0000000000415f7d in p11_test (session=6917488) at ../../../src/src/tools/pkcs11-tool.c:5466
#21 0x00000000004068b5 in main (argc=5, argv=0x7fffffffdd38) at ../../../src/src/tools/pkcs11-tool.c:1133

Note that RSA_padding_add_PKCS1_type_2 is requesting random numbers for the padding.
The RAND_DRBG_ routines are then setting up their pools, and requesting a 4096 byte block
from CRYPTO_secure_malloc is called to get 4096 bytes from the secure heap.
The routine is about to return NULL because there is no space in the heap.

Proposed Resolution

The question: "What's a reasonable amount of secure heap?" is not easily answered.
To use this feature, it appears CRYPTO_secure_malloc_init should be called at the application level,
not within OpenSC when called via PKCS#11.

The real problem is OpenSSL does not allow the heap to grow or seperate heaps for each library.

If OpenSC is going to use this feature, 4096 is too small.
838 CRYPTO_secure_malloc_init(65536, 32);
gets around the problem for now.

Steps to reproduce

Use OpenSSL 1.1.1 (built from github) with pkcs11-tool --test --login

[frankmorgner]

Thanks! Do you know a way to monitor the usage of the secure heap for approximating a good size?

In any case, please make a PR.

[dengert]

CRYPTO_secure_malloc_init.html defines all the routines. CRYPTO_secure_used() shows how much is currently used. But I don't see any routine to show the max used size.

It also says: "If a secure heap is used, then private key BIGNUM values are stored there." It is also not known at this time what other internal OpenSSL routines may use this or how other libraries or how the application may also use it.

So If I were to submit a PR, it would remove lines 835 to 840 and replace with a comment saying it will be used if already initialized.

But this may be complicated by linker options, the OS and how modules are loaded. It might be possible to have the pkcs11 module be self contained so the it has its own OpenSSL library and data separate from the calling application which would have its own OpenSSL library and data. In this case each would have its own secure heap.

Think about complicated programs like web servers or any program using PAM. Or any programs using libp11 and p11-kit. OpenSSL is used in many of these.

A more complicated PR could make initializing the secure heap an option in opensc.conf with a size.

Any OpenSC programmers out here that would like follow up on this?

[dengert]

IMHO OpenSC should not be calling CRYPTO_secure_malloc_init from any library or module. It can use the secure heap if it available. In the OpenSSL source code it is only called from /test/secmemtest.c It is never called from any library. It could be called from some tools but not from ctx.c.

There can only be one secure heap and once created it is not expandable. Any changes to make it expandable would need to be done in OpenSSL.

If other libraries start calling CRYPTO_secure_malloc_init the first library called will set the size. Think about applications that use PAM modules or applications that use p11-kit. Trying to estimate the size needed is not an easy task, and this bug report is based on OpenSSL internally starting to use it if available for random numbers, and tries to get some 4096 byte chunks.

The problem should be fixed by removing lines 835 - 840 from ctx.c. Leave its use up to the application
to create it and set it size.

[frankmorgner]

Almost... see #1525 for a fix

[dengert]

Yes I had looked at it. And will make further comments on #1525