Is there a reason to mess with secure heap from within OpenSC at all, instead of following @dengert's suggestion to leave it to the application?

Well, do you know what the secure heap is used for?

sorry for being cocky. We need to do it in OpenSC if nobody else does it (for example when we're linking OpenSSL statically) otherwise there is the risk that secrets get swapped out to disk.

I agree that writing secrets to disk is a problem and is a problem for many projects. And what we will see is other projects will also try use it too.

CRYPTO_secure_malloc man pages say: " help protect applications (particularly long-running servers)" and "The level and type of security guarantees depend on the operating system." It also goes on to say "private key BIGNUM values are stored there".

OpenSSL has made it easy to use their "secure heap", but their implementation is not flexible. It may have been good enough for a few BIGNUMs. But it looks like in 1.1.1 (or maybe earlier) OpenSSL started to store other things in there like their RAND_DRBG pools, that are much larger then a few BIGNUMs. They (or other projects) may be storing other things in there too.

I would rather see OpenSC try to get OpenSSL to improve their implementation to allow for expansion of the "secure heap". Either when requested by a library or upon a failed allocation attempt. I would like to see this become their problem with their solution and not OpenSCs problem with an OpenSC only solution.

For example: Upon an attempt to use one of the CRYPTO_secure_* routines (not sure which one) if env OPENSSL_SECURE_HEAP="<size>, <minsize>" is set, create it. This would then work for all the libraries that that might want to use it and is set at the application level.

And the problem of exposing secrets goes beyond just libraries, and the OS. Consider a virtual machine that thinks it has non paged memory, but in reality the host may still be swapping the VM's memory to disk or not clearing physical memory with secrets before ging it to another VM. Other projects may have addressed this in their own way, or tried using kernel memory that is non-paged.

Just some quick remarks on the misconceptions above:

1. Future random number need to be kept secret. Not swapping the pool/state of RNG is totally reasonable.
2. If you're doing Virtualization correct, locked memory won't be swapped.

Yes OpenSSL could do better and yes you could take the time to report this. But look at the development process of OpenSSL (or especialy the contribution about secure memory): The OpenSSL core developers don't always have the time to make every feature smooth and comfortable. Nevertheless, its useful and gets integrated and obviously it is used by some people despite the limitations. If you suggest (or even implement) a fix for OpenSSL, this will usually take years until this hits a release.

Until then we need to deal with problems (swapping) with the tools that are available (OpenSSL 1.1+). (And pushing all responsibility to the user by honoring an environment variable is not a good way of dealing with this).