X509_check_private_key() does not honor RSA_METHOD_FLAG_NO_CHECK flag

[ansasaki]

The X509_check_private_key() function ignores the RSA_METHOD_FLAG_NO_CHECK set in the RSA key structure and tries to check if the public key information of the private key matches the public key from the certificate.

This is especially problematic for keys obtained from engines that explicitly set the flag to prevent this match check from occurring.

For instance, the PKCS#11 standard does not differentiate RSA and RSASSA-PSS key objects, there is only one common type for both (CKK_RSA). OpenSSL uses different types for RSA and RSASSA-PSS keys. The engine can set the RSA_METHOD_FLAG_NO_CHECK flag to prevent OpenSSL from comparing the types of an RSASSA-PSS public key obtained from a certificate with the type contained in an RSA private key object provided by the engine.

[t8m]

The question is also whether the type mismatch for RSA private key and RSASSA-PSS public key from c certificate should not be handled on lower level in EVP_PKEY_cmp().
I'd prepare a PR but some guidance what is the appropriate way to solve it would be nice.

[mtrojnar]

Is anyone working on this issue?

[kaduk]

I don't think anyone is currently working on this.
I have a vague recollection of previous discussions around this flag indicating that it is not something anyone really likes but that has to be kept around for historical compatibility. (Note that RSA is the only key type with such a thing.) It's also only checked from libssl, so there's not much precedent for expecting it to work in PKCS#11.
That said, if we did want to make it apply to RSA-PSS keys as well (in the same way it does to "plain" RSA keys), the approach would be straightforward -- add another clause in the EVP_PKEY_id() check.
I don't know that there would be much appetite for expanding its scope to apply to more than libssl.

I will also note in closing that if you have a scheme in place that uses, e.g., an ENGINE for private key operations, you can get around the openssl consistency checks by telling openssl that the private key has the same value as the public key. (This is not a guarantee that doing so will continue to work indefinitely; just an observation about the current state of things.)

[t8m]

The version 1.1.1 is not supported anymore. Closing. If this is still an issue with currently supported releases, please open a new issue.

[mtrojnar]

So you closed this issue based on the fact that @levitte marked it as affecting OpenSSL 1.1.1, which was the latest stable branch at the time, and the master branch. Interesting.

[t8m]

Given the engines are deprecated I do not think it makes sense to keep the issue open as it will not be a priority for the team to fix it. But yeah, I can reopen it if anybody wants to try to tackle it.