Bypass private/public key check

[panedw]

I am using curl/openssl/libp11 with a HSM (that doesn't let it's private key out) for client SSL authentication. Even though curl correctly asks the HSM to encrypt server's challenge with its private key, before it gets to this point it attempts to retrieve the certificate, public key and private key from the HSM and then checks for match - even though it doesn't use the private key directly.

There is openssl method flag - RSA_METHOD_FLAG_NO_CHECK that can be used to skip the public/private check, which is supported by curl, but I don't see it supported by libp11. Instead the flag seems to be hardcoded to 0, see the function pkcs11_get_evp_key_rsa() in p11_rsa.c.

Any suggestions how to work around it? My curl complains about public/private mismatch and refuses to do the handshaking. When I use a bogus private key and force the RSA_METHOD_FLAG_NO_CHECK flag it works, but there seems to be no way to have libp11 to add this flag automatically. Shouldn't it be set when the CKA_EXTRACTABLE attribute is set to false?

[mtrojnar]

Would you submit a PR?

[ansasaki]

I've reported a related issue to OpenSSL: openssl/openssl#8767

Some time ago I tried to use something similar to be able to use RSASSA-PSS keys, but even setting the flag in the engine OpenSSL does not honour it.

I couldn't use RSASSA-PSS keys because in PKCS#11 RSA and RSASSA-PSS objects are identified by a single identifier (CKK_RSA), but for OpenSSL they are different. So the check for matching private/public keys would always fail (different OIDs). The only solution would be to set the flag AND make OpenSSL to honour it.

[mtrojnar]

Good news: @levitte has confirmed the bug in OpenSSL.