pkcs11-tool --test --login

[dengert]

Problem Description

pkcs11-tool --test --login fails with RSA-X-509: Encryption failed, returning

/opt/ossl-1.1/bin/pkcs11-tool --module /opt/ossl-1.1/lib/opensc-pkcs11.so --test --login
Using slot 0 with a present token (0x0)
Logging in to "Test Cardholder".
Please enter User PIN: 
C_SeedRandom() and C_GenerateRandom():
  seeding (C_SeedRandom) not supported
  seems to be OK
Digests:
  all 4 digest functions seem to work
  MD5: OK
  SHA-1: OK
  RIPEMD160: OK
Signatures (currently only for RSA)
  testing key 0 (PIV AUTH key) 
  all 4 signature functions seem to work
  testing signature mechanisms:
    RSA-X-509: OK
    RSA-PKCS: OK
    SHA1-RSA-PKCS: OK
    MD5-RSA-PKCS: OK
    RIPEMD160-RSA-PKCS: OK
    SHA256-RSA-PKCS: OK
  testing key 1 (2048 bits, label=SIGN key) with 1 signature mechanism
Logging in to "Test Cardholder".
Please enter context specific PIN: 
    RSA-X-509: OK
  testing key 2 (2048 bits, label=KEY MAN key) with 1 signature mechanism -- can't be used to sign/verify, skipping
  testing key 3 (2048 bits, label=CARD AUTH key) with 1 signature mechanism
    RSA-X-509: OK
Verify (currently only for RSA)
  testing key 0 (PIV AUTH key)
    RSA-X-509: OK
    RSA-PKCS: OK
    SHA1-RSA-PKCS: OK
    MD5-RSA-PKCS: OK
    RIPEMD160-RSA-PKCS: OK
  testing key 1 (SIGN key) with 1 mechanism
Logging in to "Test Cardholder".
Please enter context specific PIN: 
    RSA-X-509: OK
  testing key 2 (KEY MAN key) with 1 mechanism -- can't be used to sign/verify, skipping
  testing key 3 (CARD AUTH key) with 1 mechanism
    RSA-X-509: OK
Unwrap: not implemented
Decryption (currently only for RSA)
  testing key 0 (PIV AUTH key) 
140628147707776:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:../openssl/crypto/dso/dso_dlfcn.c:117:filename(libgost.so): libgost.so: cannot open shared object file: No such file or directory
140628147707776:error:25070067:DSO support routines:DSO_load:could not load the shared library:../openssl/crypto/dso/dso_lib.c:162:
140628147707776:error:260B6084:engine routines:dynamic_load:dso not found:../openssl/crypto/engine/eng_dyn.c:414:
140628147707776:error:04068084:rsa routines:rsa_ossl_public_encrypt:data too large for modulus:../openssl/crypto/rsa/rsa_ossl.c:131:
    RSA-X-509: Encryption failed, returning
    RSA-PKCS: OK
  testing key 1 (SIGN key) 
    RSA-X-509: Logging in to "Test Cardholder".
Please enter context specific PIN: 
...

May be related to #1089 and #1551
fails here in line 5705 pkcs11-tool.c:

5705		if (EVP_PKEY_encrypt(ctx, encrypted, &out_len, orig_data, in_len) <= 0) {
5706			ERR_print_errors_fp(stderr);    /* added for OpenSSL debugging) */
5707			EVP_PKEY_CTX_free(ctx);
5708			EVP_PKEY_free(pkey);
5709			printf("Encryption failed, returning\n");

Problem appears to be related to line:
5547 pseudo_randomize(orig_data, sizeof(orig_data));
In a GDB test case this starts with:

(gdb) x/10xb orig_data
0x7fffffffd4d0:	0xd2	0xb2	0xd0	0x85	0xfa	0x54	0xd8	0x35
0x7fffffffd4d8:	0xe8	0xd4

The first byte 0xd2 is larger then the first byte of the public key 0xd1 from my card.

the added line:
5706 ERR_print_errors_fp(stderr);
prints the OpenSSL error log that has as the last line:

140737353936768:error:04068084:rsa routines:rsa_ossl_public_encrypt:data too large for modulus:../openssl/crypto/rsa/rsa_ossl.c:131:

128     if (BN_ucmp(f, rsa->n) >= 0) {
129         /* usually the padding functions would catch this */
130         RSAerr(RSA_F_RSA_OSSL_PUBLIC_ENCRYPT,
131                RSA_R_DATA_TOO_LARGE_FOR_MODULUS);
132         goto err;
133     }

BN_ucmp is comparing the value of f (orig_data) to rsa->n the modulus and failing.

Note: in my GDB runs pseudo_randomize produces the same data each time.

Proposed Resolution

This may not be perfect, but it has a beter chance of not failing on most keys.

diff --git a/src/tools/pkcs11-tool.c b/src/tools/pkcs11-tool.c
index 655e28cb..b5164097 100644
--- a/src/tools/pkcs11-tool.c
+++ b/src/tools/pkcs11-tool.c
@@ -5545,6 +5545,7 @@ static int encrypt_decrypt(CK_SESSION_HANDLE session,
        printf("    %s: ", p11_mechanism_to_name(mech_type));
 
        pseudo_randomize(orig_data, sizeof(orig_data));
+       orig_data[0] = 0; /* Make sure it is less then modulus */
 
        pkey = get_public_key(session, privKeyObject);
        if (pkey == NULL)

Steps to reproduce

Logs