PKCS#11 testsuite

[Jakuje]

As I announced some time ago [1] I was looking into a quite standalone PKCS#11 testsuite. We run this testsuite nightly (on upstream changes) for over a year now for our CAC and Coolkey cards and produce simple status badge, that would be nice to have also for other cards and in upstream:

If others have a spare card, reader and machine, they can do the same, quite irrespective to their platform of choice. Or at least running something like this on new supported cards or suppose-supported cards before release should also improve the releases avoiding regressions.

The tests work with common RSA keys and mechanisms as well as with ECC keys. Currently, there is also support for RSA-PSS and OAEP mechanisms that was verified against soft opencryptoki and softhsm software tokens, since there are no support for these mechanisms in OpenSC yet, but it can be a good base for it in future. It can also generate JSON report, which can be simply compared against last run and notice (unwanted) changes.

The commits represent the progress over time and evolution of the tool and might not be necessarily useful, since the features, fixes and refactoring was done somehow iterative.

The build passes with current Fedora 27 (OpenSSL 1.1.0) and RHEL7 (OpenSSL 1.0.2).

Comments, ideas, improvements, additions welcomed.

[1] https://sourceforge.net/p/opensc/mailman/message/35091170/
[2] https://gitlab.com/redhat-sectech/OpenSC/pipelines

Checklist

• Documentation is added or updated
• New files have a LGPL 2.1 license statement
• Tested with the following card: CAC, Coolkey, PIV; opencryptoki and softhsm softtokens
  • tested PKCS#11

[Jakuje]

The travis build fails, because it does not have cmocka package. Not sure if it is better to introduce configure condition and do not build it in travis, or download and install it during the build time. Probably combination of both of them so it is also tested, but not mandatory for all builds.

[frankmorgner]

superfluous assignment

[Jakuje]

The paths is not used only in the pkcs11-tool, but also for others, including PKCS11SPY.

[frankmorgner]

I see, thanks.

[frankmorgner]

That's quite a lot of work done, thanks! It looks OK on first glance, but I'll need to get back to this later.

Yes, you could test for cmocka in configure.ac which enables p11test conditionally. Maybe this would be a good opportunity to create a configure switch --enable-tests/--disable-tests which allows ignoring the tests directory entirely for the build. In CI, the test cases should be built if we don't create the "light" version.

Can some of the tests be put into pkcs11-tool? This way they would also be available via the installer.

Does it make sense to call pkcs11-tool --test in runtest.sh?

[Jakuje]

Thank you for the comments. I see that it is a lot of code so I don't expect it to be reviewed overnight. --enable-tests/--disable-tests switch sounds like a good way to go. I will try to put it together.

The tests in pkcs11-tool already do quite nice job, but they are really hard to extend, lets say, with regression cases, because of quite hardcoded flow in large functions there. This also means there is hard to do any recovery from failures of specific parts. Extending it to handle EC keys and/or PSS/OAEP signatures would again make it even more complex.

We can argue that pkcs11-tool has also EC test, but it expects writable card (generating EC key at the beginning), which is also not always available.

Run the pkcs11-tool --test from runtest.sh as a "sanity" can make sense. I will check how it can work.

[Jakuje]

Now it builds fine on Linux, but it still needs some adjustments on osx, because I don't know how to install cmocka correctly there (it has probably some different default paths for pkgconfig and probably others). Pointers welcomed.

-- Installing: /usr/lib/pkgconfig/cmocka.pc

CMake Error at cmake_install.cmake:39 (file):

  file INSTALL cannot copy file "/tmp/cmocka-1.1.0/build/cmocka.pc" to

  "/usr/lib/pkgconfig/cmocka.pc".

But this error is not fatal and it falls back to building without tests.

[frankmorgner]

I think the prefix should be /usr/local on macOS. Isn't there a brew package of cmocka?

[frankmorgner]

Is there some documentation available for setting up a test worker?

[Jakuje]

I think the prefix should be /usr/local on macOS. Isn't there a brew package of cmocka?

When I was searching for it last time, I didn't find it in either Ubuntu nor OSX, but now I find in in both of them so have a look what is going to happen with the last commit.

Is there some documentation available for setting up a test worker?

It probably depends on platform, but from gitlab, with GitLab CI Runner, it is matter of install a package and set up the runner with the token from the gitlab administration:

https://docs.gitlab.com/runner/install/

How I run the testsuite there is visible in your .gitlab-ci.yml:

https://gitlab.com/redhat-sectech/OpenSC/blob/master/.gitlab-ci.yml

[frankmorgner]

is this still required?

[Jakuje]

It should not be needed, if the installation from repositories worked (the last build was success including the installation of cmocka, so I will remove this line).

[Jakuje]

ping.

I rebased this PR on the current master changes.

Would not this help in testing the new release? Is there something that could be improved?

This could be helpful for others to gather results of their testing in the common format.