benchmarks update

[franziskuskiefer]

This PR improves usabiblity of benchmarks and adds some more

• allow using ./mach build --release --benchmark to build and run benchmarks
• moved configuration for benchmarks to config.json
• added benchmarks to CI
• added p256, k256, x25519, chachapoly, blake2 benchmarks including openssl comparison

Ticks some boxes in #37. More to follow.

[coveralls]

Pull Request Test Coverage Report for Build 3181956031

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 13 unchanged lines in 2 files lost coverage.
• Overall coverage increased (+0.5%) to 52.814%

---

Files with Coverage Reduction	New Missed Lines	%
src/EverCrypt_Curve25519.c	3	91.18%
src/Hacl_Curve25519_51.c	10	95.9%

Totals
Change from base Build 3181921469:	0.5%
Covered Lines:	32458
Relevant Lines:	61457

---

💛 - Coveralls

[duesee]

Can you add some documentation how to run the benchmarks? Do I need additional dependencies?

Could you also rename the subcommand ./mach benchmarks to ./mach benchmark to make it a verb?

$ ./mach build --release --benchmarks
 [mach] Dependency checks ...
 [mach] Probing for cmake: Found
 [mach] Probing for ninja: Found

 [mach] Running config to write config.cmake and config.h ...
 [mach] Using config/config.json to configure ...
 [mach] Writing cmake config to build/config.cmake ...
 [mach] Collecting files and dependencies ...
-- The C compiler identification is GNU 11.3.0
-- The CXX compiler identification is GNU 11.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Bug 81300 check: FALSE
-- int128 support: TRUE
-- explicit_bzero support: TRUE
-- vec128 support: TRUE
-- vec256 support: TRUE
-- Detected vale support
-- Detected inline assembly support
-- Detected intrinsics support
-- Detected an x64 architecture
-- Building benchmarks
-- Found Python: /usr/bin/python3.10 (found version "3.10.6") found components: Interpreter 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Populating benchmark
-- Configuring done
-- Generating done
-- Build files have been written to: /home/duesee/repos/cryspen/hacl-packages/build/benchmark-subbuild
[1/9] Creating directories for 'benchmark-populate'
[1/9] Performing download step (git clone) for 'benchmark-populate'
Klone nach 'benchmark-src' …
HEAD ist jetzt bei 361e8d1 version bump
[2/9] Performing update step for 'benchmark-populate'
[4/9] No patch step for 'benchmark-populate'
[5/9] No configure step for 'benchmark-populate'
[6/9] No build step for 'benchmark-populate'
[7/9] No install step for 'benchmark-populate'
[8/9] No test step for 'benchmark-populate'
[9/9] Completed 'benchmark-populate'
-- Failed to find LLVM FileCheck
-- Found Git: /usr/bin/git (found version "2.34.1") 
-- git version: v1.7.0 normalized to 1.7.0
-- Version: 1.7.0
-- Looking for shm_open in rt
-- Looking for shm_open in rt - found
-- Performing Test HAVE_CXX_FLAG_STD_CXX11
-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- success
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile: Change Dir: /home/duesee/repos/cryspen/hacl-packages/build/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/ninja cmTC_b9a5a && [1/2] Building CXX object CMakeFiles/cmTC_b9a5a.dir/Debug/gnu_posix_regex.cpp.o
FAILED: CMakeFiles/cmTC_b9a5a.dir/Debug/gnu_posix_regex.cpp.o 
/usr/bin/c++ -DCMAKE_INTDIR=\"Debug\"  -std=c++11  -Wall  -Wextra  -Wshadow  -Wfloat-equal  -pedantic  -pedantic-errors  -fstrict-aliasing  -Wno-deprecated-declarations  -Wstrict-aliasing  -g -o CMakeFiles/cmTC_b9a5a.dir/Debug/gnu_posix_regex.cpp.o -c /home/duesee/repos/cryspen/hacl-packages/build/benchmark-src/cmake/gnu_posix_regex.cpp
/home/duesee/repos/cryspen/hacl-packages/build/benchmark-src/cmake/gnu_posix_regex.cpp:1:10: fatal error: gnuregex.h: Datei oder Verzeichnis nicht gefunden
    1 | #include <gnuregex.h>
      |          ^~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.

CI has these errors, too.

[franziskuskiefer]

Can you add some documentation how to run the benchmarks? Do I need additional dependencies?
CI has these errors, too.

You don't need any additional dependencies. If you don't have OpenSSL installed, you need to disable these benchmarks with --no-openssl.
These aren't errors, they are tests if something is available. You can just ignore it.

[duesee]

Thanks. The benchmarks seem to work. The output is very noisy due to the reported "errors" and deprecation warnings.

hacl-packages/benchmarks/p256.cc:137:42: warning: ‘EC_KEY* EC_KEY_new_by_curve_name(int)’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
  137 |   EC_KEY* sk_a = EC_KEY_new_by_curve_name(NID_X9_62_prime256v1);

Should we care about the deprecation warnings? If not, LGTM.

[franziskuskiefer]

Should we care about the deprecation warnings?

The OpenSSL APIs here are horrible. I opted for the legacy APIs because they're easier to use. We can silence the warnings with build flags (see CMakeLists.txt:92). But I felt like leaving them in for now so we're constantly reminded of doing something about it in some way (and we shouldn't ignore all deprecation warnings) 😉