openssl: add a more specific DES support detection

[xnox]

Improve detection for DES support in OpenSSL, to allow compilation against system OpenSSL without DES.

Despite MariaDB depreciation policy, distributions may want to turn off DES in their OpenSSL. And still desire to build MariaDB with TLS OpenSSL support. Allow such combination of builds.

• It is related to https://jira.mariadb.org/browse/MDEV-31873

Description

Support building MariaDB without DES support in system OpenSSL

Release Notes

The deprecated DES_ENCRYPT / DES_DECRYPT / --des-key-file functionality will not be available when MariaDB is compiled against system OpenSSL which itself does not provide DES support.

How can this PR be tested?

export CPPFLAGS=-DOPENSSL_NO_DES and compile against any OpenSSL.

Basing the PR against the correct MariaDB version

• This is a new feature and the PR is based against the latest MariaDB development branch.
• This is a bug fix and the PR is based against the earliest maintained branch in which the bug can be reproduced.

PR quality check

• I checked the CODING_STANDARDS.md file and my PR conforms to this where appropriate.
• For any trivial modifications to the PR, I am ok with the reviewer making the changes themselves.

[CLAassistant]

All committers have signed the CLA.

[grooverdan]

Needs addition to config.h.cmake for compilation to see the HAS_des define (making the cmake define come in as a preprocessor define). Otherwise looks pretty ok. This could rebase back to 10.5 to allow compilation there in the no-des case.

Also on the related algorithm still used - there's still md5 references as well (https://jira.mariadb.org/browse/MDEV-32368) in the calculation of views (in addition to the SQL function). Given its a bit more complex, I welcome a second PR on that one at least so it won't crash on the missing function, or require it to generate the view hash.

[vuvova]

@grooverdan , unlike DES — which is purely a user-facing function, MD5 is also used internally — for views, yes — for non-cryptographic purposes, just as a checksum. I suspect we won't remove it, but will simply bundle our own MD5 implementation for that. We can, of course, stop providing MD5() function to users if OpenSSL doesn't have it.

[xnox]

Needs addition to config.h.cmake for compilation to see the HAS_des define (making the cmake define come in as a preprocessor define).

Thank you

Otherwise looks pretty ok. This could rebase back to 10.5 to allow compilation there in the no-des case.

Yes, I would like that very much, as far back as possible.

Also on the related algorithm still used - there's still md5 references as well (https://jira.mariadb.org/browse/MDEV-32368) in the calculation of views (in addition to the SQL function). Given its a bit more complex, I welcome a second PR on that one at least so it won't crash on the missing function, or require it to generate the view hash.

For non-cryptographic use cases using xxhash64 would be the best option.
And for ambigious / user-facing it is best to remove offering MD5 function indeed, as it makes audits harder.

[xnox]

aarch64-macos - failing with ERROR: sha256sum command not found missing coreutils or maybe can just use openssl?

other linuxy errors are unrelated to the change I made

appveyor looks like a real failure.

[xnox]

https://ci.appveyor.com/project/rasmushoj/server/builds/49612312

cmake -E time cmake %GENERATOR% .. -DCMAKE_BUILD_TYPE=%BUILD_TYPE% -DMYSQL_MAINTAINER_MODE=ERR -DFAST_BUILD=1 -DBISON_EXECUTABLE=C:\cygwin64\bin\bison.exe -DPLUGIN_PERFSCHEMA=NO -DPLUGIN_FEEDBACK=NO
...
-- Looking for DES_set_key_unchecked
-- Looking for DES_set_key_unchecked - found
...
[919/920] Linking CXX shared library sql\server.dll
FAILED: sql/server.dll sql/server.lib 
C:\Windows\system32\cmd.exe /C "C:\Windows\system32\cmd.exe /C ""C:\Program Files\CMake\bin\cmake.exe" -E __create_def C:\projects\server\_build\sql\CMakeFiles\server.dir\.\exports.def C:\projects\server\_build\sql\CMakeFiles\server.dir\.\exports.def.objs && cd C:\projects\server\_build" && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_dll --intdir=sql\CMakeFiles\server.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MICROS~2\2022\COMMUN~1\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\server.rsp  /out:sql\server.dll /implib:sql\server.lib /pdb:sql\server.pdb /dll /version:0.0 /machine:x64 /INCREMENTAL:NO /debug /NODEFAULTLIB:libucrt.lib /DEFAULTLIB:ucrt.lib  /DEF:sql\CMakeFiles\server.dir\.\exports.def && cd ."
LINK: command "C:\PROGRA~1\MICROS~2\2022\COMMUN~1\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\server.rsp /out:sql\server.dll /implib:sql\server.lib /pdb:sql\server.pdb /dll /version:0.0 /machine:x64 /INCREMENTAL:NO /debug /NODEFAULTLIB:libucrt.lib /DEFAULTLIB:ucrt.lib /DEF:sql\CMakeFiles\server.dir\.\exports.def /MANIFEST:EMBED,ID=2" failed (exit code 1120) with the following output:
   Creating library sql\server.lib and object sql\server.exp
des_key_file.cc.obj : error LNK2001: unresolved external symbol "struct st_mysql_mutex LOCK_des_key_file" (?LOCK_des_key_file@@3Ust_mysql_mutex@@A)
item_strfunc.cc.obj : error LNK2001: unresolved external symbol "struct st_mysql_mutex LOCK_des_key_file" (?LOCK_des_key_file@@3Ust_mysql_mutex@@A)
mysqld.cc.obj : error LNK2001: unresolved external symbol "struct st_mysql_mutex LOCK_des_key_file" (?LOCK_des_key_file@@3Ust_mysql_mutex@@A)
mysqld.cc.obj : error LNK2001: unresolved external symbol "char * des_key_file" (?des_key_file@@3PEADEA)
sql_reload.cc.obj : error LNK2001: unresolved external symbol "char * des_key_file" (?des_key_file@@3PEADEA)
sql\server.dll : fatal error LNK1120: 2 unresolved externals
ninja: build stopped: subcommand failed.
Elapsed time (seconds): 1065.82

So des detection did work and was found.... but as if HAVE_des define is not working on cygwin build.

[xnox]

Fixed appveyor build... but i still broke something cause it fails to exec

920/920] Linking CXX executable sql\mariadbd.exe
Hardlink created for mysqld.exe <<===>> mariadbd.exe
Elapsed time (seconds): 971.804
set PATH=C:\Strawberry\perl\bin;%PATH%;C:\Program Files (x86)\Windows Kits\10\Debuggers\x64
cd %APPVEYOR_BUILD_FOLDER%\_build\mysql-test
set /A parallel=4*%NUMBER_OF_PROCESSORS%
perl mysql-test-run.pl --force --max-test-fail=10 --retry=2 --parallel=%parallel% --testcase-timeout=4 --suite=main  --skip-test-list=%APPVEYOR_BUILD_FOLDER%\win\appveyor_skip_tests.txt --mysqld=--loose-innodb-flush-log-at-trx-commit=2
Logging: C:/projects/server/mysql-test/mariadb-test-run.pl  --force --max-test-fail=10 --retry=2 --parallel=8 --testcase-timeout=4 --suite=main --skip-test-list=C:\projects\server\win\appveyor_skip_tests.txt --mysqld=--loose-innodb-flush-log-at-trx-commit=2
VS config: 
vardir: C:/projects/server/_build/mysql-test/var
Removing old var directory...
Creating var directory 'C:/projects/server/_build/mysql-test/var'...
Checking supported features...
mysql-test-run: *** ERROR: Could not find version of MariaDB

I wonder if I need to spin up cygwin environment on my windows laptop to debug this.

[grooverdan]

Don't worry about appveryor, it looks something high levelish, there's other windows builds passing. Probably not related to anything you've done.

[vuvova]

looks ok to me, see one comment, though

[xnox]

Don't worry about appveryor, it looks something high levelish, there's other windows builds passing. Probably not related to anything you've done.

It worries me, because other pull requests build fine. So I do think it is something of my own doing. As I'm also hiding lots of symbols and command-line arg that used to be there as well.

Maybe I should keep on accepting cmdline arg, and config keys - even though they will not do anything? but imho that would be missleading too. I'd rather fail on startup, rather than upon user call to des_decrypt.

[LinuxJedi]

Don't worry about appveryor, it looks something high levelish, there's other windows builds passing. Probably not related to anything you've done.

It worries me, because other pull requests build fine. So I do think it is something of my own doing. As I'm also hiding lots of symbols and command-line arg that used to be there as well.

Maybe I should keep on accepting cmdline arg, and config keys - even though they will not do anything? but imho that would be missleading too. I'd rather fail on startup, rather than upon user call to des_decrypt.

There are known issues at the moment causing some builders to randomly fail in 11.5 (and a couple to constantly fail). Appveryor is actually completely broken. These are being addressed in the earlier versions, but will take a little while to hit 11.5. I cannot see that anything you have changed has affected the required builders, I can retrigger them if needed.

Please feel free to take this out of draft too :)

[vaintroub]

Fixed appveyor build... but i still broke something cause it fails to exec

920/920] Linking CXX executable sql\mariadbd.exe
Hardlink created for mysqld.exe <<===>> mariadbd.exe
Elapsed time (seconds): 971.804
set PATH=C:\Strawberry\perl\bin;%PATH%;C:\Program Files (x86)\Windows Kits\10\Debuggers\x64
cd %APPVEYOR_BUILD_FOLDER%\_build\mysql-test
set /A parallel=4*%NUMBER_OF_PROCESSORS%
perl mysql-test-run.pl --force --max-test-fail=10 --retry=2 --parallel=%parallel% --testcase-timeout=4 --suite=main  --skip-test-list=%APPVEYOR_BUILD_FOLDER%\win\appveyor_skip_tests.txt --mysqld=--loose-innodb-flush-log-at-trx-commit=2
Logging: C:/projects/server/mysql-test/mariadb-test-run.pl  --force --max-test-fail=10 --retry=2 --parallel=8 --testcase-timeout=4 --suite=main --skip-test-list=C:\projects\server\win\appveyor_skip_tests.txt --mysqld=--loose-innodb-flush-log-at-trx-commit=2
VS config: 
vardir: C:/projects/server/_build/mysql-test/var
Removing old var directory...
Creating var directory 'C:/projects/server/_build/mysql-test/var'...
Checking supported features...
mysql-test-run: *** ERROR: Could not find version of MariaDB

I wonder if I need to spin up cygwin environment on my windows laptop to debug this.

We do not use cygwin, and we won't compile in Cygwin either. The Windows environment, that works, is MSVC / Strawberry perl. Unfortunately, Appveyor builds were broken back in January (not really appveyor, but all tests for all builds without performance schema). This will be fixed. Fortunately it was not you who broke those builds.

[xnox]

buildbot/amd64-debian-11-msan - is sort of successful, but the job failed to upload things at the end of it. Is it a broken workflow? or did something msan-ish didn't actually work?

buildbot/amd64-ubuntu-2004-debug - has an unrelated lowercase_fs test case failure, and no others.

Do I need above two to go green before this PR can be merged? Do I need to attempt to fix above issues in separate PRs first? and/or file issues?

@vuvova @vaintroub this is my first contribution to MariaDB so not sure if something else is expected from me here, or not.

[xnox]

all required checks passed upon rebase / retry. Whoop whoop.