Use vendor implementations for some Dense kernels in Ginkgo

[pratikvn]

This PR adds a strategy enum to allow runtime switching between vendor and Ginkgo provided kernels.

+ Is not kernel specific, but Dense object specific.
+ create_with_config_of and create_with_type_of set the same strategy as their parent.
+ Strategies of objects created within Ginkgo cannot be controlled, but set to default, which is currently using Ginkgo kernels.

This is still a WIP. Only dot has been implemented as a prototype. You can see how it works in the cuda/hip dense kernel tests.

The implementation switch is now inside the kernel and the single vector implementation is the vendor implementation (where available) and multi-vector always uses Ginkgo's implementation. Only the following routines are replaced:

1. Dot
2. Conjugate dot
3. Norm2

[upsj]

What is the purpose of this strategy approach? From the user side, I mainly care about performance and correctness. Opposite to Csr, both of those are not really data-dependent, as the only data with performance impact are the matrix dimensions/stride.
If we want to use the vendor libraries for #rhs = 1, why don't we do it on the kernel side?
If this is about benchmarking, we can add hipBLAS/cuBLAS wrappers to the BLAS benchmark like we did for Csr.

[Slaedr]

The general idea makes sense. However, I think we, and users, may want to use different strategies (I guess they're really blas-backends or something) for different operations. See my comments below.

[Slaedr]

@upsj You make a good point; we could go that way too. But if the idea is to give the user choice of blas backend, I'm for that as well. From time to time, we or the vendor might introduce improvements that might change the calculus for the user. We may not be able to or want to react to changes from the vendor side quickly enough.

stale

[upsj]

LGTM! Can you add your tuning results for the generic reductions here, as well?

[pratikvn]

I only ran the tuning results with single vector and with dot product in mind. I did not really verify how that affects the other kernels using this generic reduction, so I am a bit wary of changing the oversubscription directly in the code as of now. I will try to do a more comprehensive run and update that in another PR.

[yhmtsai]

LGTM. could you also add the norm2 test for one vector?
There are 1/20 vectors test for dot and conj_dot but only 20 vectors test for norm2

[sonarcloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
10 Code Smells

42.1% Coverage
44.5% Duplication

[codecov]

Codecov Report

Merging #967 (3ea6e0c) into develop (5efed26) will increase coverage by 0.00%.
The diff coverage is 92.50%.

@@           Coverage Diff            @@
##           develop     #967   +/-   ##
========================================
  Coverage    93.38%   93.39%           
========================================
  Files          480      480           
  Lines        39577    39598   +21     
========================================
+ Hits         36959    36981   +22     
+ Misses        2618     2617    -1     

Impacted Files	Coverage Δ
core/device_hooks/common_kernels.inc.cpp	0.00% <0.00%> (ø)
core/matrix/dense.cpp	94.54% <100.00%> (ø)
omp/matrix/dense_kernels.cpp	79.01% <100.00%> (+1.67%)	⬆️
omp/test/matrix/dense_kernels.cpp	99.69% <100.00%> (+<0.01%)	⬆️
reference/matrix/dense_kernels.cpp	89.06% <100.00%> (+0.17%)	⬆️
reference/base/index_set_kernels.cpp	100.00% <0.00%> (ø)
devices/machine_topology.cpp	82.71% <0.00%> (+4.93%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5efed26...3ea6e0c. Read the comment docs.