Add cusp benchmark

[tcojean]

Move CuSPARSE benchmarking capacities to the spmv benchmark.

• Add all Cusp benchmarks to a file cuda_linops.hpp,
• Make all the Cusp classes be gko::LinOp for easy integration;
• Control usage of these classes with the flag HAS_CUDA, given only when the
  benchmark is compiled with CUDA capacities;
• Use dynamic pointer cast of std::shared_ptr<const gko::Executor> to ensure
  the Cusp classes are called with a std::shared_ptr<const gko::CudaExecutor>.
• Remove the spmv_comparison_cuda benchmark.
• Replace CUDA_VERSION with actual CMAKE_CUDA_COMPILER_VERSION.
• Catch exceptions in benchmarks by const reference to get their actual type.
• Add a function with specializations to manage the cudaDataType_t such as
  CUDA_R_64F.
• Use pointer mode HOST for cuspCsrEx as it seems to be required.
• Create CuSPARSE bindings for different precisions.

[pratikvn]

LGTM!

[yhmtsai]

Before using the cuSPARSE API, it needs to set CUSPARSE_POINTER_MODE_HOST or move alpha and beta to device.
I prefer the second way because the Ginkgo kernel uses device parameter.

[tcojean]

@yhmtsai I using the handles we already have in Ginkgo for each CUDA executor, which sets the pointer to DEVICE mode and everything seems to work properly. Why should it be HOST mode?

ginkgo/cuda/base/cusparse_bindings.hpp

Lines 216 to 223 in e1ed14d

	inline cusparseHandle_t init()
	{
	cusparseHandle_t handle{};
	GKO_ASSERT_NO_CUSPARSE_ERRORS(cusparseCreate(&handle));
	GKO_ASSERT_NO_CUSPARSE_ERRORS(
	cusparseSetPointerMode(handle, CUSPARSE_POINTER_MODE_DEVICE));
	return handle;
	}

[thoasm]

Mostly nits about using const &, or hiding Constructors and _impl functions with protected.
My concern is with const gko::CudaExecutor *gpu_exec in CuspBase when using the operator=. Do you actually need the operator=? If you don't, then just delete it and everything is fine, however, if you require it, it would be better to have gpu_exec as an std::shared_ptr<CudaExecutor> to ensure it is not deleted before the class.

[yhmtsai]

Could you also add some description of these formats in spmv?

[yhmtsai]

should it add GKO_NOT_IMPLEMENTED; ?
Except for cusp_csrmm, the formats only allow nrhs = 1, so they need to handle the condition in their apply function

[tcojean]

The benchmarker benchmarks only for one nrhs so I put this as NOT_IMPLEMENTED.

[tcojean]

@yhmtsai I cannot get cusparseCsrmvEx with the CUSPARSE_ALG_MERGE_PATH algorithm to work whatever I do. I'm a bit puzzled since I believe everything should be done according to the documentation.

https://docs.nvidia.com/cuda/archive/9.2/cusparse/index.html#cusparse-csrmvEx

We use double type, CUSPARSE_OPERATION_NON_TRANSPOSE plus other default parameters, and all data is allocated through cudaMalloc (so properly aligned). All other csr functions work, and this works with the NAIVE algorithm. Since the buffer is not used, even allocating a dummy buffer and not calling cusparseCsrmvEx_bufferSize results in the same problem in cusparseCsrmvEx.

Do you have any idea ?

[yhmtsai]

@tcojean I guess this function only allows CUSPARSE_POINTER_MODE_HOST
I set the scalar in host memory and set the pointer_mode, and then the function works.

Some other try:
I set the buffersize = the value I got in the above statement. it also failed.
set the buffer pointer = nullptr. it also failed

stale

[tcojean]

@yhmtsai Thanks for the feedback, I will try with CUSPARSE_POINTER_MODE_HOST and see if it works.

[thoasm]

LGTM!

[yhmtsai]

@tcojean Could you also add the description of these cuda formats in

ginkgo/benchmark/spmv/spmv.cpp

Lines 76 to 95 in 0c60dee

	DEFINE_string(
	formats, "coo",
	"A comma-separated list of formats to run."
	"Supported values are: coo, csr, ell, sellp, hybrid, hybrid0, "
	"hybrid25, hybrid33, hybridlimit0, hybridlimit25, hybridlimit33, "
	"hybridminstorage.\n"
	"coo: Coordinate storage. The CUDA kernel uses the load-balancing approach "
	"suggested in Flegar et al.: Overcoming Load Imbalance for Irregular "
	"Sparse Matrices.\n"
	"csr: Compressed Sparse Row storage. The CUDA kernel invokes NVIDIAs "
	"cuSPARSE CSR routine.\n"
	"ell: Ellpack format according to Bell and Garland: Efficient Sparse "
	"Matrix-Vector Multiplication on CUDA.\n"
	"sellp: Sliced Ellpack uses a default block size of 32.\n"
	"hybrid: Hybrid uses ell and coo to represent the matrix.\n"
	"hybrid0, hybrid25, hybrid33: Hybrid uses the row distribution to decide "
	"the partition.\n"
	"hybridlimit0, hybridlimit25, hybrid33: Add the upper bound on the ell "
	"part of hybrid0, hybrid25, hybrid33.\n"
	"hybridminstorage: Hybrid uses the minimal storage to store the matrix.");

[tcojean]

@yhmtsai this is done, but I did a little bit more. Here is what is in the last commit. You might also want to check it, @thoasm.

Create CuSPARSE bindings for different precisions.

• Support IndexType int32.
• Support ValueType float, double, complex float and complex double.
• Add a description of the CuSPARSE benchmarks in the spmv file.

[thoasm]

LGTM!
Good job on abstracting the CuSparse calls!

[yhmtsai]

LGTM