memory access fault when linking dynamic lib that uses hipcub

[ntrost57]

I am running into a memory access fault when linking to a dynamic library that uses hipcub if I use hipcub also in the main program (that links to the package that uses hipcub).
Please see the following minimal example hipcub_test.tar.gz

Edit: This does not happen when using rocprim or cub instead of hipcub.

[jszuppe]

I'll look into it.

[jszuppe]

Does the same thing happen with rocPRIM?

[ntrost57]

No, does not happen with rocPRIM.

[jszuppe]

We identified the problem: for some reasons in said configuration (linking to a dynamic library that uses hipcub if I use hipcub also in the main program) every kernel is run twice by HIP framework. Some kernels are resistant to that, some are not (access errors). We are not sure why it happens only for hipCUB.

Right now we are thinking that probably in here should be a break after hipModuleLaunchKernel().

[jszuppe]

Here are HIP API traces for correct and incorrect (with bug) execution, where we can clearly the difference.

bug_hiptrace.txt
correct_hiptrace.txt

[jszuppe]

I guess we should open issue in HIP repo.

[jszuppe]

Should be fixed by ROCm/HIP#575. I don't know which ROCm release will include that.