ImportError: undefined symbol: __cudaPopCallConfiguration

[rajarshighoshal]

I'm new to deep learning stuff and I walk through the readme till the Execute step. Where I'm getting this error. Can anyone please guide me what I'm doing wrong?

[pfjaeger]

See here #41 . At the moment you have to downgrade to torch=0.4.1 . Sorry for that.

[rajarshighoshal]

I am using pytorch version 0.4.1 still getting this error!

[alexfoo-dw]

@rajarshighoshal

I downgraded from CUDA 10.0 to 9.0 and ran these commands from the medicaldetectiontoolkit directory again after downgrading to torch 0.4.1. Fixed my import error.

cd cuda_functions/nms_xD/src/cuda/
nvcc -c -o nms_kernel.cu.o nms_kernel.cu -x cu -Xcompiler -fPIC -arch=[arch]
cd ../../
python build.py
cd ../

cd cuda_functions/roi_align_xD/roi_align/src/cuda/
nvcc -c -o crop_and_resize_kernel.cu.o crop_and_resize_kernel.cu -x cu -Xcompiler -fPIC -arch=[arch]
cd ../../
python build.py
cd ../../

[rajarshighoshal]

So it's working for cuda 9? Ok I will try then.

…

On Thu 6 Jun, 2019, 1:44 PM alexfoodw, ***@***.***> wrote: @rajarshighoshal <https://github.com/rajarshighoshal> I downgraded from CUDA 10.0 to 9.0 and ran these commands from the medicaldetectiontoolkit directory again after downgrading to torch 0.4.1. Fixed my import error. cd cuda_functions/nms_xD/src/cuda/ nvcc -c -o nms_kernel.cu.o nms_kernel.cu -x cu -Xcompiler -fPIC -arch=[arch] cd ../../ python build.py cd ../ cd cuda_functions/roi_align_xD/roi_align/src/cuda/ nvcc -c -o crop_and_resize_kernel.cu.o crop_and_resize_kernel.cu -x cu -Xcompiler -fPIC -arch=[arch] cd ../../ python build.py cd ../../ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFIPNO6VHYRKSQ5W6MYULMDPZDBMXANCNFSM4HSCWNJA> .

[sophie-isobel]

@rajarshighoshal did changing to cuda 9 work for you?

I am getting a similar Import Error: undefined symbol: __cudaRegisterFatBinaryEnd

!nvidia-smi returns that I have cuda version 10 and tesla K80 GPU (note i am using google colab).

I have specified arch='sm_37' for the K80 gpu and although I haven't downgraded the cuda version from 10 to 9 as @alexfoodw suggested, I can see that the torch.version.cuda is '9.0.176' for the torch ==0.4.1 I have installed, so the torch should be using cuda 9.

Would you know why I am having this issue?

[rajarshighoshal]

No, I tried after changing to cuda 9. Not working! I'm thinking about trying with cuda 8! 😅

…

On Wed 14 Aug, 2019, 5:59 PM sophie-isobel, ***@***.***> wrote: @rajarshighoshal <https://github.com/rajarshighoshal> did changing to cuda 9 work for you? I am getting a similar Import Error: undefined symbol: __cudaRegisterFatBinaryEnd !nvidia-smi returns that I have cuda version 10 and tesla K80 GPU (note i am using google colab). I have specified arch='sm_37' for the K80 gpu and although I haven't downgraded the cuda version from 10 to 9 as @alexfoodw <https://github.com/alexfoodw> suggested, I can see that the torch.version.cuda is '9.0.176' for the torch ==0.4.1 I have installed, so the torch should be using cuda 9. Would you know why I am having this issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFIPNOYUFPV26BKTO6EIZQDQEP3DBANCNFSM4HSCWNJA> .

[sophie-isobel]

@rajarshighoshal so I changed to cuda 9 and also changed the default gcc version using the following code, and I am no longer getting this error.

!update-alternatives --remove-all gcc 
!update-alternatives --remove-all g++

!update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 20
!update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 20

!update-alternatives --install /usr/bin/cc cc /usr/bin/gcc 30
!update-alternatives --set cc /usr/bin/gcc

!update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++ 30
!update-alternatives --set c++ /usr/bin/g++

[rajarshighoshal]

Oh! Okkk!!! Thanks a lot! I will try this!

…

On Wed 14 Aug, 2019, 8:11 PM sophie-isobel, ***@***.***> wrote: @rajarshighoshal <https://github.com/rajarshighoshal> so I changed to cuda 9 and also changed the default gcc version using the following code, and I am no longer getting this error. !update-alternatives --remove-all gcc !update-alternatives --remove-all g++ !update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 20 !update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 20 !update-alternatives --install /usr/bin/cc cc /usr/bin/gcc 30 !update-alternatives --set cc /usr/bin/gcc !update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++ 30 !update-alternatives --set c++ /usr/bin/g++ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFIPNOYIF7PLFNNFACUJBYDQEQKSFANCNFSM4HSCWNJA> .

[heyitsguay]

Does this mean medicaldetectiontoolkit will not work with CUDA 10.2 and g++ 7.5.0 with a GTX 1080 (sm_61)? I'm getting the same "undefined symbol: __cudaRegisterFatBinaryEnd" error running python exec.py --mode train on the included toy experiment.

Update: Switched to the torch1x branch and got it working.

[henbucuoshanghai]

pytorch is 1.1？ 1.4？@h @heyitsguay