-
Notifications
You must be signed in to change notification settings - Fork 526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RuntimeError: CUDA error: an illegal memory access was encountered when evaluate the beyond bounding-box ? #190
Comments
Please add |
the same error occurred in this setting. File "E:/Experiment/目标检测实验/MMDet/OBB_Detectors/test.py", line 87, in predict |
This error is caused by incorrect input of min_area_polygons. Could you print the |
the first pts are shown here: and the second pts are shown here: I tested their performance in the function of min_area_polygons and found that the second pts will caused this error. |
It looks like a bug in |
I met the same error when testing rotate reppoint model. |
it seems a bug。。。 |
I also have same error |
I have the same problem, is there a solution? |
A successful solution: set smaller
|
@yangxue0827 That change did not solve the issue for me as mentioned in #405 (comment) |
I also have the same problem after change nms_pre. |
same error as mine, I have tried on Tesla V100-PCIE 32GB and RTX3090 26G
|
I want to know if this bug is solved? |
|
I also have the same problem after change nms_pre. I found this error when I was testing a certain image, but not other images. I want to know if this bug is solved? |
Same here. |
The same problem.
TorchVision: 0.9.1+cu102 File "./tools/train.py", line 192, in When training in epoch=50, this error is uncomfortable |
I wonder that have the this bug been solved? |
Describe the bug
I am trying to train and test the clf model which using the configs of rotated_reppoints_r50_fpn_1x_dota_oc, it is successed to finished the train process, but an error occured during the evaluation, here the log is:
File "E:/Experiment/目标检测实验/MMDet/OBB_Detectors/test.py", line 87, in predict
outputs = single_gpu_test(model, data_loader, args.show, args.show_dir, args.show_score_thr)
File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmdet\apis\test.py", line 31, in single_gpu_test
result = model(return_loss=False, rescale=True, **data)
File "E:\Anaconda\envs\mmdet2\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmcv\parallel\data_parallel.py", line 50, in forward
return super().forward(*inputs, **kwargs)
File "E:\Anaconda\envs\mmdet2\lib\site-packages\torch\nn\parallel\data_parallel.py", line 165, in forward
return self.module(*inputs[0], **kwargs[0])
File "E:\Anaconda\envs\mmdet2\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmcv\runner\fp16_utils.py", line 109, in new_func
return old_func(*args, **kwargs)
File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmdet\models\detectors\base.py", line 174, in forward
return self.forward_test(img, img_metas, **kwargs)
File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmdet\models\detectors\base.py", line 147, in forward_test
return self.simple_test(imgs[0], img_metas[0], **kwargs)
File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmrotate\models\detectors\single_stage.py", line 100, in simple_test
bbox_list = self.bbox_head.get_bboxes(*outs, img_metas, rescale=rescale)
File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmcv\runner\fp16_utils.py", line 197, in new_func
return old_func(*args, **kwargs)
File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmrotate\models\dense_heads\rotated_reppoints_head.py", line 1044, in get_bboxes
results = self._get_bboxes_single(cls_score_list, point_pred_list,
File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmrotate\models\dense_heads\rotated_reppoints_head.py", line 1136, in _get_bboxes_single
mlvl_bboxes[..., :4] /= mlvl_bboxes[..., :4].new_tensor(scale_factor)
RuntimeError: CUDA error: an illegal memory access was encountered
Details
I have tired to find reason for this error, and find this is caused a specific data.
Specifically, a function that named min_area_polygons in rotated_reppoints_head.py cannot handle the variables pts in the second iteration of processing.
The num_class in the model config is changed to 16 in my test to fit the DOTA V1.5
The data that caused this error is uploaded to baidu cloud driver if you need to test it:
链接:https://pan.baidu.com/s/1CCnthEl-kzOIXfJnU-PxRQ?pwd=l13h
提取码:l13h
DOTA_V1.5 splited with 1024 size and 500 gaps
Environment
Other images can be predict correctly and the results of mmrotate/utils/collect_env.py are shown here:
sys.platform: win32
Python: 3.8.12 (default, Oct 12 2021, 03:01:40) [MSC v.1916 64 bit (AMD64)]
CUDA available: True
GPU 0: GeForce GTX 1660 Ti
CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2
NVCC: Cuda compilation tools, release 10.2, V10.2.89
GCC: n/a
PyTorch: 1.8.1
PyTorch compiling details: PyTorch built with:
C++ Version: 199711
MSVC 192829913
Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
OpenMP 2019
CPU capability usage: AVX2
CUDA Runtime 10.2
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
CuDNN 7.6.5
Magma 2.5.4
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -DNDEBUG -DUSE_FBGEMM -DUSE_XNNPACK, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON,
TorchVision: 0.9.1
OpenCV: 4.5.5
MMCV: 1.4.5
MMCV Compiler: MSVC 192930136
MMCV CUDA Compiler: 10.2
MMRotate: 0.1.1+
The text was updated successfully, but these errors were encountered: