device-side assert triggered

[ubuntu-achieve]

Hi, When I try to use the command python demo.py --model resnet_18 --dataset voc2010_crop --label_name bird to reproduce the results of the paper, I get the error device-side assert triggered.
And this problem occurs steadily on the 13th batch of the 6th epoch. After debugging, I found that the problem occurred with the previous parameter update, which caused one of the values in mask1[0].mask_weight to be nan, which caused all subsequent values to be nan and led to the error.

But the strange thing is that the first few times the parameters were updated very normally, how can I solve this problem. 😭

[WindymanJOX]

感觉像是梯度消失，我在第5轮的时候出现了这个错误