RuntimeError: cublas runtime error during training

[MariOvO-casual]

Hello,

I have encountered an issue while running the training script with the following error:

RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THC/THCBlas.cu:249

The error seems to occur during the forward pass of the network:

File "/workspace/instagan/models/insta_gan_model.py", line 145, in forward self.fake_B_sng = self.netG_A(self.real_A_sng)

Here is my system configuration:

Operating System: Ubuntu 20.04.5 LTS
PyTorch version: torch==0.4.0. torchvision==0.2.1
CUDA version: release 11.8

I have tried various solutions, but the issue persists. Any help or suggestions on how to resolve this issue would be much appreciated. Thank you in advance!

Best regards,
Mario