OSError: [Errno 12] Cannot allocate memory

[Deeachain]

when i train cityscapes with single GPU 1080ti,encounter this error when iters=4000 and iters==8000
OSError: [Errno 12] Cannot allocate memory
I think should decrease Bath_size.So if i need to change 1img/perGPU ,how to do that? And, how to set total Epoch instead of total_iters = 40000

[xvjiarui]

Hi @Deeachain
May I know, at which stage OSError was raised?
Currently, we don't support setting the total epoch. You may convert from epochs to iterations by yourself.

[Deeachain]

Hi @xvjiarui
The error OSError occurred in the second time Validation just iter==8000;The cause of that error may be out of the GPU memory.
So i try to reduce the parameters samples_per_gpu==1 in config,but encounter a new error ，Can't set bath size==1？

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 512, 1, 1])

[xvjiarui]

I think it dues to inefficient CPU memory.
To training with sample_per_gpu=1, you may switch to MMSyncBN. You may refer to here for detailed docs.

[Deeachain]

I have solved this problem OSError, just add os swap memory.