magma_queue_create_from_cuda_internal: Assertion `queue->dBarray__ != __null' failed

[josephrocca]

There was a follow-on problem after solving #1, but I wanted to create a separate issue for this because it's likely unrelated. After running the below commands (as described in that previous issue):

docker build -t kcosta42-vqgan-clip .
docker run --gpus all -v $PWD/models:/models -v $PWD/configs:/configs -v $PWD/core:/app/core -v $PWD/scripts:/app/scripts -v $PWD/outputs:/outputs -e DEVICE=cuda -it kcosta42-vqgan-clip python -m scripts.generate -c /configs/docker.json

it gets to iteration 2 and then crashes:

  1%|█                                                                                                                                      | 2/250 [00:01<04:08,  1.00s/it]
python: /opt/conda/conda-bld/magma-cuda111_1605822518874/work/interface_cuda/interface.cpp:899: void magma_queue_create_from_cuda_internal(magma_device_t, cudaStream_t, cublasHandle_t, cusparseHandle_t, magma_queue**, const char*, const char*, int): Assertion `queue->dBarray__ != __null' failed.

My GPU is a RTX 2070 with 8GB of memory. I haven't changed any config, so it's still set to the default 256x256 size. I'm still in the process of investigating this - I'll post an update here if I solve it.

[josephrocca]

Seems like it was some sort of down-stream result of running out of memory. On repeated attempts of that same command I either get a crash like that, or a "CUDA out of memory" crash. I think this is because, although my GPU has 8GB of memory, the system/OS is using 1GB, so there's really only 7GB available.

The maximum my machine can handle is 192x192 pixels.

Thanks for your work on this!