RuntimeError: GET was unable to find an engine to execute this computation

[quarterturn]

System Info / 系統信息

declare -x CONDA_DEFAULT_ENV="cog"
declare -x CONDA_EXE="/opt/conda/bin/conda"
declare -x CONDA_PREFIX="/home/alex/.conda/envs/cog"
declare -x CONDA_PREFIX_1="/opt/conda"
declare -x CONDA_PROMPT_MODIFIER="(cog) "
declare -x CONDA_PYTHON_EXE="/opt/conda/bin/python"
declare -x CONDA_SHLVL="2"
declare -x CUDA_HOME="/usr/local/cuda-12.5/"

Python 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import torch
print(torch.version.cuda)
12.4
exit()

Python 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import torch
available_gpus = [torch.cuda.device(i) for i in range(torch.cuda.device_count())]
available_gpus
[<torch.cuda.device object at 0x7f4185909850>, <torch.cuda.device object at 0x7f42c7de8f90>, <torch.cuda.device object at 0x7f42c7de9250>, <torch.cuda.device object at 0x7f42c7de9290>, <torch.cuda.device object at 0x7f42c7de9310>]

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Jun__6_02:18:23_PDT_2024
Cuda compilation tools, release 12.5, V12.5.82
Build cuda_12.5.r12.5/compiler.34385749_0

nvidia-smi
Fri Aug 30 13:36:58 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.06 Driver Version: 555.42.06 CUDA Version: 12.5 |

Information / 问题信息

• The official example scripts / 官方的示例脚本
• My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

python3 cli_demo.py --model_path CogVideoX-5b/ --prompt "A garden comes to life as a kaleidoscope of butterflies flutters amidst the blossoms, their delicate wings casting shadows on the petals below. In the background, a grand fountain cascades water with a gentle splendor, its rhythmic sound providing a soothing backdrop. Beneath the cool shade of a mature tree, a solitary wooden chair invites solitude and reflection, its smooth surface worn by the touch of countless visitors seeking a moment of tranquility in nature's embrace."

Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.19s/it]
Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████| 5/5 [00:08<00:00, 1.79s/it]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [09:04<00:00, 10.90s/it]
Traceback (most recent call last):
File "/home/quarterturn/CogVideo/inference/cli_demo.py", line 106, in
generate_video(
File "/home/quarterturn/CogVideo/inference/cli_demo.py", line 66, in generate_video
video = pipe(
^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/pipelines/cogvideo/pipeline_cogvideox.py", line 735, in call
video = self.decode_latents(latents)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/pipelines/cogvideo/pipeline_cogvideox.py", line 360, in decode_latents
frames = self.vae.decode(latents).sample
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
return method(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 1153, in decode
decoded = self._decode(z).sample
^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 1123, in _decode
z_intermediate = self.decoder(z_intermediate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/accelerate/hooks.py", line 169, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 851, in forward
hidden_states = self.conv_in(sample)
^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/accelerate/hooks.py", line 169, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 144, in forward
output = self.conv(inputs)
^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/accelerate/hooks.py", line 169, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 64, in forward
return super().forward(input)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 608, in forward
return self._conv_forward(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 603, in _conv_forward
return F.conv3d(
^^^^^^^^^
RuntimeError: GET was unable to find an engine to execute this computation

Expected behavior / 期待表现

Should produce video.

[zRzRzRzRzRzRzR]

are cuda_12.5 compile with torch? abd what is your GPU