Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch) #57

Open
tula-liuxin opened this issue Jan 2, 2024 · 8 comments
Assignees
Labels
bug Something isn't working

Comments

@tula-liuxin
Copy link

legged_gym) leo@leo-pc:~/legged_gym$ python legged_gym/scripts/train.py --task=anymal_c_flat
Importing module 'gym_38' (/home/leo/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/leo/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.10.0+cu113
Device count 1
/home/leo/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/leo/.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Emitting ninja build file /home/leo/.cache/torch_extensions/py38_cu113/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
Setting seed: 1
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
/home/leo/miniforge3/envs/legged_gym/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Traceback (most recent call last):
File "legged_gym/scripts/train.py", line 47, in
train(args)
File "legged_gym/scripts/train.py", line 41, in train
env, env_cfg = task_registry.make_env(name=args.task, args=args)
File "/home/leo/legged_gym/legged_gym/utils/task_registry.py", line 97, in make_env
env = task_class( cfg=env_cfg,
File "/home/leo/legged_gym/legged_gym/envs/anymal_c/anymal.py", line 49, in init
super().init(cfg, sim_params, physics_engine, sim_device, headless)
File "/home/leo/legged_gym/legged_gym/envs/base/legged_robot.py", line 71, in init
super().init(self.cfg, sim_params, physics_engine, sim_device, headless)
File "/home/leo/legged_gym/legged_gym/envs/base/base_task.py", line 84, in init
self.create_sim()
File "/home/leo/legged_gym/legged_gym/envs/base/legged_robot.py", line 244, in create_sim
self._create_envs()
File "/home/leo/legged_gym/legged_gym/envs/base/legged_robot.py", line 675, in _create_envs
pos[:2] += torch_rand_float(-1., 1., (2,1), device=self.device).squeeze(1)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

@tula-liuxin tula-liuxin added the bug Something isn't working label Jan 2, 2024
@Cdfghglz
Copy link

Cdfghglz commented Jan 2, 2024

Your gpu may be too new, not supporting cuda 11.3 - this was my issue with a 4060 recently.

@Ke-Wang1017
Copy link

Actually newer version of Pytorch and torchvision is working fine for me. I suggest you check your Cuda version with nvidia-smi and install the corresponding version. For me the cuda is 12.1 and I installed the newest pytorch1

@tula-liuxin
Copy link
Author

python legged_gym/scripts/train.py --task=anymal_c_flat

Thank you, it helps! I didn't know the change the pytorch version given my cuda version, and it works well now.

@tula-liuxin
Copy link
Author

Your gpu may be too new, not supporting cuda 11.3 - this was my issue with a 4060 recently.

Thank your comments! I found it's about the imcompatibility between the pytorch and cuda.

@tula-liuxin
Copy link
Author

Solved by:
pip3 uninstall torch torchvision torchaudio
pip3 install torch torchvision torchaudio -f https://download.pytorch.org/whl/cu123/torch_stable.html #My cuda version is 12.3

@KevinGrey
Copy link

cuda 12.0
RTX4090
ubuntu22.04
python3.7.12
torch 1.8
then
I uninstalled torch
pip uninstall torch torchvision torchaudio
and
installed torch 1.13
fixed

@Andy-xiong6
Copy link

cuda 12.2
RTX4060 TI
ubuntu20.04
python3.8
torch 1.10
then
I uninstalled torch
pip uninstall torch torchvision torchaudio
and
installed torch 2.3.0+cu121
fixed

@asibatian
Copy link

RTX4060 Ti
Ubuntu20.04
Python3.8
At first, I installed cuda11.3 and install torch using pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html (This version I works well on RTX3090)
It failed.
Then, I uninstall cuda 11.3 and install cuda12.1
uninstall torch using pip uninstall torch torchvision torchaudio
install torch using pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
torch.__version__: 2.3.1+cu121
It works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

7 participants