Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Prevent infinite recursion when DS_ACCELERATOR is set to cuda (micros…
…oft#4962) When DS_ACCELERATOR is overriden to CUDA, `get_accelerator` attempts to check if `is_current_accelerator_supported`. But since that calls `get_accelerator` again and `ds_accelerator` has not been initialized, DeepSpeed runs into infinite recursion. ``` elif is_current_accelerator_supported(): File "/usr/local/lib/python3.8/dist-packages/deepspeed/accelerator/real_accelerator.py", line 48, in is_current_accelerator_supported return get_accelerator().device_name() in SUPPORTED_ACCELERATOR_LIST File "/usr/local/lib/python3.8/dist-packages/deepspeed/accelerator/real_accelerator.py", line 101, in get_accelerator elif is_current_accelerator_supported(): File "/usr/local/lib/python3.8/dist-packages/deepspeed/accelerator/real_accelerator.py", line 48, in is_current_accelerator_supported return get_accelerator().device_name() in SUPPORTED_ACCELERATOR_LIST File "/usr/local/lib/python3.8/dist-packages/deepspeed/accelerator/real_accelerator.py", line 101, in get_accelerator elif is_current_accelerator_supported(): File "/usr/local/lib/python3.8/dist-packages/deepspeed/accelerator/real_accelerator.py", line 48, in is_current_accelerator_supported return get_accelerator().device_name() in SUPPORTED_ACCELERATOR_LIST File "/usr/local/lib/python3.8/dist-packages/deepspeed/accelerator/real_accelerator.py", line 101, in get_accelerator elif is_current_accelerator_supported(): File "/usr/local/lib/python3.8/dist-packages/deepspeed/accelerator/real_accelerator.py", line 48, in is_current_accelerator_supported return get_accelerator().device_name() in SUPPORTED_ACCELERATOR_LIST File "/usr/local/lib/python3.8/dist-packages/deepspeed/accelerator/real_accelerator.py", line 59, in get_accelerator if "DS_ACCELERATOR" in os.environ.keys(): File "/usr/lib/python3.8/_collections_abc.py", line 717, in __contains__ return key in self._mapping File "/usr/lib/python3.8/_collections_abc.py", line 666, in __contains__ self[key] File "/usr/lib/python3.8/os.py", line 672, in __getitem__ value = self._data[self.encodekey(key)] RecursionError: maximum recursion depth exceeded ``` This change fixes that by comparing the accelerator directly with the supported list of accelerators.
- Loading branch information