Installation error: pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

[pappagari]

Describe the bug

Installing Data curator (from both PyPi and source) is not working. The error is

ERROR: Exception:                                                                                                                                      
Traceback (most recent call last):                                                                                                                     
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper                                
    status = _inner_run()                                                                                                                              
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run                                   
    return self.run(options, args)                                                                                                                     
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper                                                                                                                                                                                               
    return func(self, options, args)                                                                                                                                                                                                                                                                           
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 379, in run                                         
    requirement_set = resolver.resolve(                                                                                                                
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve                        
    result = self._result = resolver.resolve(                                                                                                          
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve                                   
    state = resolution.resolve(requirements, max_rounds=max_rounds)                                                                                    
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve                                   
    raise ResolutionTooDeep(max_rounds)                                                                                                                
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000    

Steps/Code to reproduce bug

First I created a separate conda env for this package

conda create -n nemo python=3.10
conda activate nemo

Then I installed cython (which worked fine) using

pip install cython

Then all of the following methods throws the error

Installation from PyPi, CPU-only --> pip install nemo-curator 
Installation from PyPi, CPU and CUDA-accelerated modules --> pip install --extra-index-url https://pypi.nvidia.com nemo-curator[cuda12x]
Installation from source, CPU and CUDA-accelerated modules --> pip install --extra-index-url https://pypi.nvidia.com ".[cuda12x]"

Expected behavior

Expected the commands to just display some kind of successful installation message.

Environment details

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:

• OS version -- Ubuntu 22.04.5 LTS (GNU/Linux 6.8.0-1015-aws x86_64)
• Python version -- 3.10.15
• pip version -- 24.2

[ayushdg]

Thanks for raising @pappagari . A recent spacy update (v3.8) created some dependency conflicts leading to these failures.

Since this update impacts our pypi releases, we're working on updating our pip packages with the fix in place.

To temporarily unblock you I have a couple of suggestions:

1. From source: We've fixed this in main with Pin to spacy<3.8 temporarily to unblock CI #276, so if you try installing from source with the latest code from main, the Installation from source approach should work for you.
2. From pypi: As a temporary workaround I would recommend the following:

   pip install cython
   pip install "spacy<3.8"
   pip install nemo-curator

Ensuring that spacy<3.8 is installed prior to Nemo curator

[pappagari]

It is working now. Thanks!

[ayushdg]

Thanks for confirming. We've also released 0.4.1 to address this issue so you would no longer need to install spacy<3.8 manually.