train shows an unexpected exception in best_download no explanation given

[nevakrien]

hey I am trying to run this repo on a big linux computer that has 8 h100 and is just sitting idol so timely help would be very appreciated.
as a test I am ruining things on my ubuntu mostly to see it runs but I got this weird error that seems like it is potentially a bug since it is an uncaught exception with no explanation

so this is probably me not setting up the environment correctly I tried pip installing it and it didn't work so I went and used anaconda for some and pip for the rest.

tokenizer worked fine then I ran the train and got:

(neox_toolkit) user@user-System-Product-Name:~/Desktop/coder reaserch/gpt-neox$ python train.py tester_config.yml tester_setup.yml
Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):
  File "train.py", line 20, in <module>
    from megatron.training import pretrain
  File "/home/user/Desktop/coder reaserch/gpt-neox/megatron/training.py", line 58, in <module>
    from eval_tasks import run_eval_harness
  File "/home/user/Desktop/coder reaserch/gpt-neox/eval_tasks/__init__.py", line 15, in <module>
    from .eval_adapter import EvalHarnessAdapter, run_eval_harness
  File "/home/user/Desktop/coder reaserch/gpt-neox/eval_tasks/eval_adapter.py", line 16, in <module>
    import best_download
  File "/home/user/anaconda3/envs/neox_toolkit/lib/python3.8/site-packages/best_download/__init__.py", line 35, in <module>
    retry_strategy = Retry(
TypeError: __init__() got an unexpected keyword argument 'method_whitelist'
(neox_toolkit) user@user-System-Product-Name:~/Desktop/coder reaserch/gpt-neox$ 

note that I did went ahead and tried literally all the available best_download version and got the same error, tokenizer itself was from hugging face polycoder and I pased it like this

python tools/preprocess_data.py --input ./tester.jsonl --tokenizer-type GPT2BPETokenizer --vocab-file ./vocab.json --merge-file ./merges.txt --output-prefix ./output 

environment was made haphazardly and its just a mockup to see I can run things. if this is a known environment bug I would love for an explanation of whats wrong so when I set it up on the real machine it would stick

[nevakrien]

apparently urllib3 needs to be a specific version and thats not shown in the requirments

[StellaAthena]

apparently urllib3 needs to be a specific version and thats not shown in the requirments

Thanks for catching this! Feel free to fork the repo and open a PR with the change :)

[mariebiscuit]

This PR in the best_downloads repo would fix this issue by updating the deprecated argument to its successor but it hasn't been merged yet. Specifying a backdated version of urllib3 in requirements.txt is not an ideal fix because a newer version of urllib3 is required by wandb.

[StellaAthena]

This PR in the best_downloads repo would fix this issue by updating the deprecated argument to its successor but it hasn't been merged yet. Specifying a backdated version of urllib3 in requirements.txt is not an ideal fix because a newer version of urllib3 is required by wandb.

Gotcha. I’ve reminded them to sign the CLA and if they don’t do so by the end of the week I’ll resubmit the same fix.

[StellaAthena]

The above PR has been merged. @nevakrien can you confirm that this solves your problem?

[mariebiscuit]

The update hasn't been pushed to PyPl but if you reinstall best_download via
pip install git+https://github.com/EleutherAI/best-download
it should work

[nevakrien]

The above PR has been merged. @nevakrien can you confirm that this solves your problem?

I am no longer working with the repo (my research group changed hardware to intel xpus)
what I can confirm is that I got it working by installing best downloads back when I tried to use this repo.
just like @mariebiscuit suggested.

its a pretty cut-and-dry case since the dependency can be tracked back to the deepspeed fork.