-
Notifications
You must be signed in to change notification settings - Fork 981
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Put Neo-X 20B weights on BitTorrent #542
Comments
How long does it take to download the weights on a 200 megabits per second internet connection? And are we talking about the full weights or the slim weights? |
To be clear, it's not realizing the full bandwidth of the connection. I haven't timed it, but wget command specified for slim weights showed something like 5MiB/sec earlier today. The total download size is 39GiB. so that's roughly two hours. This is on a connection that normally can pull down at least 20MiB/sec. Here's the command I used to download: wget --cut-dirs=5 -nH -r --no-parent --reject "index.html*" https://mystic.the-eye.eu/public/AI/models/GPT-NeoX-20B/slim_weights/ -P 20B_checkpoints |
@depthwise there's a torrent for both the slim weights and the full weights: here and here |
Is your feature request related to a problem? Please describe.
Currently, GPT Neo-X 20B weights take a loooong time to download even on a reasonably fast 200mbps fiber connection.
Describe the solution you'd like
Could the authors please share them through Academic Torrents or as a direct Torrent download?
Describe alternatives you've considered
An alternative would be to maybe cache them through Cloudflare/Akamai I guess, but that's probably more expensive and less scalable for the purpose.
Additional context
Might be a good idea for other large downloads as well.
The text was updated successfully, but these errors were encountered: