Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Put Neo-X 20B weights on BitTorrent #542

Closed
depthwise opened this issue Feb 11, 2022 · 3 comments
Closed

Put Neo-X 20B weights on BitTorrent #542

depthwise opened this issue Feb 11, 2022 · 3 comments
Labels
feature request New feature or request

Comments

@depthwise
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Currently, GPT Neo-X 20B weights take a loooong time to download even on a reasonably fast 200mbps fiber connection.

Describe the solution you'd like
Could the authors please share them through Academic Torrents or as a direct Torrent download?

Describe alternatives you've considered
An alternative would be to maybe cache them through Cloudflare/Akamai I guess, but that's probably more expensive and less scalable for the purpose.

Additional context
Might be a good idea for other large downloads as well.

@depthwise depthwise added the feature request New feature or request label Feb 11, 2022
@StellaAthena
Copy link
Member

How long does it take to download the weights on a 200 megabits per second internet connection? And are we talking about the full weights or the slim weights?

@depthwise
Copy link
Contributor Author

depthwise commented Feb 12, 2022

To be clear, it's not realizing the full bandwidth of the connection. I haven't timed it, but wget command specified for slim weights showed something like 5MiB/sec earlier today. The total download size is 39GiB. so that's roughly two hours. This is on a connection that normally can pull down at least 20MiB/sec. Here's the command I used to download:

wget --cut-dirs=5 -nH -r --no-parent --reject "index.html*" https://mystic.the-eye.eu/public/AI/models/GPT-NeoX-20B/slim_weights/ -P 20B_checkpoints

@sdtblck
Copy link
Contributor

sdtblck commented Feb 12, 2022

@depthwise there's a torrent for both the slim weights and the full weights: here and here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants