-
Notifications
You must be signed in to change notification settings - Fork 984
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Will M2 ultra be supported to use LLM of 100B paras? #381
Comments
My 192GB M2 Ultra should arrive in 1-2 weeks. |
@ggerganov Thrilled to have you kind response! Great thanks in advance 😄~ My first choice for 100+B model is Tigerbot 180B, which is based on bloom architecture, multilingual, and free for commercial use. I've got some ideas regarding family-centered LLM application and really look forward to ggml's update~~ Best, |
that's insane, where do you get such cash? |
GGML has been funded so it makes sense that a wider array (tensor) of hardware will become available. I've been pondering how far the M2 Ultra can be pushed. AMD has some interesting GPU(s) coming out with large amounts of HBM3 memory too. https://www.techpowerup.com/gpu-specs/radeon-instinct-mi300.c4019 Interesting times ahead.. |
This is amazing, Despite it having been already released, it'd hard to find anything about it, including even MSRP or where to buy one. However, assuming you want to be in the merket for one:
It seems to be almost on-par with the 4090, but the 4090 makes annoying compromises that aren't present here. For example the 4090 has much less memory capacity, a far shorter bus witch, a far lower requency and a far lower transfer rate as a result. I don't quite know how this card flew below the all the radars but this looks like en extremely good piece of home-inferencing hardware that may even be capable of training. Now all that's left is... the cost... good luck finding one. If you do, fat chance it'll be the 128gb version, and fat chance it won't cost you thousands of dollars. Sigh. |
@LoganDark Thank you so much for your professional advice. I really need more homework to fully understand it. Yes I heard about the AMD new card. Lately I noticed the llama 2-70B can run on a macbook m2 max 12-core CPU and 64 ram, whose price sounds a bit reasonable (in China it's lower than dual 3090 host )... |
lol. This MI300 card costs more than 20K USD. |
Basically called it lol. Guess it's useless then |
As titled. As apple-lover without strong CS background, I wish master Georgi could work on more projects for apple users to use M2 ultra (maybe 192G) in order to tap into the potential of real LLMs (over 100B parameters). Is it possible in near future? Thanks to any masterminds!
Best,
Sarai
The text was updated successfully, but these errors were encountered: