-
Notifications
You must be signed in to change notification settings - Fork 984
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gpt-j, starcoder, gptneox examples cause "not enough space in the context's memory pool" for batches >32 #484
Comments
The As far as I know, currently the only working example of how to use it is in |
Yes, currently only Probably it's best to start with |
Hi all - I'm working on an issue where users on M1 apple silicon get
ggml_new_tensor_impl: not enough space in the context's memory pool
when they try to use starcoder or gptneox models from turbopilot (ravenscroftj/turbopilot#47).I've managed to get my hands on a mac mini for a few days and I can see that the
mem_per_token
calculation is coming out a long way short of the real memory requirements for at least a few of the example program flavours (I tried gpt-j, gpt-neox and starcoder). This problem occurs if the batch size is big enough that accelerated matrix operations kick in (it's batch sizes of 32 I seem to recall?)I get similar issues when I compile with
-DGGML_OPENBLAS=ON
on an intel machine.Taking the gpt-neox example to illustrate the problem:
gpt_neox_eval
with 4 dummy tokens andmem_per_token = 0
I getused_mem = 1417200
so thereforemem_per_token
is set to1417200 / 4 = 354300
-b 512
, by my calculation we need512*354300=181401600
(~173MB) of bufferggml_new_object: not enough space in the context's memory pool (needed 609977808, available 536870912)
This would seem to imply that the real/actual value of mem_per_token should be
609977808 / 512 = 1191362.90625
rather than354300
bytes I guess?The only fix I've been able to identify is to multiply the mem_per_token by some fixed amount per model. However, I'm sure there must be a nicer way to do this?
I'm guessing that ggml-alloc might help but I've not quite got my head around how to use this yet. @slaren's suggestion here to update the examples to use alloc would definitely be appreciated! Especially if it helps resolve this problem.
Additional Weirdness/Partial Solution
I had a brainwave and tried running the initial call to gpt_neox_eval with 64 dummy tokens to see if that caused additional overheads related to the BLAS/Accelerate stuff to magically fall out and it actually worked when the buffer is set to 512MB.
When I set the buffer back to 256MB I get:
I feel like I'm really close to figuring this thing out but I've spent a few hours and my brain is starting to melt now.
Thanks in advance for any help guys.
The text was updated successfully, but these errors were encountered: