-
Notifications
You must be signed in to change notification settings - Fork 995
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ggml : improve memory allocation for weights and similar lists of tensors #578
Comments
Yes, we should consolidate the different ways of allocating memory.
Either this, or even just a function that returns the required memory for a context by doing a similar loop as the one in ggerganov/llama.cpp#3605 would be helpful. |
I consider this fixed with |
There are several patterns used to allocate memory for a list of fixed size tensor, such as model weights:
no-alloc
context, adding them to list or map, or obtaining them by name from aggml_context
withggml_get_tensor
, summing their sizes and finally allocating them (the last one isno-alloc
context, allocate the weights manually withggml-alloc
, first with a measure allocator and then again with the exact memory requirements (current llama.cppfinetune
)no-alloc
context, then enumerating the tensors in the context and summing their sizes (newfinetune
in ggml : add context enumeration functions llama.cpp#3605)ggml_context
with a lot of memory and hope for the bestThis becomes significantly more complicated when the weights have to be split between different backends (current llama.cpp and
ggml-backend
wip).For something so basic, this is a lot more complicated than it should, and we should have a normalized way to do this. At the most basic level, it could be simply a function to automatically allocate all the tensors created in a
no-alloc
context with the exact memory requirements. Support for multiple backends will be more complicated.This could also be useful for debugging operations in compute contexts, where it might be desirable to allocate memory for every tensor in the graph to be able to inspect the results of each op later.
The text was updated successfully, but these errors were encountered: