-
Notifications
You must be signed in to change notification settings - Fork 9.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ggml : add WebGPU backend #7773
Comments
Hi! I'm interested in bringing this backend to GGML and was wondering if there are any startup materials available for newcomers to quickly ramp up and start working on this backend? |
So I've been playing with implementation of webgpu for a few days. I got a very minimal version with working buffer management and support for some simple ops. My version is based on ggerganov/ggml#585 but with some noticeable changes:
However, I'm not very familiar with ggml backend interface so I'm having a question: I made a test cgraph to test my implementation: https://github.com/ngxson/ggml_webgpu_dev/blob/a5fcc25c359b997869b8683ab485d1d3f96b37f9/main.cpp#L70 When calling
Here is my @ggerganov Could you help me understand this part? Thank you. |
If every tensor used in the graph needed to be allocated separately, the compute buffer would be several gigabytes even for the simplest models. The point of ggml-alloc is to minimize the size of the compute buffer by allocating tensors in the same memory locations when possible based on the order of evaluation of the graph. So this behavior is completely expected. I don't understand what you are trying to do with |
@slaren Thanks for the explanation. So apparently I'm now running into another issue that both src and dest of
I'm not sure how other backends handle this (and also the |
ggml-alloc can make some operations automatically inplace if it determines that it is safe to do so, to save memory. Other backends do not need to do anything special in this case, they just pass the same pointer for both the destination and src. I am not sure why this is a problem for webGPU, in the worst case it might require making a different version of the kernels for inplace operations, but there is probably some workaround possible. |
I hope that this would be relatively easy to do since AFAIK WebGPU allows us to write kernels in a shader language, so we have experience how to create such backends.
There has been some initial work in ggerganov/ggml#585 - could be useful as a starting point
The text was updated successfully, but these errors were encountered: