-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support vec_dot_q and dequantize_row_q for GGML_TYPE_Q8_1 #200
Comments
closing because this type does not seem intended / useful for inference (RWKV/rwkv.cpp#74 (comment)) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I had a crack at implementing a quantizer for this (as part of RWKV/rwkv.cpp#74) in order to try to run rwkv.cpp with it, but it looks like vec_dot_q at the very least is not implemented, resulting in a NULL pointer dereference at runtime. AKA inference isn't fully supported with this type.
Since there are a couple TODO comments implying that these might be implementable in the future, and I couldn't find any other tracking issues mentioning this, I'm opening this issue to show that at least someone cares.
I'm not well-versed in SIMD intrinsics or the inner workings of this library so I can't implement these operators atm.
The text was updated successfully, but these errors were encountered: