-
Notifications
You must be signed in to change notification settings - Fork 994
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Quantizing GPT-J Produces Nonsense #71
Comments
I think I might not know how to communicate what I am trying to say correctly, but it looks like the output of transformer.h.0.mlp.fc_in.bias might need their outputs in the opposite order the comment Line 124 in 0bac483
and the rest of the model structure indicate the model would plausibly be described as [rows,columns] but maybe I'm wrong or confused EDIT: definitely wrong and confused. |
Hi @RaymondCrandall @zanussbaum @ggerganov I think I have figured out this issue, the f16 to f32 tables were not properly initialized in the quantize examples. This can be fixed by adding this code to main() in quanitize.cpp
Please refer to my PR #77 |
This was integrated and can be closed, yes? |
It should be. I am already using it in my fork with correct results, it looks like the quantization works. If you see the histogram outputs during quantization, you can tell if they have a bunch of different numbers, then it should be correct. |
Hey thanks for the great package! When I try to quantize an fp16 ggml file of GPT-J, the outputs from chat are nonsense. Also the outputs of the
gpt-j-quantize
bin seem to be off as I'd expect the hist to have non-zero values (as seen in other examples likellama.cpp
quantize)Is quantizing from fp16 not possible for GPT-J?
The text was updated successfully, but these errors were encountered: