-
Notifications
You must be signed in to change notification settings - Fork 995
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Some Issues with mul_mat (at least on small matrices) #478
Comments
Some operations require that the tensors are contiguous in memory. You can use auto f = ggml_transpose(ctx, ggml_mul_mat(ctx, t, ggml_cont(ctx, ggml_transpose(ctx, t2)))); |
The suggestion by @slaren should give you the expected answer. To try to explain this better, lets say you want to multiply the matrices The oddity about
In other words:
You can also check the test-mul-mat0.c example - it might be useful to understand what is going on. |
Is there any particular rational for this design? Also if that formula you posted was added as comment over mul_mat it would have saved me 10 hours of experimentation! |
When I started this project I didn't have a good idea about what is typically done in other frameworks. I mainly wanted to perform the matrix multiplication using "row-by-row" dot products because I thought it is cache-friendly this way. The result is transposed because this way it is already in the correct memory order to be passed as the second argument to another matrix multiplication: z_0 = ggml_mul_mat(ctx0, x_0, y);
z_1 = ggml_mul_mat(ctx0, x_1, z_0);
... This is a common pattern in ML - the Obviously, this has advantages and drawbacks. But it was easier to think about the dot products and not worry about trashing the L1 cache when implementing a naive matrix multiplication. Docs are in a very bad state at the moment - hopefully we will update those some time soon. |
Let me begin by apologizing if I have missed something incredibly basic, but I have tried to come at this problem from a multitude of angles. Copying the example of how to use the library out of the documentation at the top of ggml.h and then expanding it to one of the simplest matrix multiplication problems possible might lead to some code which looks something like this:
Which prints
46, 118, 190, 28, 73, 118, 10, 28, 46
but WolframAlpha says that {{1, 2, 3}, {4, 5, 6}, {7, 8, 9}} * {{9, 8, 7}, {6, 5, 4}, {3, 2, 1}} should result in {{30, 24, 18}, {84, 69, 54}, {138, 114, 90}} and no matter how you order the results in memory the two results aren't even intersecting sets!I must be handling my data wrong! Replace mul_mat with mul and the result is
9, 16, 21, 24, 25, 24, 21, 16, 9,
WolframAlpha? {{9, 16, 21}, {24, 25, 24}, {21, 16, 9}} so my data is correct!Try a simpler problem... multiply by the identity (t2 = {1, 0, 0, 0, 1, 0, 0, 0, 1}), and the result is
1, 4, 7, 2, 5, 8, 3, 6, 9,
which looks close to correct if tensors were stored in column major order... the documentation for mul_mat mentions that the second tensor is transposed internally is this false? Do we need to transpose it our self?Running t2Data through a custom row major to column major function:
before copying into t2's data we get:
30, 84, 138, 24, 69, 114, 18, 54, 90,
and1, 4, 7, 2, 5, 8, 3, 6, 9,
which are the correct column major results for each computation. Perfect so we just need to transpose t2 and the result:auto f = ggml_transpose(ctx, ggml_mul_mat(ctx, t, ggml_transpose(ctx, t2)));
Which when run gives us
GGML_ASSERT: ggml.c:11058: nb10 == sizeof(float) Aborted (core dumped)
There are asserts ensuring that neither input nor the output are permuted! Disable the check to only run the OpenBLAS path on large matrices and try again on raw row major data... same results. Try again on a 2x2 matrix with t = {{2, -1}, {1, 3}} and t2 = {{1, 2}, {3, 4}}, WolframAlpha gives us {{-1, 0}, {10, 14}}, ggml gives us
0, 7, 2, 15,
"manually" transposing t2 it gives us-1, 10, 0, 14,
I am trying to figure out how whisper.cpp is implemented with these weird transposing rules, what kind of black magic goes into its graph to make this arcane input and output system work?
Nothing?
I must have configured something weird or there is something wrong with my machine. So I asked a coworker to run my minimal replicatable code from the top of this issue built against a fresh clone of ggml... same result! Is there something incredibly simple I missed? Or is there a bug in mul_mat that only appears on small tensors?
The text was updated successfully, but these errors were encountered: