apple silicon GPU thread count correct?

[jac-cbi]

I've read with extreme interest the efforts to use the Apple Silicon GPUs with ggml. However, I think there might be some mis-understanding. I hope it's me, but if it's not, there's a HUGE performance gain possible.

Several of the ggml benchmarks floating around seem to indicate 1 thread per GPU core. which is nice and all, but we can do orders of magnitude better:

Each GPU core is split into 16 execution units, which each contain eight arithmetic logic units (ALUs).

Wikipedia: Apple M2 GPU

Which, if I understand correctly, means we should be able to launch 128 threads per GPU core. And with Apple Silicons' Unified memory, they'll all have direct access to all of system RAM.

Am I missing something here? I'm really conflicted... I want to be right, but I highly doubt this was missed.

Cc: @ggerganov

[jac-cbi]

This is the slide for the M1 Ultra, it seems marketing refers to the ALU count as the "execution units". I don't yet grok how that gets to "196,608 concurrent threads"... Well, other than multiplying by 24 :-P

[SaraiQX]

wow. As apple-lover without strong CS background, I hope master Georgi could work on more projects for apple users to use M2 ultra (maybe 192G) in order to tap into the potential of real LLMs (over 100B parameters). Is it possible in near future? Thanks to any masterminds!