CPU inferencing a lot slower than llama.cpp

[netspym]

Hi Foldl:

I found this project running Yi-34b-chat Q4 a lot slower than the latest llama.cpp, is it because it is not optimized for CPUs?

Such as not supporting AVX, AVX2 and AVX512 support for x86 architectures or
1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use?

Thanks
Yuming

[foldl]

This project is based on ggml, just the same as llama.cpp. Performance of both implementations should be similar when using CPU acceleration.

Maybe you are using GPU in llama.cpp?

[netspym]

Hi Foldl:

I'm using AMD EPYC 9654 with 96 cores, I don't have a GPU in the system.
The infer speed of this project is only 20-30% of the llama.cpp or ollama.
I guess llama.cpp project has more optimization in the past 6 months?

My Wechat ID is: 719784

Warm Regards
Yuming