evaluation extremely slow with llama_cpp/gguf

[mobicham]

Evaluation of gguf models via llama_cpp server is extremely slow. All the layers are offloaded to the GPU so normally it should work fine, but truthfulqa takes 10 hours, it should normally take ~40 minutes or even less.
lm_eval --model gguf --model_args base_url=http:https://localhost:8000 --batch_size 16 --tasks truthfulqa_mc2 --num_fewshot 0

[Andy1314Chen]

Note that for externally hosted models, configs such as --device and --batch_size should not be used and do not function.

i have the same problem. What can I do to make it support batch processing？？

[linotfan]

It seems to be an issue with llama.cpp.

[mobicham]

It seems to be an issue with llama.cpp.

So basically they say it's a problem with quantized models running with large prompts.
That sounds strange because the impact of the dequantization step should actually matter much less with longer prompts and larger batches sizes.