RAM Misuse on CPU-Only setup / Very Slow

[jacmkno]

Running on Colab with CPU only works, but I happen to notice that the RAM is not being used. Seems like the library is doing a lot of disk I/O instead of storing the whole model on RAM as happens with the standard GPU setup with hugging face transformers. As a consequence inference is obscenely slow.

I'm trying to hunt down if the issue corresponds directly to llama.cpp, GGML or ctransformers.

Here is the Colab: https://colab.research.google.com/drive/1iGifBXEaXI2JDbJG1Il7BAS8gqVWm8kR?usp=sharing

[willb0]

also seeing this when working with ggml quantized LLAMA to 4bit loaded using ctransformers in a ipynb-- when running llama.cpp stuff there is a --mlock parameter to keep the whole model loaded in memory, would be nice to know if that is possible for ctransfomers

[willb0]

from what I can tell, we would have to add mlock as a parameter when we call self._llm = self._lib.ctransformers_llm_create, and add it to the load_library loading fxn as well

[marella]

I will add mmap and mlock options in the next release. I tested with different combinations of mmap and mlock but didn't notice any performance improvement on my machine and Google Colab.
Note that Colab CPU only has 2 cores so inference will be slow. It runs ~3x faster on my machine which has 6 cores with same settings. On Colab, it is better to use GPU.
Also llm(prompt) only gives output after the entire text is generated. Try using stream=True to see output after each token is generated:

for text in llm(prompt, stream=True):
    print(text, end='', flush=False)

[willb0]

thank you @marella !!
i noticed improved inference on mac m1 8GB using GGML quantized LLAMA 7B, but that wasn't your release just on my fork where i changed mlock

[marella]

Added mmap and mlock parameters for LLaMA and Falcon models in the latest version 0.2.23

[baylitoo]

How can i use mmap and mlock when working with langchain and creating the model using CTransformers(path,config) ?

No need to reply, found it