Give too many <MID> <PRE> <SUF> inline response when load custom LLM model with llm-vscode-server

[bonuschild]

Environment

• api server project: https://github.com/wangcx18/llm-vscode-inference-server (based on vllm)
• model: Phind-CodeLlama-34B-v2-AWQ from https://huggingface.co/TheBloke/Phind-CodeLlama-34B-v2-AWQ
• plugin: llm-vscode 0.1.6, settings as below:
  • "llm.tokenizer": {"path": "C:\Users\XXX\tokenizer.json"} (which is get from model folder)

Phenomenon

Then set the endpoint to http:https://localhost:8000/generate and plugins works, but given the and other symbols which make the code completions won't work well any more.

Question

• Is something wrong that cause this error?
• Is there some exact tutorials that instructing developer to deploy a custom model that is totally available to work with llm-vscode?

Thanks for reading and thinking!

[thanhnew2001]

Hello I got a similar error, i.e. empty response even though the debug output constantly log the input:
#105

I wonder there is a way to debug so that we know what exactly happen?

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[LouiFi]

Have tried to remove this settings form the extension settings ?

[github-actions]

This issue is stale because it has been open for 30 days with no activity.