Can't load Llama Model

[himanshud2611]

I've worked with OpenAI and Mistral previously, the DSPy pipeline executed well. I've tried loading Llama-2-7b using HuggingFace and HFClient TGI but getting errors "No LM loaded" and Connection Error. I've also tried using Anyscale to load Llama-2-7b but got "Received invalid JSON response from server" error. Can anyone help the optimal way to Llama-2-7b to execute DSPy pipeline.

[arnavsinghvi11]

Hi @himanshud2611 ,

Could you share any relevant code snippets and/or stack traces? Seems like the error message you get from TGI is from not setting the model name correctly, while the Anyscale error message has a configured endpoint but potentially incorrect arguments passed in.

[himanshud2611]

Hi @arnavsinghvi11
I break my experiences in following points:

1. I've gone through the doc to use HFClientTGI https://t.co/XDadKV9vHv and I can't really understand how to set the TGI server using docker as given. I've cloned the repo but didn't get where and how to execute the docker command as no instance is given there.
2. I'd seen the SkyCamp 2023 colab file given in DSPy github and tried to understand it but didn't really get how you used ports using HFClientTGI.

At this point in time, I want to use Language Model API Client [lm = dspy.HFClientTGI(model="meta-llama/Llama-2-7b-hf", port=8080, url="https://localhost")] with same model name. Can you help me with setting ports and url here - how can I setup TGI server locally, help me with docker part.

[tom-doerr]

Can't you just run the 4 lines from the screenshot you shared?

[himanshud2611]

Hi @tom-doerr
I've asked the question where to run this? Do I need to create a docker image or what? This is my first experience with docker and I don't find a tut to set TGI server using docker

[tom-doerr]

I don't think it makes sense for us to talk you through the fundamentals of docker, there's a lot to know.
When I set up my server with docker I used GPT-4 for that, it gave me really good tips and explained to me what I need to do

[himanshud2611]

Thanks :)
I'm looking into it and hopefully will execute the pipeline soon :)

[himanshud2611]

I've tried using Ollama to load Llama and issue is resolved