[Bug]: custom_llm_provider not honored in litellm.py

[primerano]

Bug Description

I am running litellm as a front to Amazon Bedrock and I have the following code

llm = LiteLLM(model="mistralai/Mistral-7B-Instruct-v0.2",
             temperature=0,
             custom_llm_provider='openai',
             max_retries=1,
             api_key=os.environ['YOUR_API_KEY'],
             api_base='http:https://server-where-api-is/')

litellm.py correctly checks the set custom_llm_provider for verification purposes here

but it is not added to additional_kwargs so I need to manually set it for my calls to work.

llm.additional_kwargs['custom_llm_provider'] = 'openai'
Without this setting I get

litellm.exceptions.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=mistralai/Mistral-7B-Instruct-v0.2

Should litellm.py set additional_kwargs["custom_llm_provider"]?

         additional_kwargs = additional_kwargs or {}
         if "custom_llm_provider" in kwargs:
            additional_kwargs["custom_llm_provider"] = kwargs["custom_llm_provider"]

Or am I just calling this all wrong? ;-)

Version

0.10.38

Steps to Reproduce

import os
import litellm
from llama_index.legacy.llms.litellm import LiteLLM
from llama_index.legacy.llms.litellm_utils import from_openai_message_dict

llm = LiteLLM(model="mistralai/Mistral-7B-Instruct-v0.2",
             temperature=0,
             custom_llm_provider='openai',
             max_retries=1,
             api_key=os.environ['YOUR_API_KEY'],
             api_base='http:https://server-where-api-is/')


# llm.additional_kwargs['custom_llm_provider'] = 'openai'  # work around

message = from_openai_message_dict(
    {"role": "user", "content": "how tall is the empire state building"}
)

response = llm.chat(messages=[message])

Relevant Logs/Tracbacks

litellm.exceptions.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=mistralai/Mistral-7B-Instruct-v0.2

[logan-markewich]

I'm not an expert in litellm, but it sounds like you found the issue 😅 I welcome a PR!

[primerano]

I am new to litellm and I'm a bit scared to edit this code as I found some code that work for the wrong reasons.

llama_index/llama-index-legacy/llama_index/legacy/llms/litellm_utils.py

Line 202 in 8dd2268

def validate_litellm_api_key(

litellm.validate_environment() does not return an api key so and never returns None so validate_litellm_api_key will never fail.

It should probably just use validate_openai_api_key from

llama_index/llama-index-legacy/llama_index/legacy/llms/openai_utils.py

Line 379 in 8dd2268

def validate_openai_api_key(api_key: Optional[str] = None) -> None:

from llama_index.legacy.llms.openai_utils import validate_openai_api_key
...
        additional_kwargs = additional_kwargs or {}

        if "custom_llm_provider" in kwargs:
            additional_kwargs["custom_llm_provider"] = kwargs["custom_llm_provider"]
            if (
                kwargs["custom_llm_provider"] != "ollama"
                and kwargs["custom_llm_provider"] != "vllm"
            ):  # don't check keys for local models
                breakpoint()
                validate_openai_api_key(api_key)
        else:  # by default assume it's a hosted endpoint
            validate_openai_api_key(api_key)


        if api_key is not None:
            additional_kwargs["api_key"] = api_key

@krrishdholakia - you implemented #7600. do you see any issue with these changes?

[krrishdholakia]

yea this would only check if the openai api key is in the environment - but a user could be trying to call anthropic/ai21/etc.

what is the issue you're trying to solve? @primerano

[primerano]

hey @krrishdholakia ,

Unrelated to my original issue I noticed that validate_litellm_api_key

llama_index/llama-index-legacy/llama_index/legacy/llms/litellm_utils.py

Line 202 in 8dd2268

def validate_litellm_api_key(

will always return true because

litellm.validate_environment() returns a dictionary object and thus checking to see if the returned api_key is None will always be false. Also, validate_litellm_api_key has an unused api_type parameter.

I was thinking that litellm always used the OPENAI_API_KEY to pass the api key so using llama_index/llama-index-legacy/llama_index/legacy/llms/openai_utils.py would work. I am new to litellm so I may be wrong here. :-)

My original problem, which I also fixed in the init was that custom_llm_provider was not being set in the additional_kwargs and it caused my API call to fail. See the description area of this issue for the details of what I saw there.

Thanks!

[primerano]

Trying this from home with local ollama. This worked. Maybe I just need to add openai/ as a prefix to the model?

import os
import litellm
from llama_index.legacy.llms.litellm import LiteLLM
from llama_index.legacy.llms.litellm_utils import from_openai_message_dict
llm = LiteLLM(model="openai/ollama/phi3",
             temperature=0,
             max_retries=1,
             api_key='sdfsdfs',
             api_base='http:https://localhost:4000')
message = from_openai_message_dict(
    {"role": "user", "content": "how tall is the empire state building"}
)

response = llm.chat(messages=[message])

[primerano]

@krrishdholakia, I guess part of my problem is that I was thinking I should use the litellm library to talk to the litellm proxy.

But in reality

• the OpenAI library should be used to talk to the LiteLLM Proxy
• the LiteLLM library is used to talk to model providers directly using the OpenAI format.

The problem is that in LllamaIndex the OpenAI library will reject models that are not natively part of OpenAI (because it needs to find the context size of the model)

llm = OpenAI(model="mistralai/Mistral-7B-Instruct-v0.2",
             temperature=0,
             max_retries=1,
             api_key=os.environ['YOUR_API_KEY'],
             api_base='http:https://server-where-litellm-is/')

message = ChatMessage(role="user", content="how tall is the empire state building")

llm.chat(messages=[message])

*** ValueError: Unknown model 'mistralai/Mistral-7B-Instruct-v0.2'. Please provide a valid OpenAI model name in: gpt-4, gpt-4-32k, gpt-4-1106-preview, 

I think this is why I initially switched to using the litellm library.

What is the right approach to interact with a LiteLLM proxy through LlamaIndex?