Add support for Azure OpenAI, Palm, Claude, Cohere, Replicate, Sagemaker (100+LLMs) - using litellm #174

ishaan-jaff · 2023-09-05T17:47:57Z

This PR adds support for the above mentioned LLMs using LiteLLM https://github.com/BerriAI/litellm/
Example

from litellm import completion

## set ENV variables
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["COHERE_API_KEY"] = "cohere key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# cohere call
response = completion(model="command-nightly", messages)

# anthropic call
response = completion(model="claude-instant-1", messages=messages)

ishaan-jaff · 2023-09-05T17:50:01Z

Addressing:
#172
#145
#33

ishaan-jaff · 2023-09-05T17:52:07Z

@jesse-michael-han @kataqatsi can I get a review on this pr ?

happy to add docs/testing if this initial commit looks good

jesse-michael-han · 2023-09-05T21:43:09Z

Hi, thanks for making this PR! This solves a real user problem as evidenced by the multiple issues it addresses and I'd love to get this kind of functionality working in Rift.

Could you please add support for users being able to specify the API keys supported by litellm inside of Rift's VSCode settings as well, similar to how rift.openaiKey is stored and read from the VSCode configuration? The extra keys (rift.cohereKey, etc. ...) should be read from the configuration and used to manually set the corresponding environment variables, similar to how it is done in rift.agents.ThirdPartyAgent.

ishaan-jaff · 2023-09-06T01:16:15Z

editors/rift-vscode/package.json

@@ -46,6 +46,46 @@
 "default": null,
 "description": "OpenAI API key"
 },
+ "rift.cohereKey": {


@jesse-michael-han does it make sense to add all the key variables here?

if this is used by the rift UI don't want to fill up the entire section with key configs

yeah that works, now just add corresponding support for setting these env vars in the Python and we should be good to go 👍

ishaan-jaff · 2023-09-06T03:48:08Z

@jesse-michael-han done

litellm reads the keys directly from the .env so we should be good to go once it's set in the .env

Patrick10203 · 2023-09-12T12:52:13Z

There is a Quote to much in the dependencies in the pyproject.toml file
You wrote: "litellm" @ git+https://github.com/BerriAI/litellm/",
It should be "litellm @ git+https://github.com/BerriAI/litellm/",

ishaan-jaff · 2023-09-12T16:07:54Z

fixed @Patrick10203

Und3rf10w

Fixed in 24681d6

Und3rf10w · 2023-09-10T18:39:11Z

rift-engine/pyproject.toml

@@ -32,6 +32,7 @@ dependencies = [
 "aider-chat @ git+https://github.com/morph-labs/aider",
 "smol_dev @ git+https://github.com/morph-labs/smol_dev",
 "mentat-ai @ git+https://www.github.com/morph-labs/mentat", 
+ "litellm" @ git+https://github.com/BerriAI/litellm/",


This line should instead be:

"litellm @ git+https://github.com/BerriAI/litellm/",

Patrick10203 · 2023-09-14T15:06:43Z

I tried this pull request with the newest version of the main branch of rift and it doesnt work at all. I tried it with Azure Open AI and Open AI, but both dont work anymore after the change of rift-engine/rift/llm/openai_client.py
The function (chat_completions) isnt getting called anymore and i get an error that he expects an return of an async iterable and not a generator.
Also how should this even work with azure. There are parameters missing for Azure when calling the completion function. EG the deployment_id paramter has to be set for the completion function in order to use Azure. There are probably many other issues, which i didnt find because of the previous issues.

Und3rf10w · 2023-09-14T21:32:08Z

rift-engine/rift/llm/openai_client.py

- endpoint, params=params, input_type=input_type, output_type=output_type
+ response = completion(
+ params,
+ stream=True


Stream shouldn't be forced here

But if you don't force stream, I think it just doesn't work due to how non-stream responses are handled in rift

Und3rf10w · 2023-09-15T18:04:45Z

@Patrick10203,

I tried this pull request with the newest version of the main branch of rift and it doesnt work at all. I tried it with Azure Open AI and Open AI, but both dont work anymore after the change of rift-engine/rift/llm/openai_client.py The function (chat_completions) isnt getting called anymore and i get an error that he expects an return of an async iterable and not a generator. Also how should this even work with azure. There are parameters missing for Azure when calling the completion function. EG the deployment_id paramter has to be set for the completion function in order to use Azure. There are probably many other issues, which i didnt find because of the previous issues.

Try setting the following environment variables, e.g.:

export AZURE_API_KEY= 
export AZURE_API_BASE= 
export AZURE_API_VERSION=2023-07-01-preview 
export AZURE_DEPLOYMENT_NAME=

Patrick10203 · 2023-09-19T08:10:57Z

@Und3rf10w These env variables where already set before. So it doesnt do anything. I get no response from the rift server when sending a message. Does this work for you?

Patrick10203 · 2023-09-19T08:46:24Z

rift-engine/rift/llm/openai_client.py

 )
+ for chunk in response:
+ yield chunk # text content found at chunk['choices'][0]['delta']


This can not work. The function outside requires a async iterable. Here we create a generator as result of the function. You can not have returns and yields in the same function. Then the Python compiler will think this function always returns a generator. Which is not the case in the other if condition. And so the function chat_completions is never called.

Und3rf10w · 2023-09-19T18:51:09Z

@Patrick10203 You're right, I am not at all able to after trying it out some, sorry for the confusion.

smyja · 2023-11-04T19:37:11Z

is this still active?

garganshulgarg · 2023-12-29T15:41:43Z

Awaiting for these changes to be merged !

v0 litellm

2735ccc

ishaan-jaff mentioned this pull request Sep 5, 2023

Add Azure OpenAI instance as an option #33

Open

add cohere, replicate, azure setting in props

617bc04

ishaan-jaff commented Sep 6, 2023

View reviewed changes

set keys in .env

ceed0eb

fix pyproject.toml

24681d6

Und3rf10w approved these changes Sep 13, 2023

View reviewed changes

Und3rf10w reviewed Sep 15, 2023

View reviewed changes

Patrick10203 reviewed Sep 19, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Azure OpenAI, Palm, Claude, Cohere, Replicate, Sagemaker (100+LLMs) - using litellm #174

Add support for Azure OpenAI, Palm, Claude, Cohere, Replicate, Sagemaker (100+LLMs) - using litellm #174

ishaan-jaff commented Sep 5, 2023 •

edited

Loading

ishaan-jaff commented Sep 5, 2023

ishaan-jaff commented Sep 5, 2023

jesse-michael-han commented Sep 5, 2023

ishaan-jaff Sep 6, 2023

jesse-michael-han Sep 6, 2023

ishaan-jaff commented Sep 6, 2023

Patrick10203 commented Sep 12, 2023

ishaan-jaff commented Sep 12, 2023

Und3rf10w left a comment

Und3rf10w Sep 10, 2023

Patrick10203 commented Sep 14, 2023

Und3rf10w Sep 14, 2023

Und3rf10w Sep 15, 2023

Und3rf10w commented Sep 15, 2023 •

edited

Loading

Patrick10203 commented Sep 19, 2023

Patrick10203 Sep 19, 2023 •

edited

Loading

Und3rf10w commented Sep 19, 2023 •

edited

Loading

smyja commented Nov 4, 2023

garganshulgarg commented Dec 29, 2023

Add support for Azure OpenAI, Palm, Claude, Cohere, Replicate, Sagemaker (100+LLMs) - using litellm #174

Are you sure you want to change the base?

Add support for Azure OpenAI, Palm, Claude, Cohere, Replicate, Sagemaker (100+LLMs) - using litellm #174

Conversation

ishaan-jaff commented Sep 5, 2023 • edited Loading

ishaan-jaff commented Sep 5, 2023

ishaan-jaff commented Sep 5, 2023

jesse-michael-han commented Sep 5, 2023

ishaan-jaff Sep 6, 2023

Choose a reason for hiding this comment

jesse-michael-han Sep 6, 2023

Choose a reason for hiding this comment

ishaan-jaff commented Sep 6, 2023

Patrick10203 commented Sep 12, 2023

ishaan-jaff commented Sep 12, 2023

Und3rf10w left a comment

Choose a reason for hiding this comment

Und3rf10w Sep 10, 2023

Choose a reason for hiding this comment

Patrick10203 commented Sep 14, 2023

Und3rf10w Sep 14, 2023

Choose a reason for hiding this comment

Und3rf10w Sep 15, 2023

Choose a reason for hiding this comment

Und3rf10w commented Sep 15, 2023 • edited Loading

Patrick10203 commented Sep 19, 2023

Patrick10203 Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

Und3rf10w commented Sep 19, 2023 • edited Loading

smyja commented Nov 4, 2023

garganshulgarg commented Dec 29, 2023

ishaan-jaff commented Sep 5, 2023 •

edited

Loading

Und3rf10w commented Sep 15, 2023 •

edited

Loading

Patrick10203 Sep 19, 2023 •

edited

Loading

Und3rf10w commented Sep 19, 2023 •

edited

Loading