Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

I can't use two models with dspy.OpenAI #915

Closed
Su3h7aM opened this issue Apr 27, 2024 · 5 comments · May be fixed by #923
Closed

I can't use two models with dspy.OpenAI #915

Su3h7aM opened this issue Apr 27, 2024 · 5 comments · May be fixed by #923
Labels
LM Issues

Comments

@Su3h7aM
Copy link

Su3h7aM commented Apr 27, 2024

When I try to use two different models using dspy.OpenAI, it ends up using only the last declaration. As in the example below, even using llama3 it ends up using phi3.
In this case I am using dspy.OpenAI but behind the scenes there are two instances of llama.cpp (It does not appear to be a problem with llama.cpp, as using the API directly the problem does not recur).
When I try to use dspy.OpenAI for Llama-3 and dspy.OllamaLocal for Phi-3, everything works correctly.

In short: Even using two models, the responses are only generated by the last declared model.

import dspy

llama3 = dspy.OpenAI(api_base="https://localhost:8080/v1/", model_type="chat", api_key="no-key")
phi3 = dspy.OpenAI(api_base="https://localhost:4040/v1/", model_type="chat", api_key="no-key")
#phi3 = dspy.OllamaLocal(model="phi3")

#dspy.settings.configure(lm=llama3)

question = "Who are you? Who developed you?"

class QA(dspy.Signature):
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()

with dspy.context(lm=llama3):
    a = dspy.TypedPredictor(QA)(question=question)

    print("\nLlama3: ")
    #print(llama3(question))
    print(a.answer)

with dspy.context(lm=phi3):
    b = dspy.TypedPredictor(QA)(question=question)

    print("\nPhi3: ")
    #print(phi3(question))
    print(b.answer)

#llama3.inspect_history(n=1)
#phi3.inspect_history(n=1)

Env: DSP_CACHEBOOL=false

DSPy: v2.4.5
OpenAI: v1.23.4
Python: v3.11.9
OS: Fedora Linux 40 (KDE Plasma), Linux 6.8.7-300.fc40.x86_64
CPU: AMD Ryzen 7 5700X 8-Core Processor
RAM: 32010 Mi
@Su3h7aM Su3h7aM changed the title I can't use two models with dspy.OpenAI I can't use two models with dspy.OpenAI Apr 27, 2024
@arnavsinghvi11
Copy link
Collaborator

Hi @Su3h7aM , #744 is related but not mergeable yet.

the responses are only generated by the last declared model.

this is bit unclear. Are the generations only from phi3, or does the inspect_history printed value only show phi3 generations? Specifying the model within the context manager ensures the corresponding generations are from the specified model.

@Su3h7aM
Copy link
Author

Su3h7aM commented Apr 27, 2024

this is bit unclear.

Sorry, I'll try to be more specific.

llama3 = dspy.OpenAI(api_base="https://localhost:8080/v1/", model_type="chat", api_key="no-key")
phi3 = dspy.OpenAI(api_base="https://localhost:4040/v1/", model_type="chat", api_key="no-key")

Here for example, all requests go to https://localhost:4040/v1/ even when using with dspy.context(lm=llama3): which should send to https://localhost:8080/v1/

phi3 = dspy.OpenAI(api_base="https://localhost:4040/v1/", model_type="chat", api_key="no-key")
llama3 = dspy.OpenAI(api_base="https://localhost:8080/v1/", model_type="chat", api_key="no-key")

Here they all go to https://localhost:8080/v1/

@Su3h7aM
Copy link
Author

Su3h7aM commented Apr 27, 2024

Hi @Su3h7aM , #744 is related but not mergeable yet.

I just tested it on his branch and it worked as expected. Thanks

@Su3h7aM Su3h7aM mentioned this issue Apr 28, 2024
Open
@sean-hacker
Copy link

Are there any updates on this? Would love to be able to use both GPT-3.5 and GPT-4 within a single DSPy program.

@SomeoneKong SomeoneKong mentioned this issue Jul 17, 2024
Merged
@okhat okhat added the LM Issues label Aug 9, 2024
@okhat
Copy link
Collaborator

okhat commented Sep 24, 2024

Thanks for opening this! We released DSPy 2.5 yesterday. I think the new dspy.LM and the underlying dspy.ChatAdapter will probably resolve this problem.

Here's the (very short) migration guide, it should typically take you 2-3 minutes to change the LM definition and you should be good to go: https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb

Please let us know if this resolves your issue. I will close for now but please feel free to re-open if the problem persists.

@okhat okhat closed this as completed Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
LM Issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add support for multiple GPT3 clients Su3h7aM/dspy
4 participants
@okhat @Su3h7aM @arnavsinghvi11 @sean-hacker