Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding the OpenAIFunctionCaller #14

Merged
merged 18 commits into from
Jun 18, 2024
Merged

adding the OpenAIFunctionCaller #14

merged 18 commits into from
Jun 18, 2024

Conversation

TuanaCelik
Copy link
Member

Proposed Changes:

I propose adding this OpenAIFunctionCaller component that is able to accept a list of ChatMessages, and run a function if there were any tool_calls within these messages.

Some simple guardrails that have been added:
If the tool/function name in the message that we get doesn't actually exist (hallucinated) , the component returns an assistant message indicating as such.

How did you test it?

Manual tests. Here is the pipeline I'm using:

message_collector = BranchJoiner(List[ChatMessage])
chat_generator = OpenAIChatGenerator(model="gpt-3.5-turbo", generation_kwargs={'tools': tools})
function_caller = OpenAIFunctionCaller(available_functions={"rag_pipeline_func": rag_pipeline_func, 
                                                            "get_current_weather": get_current_weather})

function_calling_pipeline = Pipeline()
function_calling_pipeline.add_component("message_collector", message_collector)
function_calling_pipeline.add_component("generator", chat_generator)
function_calling_pipeline.add_component("function_caller", function_caller)

function_calling_pipeline.connect("message_collector", "generator.messages")
function_calling_pipeline.connect("generator", "function_caller")
function_calling_pipeline.connect("function_caller.function_replies", "message_collector")

messages = [
    ChatMessage.from_system(
        """If needed, break down the users question to simpler questions and follow up questions that you can use with your tools.
        Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."""
    )
]
while True:
  user_input = input("INFO: Type 'exit' or 'quit' to stop\n")
  if user_input.lower() == "exit" or user_input.lower() == "quit":
    break
  messages.append(ChatMessage.from_user(user_input))
  response = function_calling_pipeline.run({"message_collector": {"value": messages}})
  messages.extend(response['function_caller']['assistant_replies'])
  print(response['function_caller']['assistant_replies'][0])

Notes for the reviewer

You can try this pipeline here

Checklist

@TuanaCelik TuanaCelik requested a review from a team as a code owner June 14, 2024 09:30
@TuanaCelik TuanaCelik requested review from masci and removed request for a team June 14, 2024 09:30
@coveralls
Copy link

coveralls commented Jun 14, 2024

Pull Request Test Coverage Report for Build 9513906991

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 97.764%

Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

Copy link
Contributor

@masci masci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took a first pass, this will be tricky to serialize but doable

@coveralls
Copy link

coveralls commented Jun 17, 2024

Pull Request Test Coverage Report for Build 9548401674

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 17, 2024

Pull Request Test Coverage Report for Build 9548404389

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 17, 2024

Pull Request Test Coverage Report for Build 9548413341

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 17, 2024

Pull Request Test Coverage Report for Build 9548912053

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 17, 2024

Pull Request Test Coverage Report for Build 9549551101

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 17, 2024

Pull Request Test Coverage Report for Build 9550535764

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 17, 2024

Pull Request Test Coverage Report for Build 9550533685

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 17, 2024

Pull Request Test Coverage Report for Build 9550537322

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 17, 2024

Pull Request Test Coverage Report for Build 9551129065

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

Copy link
Contributor

@masci masci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good on my side

@coveralls
Copy link

coveralls commented Jun 18, 2024

Pull Request Test Coverage Report for Build 9560403086

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 18, 2024

Pull Request Test Coverage Report for Build 9560720590

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 18, 2024

Pull Request Test Coverage Report for Build 9563066747

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 18, 2024

Pull Request Test Coverage Report for Build 9563287676

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 10 unchanged lines in 1 file lost coverage.
  • Overall coverage remained the same at 97.764%

Files with Coverage Reduction New Missed Lines %
haystack_experimental/evaluation/harness/rag/harness.py 10 91.87%
Totals Coverage Status
Change from base Build 9465387731: 0.0%
Covered Lines: 612
Relevant Lines: 626

💛 - Coveralls

@TuanaCelik TuanaCelik requested a review from shadeMe June 18, 2024 11:35
@coveralls
Copy link

coveralls commented Jun 18, 2024

Pull Request Test Coverage Report for Build 9564264800

Details

  • 79 of 83 (95.18%) changed or added relevant lines in 5 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-0.3%) to 97.461%

Changes Missing Coverage Covered Lines Changed/Added Lines %
test/components/tools/openai/test_function_caller.py 34 35 97.14%
haystack_experimental/components/tools/openai/function_caller.py 39 42 92.86%
Totals Coverage Status
Change from base Build 9516119182: -0.3%
Covered Lines: 691
Relevant Lines: 709

💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 18, 2024

Pull Request Test Coverage Report for Build 9564511117

Details

  • 79 of 83 (95.18%) changed or added relevant lines in 5 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-0.3%) to 97.461%

Changes Missing Coverage Covered Lines Changed/Added Lines %
test/components/tools/openai/test_function_caller.py 34 35 97.14%
haystack_experimental/components/tools/openai/function_caller.py 39 42 92.86%
Totals Coverage Status
Change from base Build 9516119182: -0.3%
Covered Lines: 691
Relevant Lines: 709

💛 - Coveralls

@TuanaCelik TuanaCelik merged commit 3193cbd into main Jun 18, 2024
5 checks passed
@TuanaCelik TuanaCelik deleted the feat/function-caller branch June 18, 2024 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants