Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FR] Support tool calling for any arbitrary (ollama?) model #663

Open
MichaelDoyle opened this issue Jul 22, 2024 · 0 comments
Open

[FR] Support tool calling for any arbitrary (ollama?) model #663

MichaelDoyle opened this issue Jul 22, 2024 · 0 comments

Comments

@MichaelDoyle
Copy link
Member

Is your feature request related to a problem? Please describe.
Currently several other frameworks support tool calling for ollama via a "wrapper", which I think has led this to become an expected feature. As of today, we rely on model api support for whether or not we can enable tool calling. We already retrofit functionality such as "context" and "system prompts" via middleware.

Describe the solution you'd like
I have not fully thought this out, but we could add a middleware that would roughly do the following:

  • Append instructions to the prompt that include tool definitions
  • Append instructions to the prompt that specify the expected json output for a tool call request
  • Translate json tool call requests into genkit tool call requests
  • Tool calling should work as usual?
  • Translate tool call responses back into "user" messages

Describe alternatives you've considered

  • N/A

Additional context
See also: #661

Notes: Such a feature is largely dependent on models that can reliably produce json output.

@MichaelDoyle MichaelDoyle changed the title [FR] Support tool calling for any arbitrary model [FR] Support tool calling for any arbitrary (ollama?) model Jul 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

1 participant