Improve Online Search: Parallelize Search, Use Jina Reader API by default #832
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
85ad6f6
to
373eff2
Compare
sabaimran
reviewed
Jun 24, 2024
9ebf68a
to
2e1b112
Compare
2cba000
to
2e1b112
Compare
Jina AI provides a search and webpage reader API that doesn't require an API key. This provides a good default to enable online search for self-hosted readers requiring no additional setup. Jina search API also returns webpage contents with the results, so just use those directly when Jina Search API used instead of trying to read webpages separately. The extract relvant content from webpage step using a chat model is still used from the `read_webpage_and_extract_content' func in this case. Parse search results from Jina search API into same format as Serper.dev for accurate rendering of online references by clients
It is unnecessary as the OpenAI client automatically tries to use API key from OPENAI_API_KEY env var when the api_key field is unset
Update offline, openai chat actor, director tests to not require Serper to run the online command tests Update documentation for self-hosted online search to mention no setup is required by default. But improvements can be made by using Serper.dev or Olostep
2e1b112
to
d5ceff2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Khoj wil be able to do online search out of the box, even for self-hosted users
Details
Jina provides a reader API for online search and web page reading.
It requires no API key. This provides a good default to enable
online search for self-hosted readers requiring no additional setup.
Jina search API also returns webpage contents with the results, so
just use those directly when Jina Search API used instead of
trying to read webpages separately. The extract relvant content from
webpage step using a chat model is still used from the
read_webpage_and_extract_contentfunc in this case.Parse search results from Jina search API into same format as
Serper.dev for accurate rendering of online references by clients
Run online searches in parallel with AsyncIO to process multiple
queries faster