Feature/researcher

[shenchucheng]

Features

• Unified Search Engine API with support for returning structured data
• Unified WebPage object return for the WebBrowserEngine API
• Added DuckDuckGo as a Search Engine
• Added a new researcher role

Search Engine API

class SearchEngine:
    @overload
    def run(self, query: str, max_results: int = 8, as_string: Literal[True] = True) -> str:
        ...

    @overload
    def run(self, query: str, max_results: int = 8, as_string: Literal[False] = False) -> list[dict[str, str]]:
        ...

    async def run(self, query: str, max_results: int = 8, as_string: bool = True) -> str | list[dict[str, str]]:
        """Run a search query.

        Args:
            query: The search query.
            max_results: The maximum number of results to return. Defaults to 8.
            as_string: Whether to return the results as a string or a list of dictionaries. Defaults to True.

        Returns:
            The search results as a string or a list of dictionaries.
        """
        ...

WebBrowserEngine API

class WebBrowserEngine:
    @overload
    async def run(self, url: str) -> WebPage:
        ...

    @overload
    async def run(self, url: str, *urls: str) -> list[WebPage]:
        ...

    async def run(self, url: str, *urls: str) -> WebPage | list[WebPage]:
        ...

[daviskeene]

This is awesome!

[stellaHSR]

LGTM

[better629]

@shenchucheng Can you provide an execution log file as attachment of Researcher?

[shenchucheng]

@shenchucheng Can you provide an execution log file as attachment of Researcher?

logs: log.txt
report: dataiku vs. datarobot.md

[better629]

LGTM

[RonTuretzky]

I'm getting

File "/home//.local/lib/python3.10/site-packages/yarl/_url.py", line 946, in _query_var
raise TypeError(
TypeError: Invalid variable type: value should be str, int or float, got None of type <class 'NoneType'>

[shenchucheng]

I'm getting

File "/home//.local/lib/python3.10/site-packages/yarl/_url.py", line 946, in _query_var raise TypeError( TypeError: Invalid variable type: value should be str, int or float, got None of type <class 'NoneType'>

This issue arises due to the absence of a set SERPAPI_API_KEY for the SerpAPI search engine, and there's a lack of a user-friendly error message. You can locate the SERPAPI_API_KEY in config.yaml#L28 for reference.

And in pr #240, there will be a check implemented during the execution of the SearchEngine to verify if the required packages or configurations meet the criteria. If the prerequisites are not met, an exception will be triggered, accompanied by user-friendly suggestions for necessary modifications.

[RonTuretzky]

@shenchucheng Thank you, that worked!