generate_test does not work with Azure Open AI key

[terry07]

Issue Type

Bug

Source

source

Giskard Library Version

2.8.0

Giskard Hub Version

NaN

OS Platform and Distribution

Ubuntu 22.04.4 LTS

Python version

3.9.18

Installed python packages

langchain
langchain-openai
langchain_experimental
langchain_community
docarray
pydantic==1.10.8
python-dotenv
ruff
bs4
ipytest
giskard[llm]

Current Behaviour?

Trying to call the generate_testset, I get the error attached below.

My difference is the fact that I use from langchain.embeddings import AzureOpenAIEmbeddings
defining the necessary embedding model for the vector_stroe.

Standalone code OR list down the steps to reproduce the issue

I am trying to replicate that code here: https://github.com/svpino/llm/tree/main/evaluation

Relevant log output

---------------------------------------------------------------------------
AuthenticationError                       Traceback (most recent call last)
File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/giskard/llm/client/openai.py:288, in OpenAIClient._embeddings_generation(self, texts, model)
    287 try:
--> 288     out = self._client.embeddings.create(input=list(texts), model=model)
    289     embeddings = [element.embedding for element in out.data]

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/sentry_sdk/integrations/openai.py:279, in _wrap_embeddings_create.<locals>.new_embeddings_create(*args, **kwargs)
    278     _capture_exception(Hub.current, e)
--> 279     raise e from None
    281 prompt_tokens = 0

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/sentry_sdk/integrations/openai.py:276, in _wrap_embeddings_create.<locals>.new_embeddings_create(*args, **kwargs)
    275 try:
--> 276     response = f(*args, **kwargs)
    277 except Exception as e:

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/openai/resources/embeddings.py:113, in create(self, input, model, dimensions, encoding_format, user, extra_headers, extra_query, extra_body, timeout)
    103     return obj
    105 return self._post(
    106     "/embeddings",
    107     body=maybe_transform(params, embedding_create_params.EmbeddingCreateParams),
    108     options=make_request_options(
    109         extra_headers=extra_headers,
    110         extra_query=extra_query,
    111         extra_body=extra_body,
    112         timeout=timeout,
--> 113         post_parser=parser,
    114     ),
    115     cast_to=CreateEmbeddingResponse,
    116 )

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/openai/_base_client.py:1208, in post(self, path, cast_to, body, options, files, stream, stream_cls)
   1207     else:
-> 1208         timeout = DEFAULT_TIMEOUT
   1210 super().__init__(
   1211     version=version,
   1212     base_url=base_url,
   (...)
   1221     _strict_response_validation=_strict_response_validation,
   1222 )

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/openai/_base_client.py:897, in request(self, cast_to, options, remaining_retries, stream, stream_cls)
    895 if retries > 0:
    896     return self._retry_request(
--> 897         options,
    898         cast_to,
    899         retries,
    900         stream=stream,
    901         stream_cls=stream_cls,
    902         response_headers=None,
    903     )
    905 raise APIConnectionError(request=request) from err

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/openai/_base_client.py:988, in _request(self, cast_to, options, remaining_retries, stream, stream_cls)
    982     return self.request(page, options, stream=False)
    984 @overload
    985 def get(
    986     self,
    987     path: str,
--> 988     *,
    989     cast_to: Type[ResponseT],
    990     options: RequestOptions = {},
    991     stream: Literal[False] = False,
    992 ) -> ResponseT:
    993     ...

AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: *******************. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
Cell In[25], line 3
      1 from giskard.rag import generate_testset
----> 3 testset = generate_testset(
      4     knowledge_base,
      5     num_questions=60,
      6     agent_description="A chatbot answering questions about the Machine Learning School Website",
      7 )

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/giskard/rag/testset_generation.py:65, in generate_testset(knowledge_base, num_questions, question_generators, language, agent_description)
     62     question_generators = [question_generators]
     64 # Ensure topics are computed and documents populated (@TODO: remove this)
---> 65 _ = knowledge_base.topics
     67 # Generate questions
     68 
     69 # @TODO: fix this ugly way to distribute the questions across generators
     70 generator_num_questions = [
     71     num_questions // len(question_generators) + (1 if i < num_questions % len(question_generators) else 0)
     72     for i in range(len(question_generators))
     73 ]

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/giskard/rag/knowledge_base.py:224, in KnowledgeBase.topics(self)
    221 @property
    222 def topics(self):
    223     if self._topics_inst is None:
--> 224         self._topics_inst = self._find_topics()
    225     return self._topics_inst

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/giskard/rag/knowledge_base.py:235, in KnowledgeBase._find_topics(self)
    228 logger.info("Finding topics in the knowledge base.")
    229 hdbscan = HDBSCAN(
    230     min_cluster_size=self._min_topic_size,
    231     min_samples=3,
    232     metric="euclidean",
    233     cluster_selection_epsilon=0.0,
    234 )
--> 235 clustering = hdbscan.fit(self._reduced_embeddings)
    237 logger.debug(f"Found {clustering.labels_.max() + 1} clusters.")
    238 for i, doc in enumerate(self._documents):

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/giskard/rag/knowledge_base.py:193, in KnowledgeBase._reduced_embeddings(self)
    187     logger.debug("Calculating UMAP projection")
    188     reducer = umap.UMAP(
    189         n_neighbors=50,
    190         min_dist=0.5,
    191         n_components=2,
    192     )
--> 193     self._reduced_embeddings_inst = reducer.fit_transform(self._embeddings)
    194 return self._reduced_embeddings_inst

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/giskard/rag/knowledge_base.py:177, in KnowledgeBase._embeddings(self)
    175 if self._embeddings_inst is None:
    176     logger.info("Computing Knowledge Base embeddings.")
--> 177     self._embeddings_inst = self._llm_client.embeddings(
    178         [doc.content for doc in self._documents], model=self._embedding_model, chunk_size=self.chunk_size
    179     )
    180     for doc, emb in zip(self._documents, self._embeddings_inst):
    181         doc.embeddings = emb

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/giskard/llm/client/openai.py:157, in BaseOpenAIClient.embeddings(self, texts, model, chunk_size)
    155 chunks_indices = range(chunk_size, len(texts), chunk_size)
    156 chunks = np.split(texts, chunks_indices)
--> 157 embedded_chunks = [self._embeddings_generation(chunk, model) for chunk in chunks]
    158 return np.stack([emb for embeddings in embedded_chunks for emb in embeddings])

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/giskard/llm/client/openai.py:157, in <listcomp>(.0)
    155 chunks_indices = range(chunk_size, len(texts), chunk_size)
    156 chunks = np.split(texts, chunks_indices)
--> 157 embedded_chunks = [self._embeddings_generation(chunk, model) for chunk in chunks]
    158 return np.stack([emb for embeddings in embedded_chunks for emb in embeddings])

File ~/miniconda3/envs/spvino-llm/lib/python3.9/site-packages/giskard/llm/client/openai.py:297, in OpenAIClient._embeddings_generation(self, texts, model)
    291     raise ValueError(
    292         f"The embedding model: '{model}' was not found,"
    293         "make sure the model is correctly deployed on "
    294         f"the specified endpoint: {self._client._base_url}."
    295     ) from err
    296 except Exception as err:
--> 297     raise RuntimeError("Embedding creation failed.") from err
    299 return embeddings

RuntimeError: Embedding creation failed.

[pierlj]

Hello @terry07, have you check that the text-embedding-ada-002 model is deployed on your Azure OpenAI endpoint?

By default, RAGET uses gpt-4and text-embedding-ada-002 for text generation and text embedding respectively, but this can be changed if needed.

[luca-martial]

Hi @terry07, I'll close the issue for now. Let us know if the issue got resolved, if not feel free to share some updates!