You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The full code is below. As you can see the dataset has a column question but Dataset constructor is throwing an Error that it doesn't.
# Step 4: Wrap the dataset with giskard.Datasetgiskard_dataset=giskard.Dataset(
df=test_dataset,
name="Climate Change Question Answering",
target="question",
column_types={"question": "text"}
)
returns
/home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/giskard/datasets/base/__init__.py:443: UserWarning: The provided keys ['question'] in 'column_types' are not part of your dataset 'columns'. Please make sure that the column names in `column_types` refers to existing columns in your dataset.
warning(
In the step before I print out what test_dataset looks like and I get this:
2024-06-20 17:08:52,178 pid:3360791 MainThread giskard.models.automodel INFO Your 'prediction_function' is successfully wrapped by Giskard's 'PredictionFunctionModel' wrapper class.
question
0 What is climate change?
1 How does climate change affect the environment?
2 What are the causes of climate change?
3 What is climate change?
4 How does climate change affect the environment?
.. ...
94 How does climate change affect the environment?
95 What are the causes of climate change?
96 What is climate change?
97 How does climate change affect the environment?
98 What are the causes of climate change?
[99 rows x 1 columns]
If run
print(test_dataset.columns)
I get
Index(['question'], dtype='object')
I rtied changing to
# Step 4: Wrap the dataset with giskard.Datasetgiskard_dataset=giskard.Dataset(
df=test_dataset,
name="Climate Change Questions",
target="question",
column_types={"question": "object"}
)
Standalone code OR list down the steps to reproduce the issue
I am using LM Studio to host TheBloke/Llama2-chat-13B-q-*_0-GGUF with nomic-embeded-text embeddings. I succesfully modified the first example steps leveraging LM Studio.
importrequestsimportnumpyasnpimportfaissfromopenaiimportOpenAIfromlangchainimportPromptTemplatefromlangchain.chainsimportRetrievalQAfromlangchain.text_splitterimportRecursiveCharacterTextSplitterfromlangchain.vectorstoresimportFAISSfromlangchain.docstore.in_memoryimportInMemoryDocstorefromlangchain.schemaimportDocumentfromlangchain_openaiimportChatOpenAIimportgiskardimportpandasaspd# Initialize OpenAI client for LM Studio embeddingsembedding_client=OpenAI(base_url="https://localhost:5000/v1", api_key="lm-studio")
# Function to get embeddings from LM Studiodefget_embedding(text, model="model-identifier", retries=3, timeout=120):
text=text.replace("\n", " ")
data= {
"input": [text],
"model": model
}
forattemptinrange(retries):
try:
response=requests.post("https://localhost:5000/v1/embeddings", json=data, timeout=timeout)
response.raise_for_status() # Raise an error for bad status codesresponse_data=response.json()
if'data'inresponse_dataandlen(response_data['data']) >0:
returnresponse_data['data'][0]['embedding']
returnNoneexceptrequests.exceptions.RequestExceptionase:
ifattempt<retries-1:
time.sleep(2**attempt) # Exponential backoffexceptExceptionase:
ifattempt<retries-1:
time.sleep(2**attempt) # Exponential backoffreturnNone# Function to load and split PDF using PyMuPDFdefload_pdf(file_path):
importfitz# PyMuPDFdoc=fitz.open(file_path)
text=""forpage_numinrange(len(doc)):
page=doc.load_page(page_num)
text+=page.get_text("text")
returntext# Load the PDFpdf_text=load_pdf("IPCC_AR6_SYR_LongerReport.pdf")
# Split the text into chunkstext_splitter=RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100, add_start_index=True)
texts=text_splitter.split_text(pdf_text)
# Get embeddings for each text chunkembeddings= [get_embedding(text) fortextintextsifget_embedding(text)]
# Convert embeddings to a NumPy arrayembeddings=np.array(embeddings, dtype=np.float32)
# Ensure the embeddings have the desired dimensiondesired_dimension=768ifembeddings.shape[1] !=desired_dimension:
print(f"Warning: The embedding dimension is {embeddings.shape[1]}, not {desired_dimension}")
# Create a FAISS indexindex=faiss.IndexFlatL2(embeddings.shape[1])
index.add(embeddings)
# Prepare documents and docstoredocuments= [Document(page_content=text) fortextintexts]
docstore=InMemoryDocstore({str(i): docfori, docinenumerate(documents)})
# Create an index-to-docstore-id mappingindex_to_docstore_id= {i: str(i) foriinrange(len(documents))}
# Use Langchain's FAISS retrievervectorstore=FAISS(embedding_function=get_embedding, index=index, docstore=docstore, index_to_docstore_id=index_to_docstore_id)
# Prepare QA chainPROMPT_TEMPLATE="""You are the Climate Assistant, a helpful AI assistant made by Giskard.Your task is to answer common questions on climate change.You will be given a question and relevant excerpts from the IPCC Climate Change Synthesis Report (2023).Please provide short and clear answers based on the provided context. Be polite and helpful.Context:{context}Question:{question}Your answer:"""```pythonllm=ChatOpenAI(base_url="https://localhost:5000/v1", temperature=0.85, api_key="not_needed")
prompt=PromptTemplate(template=PROMPT_TEMPLATE, input_variables=["question", "context"])
climate_qa_chain=RetrievalQA.from_llm(llm=llm, retriever=vectorstore.as_retriever(), prompt=prompt)
# Example questionquestion="What are the main impacts of climate change?"answer=climate_qa_chain({"query": question})
print(answer['result'])
NOTE I'm using a local copy of the PDF.
that returns:
Hi there! As the Climate Assistant, I'd be happy to help you understand the main impacts of climate change based on the IPCC Climate Change Synthesis Report (2023). Here are some key points:
* Climate change is causing impacts across multiple sectors, including agriculture, forestry, fishery, energy, and tourism, with regional effects and substantial economic damages (high confidence).
* The extent and magnitude of climate change impacts are larger than estimated in previous assessments (high confidence).
* Biological responses to climate change, such as changes in geographic placement and shifting seasonal timing, are often not sufficient to cope with recent climate change (very high confidence).
* Hundreds of local losses of species have been driven by increases in the magnitude of heat extremes (high confidence) and mass mortality events on land and in the ocean (very high confidence).
* Impacts on some ecosystems are approaching irreversibility, such as the impacts of hydrological changes resulting from the retreat of glaciers or the changes in some mountain and Arctic ecosystems driven by permafrost thaw (high confidence).
* In urban settings, climate change has caused adverse impacts on human health, livelihoods, and key infrastructure, including hot extremes, worsened air pollution events, and compromised transportation, water, sanitation, and energy systems (high confidence).
* Climate change has caused widespread adverse impacts and related losses and damages to nature and people, with adverse effects on gender and social equity (high confidence).
Overall, the main impacts of climate change are significant and far-reaching, affecting ecosystems, human health, economies, and societies around the world. It's important for us to take action to mitigate and adapt to climate change in order to minimize these impacts and protect our planet and its inhabitants.
Step 2 I modificed a bit, Im using Jupyter notebook so there may be some redundacy here.
fromopenaiimportOpenAI# Initialize LM Studio clientclient=OpenAI(base_url="https://localhost:5000/v1", api_key="lm-studio")
deftest_lm_studio():
response=client.embeddings.create(
input="Test embedding", # Pass a single string instead of a listmodel="model-identifier"
)
print(response)
test_lm_studio()
importgiskardimportpandasaspddefmodel_predict(df: pd.DataFrame):
return [climate_qa_chain.run({"query": question}) forquestionindf["question"]]
giskard_model=giskard.Model(
model=model_predict,
model_type="text_generation",
name="Climate Change Question Answering",
description="This model answers any question about climate change based on IPCC reports",
feature_names=["question"],
base_url="https://localhost:5000/v1"# Ensure this points to your LM Studio
)
responds
2024-06-20 16:57:27,482 pid:3360791 MainThread giskard.models.automodel INFO Your 'prediction_function' is successfully wrapped by Giskard's 'PredictionFunctionModel' wrapper class.
then
importpandasaspddefgenerate_test_dataset_with_lm_studio(model, num_samples=100):
# Use LM Studio to generate synthetic data# Replace this with your actual implementation to generate data using LM Studioquestions= ["What is climate change?", "How does climate change affect the environment?", "What are the causes of climate change?"] * (num_samples//3)
data=pd.DataFrame({"question": questions})
returndata# Step 1: Test LM Studio Clientdeftest_lm_studio():
response=client.embeddings.create(
input="Test embedding",
model="model-identifier"
)
# print(response)test_lm_studio()
# Step 2: Define the Giskard modeldefmodel_predict(df: pd.DataFrame):
return [climate_qa_chain.run({"query": question}) forquestionindf["question"]]
giskard_model=giskard.Model(
model=model_predict,
model_type="text_generation",
name="Climate Change Question Answering",
description="This model answers any question about climate change based on IPCC reports",
feature_names=["question"],
base_url="https://localhost:5000/v1"
)
# Step 3: Generate the test dataset with LM Studiotest_dataset=generate_test_dataset_with_lm_studio(giskard_model)
# Ensure the test dataset has the expected structureprint(test_dataset)
returns
2024-06-20 17:08:52,178 pid:3360791 MainThread giskard.models.automodel INFO Your 'prediction_function' is successfully wrapped by Giskard's 'PredictionFunctionModel' wrapper class.
question
0 What is climate change?
1 How does climate change affect the environment?
2 What are the causes of climate change?
3 What is climate change?
4 How does climate change affect the environment?
.. ...
94 How does climate change affect the environment?
95 What are the causes of climate change?
96 What is climate change?
97 How does climate change affect the environment?
98 What are the causes of climate change?
[99 rows x 1 columns]
# Step 4: Wrap the dataset with giskard.Datasetgiskard_dataset=giskard.Dataset(
df=test_dataset,
name="Climate Change Questions",
target="question",
column_types={"question": "text"}
)
returns, however there is a column
/home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/giskard/datasets/base/__init__.py:443: UserWarning: The provided keys ['question'] in 'column_types' are not part of your dataset 'columns'. Please make sure that the column names in `column_types` refers to existing columns in your dataset.
warning(
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[61], line 2
1 # Step 4: Wrap the dataset with giskard.Dataset
----> 2 giskard_dataset = giskard.Dataset(
3 df=test_dataset,
4 name="Climate Change Questions",
5 target="question",
6 column_types={"question": "text"}
7 )
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/pydantic/validate_call_decorator.py:59, in validate_call.<locals>.validate.<locals>.wrapper_function(*args, **kwargs)
57 @functools.wraps(function)
58 def wrapper_function(*args, **kwargs):
---> 59 return validate_call_wrapper(*args, **kwargs)
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/pydantic/_internal/_validate_call.py:81, in ValidateCallWrapper.__call__(self, *args, **kwargs)
80 def __call__(self, *args: Any, **kwargs: Any) -> Any:
---> 81 res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
82 if self.__return_pydantic_validator__:
83 return self.__return_pydantic_validator__(res)
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/giskard/datasets/base/__init__.py:211, in Dataset.__init__(self, df, name, target, cat_columns, column_types, id, validation, original_id)
208 if validation:
209 from giskard.core.dataset_validation import validate_column_types
--> 211 validate_column_types(self)
213 if validation:
214 from giskard.core.dataset_validation import (
215 validate_column_categorization,
216 validate_numeric_columns,
217 )
File /home/michael/anaconda3/envs/giskard/lib/python3.9/site-packages/giskard/core/dataset_validation.py:94, in validate_column_types(ds)
89 raise ValueError(
90 f"Invalid column_types parameter: {ds.column_types}"
91 + f"Please choose types among {[column_type.value for column_type in SupportedColumnTypes]}."
92 )
93 else:
---> 94 raise ValueError(f"Invalid column_types parameter: {ds.column_types}. Please specify non-empty dictionary.")
96 df_columns_set = set(ds.columns)
97 df_columns_set.discard(ds.target)
ValueError: Invalid column_types parameter: {}. Please specify non-empty dictionary.
### Relevant log output
_No response_
The text was updated successfully, but these errors were encountered:
Issue Type
Bug
Source
source
Giskard Library Version
2.14.0
Giskard Hub Version
not running hub
OS Platform and Distribution
Ubuntu 22.04.4 LTS
Python version
Python 3.9.19
Installed python packages
Current Behaviour?
The full code is below. As you can see the dataset has a column
question
but Dataset constructor is throwing an Error that it doesn't.returns
In the step before I print out what test_dataset looks like and I get this:
If run
I get
I rtied changing to
returns
Standalone code OR list down the steps to reproduce the issue
I am using LM Studio to host TheBloke/Llama2-chat-13B-q-*_0-GGUF with nomic-embeded-text embeddings. I succesfully modified the first example steps leveraging LM Studio.
NOTE I'm using a local copy of the PDF.
that returns:
Step 2 I modificed a bit, Im using Jupyter notebook so there may be some redundacy here.
responds
responds
then
returns
returns, however there is a column
The text was updated successfully, but these errors were encountered: