'Predict' object has no attribute 'new_signature'.

[tom-doerr]

I'm getting this error when running MIPRO:

AttributeError: 'Predict' object has no attribute 'new_signature'. Did you mean: 'signature'?

This is how I run MIPRO:

            teleprompter = dspy.teleprompt.MIPRO(metric = great_tweet_metric, verbose=True)
            kwargs = dict(num_threads=16, display_progress=True, display_table=0)
            tweet_generator_compiled = teleprompter.compile(tweet_generator, trainset=trainset,  num_trials=2, max_bootstrapped_demos=4, max_labeled_demos=5, eval_kwargs=kwargs, requires_permission_to_run=False)

More output:

  0%|                                             | 0/50 [00:00<?, ?it/s]2024-06-06T01:00:54.964938Z [error    ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: 2\n---\n\n# retrieve.ChromadbRM\n\n### Constructor\n\nInitialize an instance of the `ChromadbRM` class, with the option to use OpenAI\'s embeddings or any alternative supported by chromadb, as detailed in the official [chromadb embeddings documentation](https://docs.trychroma.com/embeddings).\n\n```python\nChromadbRM(\n    collection_name: str,\n    persist_directory: str,\n    embedding_function: Optional[EmbeddingFunction[Embeddable]] = OpenAIEmbeddingFunction(),\n    k: int = 7,\n)\n```\n\n**Parameters:**\n- `collection_name` (_str_): The name of the chromadb collection.\n- `persist_directory` (_str_): Path to the directory where chromadb data is persisted.\n- `embedding_function` (_Optional[EmbeddingFunction[Embeddable]]_, _optional_): The function used for embedding documents and queries. Defaults to `DefaultEmbeddingFunction()` if not specified.\n- `k` (_int_, _optional_): The number of top passages to retrieve. Defaults to 7.\n\n### Methods\n\n#### `forward(self, query_or_queries: Union[str, List[str]], k: Optional[int] = None) -> dspy.Prediction`\n\nSearch the chromadb collection for the top `k` passages matching the given query or queries, using embeddings generated via the specified `embedding_function`.\n\n**Parameters:**\n- `query_or_queries` (_Union[str, List[str]]_): The query or list of queries to search for.\n- `k` (_Optional[int]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization.\n\n**Returns:**\n- `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with schema `[{"id": str, "score": float, "long_text": str, "metadatas": dict }]`\n\n### Quickstart with OpenAI Embeddings\n\nChromadbRM have the flexibility from a variety of embedding functions as outlined in the [chromadb embeddings documentation](https://docs.trychroma.com/embeddings). While different options are available, this example demonstrates how to utilize OpenAI embeddings specifically.\n\n```python\nfrom dspy.retrieve.chromadb_rm import ChromadbRM\nimport os\nimport openai\nfrom chromadb.utils.embedding_functions import OpenAIEmbeddingFunction\n\nembedding_function = OpenAIEmbeddingFunction(\n    api_key=os.environ.get(\'OPENAI_API_KEY\'),\n    model_name="text-embedding-ada-002"\n)\n\nretriever_model = ChromadbRM(\n    \'your_collection_name\',\n    \'/path/to/your/db\',\n    embedding_function=embedding_function,\n    k=5\n)\n\nresults = retriever_model("Explore the significance of quantum computing", k=5)\n\nfor result in results:\n    print("Document:", result.long_text, "\\n")\n```\n'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=204
2024-06-06T01:00:54.969715Z [error    ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: \n---\n\n# dspy.Snowflake\n\n### Usage\n\n```python\nimport dspy\nimport os\n\nconnection_parameters = {\n\n    "account": os.getenv(\'SNOWFLAKE_ACCOUNT\'),\n    "user": os.getenv(\'SNOWFLAKE_USER\'),\n    "password": os.getenv(\'SNOWFLAKE_PASSWORD\'),\n    "role": os.getenv(\'SNOWFLAKE_ROLE\'),\n    "warehouse": os.getenv(\'SNOWFLAKE_WAREHOUSE\'),\n    "database": os.getenv(\'SNOWFLAKE_DATABASE\'),\n    "schema": os.getenv(\'SNOWFLAKE_SCHEMA\')}\n\nlm = dspy.Snowflake(model="mixtral-8x7b",credentials=connection_parameters)\n```\n\n### Constructor\n\nThe constructor inherits from the base class `LM` and verifies the `credentials` for using Snowflake API.\n\n```python\nclass Snowflake(LM):\n    def __init__(\n        self, \n        model,\n        credentials,\n        **kwargs):\n```\n\n**Parameters:**\n- `model` (_str_): model hosted by [Snowflake Cortex](https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions#availability).\n- `credentials`  (_dict_): connection parameters required to initialize a [snowflake snowpark session](https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/latest/api/snowflake.snowpark.Session)\n\n### Methods\n\nRefer to [`dspy.Snowflake`](https://dspy-docs.vercel.app/api/language_model_clients/Snowflake) documentation.\n'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=2042024-06-06T01:00:54.990462Z [error    ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: 3\n---\n\n# teleprompt.Ensemble\n\n### Constructor\n\nThe constructor initializes the `Ensemble` class and sets up its attributes. This teleprompter is designed to create ensembled versions of multiple programs, reducing various outputs from different programs into a single output.\n\n```python\nclass Ensemble(Teleprompter):\n    def __init__(self, *, reduce_fn=None, size=None, deterministic=False):\n```\n\n**Parameters:**\n- `reduce_fn` (_callable_, _optional_): Function used to reduce multiple outputs from different programs into a single output. A common choice is `dspy.majority`. Defaults to `None`.\n- `size` (_int_, _optional_): Number of programs to randomly select for ensembling. If not specified, all programs will be used. Defaults to `None`.\n- `deterministic` (_bool_, _optional_): Specifies whether ensemble should operate deterministically. Currently, setting this to `True` will raise an error as this feature is pending implementation. Defaults to `False`.\n\n### Method\n\n#### `compile(self, programs)`\n\nThis method compiles an ensemble of programs into a single program that when run, can either randomly sample a subset of the given programs to produce outputs or use all of them. The multiple outputs can then be reduced into a single output using the `reduce_fn`.\n\n**Parameters:**\n- `programs` (_list_): List of programs to be ensembled.\n\n**Returns:**\n- `EnsembledProgram` (_Module_): An ensembled version of the input programs.\n\n### Example\n\n```python\nimport dspy\nfrom dspy.teleprompt import Ensemble\n\n# Assume a list of programs\nprograms = [program1, program2, program3, ...]\n\n# Define Ensemble teleprompter\nteleprompter = Ensemble(reduce_fn=dspy.majority, size=2)\n\n# Compile to get the EnsembledProgram\nensembled_program = teleprompter.compile(programs)\n```'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=204
2024-06-06T01:00:54.995006Z [error    ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: 5\n---\n\n# dspy.HFClientVLLM\n\n### Usage\n\n```python\nlm = dspy.HFClientVLLM(model="meta-llama/Llama-2-7b-hf", port=8080, url="http:https://localhost")\n```\n\n### Prerequisites\n\nRefer to the [vLLM Server](https://dspy-docs.vercel.app/api/language_model_clients/HFClientVLLM) section of the `Using Local Models` documentation.\n\n### Constructor\n\nRefer to [`dspy.TGI`](https://dspy-docs.vercel.app/api/language_model_clients/TGI) documentation. Replace with `HFClientVLLM`.\n\n### Methods\n\nRefer to [`dspy.OpenAI`](https://dspy-docs.vercel.app/api/language_model_clients/OpenAI) documentation.'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=204
 14%|█████                               | 7/50 [00:00<00:00, 147.64it/s]
Traceback (most recent call last):
  File "/home/tom/dspy/dspy/primitives/assertions.py", line 220, in wrapper
    result = func(*args, **kwargs)
  File "/home/tom/tweet_generation/./main.py", line 127, in forward
    run_all_checks(tweet_text)
  File "/home/tom/tweet_generation/./main.py", line 110, in run_all_checks
    suggest_check(generated_text, "Does the assessed tweet sound natural? Say no if it sounds like an ad.", "The tweet does not sound natural. Please revise for quality.")
  File "/home/tom/tweet_generation/./main.py", line 100, in suggest_check
    dspy.Suggest(assessment_bool, suggestion)
  File "/home/tom/dspy/dspy/primitives/assertions.py", line 74, in __init__
    self.__call__()
  File "/home/tom/dspy/dspy/primitives/assertions.py", line 112, in __call__
    raise DSPySuggestionError(
dspy.primitives.assertions.DSPySuggestionError: The tweet does not sound natural. Please revise for quality.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tom/tweet_generation/./main.py", line 189, in <module>
    tweet_generator_compiled = teleprompter.compile(tweet_generator, trainset=trainset,  num_trials=2, max_bootstrapped_demos=4, max_labeled_demos=5, eval_kwargs=kwargs, requires_permission_to_run=False)
  File "/home/tom/dspy/dspy/teleprompt/mipro_optimizer.py", line 452, in compile
    candidate_program = tp.compile(student=module.deepcopy(), trainset=shuffled_trainset)
  File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 84, in compile
    self._bootstrap()
  File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 147, in _bootstrap
    success = self._bootstrap_one_example(example, round_idx)
  File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 203, in _bootstrap_one_example
    raise e
  File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 183, in _bootstrap_one_example
    prediction = teacher(**example.inputs())
  File "/home/tom/dspy/dspy/primitives/program.py", line 26, in __call__
    return self.forward(*args, **kwargs)
  File "/home/tom/dspy/dspy/primitives/assertions.py", line 294, in forward
    return wrapped_forward(*args, **kwargs)
  File "/home/tom/dspy/dspy/primitives/assertions.py", line 260, in wrapper
    output_fields = error_state[0].new_signature.output_fields
AttributeError: 'Predict' object has no attribute 'new_signature'. Did you mean: 'signature'?

[okhat]

MIPRO and assertions not compatible currently unfortunately

[tom-doerr]

Ah okay. Would it make sense to throw that as the error message? I spent quite a bit of time trying to solve this myself

[arnavsinghvi11]

Feel free to add it @tom-doerr ! Assertions are currently compatible with the FewShot optimizers, and we'll eventually get them integrated with the new Signature optimizers coming out.