3 bugs in one testcase: Hang, broken Retry on v2.4.9 and some other bonus bug on main

[tarasglek]

Inspired by "using ai for your metric" and nested dspy pipelines, I found a few bugs that are blocking me

1. 2.4.9 and main both hang. due to lock in

   dspy/dspy/primitives/assertions.py

   Line 189 in 78d95df

   with dspy.settings.lock:

   
   When I try to use a nested DSPy module with num_detector = CoT_NumberDetector().activate_assertions()

2. Once i get rid of 2.4.9 .activate_assertions() on NumberDetector, the Suggestions get ignored during retries(eg we do 3 identical llm calls, and hit the cache). Cause is

   dspy/dspy/predict/retry.py

   Line 59 in 78d95df

   if dspy.settings.backtrack_to == self:

   but not sure what the proper fix is. Changing if dspy.settings.backtrack_to == self: to if dspy.settings.backtrack_to is not None makes Suggestion()s get incorporated into retries.

3. With num_detector = CoT_NumberDetector() below, on main I get

Prediction(
    rationale='determine if the phrase has numbers. The phrase contains the number "12" spelled out as "twelve", so it does not have any digits.',
    has_numbers='No'
)
Traceback (most recent call last):
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 220, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/suggestion_bug.py", line 51, in forward
    dspy.Suggest(nested == "No", "Use words instead of numbers")
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 74, in __init__
    self.__call__()
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 112, in __call__
    raise DSPySuggestionError(
dspy.primitives.assertions.DSPySuggestionError: Use words instead of numbers

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/taras/Documents/DeepStructure/dspy/suggestion_bug.py", line 56, in <module>
    ans = qa_unoptimized(question="When was Lincoln born")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/program.py", line 26, in __call__
    return self.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 294, in forward
    return wrapped_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 260, in wrapper
    output_fields = error_state[0].new_signature.output_fields
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ChainOfThought' object has no attribute 'new_signature'. Did you mean: 'signature'?

Minimal testcase

Sorry this isn't more minimal. Not sure why github wont let me file it as attachment.

This is a contrived example where I do a second llm call to produce Suggestion()s to result of first llm call. This is the most faithful example I could come with from docs for using one DSPy pipeline to test another

import re
import os
# os.environ["DSP_CACHEBOOL"] = "False" # for dspy
os.environ["LOG_LEVEL"]="DEBUG" # for dspy

import dspy
from dspy.primitives.assertions import assert_transform_module, backtrack_handler
from dspy.predict import Retry
import json

os.environ["LOG_LEVEL"]="DEBUG" # for dspy
llm = dspy.OpenAI(model="gpt-3.5-turbo")
dspy.settings.configure(lm=llm)

class NumDetector(dspy.Signature):
    """has_numbers=bool(re.search(r'\d+', phrase))"""
    phrase = dspy.InputField()
    rval: str = dspy.OutputField(desc="True or False")

class CoT_NumberDetector(dspy.Module):
    def __init__(self):
        super().__init__()
        self.prog = dspy.ChainOfThought(NumDetector)

    def forward(self, *args, **kwargs):
        ret = self.prog(*args,**kwargs)
        dspy.Suggest(ret.rval == "True" or ret.rval == "False", "Only output True Or False as your answer")
        return ret

# num_detector = CoT_NumberDetector().activate_assertions() # .activate_assertions() causes hang due to following lock:
"""
def backtrack_handler(func, bypass_suggest=True, max_backtracks=2):
    def wrapper(*args, **kwargs):
        error_msg, result = None, None
        with dspy.settings.lock:
"""
num_detector = CoT_NumberDetector() # no hang

class CoT_Answer(dspy.Module):
    def __init__(self):
        super().__init__()
        self.prog = dspy.ChainOfThought("question -> answer")

    def forward(self,  *args, **kwargs):
        ret = self.prog(*args, **kwargs)
        with dspy.context(lm=llm):
            # this gonna hang
            nested = num_detector(phrase=ret.answer)
        print(ret.answer,nested)
        dspy.Suggest(nested.rval == "False", "Use words instead of numbers")
        return ret

qa_unoptimized = CoT_Answer().activate_assertions()

ans = qa_unoptimized(question="When was Lincoln born")
print(ans)
# print(llm.history)

On 2.4.9 with is not None change I'm able to get the following output:

Prediction(
    past_answer='Question: When was Lincoln born\n\nPast Answer: February 12, 1809',
    answer='February twelfth, eighteen hundred and nine'
)

[okhat]

Thank you @tarasglek ! This is indeed composing quite a few things, and I want to understand if the issue is just in the assertions in particular. So we can narrow this down, I suspect everything else is OK?

[tarasglek]

How can I help you narrow these down. There are definitely 2 clear bugs.

1. Retry getting confused and not changing llm params on retries (with code as is above) (number 2 in above list)
2. Deadlock in assertions (with # num_detector = CoT_NumberDetector().activate_assertions() # .activate_assertions() uncommented) (number 1 in above list)
3. This bug i am not able to debug further as a dspy newbie since it's hard to tell what types DSPy expects at different points in code without mypy

I've refactored code to use multiple processes to get around this and am able to use v2.4.9 with assertions and to use DSPy signatures/suggestions as written so long as they run in separate proceses

[arnavsinghvi11]

Hi @tarasglek , there was a recent PR pushed to main for assertions. Can you check if this error is resolved with the latest changes?

[tarasglek]

if i run above code without change with current dspy git:

python suggestion_bug.py 

It fails:

Lincoln was born on February 12, 1809. Prediction(
    rationale='produce the rval. We check if there are any numbers present in the phrase using a regular expression search.',
    rval='True'
)
Traceback (most recent call last):
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 220, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/suggestion_bug.py", line 50, in forward
    dspy.Suggest(nested.rval == "False", "Use words instead of numbers")
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 74, in __init__
    self.__call__()
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 112, in __call__
    raise DSPySuggestionError(
dspy.primitives.assertions.DSPySuggestionError: Use words instead of numbers

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/taras/Documents/DeepStructure/dspy/suggestion_bug.py", line 55, in <module>
    ans = qa_unoptimized(question="When was Lincoln born")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/program.py", line 26, in __call__
    return self.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 294, in forward
    return wrapped_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/taras/Documents/DeepStructure/dspy/.pixi/envs/default/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 260, in wrapper
    output_fields = error_state[0].new_signature.output_fields
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ChainOfThought' object has no attribute 'new_signature'. Did you mean: 'signature'?

[Drzhivago264]

Hello,
I am not quite sure what is going on and whether my issue relates to this or not.
However, dspy.HFClientVLLM is currently hanging indefinitely if the vllm server is down. I cannot even Ctrl-C out of the process, there is no error code to debug.
This is quite problematic for me because I run it on webserver and if the vllm is down then the frontend is also down.

I am currently on dspy-ai 2.4.10

client = dspy.HFClientVLLM(model="casperhansen/llama-3-8b-instruct-awq", port=80, url=split_url.scheme+":https://"+split_url.hostname)
dspy.configure(lm=client)
predict = dspy.Predict('document -> sentiment')
response = predict(document=message)  ###<-- it hangs here###