DSPY Copro Tutorial for Hotpot QA doesn't work with Bedrock Claude Sonnet Model

[suesther]

I word for word copied the tutorial for using DSPY Copro to optimize the hotpot example on the dspy website, but it didn't work.

from dsp import AWSAnthropic
import dspy
from dspy.datasets import HotPotQA
from dspy.evaluate import Evaluate
from dspy.teleprompt import COPRO


dataset = HotPotQA(train_seed=1, train_size=20, eval_seed=2023, dev_size=50, test_size=0)

trainset, devset = dataset.train, dataset.dev
print(trainset)
print(devset)
sonnet = AWSAnthropic(
    aws_provider = dspy.Bedrock(region_name="us-west-2"),
    model="anthropic.claude-3-sonnet-20240229-v1:0",
)
dspy.configure(lm=sonnet)
class CoTSignature(dspy.Signature):
    """Answer the question and give the reasoning for the same."""

    question = dspy.InputField(desc="question about something")
    answer = dspy.OutputField(desc="often between 1 and 5 words")

class CoTPipeline(dspy.Module):
    def __init__(self):
        super().__init__()

        self.signature = CoTSignature
        self.predictor = dspy.ChainOfThought(self.signature)

    def forward(self, question):
        result = self.predictor(question=question)
        return dspy.Prediction(
            answer=result.answer,
            reasoning=result.rationale,
        )
    


def validate_context_and_answer(example, pred, trace=None):
    answer_EM = dspy.evaluate.answer_exact_match(example, pred)
    return answer_EM

NUM_THREADS = 5
evaluate = Evaluate(devset=devset, metric=validate_context_and_answer, num_threads=NUM_THREADS, display_progress=True, display_table=False)
cot_baseline = CoTPipeline()

devset_with_input = [dspy.Example({"question": r["question"], "answer": r["answer"]}).with_inputs("question") for r in devset]
evaluate(cot_baseline, devset=devset_with_input)
kwargs = dict(num_threads=64, display_progress=True, display_table=0) # Used in Evaluate class in the optimization process
teleprompter = COPRO(
    metric=validate_context_and_answer,
    verbose=True,
)
compiled_prompt_opt = teleprompter.compile(cot_baseline, trainset=devset, eval_kwargs=kwargs)

Here is the error message I received.

Traceback (most recent call last):
  File "/Users/esthersu/workplace/p-chatbot-ws/src/P-Chatbot-Bedrock/src/p_chatbot_bedrock/sample_prompt_engineering.py", line 55, in <module>
    compiled_prompt_opt = teleprompter.compile(cot_baseline, trainset=devset, eval_kwargs=kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esthersu/workplace/p-chatbot-ws/src/P-Chatbot-Bedrock/src/p_chatbot_bedrock/path/to/venv/lib/python3.12/site-packages/dspy/teleprompt/copro_optimizer.py", line 171, in compile
    instruct = dspy.Predict(
               ^^^^^^^^^^^^^
  File "/Users/esthersu/workplace/p-chatbot-ws/src/P-Chatbot-Bedrock/src/p_chatbot_bedrock/path/to/venv/lib/python3.12/site-packages/dspy/predict/predict.py", line 61, in __call__
    return self.forward(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esthersu/workplace/p-chatbot-ws/src/P-Chatbot-Bedrock/src/p_chatbot_bedrock/path/to/venv/lib/python3.12/site-packages/dspy/predict/predict.py", line 103, in forward
    x, C = dsp.generate(template, **config)(x, stage=self.stage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esthersu/workplace/p-chatbot-ws/src/P-Chatbot-Bedrock/src/p_chatbot_bedrock/path/to/venv/lib/python3.12/site-packages/dsp/primitives/predict.py", line 113, in do_generate
    completions: list[Example] = [template.extract(example, p) for p in completions]
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/esthersu/workplace/p-chatbot-ws/src/P-Chatbot-Bedrock/src/p_chatbot_bedrock/path/to/venv/lib/python3.12/site-packages/dsp/templates/template_v2.py", line 152, in extract
    raw_pred = raw_pred.strip()
               ^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'strip'

[arnavsinghvi11]

Hi @suesther , does this work now with the latest commits in the repo. Just merged #843 that impacts AWS providers/models, which was in the backlogs

[suesther]

No, it unfortunately has the same error.

[lebsral]

No, it unfortunately has the same error.

@suesther can you specify which version of DSPy you used when you tried it again?

[suesther]

I ran this command:
pip install git+https://github.com/stanfordnlp/dspy.git

[suesther]

@lebsral @arnavsinghvi11 Do you guys have any ideas for what could be the issue?

[lebsral]

I have no idea. But if I was going to go about trying to figure it out I would probably start with "how is the wrong data type, a list instead of a string ending up at this point?"

dspy/dsp/templates/template_v2.py

Line 152 in 55510ee

raw_pred = raw_pred.strip()

[yhtest1]

Is there any update regarding this issue? I am still facing this issue today.