TypedChainOfThought throw ValueError: 'Too many retries trying to get the correct output format. Try simplifying the requirements.'

[StevenGuo42]

In some cases the TypedChainOfThought would throw the following error, but TypedPredictor would work fine.

ValueError: ('Too many retries trying to get the correct output format. Try simplifying the requirements.', {'output': "JSONDecodeError('Trailing data')"})

Reproducible example:

input_dict = {
    "role": (str, Field(description="The role and task of the assistant")),
    "items": (list, Field(description="List of items to be mapped")),
    "categories": (list, Field(description="List of categories to map the items to")),
    "output_info": (str, Field(description="The requirement of the output")),
}

output_dict = {"mapping": (dict, Field(description="The mapping from the first variable to the second variable"))}

Input = create_model('Input', **input_dict)
Output = create_model('Output', **output_dict)

class QASignature(dspy.Signature):
    input: Input = InputField()
    output: Output = OutputField()

cot_predictor = dspy.TypedChainOfThought(QASignature)

input_data = Input(
    role = "You are a helpful assistant designed to remap coded items to coded categories. Your task is to map each item to one of the categories. ",
    items = ['apple', 'banana', 'tomato', 'cabbage', 'human'],
    categories = ['fruit', 'vegetable'],
    output_info = "each item should be mapped to one of the categories or `null` if mapping is not possible."
)

prediction = cot_predictor(input=input_data)

print(prediction.output)

Error:

ValueError: ('Too many retries trying to get the correct output format. Try simplifying the requirements.', {'output': "JSONDecodeError('Trailing data')"})

Using TypedPredictor:

-cot_predictor = dspy.TypedChainOfThought(QASignature)
+cot_predictor = dspy.TypedPredictor(QASignature)

Result:

mapping={'apple': 'fruit', 'banana': 'fruit', 'tomato': 'fruit', 'cabbage': 'vegetable', 'human': None}

[demian-overflow]

I've been able to reproduce your problem and it seems the results are highly dependent on the model

import dspy
from typing import Dict, List
from pydantic import BaseModel, Field

ollama_model = dspy.OllamaLocal(model="dolphin-mixtral:8x7b-v2.6", model_type="text")
dspy.settings.configure(lm=ollama_model)


class Input(BaseModel):
    role: str = Field(description="The role and task of the assistant")
    items: List[str] = Field(description="List of items to be mapped")
    categories: List[str] = Field(description="List of categories to map the items to")
    output_info: str = Field(description="The requirement of the output")


class Output(BaseModel):
    mapping: Dict[str, str | None] = Field(
        description="The mapping from the first variable to the second variable"
    )


class QASignature(dspy.Signature):
    input: Input = dspy.InputField()
    output: Output = dspy.OutputField()


cot_predictor = dspy.TypedChainOfThought(QASignature)
predictor = dspy.TypedPredictor(QASignature)

input_data = Input(
    role="You are a helpful assistant designed to remap coded items to coded categories. Your task is to map each item to one of the categories. ",
    items=["apple", "banana", "tomato", "cabbage", "human"],
    categories=["fruit", "vegetable"],
    output_info="each item should be mapped to one of the categories or `null` if mapping is not possible.",
)

prediction = cot_predictor(input=input_data, options={"format": "json"})
# prediction = predictor(input=input_data)

print(prediction.output)

dolphin-mixtral:8x7b-v2.6 works extremely well from the first go.
On the other hand llama3 8b seems to get the task yet persists trying to come up with new schema and doesn't work even with default predictor.
I had to redefine forward of the predictor to store all the completions made in process to get the results llama3 was making.

[Prediction(
    reasoning='Input: {"role":"You are a helpful assistant designed to remap coded items to coded categories. Your task is to map each item to one of the categories. ","items":["apple","banana","tomato","cabbage","human"],"categories":["fruit","vegetable"],"output_info":"each item should be mapped to one of the categories or `null` if mapping is not possible."}\nReasoning: Let\'s think step by step in order to map each item to one of the categories. We will iterate over the items and check which category they belong to. For example, "apple" and "banana" are fruits, so we will map them to "fruit". "tomato" is a vegetable, so we will map it to',
    output='Here\'s my response:\n\n```json\n{\n  "mapping": {\n    "apple": "fruit",\n    "banana": "fruit",\n    "tomato": "vegetable",\n    "cabbage": "vegetable",\n    "human": null\n  }\n}\n```\n\nI hope this is what you were looking for!'
), Prediction(
    reasoning='Here is the JSON object representing the output:\n\n```\n{\n  "output": {\n    "apple": {"category": "fruit"},\n    "banana": {"category": "fruit"},\n    "tomato": {"category": "vegetable"},\n    "cabbage": {"category": "vegetable"},\n    "human": null\n  }\n}\n```',
    output='{\n  "output": {\n    "apple": {"category": "fruit"},\n    "banana": {"category": "fruit"},\n    "tomato": {"category": "vegetable"},\n    "cabbage": {"category": "vegetable"},\n    "human": null\n  }\n}'
), Prediction(
    reasoning='Here is the JSON object representing the output:\n\n```\n{\n  "mapping": {\n    "apple": {"category": "fruit"},\n    "banana": {"category": "fruit"},\n    "tomato": {"category": "vegetable"},\n    "cabbage": {"category": "vegetable"},\n    "human": null\n  }\n}\n```',
    output='{\n  "mapping": {\n    "apple": {"category": "fruit"},\n    "banana": {"category": "fruit"},\n    "tomato": {"category": "vegetable"},\n    "cabbage": {"category": "vegetable"},\n    "human": null\n  }\n}'
)]

[demian-overflow]

I think making the model return correct schemas should be a part of optimizers processes. I'll try to make an example in a few days.

[StevenGuo42]

I'm wondering if there is a way to optimize the output schema only, so I won't need provide any training data for it.

[StevenGuo42]

It seams that DSPy only outputs JSON as plain text, even the model supports JSON schema. This is quite problematic, even the simple example from the documentation can't generate JSON properly on first try.

History of the Typed Predictors example from the documentation using GPT-4:

Output: {"answer": "The lazy dog", "confidence": 1.0}. Respond with a single JSON object. JSON Schema: {"properties": {"answer": {"description": "The answer for the question", "title": "Answer", "type": "string"}, "confidence": {"description": "The confidence score for the answer", "maximum": 1.0, "minimum": 0.0, "title": "Confidence", "type": "number"}}, "required": ["answer", "confidence"], "title": "Output", "type": "object"}

Output:
{"answer": "the lazy dog", "confidence": 1.0}

Answer the question based on the context and query provided, and on the scale of 10 tell how confident you are about the answer.

---

Follow the following format.

Input: ${input}
Reasoning: Let's think step by step in order to ${produce the output}. We ...
Output: ${output}. Respond with a single JSON object. JSON Schema: {"properties": {"answer": {"description": "The answer for the question", "title": "Answer", "type": "string"}, "confidence": {"description": "The confidence score for the answer", "maximum": 1.0, "minimum": 0.0, "title": "Confidence", "type": "number"}}, "required": ["answer", "confidence"], "title": "Output", "type": "object"}

---

Input: {"context":"The quick brown fox jumps over the lazy dog","query":"What does the fox jumps over?"}
Reasoning: Let's think step by step in order to produce the output. We can see from the context that the fox jumps over the lazy dog.
Output: {"answer": "The lazy dog", "confidence": 1.0}. Respond with a single JSON object. JSON Schema: {"properties": {"answer": {"description": "The answer for the question", "title": "Answer", "type": "string"}, "confidence": {"description": "The confidence score for the answer", "maximum": 1.0, "minimum": 0.0, "title": "Confidence", "type": "number"}}, "required": ["answer", "confidence"], "title": "Output", "type": "object"}







Make a very succinct json object that validates with the following schema

---

Follow the following format.

Json Schema: ${json_schema}
Json Object: ${json_object}

---

Json Schema: {"properties": {"answer": {"description": "The answer for the question", "title": "Answer", "type": "string"}, "confidence": {"description": "The confidence score for the answer", "maximum": 1.0, "minimum": 0.0, "title": "Confidence", "type": "number"}}, "required": ["answer", "confidence"], "title": "Output", "type": "object"}
Json Object: {"answer": "42", "confidence": 0.9}







Answer the question based on the context and query provided, and on the scale of 10 tell how confident you are about the answer.

---

Follow the following format.

Input: ${input}

Past Error (output): An error to avoid in the future

Reasoning: Let's think step by step in order to ${produce the output}. We ...

Output:
${output}. Respond with a single JSON object. 
You MUST use this format: {"answer": "42", "confidence": 0.9}
JSON Schema: {"properties": {"answer": {"description": "The answer for the question", "title": "Answer", "type": "string"}, "confidence": {"description": "The confidence score for the answer", "maximum": 1.0, "minimum": 0.0, "title": "Confidence", "type": "number"}}, "required": ["answer", "confidence"], "title": "Output", "type": "object"}

---

Input: {"context":"The quick brown fox jumps over the lazy dog","query":"What does the fox jumps over?"}

Past Error (output): JSONDecodeError('Trailing data')

Reasoning: Let's think step by step in order to produce the output. We need to understand the context and the query. The context is a sentence "The quick brown fox jumps over the lazy dog". The query is asking "What does the fox jumps over?". From the context, we can see that the fox jumps over the lazy dog. So, the answer is "the lazy dog". The past error was a JSONDecodeError which means the JSON format was not correct. We need to ensure that the output is in the correct JSON format as specified in the instructions.

Output:
{"answer": "the lazy dog", "confidence": 1.0}

History of this example using GPT-4:

Output: {"mapping":{"apple":"fruit","banana":"fruit","tomato":"vegetable","cabbage":"vegetable","human":null}}. Respond with a single JSON object. JSON Schema: {"properties": {"mapping": {"description": "The mapping from the first variable to the second variable", "title": "Mapping", "type": "object"}}, "required": ["mapping"], "title": "Output", "type": "object"}

Given the fields `input`, produce the fields `output`.

---

Follow the following format.

Input: ${input}
Reasoning: Let's think step by step in order to ${produce the output}. We ...
Output: ${output}. Respond with a single JSON object. JSON Schema: {"properties": {"mapping": {"description": "The mapping from the first variable to the second variable", "title": "Mapping", "type": "object"}}, "required": ["mapping"], "title": "Output", "type": "object"}

---

Input: {"role":"You are a helpful assistant designed to remap coded items to coded categories. Your task is to map each item to one of the categories. ","items":["apple","banana","tomato","cabbage","human"],"categories":["fruit","vegetable"],"output_info":"each item should be mapped to one of the categories or to `null` if mapping is not possible.","output_format":"strict json"}
Reasoning: Let's think step by step in order to map each item to one of the categories or to `null` if mapping is not possible. We start by identifying which category each item belongs to based on common knowledge. Apples and bananas are fruits, so they map to the "fruit" category. Tomatoes, although often used in cooking like a vegetable, are botanically fruits, but in common categorization, they are often considered vegetables. Cabbage is a vegetable. Humans do not fit into the categories of fruit or vegetable, so they map to `null`.
Output: {"mapping":{"apple":"fruit","banana":"fruit","tomato":"vegetable","cabbage":"vegetable","human":null}}. Respond with a single JSON object. JSON Schema: {"properties": {"mapping": {"description": "The mapping from the first variable to the second variable", "title": "Mapping", "type": "object"}}, "required": ["mapping"], "title": "Output", "type": "object"}







Make a very succinct json object that validates with the following schema

---

Follow the following format.

Json Schema: ${json_schema}
Json Object: ${json_object}

---

Json Schema: {"properties": {"mapping": {"description": "The mapping from the first variable to the second variable", "title": "Mapping", "type": "object"}}, "required": ["mapping"], "title": "Output", "type": "object"}
Json Object: Json Schema: {"properties": {"mapping": {"description": "The mapping from the first variable to the second variable", "title": "Mapping", "type": "object"}}, "required": ["mapping"], "title": "Output", "type": "object"}
Json Object: {"mapping": {}}







Given the fields `input`, produce the fields `output`.

---

Follow the following format.

Input: ${input}

Past Error (output): An error to avoid in the future

Reasoning: Let's think step by step in order to ${produce the output}. We ...

Output: ${output}. Respond with a single JSON object. JSON Schema: {"properties": {"mapping": {"description": "The mapping from the first variable to the second variable", "title": "Mapping", "type": "object"}}, "required": ["mapping"], "title": "Output", "type": "object"}

---

Input: {"role":"You are a helpful assistant designed to remap coded items to coded categories. Your task is to map each item to one of the categories. ","items":["apple","banana","tomato","cabbage","human"],"categories":["fruit","vegetable"],"output_info":"each item should be mapped to one of the categories or to `null` if mapping is not possible.","output_format":"strict json"}

Past Error (output): JSONDecodeError('Trailing data')

Reasoning: Let's think step by step in order to produce the correct mapping of items to categories. We need to categorize each item based on common knowledge about fruits and vegetables:

1. "apple" and "banana" are commonly known as fruits.
2. "tomato" is botanically a fruit but commonly classified as a vegetable in culinary contexts.
3. "cabbage" is a vegetable.
4. "human" does not fit into the categories of fruit or vegetable, so it should be mapped to `null`.

Output: {"mapping":{"apple":"fruit","banana":"fruit","tomato":"vegetable","cabbage":"vegetable","human":null}}. Respond with a single JSON object. JSON Schema: {"properties": {"mapping": {"description": "The mapping from the first variable to the second variable", "title": "Mapping", "type": "object"}}, "required": ["mapping"], "title": "Output", "type": "object"}







Make a very succinct json object that validates with the following schema

---

Follow the following format.

Json Schema: ${json_schema}
Json Object: ${json_object}

---

Json Schema: {"properties": {"mapping": {"description": "The mapping from the first variable to the second variable", "title": "Mapping", "type": "object"}}, "required": ["mapping"], "title": "Output", "type": "object"}
Json Object: Json Schema: {"properties": {"mapping": {"description": "The mapping from the first variable to the second variable", "title": "Mapping", "type": "object"}}, "required": ["mapping"], "title": "Output", "type": "object"}
Json Object: {"mapping": {}}







Given the fields `input`, produce the fields `output`.

---

Follow the following format.

Input: ${input}

Past Error (output): An error to avoid in the future

Past Error (output, 2): An error to avoid in the future

Reasoning: Let's think step by step in order to ${produce the output}. We ...

Output: ${output}. Respond with a single JSON object. JSON Schema: {"properties": {"mapping": {"description": "The mapping from the first variable to the second variable", "title": "Mapping", "type": "object"}}, "required": ["mapping"], "title": "Output", "type": "object"}

---

Input: {"role":"You are a helpful assistant designed to remap coded items to coded categories. Your task is to map each item to one of the categories. ","items":["apple","banana","tomato","cabbage","human"],"categories":["fruit","vegetable"],"output_info":"each item should be mapped to one of the categories or to `null` if mapping is not possible.","output_format":"strict json"}

Past Error (output): JSONDecodeError('Trailing data')

Past Error (output, 2): JSONDecodeError('Trailing data')

Reasoning: Let's think step by step in order to produce the correct mapping of items to categories. We need to categorize each item based on its nature:

1. "apple" and "banana" are fruits, so they should be mapped to the "fruit" category.
2. "tomato" and "cabbage" are vegetables, so they should be mapped to the "vegetable" category.
3. "human" does not fit into either "fruit" or "vegetable" categories, so it should be mapped to `null`.

We must ensure that the JSON output is correctly formatted to avoid errors like 'Trailing data'. This means checking that the JSON structure adheres strictly to the specified schema, with no extra commas, no missing brackets, and proper use of quotes.

Output: {"mapping":{"apple":"fruit","banana":"fruit","tomato":"vegetable","cabbage":"vegetable","human":null}}. Respond with a single JSON object. JSON Schema: {"properties": {"mapping": {"description": "The mapping from the first variable to the second variable", "title": "Mapping", "type": "object"}}, "required": ["mapping"], "title": "Output", "type": "object"}

[ahmed-moubtahij]

Why rely on model performance for adhering to structured output instead of constrained generation? Why not integrate with Outlines or SGLang?

[JayGhiya]

+1 happens for normal usecases even using gpt4o. so definitely optimisations are needed.

[mauceri]

I don't know if this can be useful to anyone, but here is how I proceeded to solve this problem.

First, I noticed that the outputs of LLMs are often truncated, so I systematically increased the value of max_tokens (I set it to 2048, which is sufficient for most of my use cases).
I also noticed that the outputs of some LLMs (for example, Mixtral-8x7B-Instruct-v0.1 on Anyscale) start with a valid JSON object followed by unnecessary text. So I asked ChatGPT to write a function that extracts the first occurrence of a valid JSON object from a string. I added this function as a method to the TypedPredictor module as follows:

def extract_valid_json(self, input_string):
"""
Patch Christian Mauceri
This function extracts a valid JSON object at the beginning of a string.

:param input_string: The input string
:return: A tuple containing the extracted JSON object and the remaining part of the string
"""
json_obj = None
end_index = -1

# Look for the end position of the valid JSON object
for i in range(len(input_string)):
    try:
        json_obj = json.loads(input_string[:i+1])
        end_index = i + 1
    except json.JSONDecodeError:
        continue

# If a JSON object was found
if json_obj is not None:
    return json_obj, input_string[end_index:]
else:
    raise ValueError("No valid JSON object found at the beginning of the string.")

Then I added the following lines in the first 'try' block of the forward method, right after 'value = completion[name]':

valid_json_object, remaining = self.extract_valid_json(value)
#print(f"****** {valid_json_object}")
#print(f"****** {remaining}")
value = json.dumps(valid_json_object)
I am aware that this is quite an ugly patch, but it allows me to move forward with:

Claude
OpenAI (GPT-4 and Turbo)
Various serverless LLMs provided by Anyscale