inner monologue example #610
Conversation
| self.cot_template = "\nBefore answering, I will reason in a step-by-step manner as to get the right answer, then conclude with the answer." | ||
| self.extract_answer_template = ( |
There was a problem hiding this comment.
These two variables could come from the constructor, which would make it much more customizable.
| self.extract_answer_template = ( | ||
| "\nGiven the above reasoning, the answer in the format requested by the question is:" | ||
| ) | ||
| self.openai_completion_fn = OpenAICompletionFn(**kwargs) |
There was a problem hiding this comment.
Should we allow using any CompletionFn from the registry here?
| return [self.response.strip()] | ||
|
|
||
|
|
||
| class OpenAIInnerMonologueCompletionFn: |
There was a problem hiding this comment.
Should we name this something like OpenAIChainOfThoughtCompletionFn? Usually see this technique referred to as simply "chain of thought".
|
How do you parse the answer? |
I use the model with an extraction prompt: |
evals/completion_fns/cot.py
Outdated
| cot_completion_fn=None, | ||
| extract_completion_fn=None, | ||
| registry_path=None, | ||
| extra_options=None, |
There was a problem hiding this comment.
Add type hints for these args
|
|
||
| # This model will extract the answer from the chain of thought | ||
| self.extract_answer_template = extract_answer_template | ||
| self.extract_completion_fn_instance = registry.make_completion_fn(extract_completion_fn) |
There was a problem hiding this comment.
For simplicity, should we just have one completion_fn variable for both cot and extract? (until there's a need for using different instances)
evals/completion_fns/cot.py
Outdated
|
|
||
| self.extra_options = extra_options | ||
|
|
||
| registry = Registry() |
There was a problem hiding this comment.
Ideally this would be the same instance that is created by oaieval rather than having to reconstruct it here. One way to do it could be to pass registry through to the CompletionFn constructor as a kwarg via Registry.make_completion_fn. But then it seems we would have to rely on the correct naming which seems not great.
Any other ideas?
Inner monologue CoT increase 3.5 accuracy on
born-firstfrom 63% -> 93%