Skip to content

shirley-wu/cot_decoding

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

A personal reproduction of DeepMind's recent great work: Chain-of-Thought Reasoning Without Prompting

Reproduction Results

Results with mistralai/Mistral-7B-Instruct-v0.1 and mistralai/Mistral-7B-v0.1 on GSM-8k dataset:

Mistral-base Mistral-Instruct
Greedy 10.24 32.22
Self consistency 14.63 46.02
CoT decoding (agg path) 20.17 46.40
CoT decoding (max path) 13.72 39.27

For comparison, the results reported in the original paper are:

Mistral-base Mistral-Instruct
Greedy 9.9 31.2
CoT decoding (agg path) 25.1 38.2

How to Run

python main.py --model_name_or_path mistralai/Mistral-7B-Instruct-v0.1 --encode_format instruct --max_new_tokens 512 --decoding cot --output_fname outputs/mistral-instruct.jsonl
python main.py --model_name_or_path mistralai/Mistral-7B-v0.1 --encode_format qa --max_new_tokens 256 --decoding cot --output_fname outputs/mistral-base.jsonl

Please adjust batch size by --batch_size xxx based on your own GPU configuraion.

Dependency

Install transformers==4.38.1. It's crucial to have >=4.38.0!

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages