The script relys on Slurm, If you run it on non-slurm envoitments,

Just use VLLM to create a openai compatible server, and insert to 'server.csv'

IP,PORT,MODEL_NAME

If you run it on slurm envoirment, change the partition name to your own partition in make_n_server.py

then, you can run the run_with_earlystopping.py for datasets.

python run_with_earlystopping.py MODEL_NAME DATA_DIR_NAME

datasets were given by the first part of DATA_DIR_NAME arguments, like gsm8k-llama3-8b-new-mcts-8 for gsm8k , can selected in,

        'gsm8k-llama3-8b-new-mcts-8',
         'gsmhard-llama3-8b-new-mcts-8',
         'olympiadbench-llama3-8b-new-mcts-8',
         'GAIC-llama3-8b-new-mcts-8',
         'MATH-llama3-8b-new-mcts-8',
         'AIME-llama3-8b-mcts-2'

Using run_olympics.py to run all of them.

Alert: That would consume a long time.

This project were still in a very earlier stage for explore, pay attentions for algorithm's output, and do not deploy it to real-world product without fully test.

https://arxiv.org/abs/2406.07394