Eval Harness Refactor Help

[StellaAthena]

We recently pushed v0.4.0 of the LM Eval Harness (previously referred to as the big-refactor branch). Large refactors always come with some transition pains, so the purpose of this issue is to document issues people have and what the best solutions are. This post will be continuously updated with recommendations for fixing common problems.

To update your local copy of this repo

git branch -m big-refactor main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

For people using a fork of our library
Before doing the above, go to Settings. In the Default branch section, select Switch to another branch and choose big-refactor. Then select Rename branch to and set it to main.

[haileyschoelkopf]

To update your local clone of this repository:

git branch -m big-refactor main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

[pminervini]

Hi all! What should we use instead of lm_eval.evaluator.simple_evaluate ?

[haileyschoelkopf]

That should still work! But is now also accessible in lm_eval.simple_evaluate() directly

[haileyschoelkopf]

Closing this as v0.4.0 has been live for a while and we just pushed v0.4.1 !

Further questions can be directed to Issues (use the 'asking questions' label) or to EleutherAI discord's #lm-thunderdome channel.