-
Notifications
You must be signed in to change notification settings - Fork 156
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix currently uploaded eval-harness numbers for 1.3B ; 6.7B #37
Comments
Crossposting from the Discord for transparency: we found a bug in our code introduced by a new feature we added between the training and evaluation of the models. It has been corrected but many, if not all, of the evaluations need to be rerun. |
Yup! I believe all but 1.3B are now corrected, though I need to do another pass through them all and delete bad ones. Should have those soon, the cluster has just been giving me trouble with freezes + not accepting my jobs for some reason. Also todo:
|
All evals in this repo should be correct now, as far as I know! |
Currently some of the 0 and 5 shot evals I ran appear to be wrong. (the 6.7B and 1.3B evals, for sure.) Not sure what went wrong but rerunning is quick.
I'll pull the ones that may be bad from the repo asap! We'll need to rerun these.
The text was updated successfully, but these errors were encountered: