Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix currently uploaded eval-harness numbers for 1.3B ; 6.7B #37

Closed
haileyschoelkopf opened this issue Dec 20, 2022 · 3 comments
Closed
Assignees
Labels
bug Something isn't working

Comments

@haileyschoelkopf
Copy link
Collaborator

Currently some of the 0 and 5 shot evals I ran appear to be wrong. (the 6.7B and 1.3B evals, for sure.) Not sure what went wrong but rerunning is quick.

I'll pull the ones that may be bad from the repo asap! We'll need to rerun these.

@haileyschoelkopf haileyschoelkopf added the bug Something isn't working label Dec 20, 2022
@StellaAthena
Copy link
Member

Crossposting from the Discord for transparency: we found a bug in our code introduced by a new feature we added between the training and evaluation of the models. It has been corrected but many, if not all, of the evaluations need to be rerun.

@haileyschoelkopf
Copy link
Collaborator Author

Yup! I believe all but 1.3B are now corrected, though I need to do another pass through them all and delete bad ones. Should have those soon, the cluster has just been giving me trouble with freezes + not accepting my jobs for some reason.

Also todo:

  • Create .feather file + instructions to use for plots with Igor's code
  • Rename lambada task to lambada_openai everywhere?

@haileyschoelkopf
Copy link
Collaborator Author

All evals in this repo should be correct now, as far as I know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants