Skip to content

Commit

Permalink
Fix pip install path in mmlu.ipynb (openai#121)
Browse files Browse the repository at this point in the history
# Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, __failure to follow
the guidelines below will result in the PR being closed automatically__.
Note that even if the criteria are met, that does not guarantee the PR
will be merged nor GPT-4 access granted. 🚨

__PLEASE READ THIS__:

In order for a PR to be merged, it must fail on GPT-4. We are aware that
right now, users do not have access, so you will not be able to tell if
the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep
in mind as we run the eval, if GPT-4 gets higher than 90% on the eval,
we will likely reject since GPT-4 is already capable of completing the
task.

We plan to roll out a way for users submitting evals to see the eval
performance on GPT-4 soon. Stay tuned! Until then, you will not be able
to see the eval performance on GPT-4. We encourage partial PR's with
~5-10 example that we can then run the evals on and share the results
with you so you know how your eval does with GPT-4 before writing all
100 examples.

## Eval details 📑
The current pip command in `mmlu.ipynb` will raise an error as blow.
```shell
ERROR: file:https:///home/ubuntu/repo/evals/examples does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.
```
This PR modifies the command to be consistent with `lafand-mt.ipynb`
  • Loading branch information
RangeKing committed Apr 21, 2023
1 parent abea36c commit b138b83
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion examples/mmlu.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"outputs": [],
"source": [
"# Install, and download MMLU if you haven't already\n",
"%pip install -e .\n",
"%pip install -e ../.\n",
"\n",
"!curl -O https://people.eecs.berkeley.edu/~hendrycks/data.tar\n",
"!tar -xf data.tar\n",
Expand Down

0 comments on commit b138b83

Please sign in to comment.