Building an MMLU Eval issue

[ToadTWP666]

I am trying to execute the Building an MMLU Eval jupyter notebooks
all of the cells execute correctly until I execute the following code:

!oaieval gpt-3.5-turbo match_mmlu_anatomy

I receive the following error:

Traceback (most recent call last):
File "/anaconda/envs/azureml_py38/bin/oaieval", line 5, in
from evals.cli.oaieval import main
File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/init.py", line 1, in
from .api import check_sampled_text, completion_query, sample_freeform
File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/api.py", line 9, in
from evals.base import ModelSpec
File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/base.py", line 93, in
class ModelSpecs:
File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/base.py", line 125, in ModelSpecs
def names(self) -> dict[str, Sequence[str]]:
TypeError: 'type' object is not subscriptable

[ToadTWP666]

I tried executing using python 3.10 and still had the issue.

[andremafei]

I got several of similar error while trying to run it in Python 3.8, I quit trying to fix it because several other places were also not compatible with 3.8. There is a bug in the file "pyproject.toml" where it says:
requires-python = ">=3.8"
it should be:
requires-python = ">=3.9"

[andrew-openai]

It's a bit hard to diagnose the issue - When you switch between python versions, did you ensure to reinstall the CLI - oaieval is pointing to the right path?

[etr2460]

Closing as stale, please file a new issue if you're still seeing this