FileNotFoundError: Couldn't find a module script at exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.

[xinghuang2050]

Hello,

I've encountered an issue when running the provided usage example on both versions 0.4.0 (main branch) and 1.0.0 (big-refactor branch) of your project. The example command I'm using is:

lm_eval --model hf
--model_args pretrained=EleutherAI/gpt-j-6B
--tasks hellaswag
--device cuda:0
--batch_size 8

However, this results in a FileNotFoundError as shown below:

File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/metrics.py", line 142, in
exact_match = evaluate.load("exact_match")
FileNotFoundError: Couldn't find a module script at /models/software/lm-evaluation-harness/exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.

Could you please help to address this issue? Any guidance would be much appreciated.

Thank you.

[baberabb]

Hi! looks like a problem with the evaluate library. I would check if it's working correctly using something like

import evaluate
print(evaluate.load('exact_match').compute(references=['hello'], predictions=['hello']))

Could also be a firewall issue maybe? It checks for the metric on the Hugging Face Hub when you call it.

[xinghuang2050]

Hi! looks like a problem with the evaluate library. I would check if it's working correctly using something like你好！看起来像是评估库的问题。我会使用类似的方法检查它是否正常工作

import evaluate
print(evaluate.load('exact_match').compute(references=['hello'], predictions=['hello']))

Could also be a firewall issue maybe? It checks for the metric on the Hugging Face Hub when you call it.也可能是防火墙问题？当您调用 Hugging Face Hub 时，它会检查 Hugging Face Hub 上的指标。

Thank you for your response.

Indeed, you're correct in your assumption that my local machine is unable to connect to the Hugging Face Hub. Given this situation, would it be possible for us to download the necessary files or perform checks in my local environment instead?

[baberabb]

I think installing from source might work:

git clone https://github.com/huggingface/evaluate.git
cd evaluate
pip install -e .

[Davidwhw]

I copied the exact_match.py file it required to the path where the error occurred, and now the error is gone, although this approach is not very elegant.

[baberabb]

I copied the exact_match.py file it required to the path where the error occurred, and now the error is gone, although this approach is not very elegant.

Did you have trouble downloading the models as well or was it just this library?

[Davidwhw]

I copied the exact_match.py file it required to the path where the error occurred, and now the error is gone, although this approach is not very elegant.

Did you have trouble downloading the models as well or was it just this library?

My situation is the same as @xinghuang2050 . I installed the "evaluate" module using pip install evaluate, and then copied the exact_match.py file from the source code of "evaluate" to the ... /lm-evaluation-harness/exact_match/.

[baberabb]

My situation is the same as @xinghuang2050 . I installed the "evaluate" module using pip install evaluate, and then copied the exact_match.py file from the source code of "evaluate" to the ... /lm-evaluation-harness/exact_match/.

I just meant evaluate is also a Huggingface library and if you had trouble downloading and loading the model as well. Just trying to identify if the problem is specific to the evaluate library or is related to the broader Huggingface ecosystem.

[xinghuang2050]

I copied the exact_match.py file it required to the path where the error occurred, and now the error is gone, although this approach is not very elegant.

Did you have trouble downloading the models as well or was it just this library?

My situation is the same as @xinghuang2050 . I installed the "evaluate" module using pip install evaluate, and then copied the exact_match.py file from the source code of "evaluate" to the ... /lm-evaluation-harness/exact_match/.

This approach is indeed effective and resolved my issue!

[xinghuang2050]

My situation is the same as @xinghuang2050 . I installed the "evaluate" module using pip install evaluate, and then copied the exact_match.py file from the source code of "evaluate" to the ... /lm-evaluation-harness/exact_match/.

I just meant evaluate is also a Huggingface library and if you had trouble downloading and loading the model as well. Just trying to identify if the problem is specific to the evaluate library or is related to the broader Huggingface ecosystem.

I copied the "exact_match" directory from the evaluate module to ./lm-evaluation-harness/, which resolved the FileNotFoundError issue. It seems likely that the problem was caused by my inability to access Huggingface, leading to the failure to download the 'exact_match' module. After fixing this, I was able to run the example case and the GSM8K evaluation task normally.

My suggestion would be to integrate the exact_match module directly into lm-evaluation-harness for more convenient use by others.

[haileyschoelkopf]

Reopening as a reminder that we should document this (and how to pre-download HF datasets/models/metrics in general with the TRANSFORMERS_OFFLINE=1 env vars, etc.), and/or remove a dependency on evaluate for exact_match when we don't need any of its more advanced kwargs.

[jijivski]

Just to record some information about this error.
datasets==2.14.5, transformers==4.36.2, pyarrow==10.0.0 will need this copy(copied the "exact_match" directory from evaluate module to ./lm-evaluation-harness/) but latter I encountered : lib/python3.8/site-packages/evaluate/module.py", line 443, in compute raise ValueError( ValueError: Evaluation module inputs are missing: ['predictions1', 'predictions2']. All required inputs are ['predictions1', 'predictions2'] , upgrade to pyarrow==11.0.0 worked for me.

[linjiadegou2]

I copied the exact_match.py file it required to the path where the error occurred, and now the error is gone, although this approach is not very elegant.

I solved the problem according to your method, thank you very much!

[zzhoo8]

git clone https://github.com/huggingface/evaluate.git

Same problem, it works for me

[junkangwu]

Just to record some information about this error. datasets==2.14.5, transformers==4.36.2, pyarrow==10.0.0 will need this copy(copied the "exact_match" directory from evaluate module to ./lm-evaluation-harness/) but latter I encountered : lib/python3.8/site-packages/evaluate/module.py", line 443, in compute raise ValueError( ValueError: Evaluation module inputs are missing: ['predictions1', 'predictions2']. All required inputs are ['predictions1', 'predictions2'] , upgrade to pyarrow==11.0.0 worked for me.

@jijivski I downgraded to pyarrow version 11.0.0, but it is still incompatible with lm-evaluation-harness v0.4.0. Despite the change, I'm encountering a persistent error: ValueError: Evaluation module inputs are missing: ['predictions1', 'predictions2']. All required inputs are ['predictions1', 'predictions2'].

[LuJunru]

My situation is the same as @xinghuang2050 . I installed the "evaluate" module using pip install evaluate, and then copied the exact_match.py file from the source code of "evaluate" to the ... /lm-evaluation-harness/exact_match/.

I just meant evaluate is also a Huggingface library and if you had trouble downloading and loading the model as well. Just trying to identify if the problem is specific to the evaluate library or is related to the broader Huggingface ecosystem.

I copied the "exact_match" directory from the evaluate module to ./lm-evaluation-harness/, which resolved the FileNotFoundError issue. It seems likely that the problem was caused by my inability to access Huggingface, leading to the failure to download the 'exact_match' module. After fixing this, I was able to run the example case and the GSM8K evaluation task normally.

My suggestion would be to integrate the exact_match module directly into lm-evaluation-harness for more convenient use by others.

Set HF mirrors can fix inability issue: export HF_ENDPOINT=https://hf-mirror.com

[zxia545]

Just to record some information about this error. datasets==2.14.5, transformers==4.36.2, pyarrow==10.0.0 will need this copy(copied the "exact_match" directory from evaluate module to ./lm-evaluation-harness/) but latter I encountered : lib/python3.8/site-packages/evaluate/module.py", line 443, in compute raise ValueError( ValueError: Evaluation module inputs are missing: ['predictions1', 'predictions2']. All required inputs are ['predictions1', 'predictions2'] , upgrade to pyarrow==11.0.0 worked for me.

@jijivski I downgraded to pyarrow version 11.0.0, but it is still incompatible with lm-evaluation-harness v0.4.0. Despite the change, I'm encountering a persistent error: ValueError: Evaluation module inputs are missing: ['predictions1', 'predictions2']. All required inputs are ['predictions1', 'predictions2'].

I have the same issue, have you fixed it?

[Harry-mic]

Just to record some information about this error. datasets==2.14.5, transformers==4.36.2, pyarrow==10.0.0 will need this copy(copied the "exact_match" directory from evaluate module to ./lm-evaluation-harness/) but latter I encountered : lib/python3.8/site-packages/evaluate/module.py", line 443, in compute raise ValueError( ValueError: Evaluation module inputs are missing: ['predictions1', 'predictions2']. All required inputs are ['predictions1', 'predictions2'] , upgrade to pyarrow==11.0.0 worked for me.

@jijivski I downgraded to pyarrow version 11.0.0, but it is still incompatible with lm-evaluation-harness v0.4.0. Despite the change, I'm encountering a persistent error: ValueError: Evaluation module inputs are missing: ['predictions1', 'predictions2']. All required inputs are ['predictions1', 'predictions2'].

I have the same issue, have you fixed it?

I meet the same issue. I find it a very strange situation, that the file exact_math.py we clone from the https://github.com/huggingface/evaluate is different from the file in https://huggingface.co/spaces/evaluate-metric/exact_match/blob/main/exact_match.py . The former has two items:predictions1,predictions2 while the later has another two: predictions, references. Therefore, I simply replace the exact_math.py with the one from https://huggingface.co/spaces/evaluate-metric/exact_match/blob/main/exact_match.py and it works for me.

[haileyschoelkopf]

patched in #2045