Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Single doc results of model inference not found #1718

Open
EugeneYCSong1212 opened this issue Apr 18, 2024 · 0 comments
Open

Single doc results of model inference not found #1718

EugeneYCSong1212 opened this issue Apr 18, 2024 · 0 comments

Comments

@EugeneYCSong1212
Copy link

Hi~Referring to the doc, I think if I add '--output' and '--log_samples' argument I could get every inference result from one doc. I have executed the basic demo with these two arguments like this:lm_eval --model hf --model_args pretrained=EleutherAI/gpt-j-6B --tasks hellaswag --device cuda:0 --batch_size 8 --output_path /home/work/ais-harness-data-test/harness-res/0418/ --log_samples
I have found 'pretrained=EleutherAI__gpt-j-6B_hellaswag.jsonl' under the result output dir but I am confused of the result. The attachment is a result demo of one doc.
result_demo.txt
I have found the dataset doc details and some results after computation. However, I do not find the inference result of the model. Has it been provided? If not, is there any way to obtain it or will it be supported in the future. I am looking forward to your reply, thanks a lot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant