-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to eval output with ideal_answer directly without having to define the completion_fn ? #1342
Comments
Hey @liuyaox, I'm not entirely sure if I've grasped your question accurately, but I'll endeavor to provide the best assistance possible. I am assuming this is intended for your personal use case on a fork of the repo, and not with an aim to contribute to the main repository. The guidelines will be virtually the same in either case, but I will not delve deeply into contribution conventions in this response for the sake of brevity. You can find more detailed information in the documentation. To run this and obtain an evaluation score based on the models' responses, follow these steps:
NOTE: Please record the folder name and file name as they will be required shortly.
In your scenario, it would be:
Provided your environment is configured correctly and all files are correctly placed, executing the above command should initiate the evaluation process. Lastly, please direct future inquiries of this nature to the Discussion Tab, as it is a more appropriate platform for seeking guidance on understanding or running the repo, where as this tab is meant for reporting implementation issues. You're more likely to receive a response to your question there. I hope this assists you, and I am here for any further queries you may have! |
Describe the feature or improvement you're requesting
I have already had the output (generated from LLM) and ideal_answers in my jsonl file. For a look:
I don't need to define the completion_fn, because it's used to generate output which I have already had.
So, how can I eval output with ideal_answer directly ?
Thanks a lot.
Additional context
No response
The text was updated successfully, but these errors were encountered: