-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Idea for Evals: GPT matches lyrics with song name #390
Labels
Idea for Eval
These issues keep track of requests for different kinds of eval PRs
Comments
Converted to a discussion: #391 |
Ein-Tim
changed the title
Eval idea: GPT matches lyrics with song name
Idea for Evals: GPT matches lyrics with song name
Apr 11, 2023
As written by @andrew-openai in #632, I'm reopening this issue and have changed the title accordingly. Please also apply the
Idea for Eval
|
andrew-openai
added
the
Idea for Eval
These issues keep track of requests for different kinds of eval PRs
label
Apr 11, 2023
12 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Eval description
An eval checking whether GPT-3.5 & GPT-4 can accurately match given lyrics to the song name.
Problem and motivation
After some testing with GPT-3.5 & GPT-4 (via ChatGPT+) I was honestly disappointed at how bad the LLM performed on these tasks, especially because a simple Google search of the lyrics nearly always brought up the correct song.
Examples
Example one: ❌❌
Example two: ❌❌
Example three: ❌✅
Example eval prompt
Is this something you're interested in working on
I'd really like to provide this eval however I currently neither have the time to do so nor sufficient technical skills. Thus, I'm sharing my idea here and hope that someone will provide a PR based on this idea!
The text was updated successfully, but these errors were encountered: