-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ASR task for SUPERB #2619
Add ASR task for SUPERB #2619
Conversation
Wait until #2620 is merged before pushing the README tags in this PR |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#2620 is already merged into master. You can merge master into this branch.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
One question: aren't you adding task_templates
to the _info
method (and to the dataset_infos.json
?
great catch! i've now added the asr task template (along with a mapping from superb task -> template) and updated the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good!
I have a suggested refactoring... Tell me what you think! :)
Co-authored-by: Albert Villanova del Moral <[email protected]>
your approach is much more elegant - i've included your suggestions 🙏 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you.
This PR starts building up the SUPERB benchmark by including the ASR task as described in the SUPERB paper and
s3prl
instructions.Usage:
I've used the GLUE benchmark as a guide for filling out the README.
To move fast during the evaluation PoC I propose to merge one task at a time, so we can continue building the training / evaluation framework in parallel.
Note: codewise this PR is ready for review - I'll add the missing YAML tags once #2620 is merged :)