Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FLAN and T0 finetuning data #486

Closed
StellaAthena opened this issue Dec 31, 2021 · 2 comments
Closed

Add FLAN and T0 finetuning data #486

StellaAthena opened this issue Dec 31, 2021 · 2 comments
Assignees
Labels
feature request New feature or request
Projects

Comments

@StellaAthena
Copy link
Member

Is your feature request related to a problem? Please describe.
FLAN and T0 are two frameworks for finetuning language models on task-structured data. Both papers show significant improvement in LM capabilities when finetuned on their datasets, which may prove useful to us. Additionally, I want to do experiments comparing the two methodologies.

Describe the solution you'd like
Process the data in a megatron-compliant fashion and create downloaders for each dataset.

@StellaAthena StellaAthena added the feature request New feature or request label Dec 31, 2021
@StellaAthena
Copy link
Member Author

@uSaiPrashanth is working on T0
@Vaibhavs10 is working on FLAN

@StellaAthena StellaAthena added this to To do in 1T or BUST via automation Dec 31, 2021
@StellaAthena StellaAthena moved this from To do to In progress in 1T or BUST Dec 31, 2021
@uSaiPrashanth
Copy link
Member

Update: I am currently working on grabbing data from p3 and trying to shape it in a format accepted by neox. The plan is to concatenate input and target of each prompt and save it in a jsonl format. Following that, the data will be preprocessed using tools/preprocess_data.py and would be converted to a version compatible with megatron

@StellaAthena StellaAthena linked a pull request Sep 18, 2022 that will close this issue
1T or BUST automation moved this from In progress to Done Apr 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
Development

Successfully merging a pull request may close this issue.

3 participants