Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multitask finetuning #676

Closed
wants to merge 146 commits into from
Closed

Multitask finetuning #676

wants to merge 146 commits into from

Conversation

lintangsutawika
Copy link
Contributor

I've re-implemented the original Super Glue prompts from the T5 Paper. Currently reconciling MTF training process with improved-t5-2.0 (Since the finetuning on SuperGLUE for T5 is exactly the same as Multitask Finetuning)

cc: @haileyschoelkopf @AranKomat

lintangsutawika and others added 30 commits June 8, 2022 21:43
@lintangsutawika
first commit, adding non causal mlm dataset
8789450
@lintangsutawika
Added prefix_len to get_batch
646578d
@lintangsutawika
changes
8e726cb
@lintangsutawika
added enums.py
25dec8e
@lintangsutawika
resolved merge conflicts
b7d6783
@lintangsutawika
added prefix attention calculation to utils
035ae8b
@lintangsutawika
removed enums
a1c14a7
@lintangsutawika
added config for mlm
7d95765
@lintangsutawika
added new args for mlm and added mlm dataset to data_utils
b528969
@lintangsutawika
added tokenizer as an arg
c3830f7
@lintangsutawika
remove get_tokenizer
9e3458c
@lintangsutawika
prefix_len to prefix
d1fabea
@lintangsutawika
added NonCausalMLMDataset to build_train_valid_test_datasets
9059902
@lintangsutawika
fixed typo in conditional
cd22016
@lintangsutawika
forgot conditional
6d4be58
@lintangsutawika
change mpu functions from Megatron to GPTNeoX
b729c0b
@lintangsutawika
tokenizer
dca68d0
@lintangsutawika
removed unused args
0be06f1
@lintangsutawika
removed bos_id
51bcc46
@lintangsutawika
added prefix key in get_batch_pipe
c4bb330
@lintangsutawika
set attention_mask to have batch size same as input batch if using pr…
835a0f7 
…efix attention
@lintangsutawika
fixed the sampling
b37f5d9
@lintangsutawika
fixed the sampling arg
6be1b67
@lintangsutawika
minor fix
86d0bd0
@lintangsutawika
fixed typo
b43f35c
@lintangsutawika
edit mlm dataset to not need input-seq-length and just rely on the or…
d23d333 
…iginal seq-length.
@haileyschoelkopf
[cleanup] remove args in buld_train_valid_test_datasets
964fe83
@haileyschoelkopf
store seed in gpt2dataset class
7676b53
@haileyschoelkopf
rename: NonCausalMLMDataset -> MLMDataset
46fccb7
@haileyschoelkopf
partial first pass: add GPT2Dataset dependency
fd658e3
@lintangsutawika lintangsutawika marked this pull request as draft September 25, 2022 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@StellaAthena StellaAthena StellaAthena left review comments

@Quentin-Anthony Quentin-Anthony Awaiting requested review from Quentin-Anthony Quentin-Anthony is a code owner automatically assigned from EleutherAI/pm-gptneo

@ShivanshuPurohit ShivanshuPurohit Awaiting requested review from ShivanshuPurohit ShivanshuPurohit is a code owner automatically assigned from EleutherAI/pm-gptneo

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

SuperGLUE / multitask finetuning Add FLAN and T0 finetuning data
3 participants
@lintangsutawika @StellaAthena @haileyschoelkopf