BART MLM adapter

[san-deep-reddy]

I want to train BART adapter1 & the Fusion Layer using MLM and freeze adapter2 & adapter3.

model_name = 'mse30/bart-base-finetuned-pubmed' #PubmedBART
model = BartForConditionalGeneration.from_pretrained(model_name)
tokenizer = BartTokenizer.from_pretrained(model_name)
config = BartConfig.from_pretrained(model_name)
adapters.init(model)
model.add_adapter("adapter1", config="seq_bn")
model.add_adapter("adapter2", config="seq_bn")
model.add_adapter("adapter3", config="seq_bn")
model.add_adapter_fusion(["adapter1", "adapter2", "adapter3"])
model.active_adapters = ac.Fuse("adapter1", "adapter2", "adapter3") #Fusion layer after the adapters
model.train_adapter("adapter1") # Train only adapter1 on MLM and the Fusion Layer, freeze adapter2 and adapter3

How do I pass this model along with the adapters to MLM (run_mlm.py) arguments?

!python "/content/adapters/examples/pytorch/language-modeling/run_mlm.py"
--model_name_or_path "/content/adapter_files"
--train_file "/content/train.txt"
--validation_file "/content/val.txt"
--do_train
--do_eval
--learning_rate 1e-4
--num_train_epochs 10.0
--output_dir "/content/output"
--train_adapter
--adapter_config "seq_bn"

[hSterz]

Hi @san-deep-reddy,

Unfortunately you can't pass this along to the script without modifying it. You can simply replace the setup_adapter_trainingcall with the your code for adapter creation, activation, and training preparation.

With model.train_adapterbe aware that it also sets the passed adapter setup as the active setup (see here) so make sure that the active setup is as you want it after freezing the model weights.

Hope this helps!

[san-deep-reddy]

Replacing setup_adapter_training with the below code did the job

model.add_adapter("adapter1", config="seq_bn")
model.add_adapter("adapter2", config="seq_bn")
model.add_adapter("adapter3", config="seq_bn")
adapter_setup = Fuse("adapter1", "adapter2", "adapter3")
model.add_adapter_fusion(adapter_setup)
model.train_adapter_fusion([adapter_setup, 'adapter1']) #Activate fusion layer and just 'adapter 1'