Fusion Training Question

[paen27]

Hi!

I've become quite interested in using this library for some of my work, however I'm a bit confused on how AdapterFusion training is supposed to be conducted for the best effect. The paper AdapterFusion mentions that you essentially use a dataset twice if you want to train single task adapters and the fusion layer on top, since the adapter first needs to learn the data and then you have to weigh the individual adapters against each other.
But these two parts are not trained on the exact same split of the data, are they? The paper and the example in the docs makes it seem like you just train on the same data twice, without doing any kind of hold out from the adapter for the fusion layer.
As far as I understand, if they are trained on the same, the adapter is just going to activate all the time, since it has seen the context of the input before, drowning out the others in the fusion layer and making this kind of pointless.

Am I wrong?

[calpt]

Hey @paen27,

there are multiple options how you can use AdapterFusion for transfer learning:

• similar to what you described, you could have a task adapter and the fusion layer trained on the same downstream task data. While this often results in the adapter of this dataset being activated predominantly in fusion, this is not necessarily the case. Especially low-resource tasks profit from knowledge of adapters trained on different, higher-resource datasets. You can see examples for this e.g. in Fig 4 of the AdapterFusion paper, where the adapters for QQP and MNLI are activated for various downstream tasks.
• alternatively, it's perfectly fine to transfer knowledge to a new task for which no adapter is in the set of fused adapters. Based on the findings in the paper, it can be expected that transfer to a new task works adequatly if the set of fused adapters contains adapters trained on similar & high-resource datasets.

Hope this helps!