UNIPELT, UNIPELT (AP) and UNIPELT (APL)

[km5ar]

1. I'm wondering what's the different between UNIPELT implementation in Adapters and UNIPELT (AP)/UNIPELT (APL) in the original paper(Mao et al. (2022))?

2. are there any AP and APL version of UNIPELT in Adapters already?

3. is AP and APL version just two/three UNIPELT stack on top of each others?

[calpt]

Hey, the UniPELT AP and APL variants in the paper compose different types of adapters. Specifically:

UniPELT APL combines bottleneck adapters (A), prefix tuning (P) and LoRA (L). This is the default version in Adapters if you use UniPELTConfig():

config = UniPELTConfig()
model.add_adapter("unipelt_apl", config=config)

To make this clearer, we can define the same config via ConfigUnion():

config = ConfigUnion(
    PrefixTuningConfig(prefix_length=10, use_gating=True),
    SeqBnConfig(reduction_factor=16, use_gating=True),
    LoRAConfig(r=8, alpha=2, use_gating=True)
)
model.add_adapter("unipelt_apl", config=config)

This is identical to using UniPELTConfig()

UniPELT AP combines bottleneck adapters (A), prefix tuning (P). We can also define this using ConfigUnion() in Adapters:

config = ConfigUnion(
    PrefixTuningConfig(prefix_length=10, use_gating=True),
    SeqBnConfig(reduction_factor=16, use_gating=True)
)
model.add_adapter("unipelt_ap", config=config)

Hope this helps!

[km5ar]

thank you a lot !!!!!!!!!!!!!!!!!!!!!!!