This is the code implementation of our paper accepted in EMNLP 2022:

Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, Shangsong Liang. Revisiting Parameter-Efficient Tuning: Are We Really There Yet?.

We provide a comprehensive study for existing famous Parameter-Efficient Tuning (PETuning) methods, i.e., Adapter, Prompt, LoRA, and BitFit, focusing on their performance and stability.

The code structure is based in part on P-tuning v2. (Thanks for their awesome work.)

• model: codes to implement PETuning methods.
• tasks: codes to preprocess datasets and choose model for each task.
• training: codes to define the trainer for training.
• scripts: scripts to run training, evaluation, and prediction for each task.
  • search_scripts: the scripts to perform grid search for each task.
  • multiruns_scripts: the scripts to conduct multi runs for each task, where each script contains the best hyper-parameters for corresponding task.
• arguments.py & run.py: the arguments and running codes for training, evaluation, and prediction.

torch==1.8.1
transformers==4.5.0
adapter-transformers==2.2.0

Please view requirements.txt for more details.

All datasets in GLUE and SuperGLUE will be automatically downloaded (from Huggingface Datasets APIs) when running the scripts.

To search the best hyper-parameters for each task, you can run the scripts in the scripts/search_scripts/ folder. For example, you can run the CB tasks with adapter by the command:

bash scripts/search_scripts/superglue/search_adapter.sh cb

To conduct multiple runs for one task, you can run the scripts in the scripts/multiruns_scripts/ folder. For example, you can run the CB tasks with adapter by the command:

bash scripts/multiruns_scripts/adapter/run_cb_roberta_both.sh

We provide the best hyper-parameters for each task in corresponding multi-runs scripts. If you cannot reproduce our reported results, please check the environment (package version) and conduct the grid search in your environment.

P-tuning v2

Hugging Face Transformers

Adapter-Hub

LoRA

BitFit

If you find our paper and resources useful, please kindly cite our paper:

@inproceedings{chen-etal-2022-revisiting,
    title = "Revisiting Parameter-Efficient Tuning: Are We Really There Yet?",
    author = "Chen, Guanzheng  and
      Liu, Fangyu  and
      Meng, Zaiqiao  and
      Liang, Shangsong",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.emnlp-main.168",
    pages = "2612--2626",
}