🔥 🔥 This is a curated list of research resources in Parameter-Efficient Fine-tuning (PEFT), which also goes by other names, such as Parameter-Efficient Transfer Learning (PETL).
🔥 🔥 In this repository, we have systematically examined over 100 papers. If you are interested, consider reading our recent survey paper Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey.
🔥 🔥 This repository is actively maintained. Pull requests or direct messages are welcome.
🔥 🔥 Visual PEFT Library/Benchmark is coming soon.
Pre-trained vision models (PVMs) have shown great potential for adaptability across various downstream vision tasks. However, with state-of-the-art PVMs growing to billions or even trillions of parameters, the standard full fine-tuning paradigm is becoming unsustainable due to high computational and storage demands. In response, researchers are exploring Parameter-Efficient Fine-Tuning (PEFT), which seeks to exceed the performance of full fine-tuning with minimal parameter modifications. This repository provides a comprehensive overview and offer a systematic review of the latest advancements. It introduces a categorization criterion that classifies existing methods into three categories: Addition-based Tuning, Partial-based Tuning, and Unified-based Tuning. This repository also introduces commonly used datasets and applications.
-
[1] AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition, NeurIPS 2022.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, Ping Luo.
-
[2] Convolutional Bypasses are Better Vision Transformer Adapters, Arxiv 2022.
Jie, Shibo and Deng, Zhi-Hong.
-
[3] ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning, NeurIPS 2022.
Pan, Junting and Lin, Ziyi and Zhu, Xiatian and Shao, Jing and Li, Hongsheng.
-
[4] AIM: Adapting Image Models for Efficient Video Action Recognition, ICLR 2023.
Yang, Taojiannan and Zhu, Yi and Xie, Yusheng and Zhang, Aston and Chen, Chen and Li, Mu.
-
[5] Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation, ICLR 2023.
Sharma, Mohit and Fantacci, Claudio and Zhou, Yuxiang and Koppula, Skanda and others.
-
[6] 1% VS 100%: Parameter-Efficient Low Rank Adapter for Dense Predictions, CVPR 2023.
Yin, Dongshuo and Yang, Yiran and Wang, Zhechao and Yu, Hongfeng and Wei, Kaiwen and Sun, Xian.
[Paper][Code]
-
[7] Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Tasks, NeurIPS 2022.
Yen-Cheng Liu, Chih-Yao Ma, Junjiao Tian, Zijian He, Zsolt Kira.
[Paper][Code]
-
[8] VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding, AAAI 2024.
Yi Xin, Junlong Du, Qiang Wang, Zhiwen Lin, Ke Yan.
-
[9] SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channels, IJCV 2023.
Henry Hengyuan Zhao, Pichao Wang, Yuyang Zhao, Hao Luo, Fan Wang, Mike Zheng Shou.
-
[10] Important Channel Tuning, Openreview 2023.
Hengyuan Zhao, Pichao WANG, Yuyang Zhao, Fan Wang, Mike Zheng Shou.
[Paper][Code]
-
[11] Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm, Arxiv 2023.
Zhao, Hengyuan and Luo, Hao and Zhao, Yuyang and Wang, Pichao and Wang, Fan and Shou, Mike Zheng.
[Paper][Code]
-
[12] Compacter: Efficient Low-Rank Hypercomplex Adapter Layer, NeurIPS 2021.
Karimi Mahabadi, Rabeeh and Henderson, James and Ruder, Sebastian.
-
[13] Parameter-efficient and student-friendly knowledge distillation, NeurIPS 2022.
Rao, Jun and Meng, Xv and Ding, Liang and Qi, Shuhan and Tao, Dacheng.
[Paper][Code]
-
[14] VL-adapter: Parameter-efficient transfer learning for vision-and-language tasks, CVPR 2022.
Sung, Yi-Lin and Cho, Jaemin and Bansal, Mohit.
-
[15] UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling, AAAI 2024.
Haoyu Lu, Mingyu Ding, Yuqi Huo, Guoxing Yang, Zhiwu Lu, Masayoshi Tomizuka, Wei Zhan.
-
[16] Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model, Arxiv 2023.
Zelin Peng, Zhengqin Xu, Zhilin Zeng, Lingxi Xie, Qi Tian, and Wei Shen.
[Paper][Code]
-
[17] Hydra: Multi-head Low-rank Adaptation for Parameter Efficient Fine-tuning, Arxiv 2023.
Sanghyeon Kim, Hyunmo Yang, Younghyun Kim, Youngjoon Hong, Eunbyung Park.
-
[18] MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answering, CVPR 2023.
Jingjing Jiang, Nanning Zheng.
-
[19] Vision Transformers are Parameter-Efficient Audio-Visual Learners, CVPR 2023.
Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, Gedas Bertasius.
-
[20] T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models, AAAI 2024.
Mou, Chong and Wang, Xintao and Xie, Liangbin and Zhang, Jian and Qi, Zhongang and others.
-
[21] I2V-Adapter: A General Image-to-Video Adapter for Video Diffusion Models, Arxiv 2023.
Guo, Xun and Zheng, Mingwu and Hou, Liang and Gao, Yuan and Deng, Yufan and others.
-
[22] AdaptIR: Parameter Efficient Multi-task Adaptation for Pre-trained Image Restoration Models, Arxiv 2023.
Hang Guo, Tao Dai, Yuanchao Bai, Bin Chen, Shu-Tao Xia, Zexuan Zhu.
-
[23] A Closer Look at Parameter-Efficient Tuning in Diffusion Models, Arxiv 2023.
Chendong Xiang, Fan Bao, Chongxuan Li, Hang Su, Jun Zhu.
-
[24] Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis, CVPR 2024.
Xin Zhou , Dingkang Liang , Wei Xu, Xingkui Zhu ,Yihan Xu, Zhikang Zou, Xiang Bai.
-
[1] Visual Prompt Tuning, ECCV 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, Ser-Nam Lim.
-
[2] Visual Prompt Tuning for Test-time Domain Adaptation, Arxiv 2022.
Gao, Yunhe and Shi, Xingjian and Zhu, Yi and Wang, Hao and Tang, Zhiqiang and Zhou, Xiong and others.
[Paper][Code]
-
[3] LPT: Long-tailed Prompt Tuning for Image Classification, ICLR 2023.
Dong, Bowen and Zhou, Pan and Yan, Shuicheng and Zuo, Wangmeng.
-
[4] Pro-tuning: Unified Prompt Tuning for Vision Tasks, TCSVT 2023.
Nie, Xing and Ni, Bolin and Chang, Jianlong and Meng, Gaofeng and Huo, Chunlei and others.
[Paper][Code]
-
[5] Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models, ICCV 2023.
Zha, Yaohua and Wang, Jinpeng and Dai, Tao and Chen, Bin and Wang, Zhi and Xia, Shu-Tao.
-
[6] Visual Prompt Multi-Modal Tracking, CVPR 2023.
Zhu, Jiawen and Lai, Simiao and Chen, Xin and Wang, Dong and Lu, Huchuan.
-
[7] LION: Implicit Vision Prompt Tuning, AAAI 2024.
Wang, Haixin and Chang, Jianlong and Luo, Xiao and Sun, Jinan and Lin, Zhouchen and Tian, Qi.
[Paper][Code]
-
[8] Convolutional Visual Prompt for Robust Visual Perception, NeurIPS 2023.
Tsai, Yun-Yun and Mao, Chengzhi and Yang, Junfeng.
[Paper][Code]
-
[9] ProSFDA: Prompt Learning based Source-free Domain Adaptation for Medical Image Segmentation, Arxiv 2023.
Hu, Shishuai and Liao, Zehui and Xia, Yong.
-
[10] Explicit Visual Prompting for Low-Level Structure Segmentations, CVPR 2023.
Liu, Weihuang and Shen, Xi and Pun, Chi-Man and Cun, Xiaodong.
-
[11] P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting, NeurIPS 2022.
Wang, Ziyi and Yu, Xumin and Rao, Yongming and Zhou, Jie and Lu, Jiwen.
-
[12] Exploring Visual Prompts for Adapting Large-Scale Models, Arxiv 2022.
Bahng, Hyojin and Jahanian, Ali and Sankaranarayanan, Swami and Isola, Phillip.
-
[13] Unleashing the Power of Visual Prompting At the Pixel Level, Arxiv 2023.
Wu, Junyang and Li, Xianhang and Wei, Chen and Wang, Huiyu and Yuille, Alan and Zhou, Yuyin and Xie, Cihang.
-
[14] Understanding and Improving Visual Prompting: A Label-Mapping Perspective, CVPR 2023.
Chen, Aochuan and Yao, Yuguang and Chen, Pin-Yu and Zhang, Yihua and Liu, Sijia.
-
[15] Learning to Prompt for Vision-Language Models, IJCV 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu.
-
[16] Hyperprompt: Prompt-based task-conditioning of transformers, ICML 2022.
He, Yun and Zheng, Steven and Tay, Yi and Gupta, Jai and Du, Yu and Aribandi, Vamsi and others.
[Paper][Code]
-
[17] MaPLe: Multi-modal Prompt Learning, CVPR 2023.
Khattak, Muhammad Uzair and Rasheed, Hanoona and Maaz, Muhammad and others.
-
[18] Hierarchical Prompt Learning for Multi-Task Learning, CVPR 2023.
Liu, Yajing and Lu, Yuning and Liu, Hao and An, Yaozu and Xu, Zhuoran and Yao, Zhuokun and others.
[Paper][Code]
-
[19] Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Driving, CVPR 2023.
Liang, Xiwen and Niu, Minzhe and Han, Jianhua and Xu, Hang and Xu, Chunjing and Liang, Xiaodan.
[Paper][Code]
-
[20] Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model, TMM 2023.
Xing, Yinghui and Wu, Qirui and Cheng, De and Zhang, Shizhou and Liang, Guoqiang and others.
-
[21] Tokenize Anything via Prompting, Arxiv 2023.
Pan, Ting and Tang, Lulu and Wang, Xinlong and Shan, Shiguang.
-
[22] MmAP : Multi-modal Alignment Prompt for Cross-domain Multi-task Learning, AAAI 2024.
Yi Xin, Junlong Du, Qiang Wang, Ke Yan, Shouhong Ding.
[Paper][Code]
-
[23] Diversity-Aware Meta Visual Prompting, CVPR 2023.
Qidong Huang, Xiaoyi Dong, Dongdong Chen, Weiming Zhang, Feifei Wang, Gang Hua, Nenghai Yu.
-
[24] Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Models, AAAI 2024.
Yiwen Tang, Ray Zhang, Zoey Guo, Xianzheng Ma, Dong Wang, Zhigang Wang, Bin Zhao, Xuelong Li.
[Paper][Code]
-
[1] Prefix-Tuning: Optimizing Continuous Prompts for Generation, ACL 2021.
Li, Xiang Lisa and Liang, Percy.
-
[2] Towards a Unified View on Visual Parameter-Efficient Transfer Learning, Arxiv 2023.
Yu, Bruce XB and Chang, Jianlong and Liu, Lingbo and Tian, Qi and Chen, Chang Wen.
-
[3] Exploring Efficient Few-shot Adaptation for Vision Transformers, TMLR 2023.
Xu, Chengming and Yang, Siqian and Wang, Yabiao and Wang, Zhanxiong and Fu, Yanwei and Xue, Xiangyang.
-
[4] Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learning, CVPR 2023.
Tu, Cheng-Hao and Mai, Zheda and Chao, Wei-Lun.
-
[5] A Unified Continual Learning Framework with General Parameter-Efficient Tuning, ICCV 2023.
Tu, Cheng-Hao and Mai, Zheda and Chao, Wei-Lun.
-
[1] Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks, ECCV 2020.
Zhang, Jeffrey O and Sax, Alexander and Zamir, Amir and Guibas, Leonidas and Malik, Jitendra.
[Paper][Code]
-
[2] LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning, NeurIPS 2022.
Sung, Yi-Lin and Cho, Jaemin and Bansal, Mohit.
-
[3] Vision transformer adapter for dense predictions. ICLR 2023.
Chen, Zhe and Duan, Yuchen and Wang, Wenhai and He, Junjun and Lu, Tong and Dai, Jifeng and Qiao, Yu.
-
[4] Side Adapter Network for Open-Vocabulary Semantic Segmentation, CVPR 2023.
Xu, Mengde and Zhang, Zheng and Wei, Fangyun and Hu, Han and Bai, Xiang.
-
[5] DTL: Disentangled Transfer Learning for Visual Recognition, AAAI 2024.
Fu, Minghao and Zhu, Ke and Wu, Jianxin.
[Paper][Code]
-
[6] Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictions, NeurIPS 2023.
Yin, Dongshuo and Han, Xueting and Li, Bin and Feng, Hao and Bai, Jing.
[Paper][Code]
-
[7] Ladder Fine-tuning approach for SAM integrating complementary network, Arxiv 2023.
Chai, Shurong and Jain, Rahul Kumar and Teng, Shiyu and Liu, Jiaqing and Li, Yinhao and others.
-
[1] Do Better ImageNet Models Transfer Better?, CVPR 2019.
Kornblith, Simon and Shlens, Jonathon and Le, Quoc V.
-
[2] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. ACL 2022.
Zaken, Elad Ben and Ravfogel, Shauli and Goldberg, Yoav.
-
[3] Differentially Private Bias-Term only Fine-tuning of Foundation Models, Arxiv 2022.
Bu, Zhiqi and Wang, Yu-Xiang and Zha, Sheng and Karypis, George.
[Paper][Code]
-
[4] AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks, NAACL 2022.
Fu, Chin-Lun and Chen, Zih-Ching and Lee, Yun-Ru and Lee, Hung-yi.
-
[5] Strong Baselines for Parameter Efficient Few-Shot Fine-tuning, AAAI 2024.
Basu, Samyadeep and Massiceti, Daniela and Hu, Shell Xu and Feizi, Soheil.
[Paper][Code]
-
[6] DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning, ICCV 2023.
Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, Zhenguo Li.
-
[7] Gradient-based Parameter Selection for Efficient Fine-Tuning, Arxiv 2023.
Zhi Zhang, Qizhe Zhang, Zijun Gao, Renrui Zhang, Ekaterina Shutova, Shiji Zhou, Shanghang Zhang.
-
[8] Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning, ICCV 2023.
Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, Bohan Zhuang.
-
[1] LoRA: Low-Rank Adaptation of Large Language Models. NeurIPS 2021.
Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and others.
-
[2] Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning, NeurIPS 2022.
Dongze Lian, Daquan Zhou, Jiashi Feng, Xinchao Wang.
-
[3] KronA: Parameter Efficient Tuning with Kronecker Adapter, Arxiv 2023.
Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J. Clark, Mehdi Rezagholizadeh.
[Paper)][Code]
-
[4] FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer, AAAI 2023.
Jie, Shibo and Deng, Zhi-Hong.
[Paper][Code]
-
[5] Aggregate, Decompose, and Fine-Tune: A Simple Yet Effective Factor-Tuning Method for Vision Transformer, Arxiv 2023.
Chen, Dongping.
-
[6] Strong Baselines for Parameter Efficient Few-Shot Fine-tuning, AAAI 2024.
Basu, Samyadeep and Massiceti, Daniela and Hu, Shell Xu and Feizi, Soheil.
[Paper][Code]
-
[7] Parameter-efficient Model Adaptation for Vision Transformers, AAAI 2023.
He, Xuehai and Li, Chunyuan and Zhang, Pengchuan and Yang, Jianwei and Wang, Xin Eric.
-
[8] DnA: Improving Few-Shot Transfer Learning with Low-Rank Decomposition and Alignment, ECCV 2022.
Jiang, Ziyu and Chen, Tianlong and Chen, Xuxi and Cheng, Yu and Zhou, Luowei and Yuan, Lu and others.
-
[9] Towards Efficient Visual Adaption via Structural Re-parameterization, Arxiv 2023.
Luo, Gen and Huang, Minglang and Zhou, Yiyi and Sun, Xiaoshuai and Jiang, Guannan and Wang, Zhiyu and Ji, Rongrong.
-
[10]SAM-PARSER: Fine-tuning SAM Efficiently by Parameter Space Reconstruction, AAAI 2024.
Zelin Peng, Zhengqin Xu, Zhilin Zeng, Xiaokang Yang, Wei Shen.
-
[10]DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Models, Arxiv 2023.
Shyam Marjit, Harshit Singh, Nityanand Mathur, Sayak Paul, Chia-Mu Yu, Pin-Yu Chen.
-
[1] Towards a Unified View of Parameter-Efficient Transfer Learning, ICLR 2022.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig.
-
[2] Towards a Unified View on Visual Parameter-Efficient Transfer Learning, Arxiv 2023.
Yu, Bruce XB and Chang, Jianlong and Liu, Lingbo and Tian, Qi and Chen, Chang Wen.
-
[3] Neural Prompt Search, Arxiv 2022.
Zhang, Yuanhan and Zhou, Kaiyang and Liu, Ziwei.
-
[4] Rethinking Efficient Tuning Methods from a Unified Perspective, Arxiv 2023.
Jiang, Zeyinzi and Mao, Chaojie and Huang, Ziyuan and Lv, Yiliang and Zhao, Deli and Zhou, Jingren.
[Paper][Code]
-
[5] A Unified Continual Learning Framework with General Parameter-Efficient Tuning, ICCV 2023.
Gao, Qiankun and Zhao, Chen and Sun, Yifan and Xi, Teng and Zhang, Gang and Ghanem, Bernard and Zhang, Jian.
-
[6] GIST: Improving Parameter Efficient Fine Tuning via Knowledge Interaction, Arxiv 2023.
Jiacheng Ruan, Jingsheng Gao, Mingye Xie, Suncheng Xiang, Zefang Yu, Ting Liu, Yuzhuo Fu.
[Paper][Code]
If you find our survey and repository useful for your research, please consider citing our paper:
@article{xin2024parameter,
title={Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey},
author={Xin, Yi and Luo, Siqi and Zhou, Haodi and Du, Junlong and Liu, Xiaohong and Fan, Yue and Li, Qing and Du, Yuntao},
journal={arXiv preprint arXiv:2402.02242},
year={2024}
}