This repository is the official implementation of X-Adapter.

X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model
Lingmin Ran, Xiaodong Cun, Jia-Wei Liu, Rui Zhao, Song Zijie, Xintao Wang, Jussi Keppo, Mike Zheng Shou

X-Adapter enable plugins pretrained on old version (e.g. SD1.5) directly work with the upgraded Model (e.g., SDXL) without further retraining.

• [17/02/2024] Inference code released

conda create -n xadapter python=3.10
conda activate xadapter

pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116
pip install -r requirements.txt

Installing xformers is highly recommended for high efficiency and low GPU cost.

[Stable Diffusion] Stable Diffusion is a latent text-to-image diffusion model capable of generating photo-realistic images given any text input. The pre-trained Stable Diffusion models can be downloaded from Hugging Face (e.g., Stable Diffusion v1-5). You can also use fine-tuned Stable Diffusion models trained on different styles (e.g., Anything V4.0, Redshift, etc.).

[ControlNet] Controlnet is a method to control diffusion model with spatial conditions. You can download ControlNet family here.

[LoRA] LoRA is a lightweight adapter to fine-tune large-scale pretrained model. It is widely used for style or identity customization in diffusion model. You can download LoRA from diffusion community(e.g., civitai).

Models can be downloaded from our HuggingFace page. Put the checkpoint in folder ./checkpoint/X-Adapter.

After preparing all checkpoints, we can run inference code using different plugins. You can refer to this tutorial to quickly get started with X-Adapter.

Set --controlnet_canny_path or --controlnet_depth_path to ControlNet's path in the bash script. The default value is its huggingface model card.

sh ./bash_scripts/canny_controlnet_inference.sh
sh ./bash_scripts/depth_controlnet_inference.sh

Set --lora_model_path to LoRA's checkpoint in the bash script. In this example we use MoXin, and we put it in folder ./checkpoint/lora.

sh ./bash_scripts/lora_inference.sh

Set --controlnet_tile_path to ControlNet-tile's path in the bash script. The default value is its huggingface model card.

sh ./bash_scripts/controlnet_tile_inference.sh

If you find X-Adapter useful for your research and applications, please cite us using this BibTeX:

@article{ran2023xadapter,
  title={X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model},
  author={Lingmin Ran and Xiaodong Cun and Jia-Wei Liu and Rui Zhao and Song Zijie and Xintao Wang and Jussi Keppo and Mike Zheng Shou},
  journal={arXiv preprint arXiv:2312.02238},
  year={2023}
}