Nick Stracke¹ · Stefan A. Baumann¹ · Josh Susskind² · Miguel A. Bautista² · Björn Ommer¹

¹ CompVis Group @ LMU Munich
² Apple

This repository contains an implementation of the paper "CTRLorALTer: Conditional LoRAdapter for Efficient 0-Shot Control & Altering of T2I Models".

We present LoRAdapter, an approach that unifies both style and structure conditioning under the same formulation using a novel conditional LoRA block that enables zero-shot control. LoRAdapter is an efficient, powerful, and architecture-agnostic approach to condition text-to-image diffusion models, which enables fine-grained control conditioning during generation and outperforms recent state-of-the-art approaches.

• Implemented B-LoRA implicit content and style disentangle using LoRAdapter
• Released Code and Weights for inference

• Add training Code
• Add more structure conditioning checkpoints (including SDXL)
• Experiment with SD3

Create the conda environment

conda env create -f environment.yaml

Activate the conda environment

conda activate loradapter

All weights are available on HuggingFace.

For ease of you, you can also use the provided bash script download_weights.sh to automatically download all available weights and place them in the the right directory.

Sampling works according to the following schema:

python sample.py experiment=<check ./config/experiments>

All currently available experiments are listed in /config/experiments. Feel free to adjust the configs according to you own needs.

Sampling using the B-LoRA LoRAdapter is possible using the config sample_b-lora_sdxl.yaml. By default this will condition on both content and style of the image. For conditioning on only content or only style, change the adaption_mode to either b-lora_content or b-lora_style. Also set ignore_check to true as we are only loading the checkpoint partially.

For best results provide information about the missing modality via the text prompt or using another LoRAdapter.

If you use this codebase or otherwise found our work valuable, please cite our paper:

@misc{stracke2024loradapter,
  title={CTRLorALTer: Conditional LoRAdapter for Efficient 0-Shot Control & Altering of T2I Models},
  author={Nick Stracke and Stefan Andreas Baumann and Joshua Susskind and Miguel Angel Bautista and Björn Ommer},
  year={2024},
  eprint={2405.07913},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}