# pip install mulankit
from diffusers import StableDiffusionPipeline
+ import mulankit
pipe = StableDiffusionPipeline.from_pretrained('Lykon/dreamshaper-8')
+ pipe = mulankit.transform(pipe, 'mulanai/mulan-lang-adapter::sd15_aesthetic.pth')
image = pipe('一只蓝色的🐶 in the 바다').images[0]
一只蓝色的 🐶 in the 바다 (Dreamshaper-8) | レゴシュワルツェネッガー (SDXL-lightning) | 一只可爱的猫头鹰 (MVDream) | 海浪风景 (AnimateDiff) |
---|---|---|---|
We present MuLan, a versatile framework to equip any diffusion model with multilingual generation abilities natively by up to 110+ languages around the world. With properly trained text encoder from noisy data, we demonstrate that MuLan could be trained on English only data and support other languages zero-shot. Additionally, we introduce Language Adapter. A language adapter with less than 20M parameters, trained against a frozen denoiser and a text encoder, can be readily combined with any homologous community models/tools, such as LoRA, LCM, ControlNet, and IP-Adapter, without any finetuning.
- release technical report
- 2024-5-14: release code and models
MuLan supports
- Base models: Stable Diffusion 1.5, 2.1, XL, Pixart-Alpha/Sigma.
- Downstream models: ControlNet, LCM, LoRA, finetuned models and etc.
- Video models: AnimateDiff.
- 3D models: MVDream.
Please refer to the USAGE.md and examples for more details.
Model | Description | Link |
---|---|---|
MuLan-Language-Adapter | Adapters for SDXL, SD1.5/2.1, Pixart | hf-model |
MuLan-Pixart | Full finetuned model | hf-model |
See more at our Huggingface 🌻 Homepage.
If you find this repo helpful, please considering citing us.
@article{lai2024mulan,
title={MuLan: Adapting Multilingual Diffusion Models for 110 + Languages},
year={2024}
}
Our work is made possible by the open-source of these great works.