Post | Medium Post | Project Page | Generate | Telegram-bot | Technical Report
Kandinsky 3.0 is an open-source text-to-image diffusion model built upon the Kandinsky2-x model family. In comparison to its predecessors, Kandinsky 3.0 incorporates more data and specifically related to Russian culture, which allows to generate pictures related to Russin culture. Furthermore, enhancements have been made to the text understanding and visual quality of the model, achieved by increasing the size of the text encoder and Diffusion U-Net models, respectively.
For more information: details of training, example of generations check out our post. The english version will be released in a couple of days.
Architecture consists of three parts:
- Text encoder Flan-UL2 (encoder part) - 8.6B
- Latent Diffusion U-Net - 3B
- MoVQ encoder/decoder - 267M
We release our two models:
- Base: Base text-to-image diffusion model. This model was trained over 2M steps on 400 A100
- Inpainting: Inpainting version of the model. The model was initialized from final checkpoint of base model and trained 250k steps on 300 A100.
To install repo first one need to create conda environment:
conda create -n kandinsky -y python=3.8;
source activate kandinsky;
pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu113/torch_stable.html;
pip install -r requirements.txt;
The exact dependencies is got using pip freeze
and can be found in exact_requirements.txt
Check our jupyter notebooks with examples in ./examples
folder
from kandinsky3 import get_T2I_pipeline
t2i_pipe = get_T2I_pipeline('cuda', fp16=True)
image = t2i_pipe( "A cute corgi lives in a house made out of sushi.")
from kandinsky3 import get_inpainting_pipeline
inp_pipe = get_inpainting_pipeline('cuda', fp16=True)
image = ... # PIL Image
mask = ... # Numpy array (HxW). Set 1 where image should be masked
image = inp_pipe( "A cute corgi lives in a house made out of sushi.", image, mask)
- Vladimir Arkhipkin: Github
- Anastasia Maltseva Github
- Andrei Filatov Github,
- Igor Pavlov: Github
- Julia Agafonova
- Arseniy Shakhmatov: Github, Blog
- Andrey Kuznetsov: Github, Blog
- Denis Dimitrov: Github, Blog
@misc{arkhipkin2023kandinsky,
title={Kandinsky 3.0 Technical Report},
author={Vladimir Arkhipkin and Andrei Filatov and Viacheslav Vasilev and Anastasia Maltseva and Said Azizov and Igor Pavlov and Julia Agafonova and Andrey Kuznetsov and Denis Dimitrov},
year={2023},
eprint={2312.03511},
archivePrefix={arXiv},
primaryClass={cs.CV}
}