To train a model, you need to run the main.py
script with the appropriate configuration file. For example, to train a model to generate keyframes you can run the following command:
python main.py --base configs/example_training/svd_image_reference.yaml --wandb True lightning.trainer.num_nodes \ lightning.strategy=deepspeed_stage_1 lightning.trainer.precision=32 model.base_learning_rate=1.e-5 \
data.params.train.datapipeline.filelist=/data/home/antoni/datasets/filelist_celebhq_hdtf.txt \
data.params.train.datapipeline.video_folder=video_crop \
data.params.train.datapipeline.audio_folder=audio_emb \
data.params.train.datapipeline.latent_folder=video_crop_emb \
data.params.train.loader.num_workers=6 \
data.params.train.datapipeline.audio_in_video=False \
data.params.train.datapipeline.load_all_possible_indexes=False \
lightning.trainer.devices=1 lightning.trainer.accumulate_grad_batches=1 data.params.train.datapipeline.virtual_increase=1000 \
model.params.network_config.params.audio_cond_method=bite data.params.train.loader.batch_size=28 model.params.loss_fn_config.params.lambda_lower=2. \
model.params.network_config.params.skip_time=True \
data.params.train.datapipeline.change_file_proba=1.
Ensure that the data configuration is correctly set in the configuration file or via command line.
If need to resume the training you can use the --resume
flag with the path to the checkpoint you want to resume from.
E.g.:
python main.py --resume logs/2024-07-23T14-34-12_example_training-svd_image --base configs/example_training/svd_image.yaml --wandb True lightning.trainer.num_nodes 8 \
........
To run inference on a trained model, you can use the scripts/sampling/simple_image_sample.py
script. For example, to generate a sequence of frames from a single image, you can run the following command:
python scripts/sampling/simple_image_sample.py --video_path=/fsx/rs2517/data/HDTF/video_crop/WDA_BarackObama_001.mp4 --cond_aug 0. --audio_path=/fsx/rs2517/data/HDTF/ \ audio_emb/WDA_BarackObama_001_wav2vec2_emb.pt --model_config=scripts/sampling/configs/svd_image.yaml --resize_size=512 --use_latent=True --num_steps=10 \ --max_seconds=10 '--force_uc_zero_embeddings=[cond_frames, audio_emb]' --video_folder=/fsx/behavioural_computing_data/voxceleb2 --latent_folder=/fsx/rs2517/data/ \ voxceleb2_sd_latent --latent_folder=video_crop_emb --video_folder=video_crop
You can check the script for more parameters and options.
Current best model path for image generation: /data/home/antoni/code/generative-models/logs/2024-07-23T14-34-12_example_training-svd_image/checkpoints/last.ckpt/checkpoint/mp_rank_00_model_states.pt
- config at /data/home/antoni/code/generative-models/scripts/sampling/configs/svd_image.yaml
To infer the model, you can run the following command:
python scripts/sampling/simple_image_sample.py --video_path=/fsx/rs2517/data/HDTF/video_crop/WDA_BarackObama_001.mp4 --cond_aug 0. --audio_path=/fsx/rs2517/data/HDTF/ \ audio_emb/WDA_BarackObama_001_wav2vec2_emb.pt --model_config=scripts/sampling/configs/svd_image.yaml --resize_size=512 --use_latent=True --num_steps=10 \ --max_seconds=10 '--force_uc_zero_embeddings=[cond_frames, audio_emb]' --video_folder=/fsx/behavioural_computing_data/voxceleb2 --latent_folder=/fsx/rs2517/data/ \ voxceleb2_sd_latent --latent_folder=video_crop_emb --video_folder=video_crop
It has been trained with this command:
python main.py --resume logs/2024-07-23T14-34-12_example_training-svd_image --base configs/example_training/svd_image.yaml --wandb True lightning.trainer.num_nodes 8 \
lightning.strategy=deepspeed_stage_1 lightning.trainer.precision=32 model.base_learning_rate=1.e-5 \
data.params.train.datapipeline.filelist=/data/home/antoni/datasets/filelist_celebhq_hdtf.txt \
data.params.train.datapipeline.video_folder=video_crop \
data.params.train.datapipeline.audio_folder=audio_emb \
data.params.train.datapipeline.latent_folder=video_crop_emb \
data.params.train.loader.num_workers=6 \
data.params.train.datapipeline.audio_in_video=False \
data.params.train.datapipeline.load_all_possible_indexes=False \
lightning.trainer.devices=4 lightning.trainer.accumulate_grad_batches=1 data.params.train.datapipeline.virtual_increase=1000 \
model.params.network_config.params.audio_cond_method=to_time_emb_image data.params.train.loader.batch_size=32 \
model.params.loss_fn_config.params.lambda_lower=2. model.params.network_config.params.skip_time=False \
data.params.train.datapipeline.change_file_proba=1.
March 18, 2024
- We are releasing SV3D, an image-to-video model for novel multi-view synthesis, for research purposes:
- SV3D was trained to generate 21 frames at resolution 576x576, given 1 context frame of the same size, ideally a white-background image with one object.
- SV3D_u: This variant generates orbital videos based on single image inputs without camera conditioning..
- SV3D_p: Extending the capability of SVD3_u, this variant accommodates both single images and orbital views allowing for the creation of 3D video along specified camera paths.
- We extend the streamlit demo
scripts/demo/video_sampling.py
and the standalone python scriptscripts/sampling/simple_video_sample.py
for inference of both models. - Please check our project page, tech report and video summary for more details.
To run SV3D_u on a single image:
- Download
sv3d_u.safetensors
from https://huggingface.co/stabilityai/sv3d tocheckpoints/sv3d_u.safetensors
- Run
python scripts/sampling/simple_video_sample.py --input_path <path/to/image.png> --version sv3d_u
To run SV3D_p on a single image:
- Download
sv3d_p.safetensors
from https://huggingface.co/stabilityai/sv3d tocheckpoints/sv3d_p.safetensors
- Generate static orbit at a specified elevation eg. 10.0 :
python scripts/sampling/simple_video_sample.py --input_path <path/to/image.png> --version sv3d_p --elevations_deg 10.0
- Generate dynamic orbit at a specified elevations and azimuths: specify sequences of 21 elevations (in degrees) to
elevations_deg
([-90, 90]), and 21 azimuths (in degrees) toazimuths_deg
[0, 360] in sorted order from 0 to 360. For example:python scripts/sampling/simple_video_sample.py --input_path <path/to/image.png> --version sv3d_p --elevations_deg [<list of 21 elevations in degrees>] --azimuths_deg [<list of 21 azimuths in degrees>]
To run SVD or SV3D on a streamlit server:
streamlit run scripts/demo/video_sampling.py
November 30, 2023
- Following the launch of SDXL-Turbo, we are releasing SD-Turbo.
November 28, 2023
-
We are releasing SDXL-Turbo, a lightning fast text-to image model. Alongside the model, we release a technical report
- Usage:
- Follow the installation instructions or update the existing environment with
pip install streamlit-keyup
. - Download the weights and place them in the
checkpoints/
directory. - Run
streamlit run scripts/demo/turbo.py
.
- Follow the installation instructions or update the existing environment with
- Usage:
November 21, 2023
-
We are releasing Stable Video Diffusion, an image-to-video model, for research purposes:
- SVD: This model was trained to generate 14
frames at resolution 576x1024 given a context frame of the same size.
We use the standard image encoder from SD 2.1, but replace the decoder with a temporally-aware
deflickering decoder
. - SVD-XT: Same architecture as
SVD
but finetuned for 25 frame generation. - You can run the community-build gradio demo locally by running
python -m scripts.demo.gradio_app
. - We provide a streamlit demo
scripts/demo/video_sampling.py
and a standalone python scriptscripts/sampling/simple_video_sample.py
for inference of both models. - Alongside the model, we release a technical report.
- SVD: This model was trained to generate 14
frames at resolution 576x1024 given a context frame of the same size.
We use the standard image encoder from SD 2.1, but replace the decoder with a temporally-aware
July 26, 2023
- We are releasing two new open models with a
permissive
CreativeML Open RAIL++-M
license (see Inference for file hashes):- SDXL-base-1.0: An improved version
over
SDXL-base-0.9
. - SDXL-refiner-1.0: An improved version
over
SDXL-refiner-0.9
.
- SDXL-base-1.0: An improved version
over
July 4, 2023
- A technical report on SDXL is now available here.
June 22, 2023
- We are releasing two new diffusion models for research purposes:
SDXL-base-0.9
: The base model was trained on a variety of aspect ratios on images with resolution 1024^2. The base model uses OpenCLIP-ViT/G and CLIP-ViT/L for text encoding whereas the refiner model only uses the OpenCLIP model.SDXL-refiner-0.9
: The refiner has been trained to denoise small noise levels of high quality data and as such is not expected to work as a text-to-image model; instead, it should only be used as an image-to-image model.
If you would like to access these models for your research, please apply using one of the following links: SDXL-0.9-Base model, and SDXL-0.9-Refiner. This means that you can apply for any of the two links - and if you are granted - you can access both. Please log in to your Hugging Face Account with your organization email to request access. We plan to do a full release soon (July).
Modularity is king. This repo implements a config-driven approach where we build and combine submodules by
calling instantiate_from_config()
on objects defined in yaml configs. See configs/
for many examples.
For training, we use PyTorch Lightning, but it should be easy to use other
training wrappers around the base modules. The core diffusion model class (formerly LatentDiffusion
,
now DiffusionEngine
) has been cleaned up:
- No more extensive subclassing! We now handle all types of conditioning inputs (vectors, sequences and spatial
conditionings, and all combinations thereof) in a single class:
GeneralConditioner
, seesgm/modules/encoders/modules.py
. - We separate guiders (such as classifier-free guidance, see
sgm/modules/diffusionmodules/guiders.py
) from the samplers (sgm/modules/diffusionmodules/sampling.py
), and the samplers are independent of the model. - We adopt the "denoiser framework" for both training and inference (most notable
change is probably now the option to train continuous time models):
- Discrete times models (denoisers) are simply a special case of continuous time models (denoisers);
see
sgm/modules/diffusionmodules/denoiser.py
. - The following features are now independent: weighting of the diffusion loss
function (
sgm/modules/diffusionmodules/denoiser_weighting.py
), preconditioning of the network (sgm/modules/diffusionmodules/denoiser_scaling.py
), and sampling of noise levels during training (sgm/modules/diffusionmodules/sigma_sampling.py
).
- Discrete times models (denoisers) are simply a special case of continuous time models (denoisers);
see
- Autoencoding models have also been cleaned up.
git clone https://github.com/Stability-AI/generative-models.git
cd generative-models
This is assuming you have navigated to the generative-models
root after cloning it.
NOTE: This is tested under python3.10
. For other python versions, you might encounter version conflicts.
PyTorch 2.0
# install required packages from pypi
python3 -m venv .pt2
source .pt2/bin/activate
pip3 install -r requirements/pt2.txt
pip3 install .
pip3 install -e git+https://github.com/Stability-AI/datapipelines.git@main#egg=sdata
This repository uses PEP 517 compliant packaging using Hatch.
To build a distributable wheel, install hatch
and run hatch build
(specifying -t wheel
will skip building a sdist, which is not necessary).
pip install hatch
hatch build -t wheel
You will find the built package in dist/
. You can install the wheel with pip install dist/*.whl
.
Note that the package does not currently specify dependencies; you will need to install the required packages, depending on your use case and PyTorch version, manually.
We provide a streamlit demo for text-to-image and image-to-image sampling
in scripts/demo/sampling.py
.
We provide file hashes for the complete file as well as for only the saved tensors in the file (
see Model Spec for a script to evaluate that).
The following models are currently supported:
- SDXL-base-1.0
File Hash (sha256): 31e35c80fc4829d14f90153f4c74cd59c90b779f6afe05a74cd6120b893f7e5b Tensordata Hash (sha256): 0xd7a9105a900fd52748f20725fe52fe52b507fd36bee4fc107b1550a26e6ee1d7
- SDXL-refiner-1.0
File Hash (sha256): 7440042bbdc8a24813002c09b6b69b64dc90fded4472613437b7f55f9b7d9c5f Tensordata Hash (sha256): 0x1a77d21bebc4b4de78c474a90cb74dc0d2217caf4061971dbfa75ad406b75d81
- SDXL-base-0.9
- SDXL-refiner-0.9
- SD-2.1-512
- SD-2.1-768
Weights for SDXL:
SDXL-1.0:
The weights of SDXL-1.0 are available (subject to
a CreativeML Open RAIL++-M
license) here:
- base model: https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/
- refiner model: https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/
SDXL-0.9: The weights of SDXL-0.9 are available and subject to a research license. If you would like to access these models for your research, please apply using one of the following links: SDXL-base-0.9 model, and SDXL-refiner-0.9. This means that you can apply for any of the two links - and if you are granted - you can access both. Please log in to your Hugging Face Account with your organization email to request access.
After obtaining the weights, place them into checkpoints/
.
Next, start the demo using
streamlit run scripts/demo/sampling.py --server.port <your_port>
Images generated with our code use the invisible-watermark library to embed an invisible watermark into the model output. We also provide a script to easily detect that watermark. Please note that this watermark is not the same as in previous Stable Diffusion 1.x/2.x versions.
To run the script you need to either have a working installation as above or try an experimental import using only a minimal amount of packages:
python -m venv .detect
source .detect/bin/activate
pip install "numpy>=1.17" "PyWavelets>=1.1.1" "opencv-python>=4.1.0.25"
pip install --no-deps invisible-watermark
To run the script you need to have a working installation as above. The script
is then useable in the following ways (don't forget to activate your
virtual environment beforehand, e.g. source .pt1/bin/activate
):
# test a single file
python scripts/demo/detect.py <your filename here>
# test multiple files at once
python scripts/demo/detect.py <filename 1> <filename 2> ... <filename n>
# test all files in a specific folder
python scripts/demo/detect.py <your folder name here>/*
We are providing example training configs in configs/example_training
. To launch a training, run
python main.py --base configs/<config1.yaml> configs/<config2.yaml>
where configs are merged from left to right (later configs overwrite the same values). This can be used to combine model, training and data configs. However, all of them can also be defined in a single config. For example, to run a class-conditional pixel-based diffusion model training on MNIST, run
python main.py --base configs/example_training/toy/mnist_cond.yaml
NOTE 1: Using the non-toy-dataset
configs configs/example_training/imagenet-f8_cond.yaml
, configs/example_training/txt2img-clipl.yaml
and configs/example_training/txt2img-clipl-legacy-ucg-training.yaml
for training will require edits depending on the
used dataset (which is expected to stored in tar-file in
the webdataset-format). To find the parts which have to be adapted, search
for comments containing USER:
in the respective config.
NOTE 2: This repository supports both pytorch1.13
and pytorch2
for training generative models. However for
autoencoder training as e.g. in configs/example_training/autoencoder/kl-f4/imagenet-attnfree-logvar.yaml
,
only pytorch1.13
is supported.
NOTE 3: Training latent generative models (as e.g. in configs/example_training/imagenet-f8_cond.yaml
) requires
retrieving the checkpoint from Hugging Face and replacing
the CKPT_PATH
placeholder in this line. The same is to be done
for the provided text-to-image configs.
The GeneralConditioner
is configured through the conditioner_config
. Its only attribute is emb_models
, a list of
different embedders (all inherited from AbstractEmbModel
) that are used to condition the generative model.
All embedders should define whether or not they are trainable (is_trainable
, default False
), a classifier-free
guidance dropout rate is used (ucg_rate
, default 0
), and an input key (input_key
), for example, txt
for
text-conditioning or cls
for class-conditioning.
When computing conditionings, the embedder will get batch[input_key]
as input.
We currently support two to four dimensional conditionings and conditionings of different embedders are concatenated
appropriately.
Note that the order of the embedders in the conditioner_config
is important.
The neural network is set through the network_config
. This used to be called unet_config
, which is not general
enough as we plan to experiment with transformer-based diffusion backbones.
The loss is configured through loss_config
. For standard diffusion model training, you will have to
set sigma_sampler_config
.
As discussed above, the sampler is independent of the model. In the sampler_config
, we set the type of numerical
solver, number of steps, type of discretization, as well as, for example, guidance wrappers for classifier-free
guidance.
For large scale training we recommend using the data pipelines from our data pipelines project. The project is contained in the requirement and automatically included when following the steps from the Installation section. Small map-style datasets should be defined here in the repository (e.g., MNIST, CIFAR-10, ...), and return a dict of data keys/values, e.g.,
example = {"jpg": x, # this is a tensor -1...1 chw
"txt": "a beautiful image"}
where we expect images in -1...1, channel-first format.