Code Tutorial + Implementation Tutorial
![Qries](https://user-images.githubusercontent.com/61938694/154516539-90e2d4d0-4383-41f4-ad32-4c6d67bd2442.jpg)
![Qries](https://user-images.githubusercontent.com/61938694/154628085-eede604f-442d-4bdb-a1ed-5ad3264e5aa0.jpg)
Vector Quantized Generative Adversarial Networks (VQGAN) is a generative model for image modeling. It was introduced in Taming Transformers for High-Resolution Image Synthesis. The concept is build upon two stages. The first stage learns in an autoencoder-like fashion by encoding images into a low-dimensional latent space, then applying vector quantization by making use of a codebook. Afterwards, the quantized latent vectors are projected back to the original image space by using a decoder. Encoder and Decoder are fully convolutional. The second stage is learning a transformer for the latent space. Over the course of training it learns which codebook vectors go along together and which not. This can then be used in an autoregressive fashion to generate before unseen images from the data distribution.
conda init
conda create -n vqgan
conda activate vqgan
conda install -y -c conda-forge pytorch-gpu torchvision cudatoolkit=11.3
conda install -y -c anaconda notebook nb_conda_kernels ipywidgets ipykernel tqdm
pip install albumentations matplotlib
Original Left | Reconstruction Middle Left | Completion Middle Right | New Image Right
Note: Let the model train for even longer to get better results.
- (optional) Configure Hyperparameters in
training_vqgan.py
- Set path to dataset in
training_vqgan.py
python training_vqgan.py
- (optional) Configure Hyperparameters in
training_transformer.py
- Set path to dataset in
training_transformer.py
python training_transformer.py
@misc{esser2021taming,
title={Taming Transformers for High-Resolution Image Synthesis},
author={Patrick Esser and Robin Rombach and Björn Ommer},
year={2021},
eprint={2012.09841},
archivePrefix={arXiv},
primaryClass={cs.CV}
}