NOTE: REFERENCES TO 'gs:https://nmt-models' have been removed as we've migrated from GCP to Azure. The code provided may not work. Pull requests are welcome.
This is the code repository for Distribution Augmentation for Generative Modeling, presented at ICML 2020.
Paper link: https://proceedings.icml.cc/static/paper_files/icml/2020/6095-Paper.pdf
Unconditional samples from our autoregressive CIFAR-10 model. (150m params, t=0.94)
This code is tested on Python 3.7.3, Ubuntu 16.04, Anaconda 4.7.11, Tensorflow version 1.13.1, and CUDA 10. It requires V100 GPUs.
It requires installing blocksparse from source:
git clone [email protected]:openai/blocksparse.git
cd blocksparse
git checkout 89074c5ccf78e3a88b4aa2aefc9e208d4773dcbc # master at time of release
make compile
pip install dist/*.whlHyperparameters for experiments live in hyperparams.py. They can be selected in a training run using the --hps [name] flag. For instance:
1 gpu runs can be run as so:
CUDA_VISIBLE_DEVICES=0 python train.py --hps c10_small
8 gpu runs can be run as so:
mpiexec -n 8 python train.py --hps c10_150m_baseline
64 gpu runs depend on the specifics of your network and system. We use the -f flag with a hostfile, but see options in mpiexec for whatever works for you.
If you run imagenet64 or imagenet32, you need to download the datasets. See the corresponding class in datasets.py to see how to do that. We dumped copies of the datasets in Azure buckets for convenience.
The specific code for experiments we reported in the paper live in the experiments/ folder.