This project aims to simplify the process of creating a custom DeepDream model by using pretrained GoogleNet models and custom image datasets.
Here are some example visualizations created with custom DeepDream models trained on summer themed images:
-
-
Collect Images
-
Sort images into the required format.
-
Remove any corrupt images.
-
Ensure that any duplicates are removed if you have not done so already
-
Resize the dataset to speed up training.
-
Calculate the mean and standard deviation of your dataset.
-
-
If the results are not great, then you may have to go back to step 1-2 and make some changes with what images, categories, and training parameters are used.
It can take as little as 5 epochs to create visualizations that resemble your training data using the main FC/Logits layer. In order to speed up training and create better looking results, the pretrained BVLC model used is partially frozen in order to protect the lower layers from changing.
Dependencies:
You can find detailed installation instructions for Ubuntu and Windows in the installation guide.
In order to train a custom DeepDream model, you will need to create a dataset composed of images that you wish to use for training. There are a variety of ways that you can aquire images for your dataset, and you will need at least a couple hundred images for each category/class.
DeepDream is most often performed with image classification models trained on image datasets that are composed of different categories/classes. Image classification models attempt to learn the difference between different image classes and in doing so the neurons gain the ability to create dream-like hallucinations. The images you choose, the differences between them, the differences between your chosen classes, and the number of images used will greatly affect the visualizations that can be produced.
PyTorch image datasets are to be structured where the main directory/folder contains subfolders/directories for each category/class. Below an example of the required dataset structure is shown:
dataset_dir
│
└───category1
│ │ image1.jpg
│ │ image2.jpg
│ │ image3.jpg
│
└───category2
│ image1.jpg
│ image2.jpg
│ image3.jpg
Once you have created your dataset in the proper format, make sure that you remove any duplicate images if you have not done so already. There are a variety of tools that you can use for this task, including free and open source software.
If you have not done so already, you may wish to create a backup copy of your dataset.
Next you will need to verify that none of the images are corrupt in such a way that prevents PyTorch from loading them. To automatically remove any corrupt images from your dataset, use the following command:
python data_tools/remove_bad.py -delete_bad -data_path <training_data>
Next you will likely want to resize your dataset to be closer to the training image size in order to speed up training. Resizing your dataset will not prevent you from creating larger DeepDream images with the resulting model. The included resizing script will only modify images that go above the specified image size with their height or width.
To resize the images in your dataset, use the following command:
python data_tools/resize_data.py -data_path <training_data> -max_size 500
Now with your newly resized dataset, you can calculate the mean and standard deviation of your dataset for use in training, and DeepDreaming. Make sure to recalculate the mean and standard deviation again if you modify the dataset by adding or removing images.
To calculate the mean and standard deviation of your dataset, use the following command and save the output for the next step:
python data_tools/calc_ms.py -data_path <training_data>
Now you can start training your DeepDream model by running the GoogleNet training script. It's recommended that you save the model every 5-10 epochs in order to monitor the quality of the visualizations.
Basic training command:
python train_googlenet.py -data_path <training_data> -balance_classes -batch_size 96 -data_mean <mean> -data_sd <sd>
Input options:
-data_path
: Path to the dataset directory/folder that you wish to use.-data_mean
: Your precalculated list of mean values for your chosen dataset.-data_sd
: Your precalculated list of standard deviation values for your chosen dataset.
Training options:
-num_epochs
: The number of training epochs to use. Default is120
.-batch_size
: The number of training and validation images to put through the network at the same time. Default is32
.-learning_rate
: Learning rate to use with the ADAM or SGD optimizer. Default is1e-2
.-optimizer
: The optimization algorithm to use; eithersgd
oradam
; default issgd
.-train_workers
: How many workers to use for training. Default is0
.-val_workers
: How many workers to use for validation. Default is0
.-balance_classes
: Enabling this flag will balance training for each class based on class size.
Model options:
-freeze_to
: Which layer to freeze the model up to; one ofnone
,conv1
,conv2
,conv3
,mixed3a
,mixed3b
,mixed4a
,mixed4b
,mixed4c
,mixed4d
,mixed4e
,mixed5a
, ormixed5b
. Default ismixed3b
.-freeze_aux1_to
: Which layer to freeze the first auxiliary branch up to; one ofnone
,loss_conv
,loss_fc
, orloss_classifier
. Default isnone
.-freeze_aux2_to
: Which layer to freeze the second auxiliary branch up to; one ofnone
,loss_conv
,loss_fc
, orloss_classifier
. Default isnone
.-delete_branches
: If this flag is enabled, no auxiliary branches will be used in the model.
Output options:
-save_epoch
: Save the model everysave_epoch
epochs. Default is10
. Set to0
to disable saving intermediate models.-output_name
: Name of the output model. Default isbvlc_out.pth
.-individual_acc
: Enabling this flag will print the individual accuracy of each class.-save_csv
: Enabling this flag will save loss and accuracy data to txt files.
Other options:
-use_device
: Zero-indexed ID of the GPU to use pluscuda:
. Default iscuda:0
.-seed
: An integer value that you can specify for repeatable results. By default this value is random for each run.
Dataset options:
-val_percent
: The percentage of images from each class to use for validation. Default is0.2
.
After training a new DeepDream model, you'll need to test it's visualizations. The best visualizations are found in the main FC layer also known as the 'logits' layer. This script helps you quickly and easily visualize all FC layer channels for a particular model epoch, by generating a separate image for each channel.
Input options:
-model_file
: Path to the pretrained GoogleNet model that you wish to use.-model_epoch
: The training epoch that the model was saved from, to use for the output image names. Default is120
.-learning_rate
: Learning rate to use with the ADAM or L-BFGS optimizer. Default is1.5
.-optimizer
: The optimization algorithm to use; eitherlbfgs
oradam
; default isadam
.-num_iterations
: Default is 500.
Only Required If Model Doesn't Contain Them, Options:
-data_mean
: Your precalculated list of mean values that was used to train the model, if they weren't saved inside the model.-num_classes
: The number of classes that the model was trained on. Default is120
.
Output options:
-print_iter
: Print progress everyprint_iter
iterations. Set to0
to disable printing.-save_iter
: Save the images everysave_iter
iterations. Default is to0
to disable saving intermediate results.-print_iter
: Print progress everyprint_iter
iterations. Set to0
to disable printing.
Other options:
-use_device
: Zero-indexed ID of the GPU to use pluscuda:
. Default iscuda:0
.-seed
: An integer value that you can specify for repeatable results. By default this value is random for each run.
Basic FC (logits) layer visualization:
python vis_fc.py -model_file <bvlc_out120>.pth -model_epoch <120>
This script lets you create DeepDream hallucinations with trained GoogleNet models.
Input options:
-model_file
: Path to the pretrained GoogleNet model that you wish to use.-learning_rate
: Learning rate to use with the ADAM or L-BFGS optimizer. Default is1.5
.-optimizer
: The optimization algorithm to use; eitherlbfgs
oradam
; default isadam
.-num_iterations
: Default is 500.-content_image
: Path to your input image. If no input image is specified, random noise is used instead.-layer
: The specific layer you wish to use. Default is set tomixed5a
.-channel
: The specific layer channel you wish to use. Default is set to-1
to disable specific channel selection.-image_size
: A comma separated list of<height>,<width>
to use for the output image. Default is set to224,224
.
Only Required If Model Doesn't Contain Them, Options:
-data_mean
: Your precalculated list of mean values that was used to train the model, if they weren't saved inside the model.-num_classes
: The number of classes that the model was trained on, if it wasn't saved inside the model.
Output options:
-print_iter
: Print progress everyprint_iter
iterations. Set to0
to disable printing.-save_iter
: Save the images everysave_iter
iterations. Default is to0
to disable saving intermediate results.-print_iter
: Print progress everyprint_iter
iterations. Set to0
to disable printing.
Other options:
-use_device
: Zero-indexed ID of the GPU to use pluscuda:
. Default iscuda:0
.-seed
: An integer value that you can specify for repeatable results. By default this value is random for each run.
Basic DeepDream:
python vis.py -model_file <bvlc_out120>.pth -layer mixed5a
See here for more information on all the included scripts/tools relating to dataset creation, cleaning, and preparation.