This repository attempts to optimize the hyperparameters of Cutout: Cutout length and number of cutouts(no_holes). It is based on code from the Cutout repository.
pip3 install -r requirements.txt
pip3 install https://download.pytorch.org/whl/cu80/torch-0.3.1-cp35-cp35m-linux_x86_64.whl
pip3 install torchvision
It uses the hyperparameter optimizer: hyperband and Bayesian Optimization-Hyperband
Bibtex:
@INPROCEEDINGS{bohb17,
author = {S. Falkner and A. Klein and F. Hutter},
title = {Combining Hyperband and Bayesian Optimization},
booktitle = {NIPS 2017 Bayesian Optimization Workshop},
year = {2017},
month = dec,
}
To optimize ResNet18 on CIFAR10 with data augmentation and cutout:
`python cutout_configspace.py --dataset=cifar10 --model=resnet18 --max-epochs=160 --optimizer=BOHB --run-id=1
This repository contains the code for the paper Improved Regularization of Convolutional Neural Networks with Cutout.
Cutout is a simple regularization method for convolutional neural networks which consists of masking out random sections of input images during training. This technique simulates occluded examples and encourages the model to take more minor features into consideration when making decisions, rather than relying on the presence of a few major features.
Bibtex:
@article{devries2017cutout,
title={Improved Regularization of Convolutional Neural Networks with Cutout},
author={DeVries, Terrance and Taylor, Graham W},
journal={arXiv preprint arXiv:1708.04552},
year={2017}
}
Test error (%, flip/translation augmentation, mean/std normalization, mean of 5 runs)
Network | CIFAR-10 | CIFAR-100 |
---|---|---|
ResNet18 | 4.72 | 22.46 |
ResNet18 + cutout | 3.99 | 21.96 |
To train ResNet18 on CIFAR10 with data augmentation and cutout:
python train.py --dataset cifar10 --model resnet18 --data_augmentation --cutout --length 16
To train ResNet18 on CIFAR100 with data augmentation and cutout:
python train.py --dataset cifar100 --model resnet18 --data_augmentation --cutout --length 8
WideResNet model implementation from https://github.com/xternalz/WideResNet-pytorch
Test error (%, flip/translation augmentation, mean/std normalization, mean of 5 runs)
Network | CIFAR-10 | CIFAR-100 | SVHN |
---|---|---|---|
WideResNet | 3.87 | 18.8 | 1.60 |
WideResNet + cutout | 3.08 | 18.41 | 1.30 |
To train WideResNet 28-10 on CIFAR10 with data augmentation and cutout:
python train.py --dataset cifar10 --model wideresnet --data_augmentation --cutout --length 16
To train WideResNet 28-10 on CIFAR100 with data augmentation and cutout:
python train.py --dataset cifar100 --model wideresnet --data_augmentation --cutout --length 8
To train WideResNet 16-8 on SVHN with cutout:
python train.py --dataset svhn --model wideresnet --learning_rate 0.01 --epochs 160 --cutout --length 20
Shake-shake regularization model implementation from https://github.com/xgastaldi/shake-shake
Test error (%, flip/translation augmentation, mean/std normalization, mean of 3 runs)
Network | CIFAR-10 | CIFAR-100 |
---|---|---|
Shake-shake | 2.86 | 15.58 |
Shake-shake + cutout | 2.56 | 15.20 |
See README in shake-shake folder for usage instructions.