"Bayesian posterior approximation with stochastic ensembles"
Oleksandr Balabanov, Bernhard Mehlig, and Hampus Linander, CVPR 2023.
-
Train and evaluate a stochastic ensemble of neural networks. The ensemble types cover Regular (Non-Stochastic), Monte Carlo Dropout, DropConnect, Non-Parametric Dropout.
-
Two classification tasks are considered: (1) a toy problem solved using a fully-connected network and (2) classification of CIFAR images with ResNet20-FRN model.
-
The considered evaluation metrics: (1) predictive entropy and mutual information for the toy model and (2) test accuracy, loss, ECE, out-of-domain detection, predictive entropy, mutual information, calibration curves for CIFAR. For CIFAR10 we also looked at resilience to distribution shift by evaluating on CIFAR10-C dataset.
Python 3 dependencies in container/requirements.txt
Singularity and Docker definitions provided.
./build_docker.sh
or
# Might need sudo
./build_singularity./shell_docker.shor
./shell_singularity.shTo enable WandB logging for the training examples:
--enable_wandbStart at toy model root
cd toy/python3 run_create_data.py --output_dir ./output --num_data_train 2000 --num_data_eval 2000 --scaling_factors 0 1 2 3 4 5 6 7 8 9
Example 1:
# --method covers HMC, regular, dropout, dropconnect, np_dropout options
python3 run_train.py --output_dir ./output --method dropout --num_nets 1024 --drop_rate 0.1
Example 2:
python3 run_train.py --output_dir ./output --method HMC
Example 1: Produce plots of entropy and mutual information for nonparametric dropout ensemble
python3 run_eval.py --output_dir ./output --method dropout --drop_rate 0.1 --num_samples_ens 1024 --compute_save_softmax_probs True --plot_scaling_factor 1
Example 2: Produce plots of entropy and mutual information for HMC
python3 run_eval.py --output_dir ./output --method HMC --num_samples_HMC 1024 --compute_save_softmax_probs True --plot_scaling_factor 1
Example 3: Produce mean abs error to HMC plots (beforehand need to produce HMC softmax_probs)
python3 run_eval.py --output_dir ./output --method dropout --drop_rate 0.1 --compute_save_softmax_probs False --plot_scaling_factors 0 1 2 3 4 5 6 7 8
Start at CIFAR root
cd CIFAR/Download the datasets from
CIFAR10: https://www.cs.toronto.edu/~kriz/cifar.html
CIFAR100: https://www.cs.toronto.edu/~kriz/cifar.html
SVHN: from torchvision with torchvision.datasets.SVHN(root=SVHN_dir, split = "test", download=True)
CIFAR10C: https://zenodo.org/record/2535967#.Y5MtEHbMJD8
and rearrange them as follows
CIFAR10: PATH_CIFAR / CIFAR / cifar-10-batches-py / (data_batch_{i} and test_batch) with i = 1, ..., 5.
CIFAR100: PATH_CIFAR / CIFAR / cifar-100-python / (test and train)
SVHN: PATH_SVHN / (test_32x32.mat and train_32x32.mat)
CIFAR10C: PATH_CIFAR10C / CIFAR-10-C / (20 .numpy files with CIFAR-10-C corruption names)
with
--data_dir_CIFAR PATH_CIFAR (for run_train.py and run_eval.py)
--data_dir_SVHN PATH_SVHN (for run_eval.py)
--data_dir_CIFARC PATH_CIFAR10C (for run_eval.py)
# --method covers regular, dropout, dropconnect, np_dropout options
# --cifar_mode = CIFAR10 or CIFAR100
python3 ./run_train.py --output_dir ./output --cifar_mode CIFAR100 --method dropout --num_nets 20 --drop_rate_conv 0.3 --drop_rate_linear 0
python3 ./run_eval.py --output_dir ./output --cifar_mode CIFAR100 --method dropout --num_nets 20 --drop_rate_conv 0.3 --drop_rate_linear 0 --compute_save_softmax_probs True