Submitted to ICML 2023.

To reproduce the results, you will need to install the packages listed in requirements.txt and have python>=3.7 installed.

Please run pip install -r requirements.txt to install them.

This section will explain how to reproduce results.

Simply run the code:

python -m src.${method}.train \
    --model_name ${model}_${dataset} \
    --seed ${seed}

Where the variables you can choose from:

• method:

  • GiniSelector, which will train the model with the cross-entropy (CE) loss.
  • SelectiveNet.
  • ConfidNet.
  • DeepGamblers.
  • SelfAdaptiveTraining.

• model:

  • vgg16
  • densenet121
  • resnet34

• dataset:

  • cifar-10
  • cifar-100
  • svhn

• seed: an integer

The training will save the best weights as well as logs, logits, and scores in the checkpoints/${method}/${model}_${dataset}/${seed} folder.

Abiding by the principles of open science, we provide the checkpoints for most methods and models on the release page. PS.: The checkpoints for SelectiveNet are not available due to the size of the files (~30GB in total).

Run the following code to download and unzip a checkpoint from the release:

mkdir checkpoints/
cd checkpoints/
wget -c https://github.com/giniselector/giniselector/releases/download/checkpoints/${method}-${model}_${dataset}.zip
unzip ./${method}-${model}_${dataset}.zip

Or go to the release page and download the checkpoints manually.

Then you can save the logits and scores from a specific method and model by running:

python -m src.${method}.test \
    --model_name ${model}_${dataset} \
    --seed ${seed}

To evaluate the benchmark, please run:

python -m src.analysis.save_results \
    --model_name ${model}_${dataset}

The results will be exported to .csv in the results/ repository, and the evaluation metrics are defined in src/utils/eval.py

Run the code:

python -m src.analysis.plots \
    --model_name ${model}_${dataset}

The images will be saved to the images/ folder.

The post-hoc Gini score can be plugged into any deep classifier trained with the cross-entropy loss. The function implemented in python that takes the logits as input is defined below.

import torch


def gini(logits: torch.Tensor):
    g = torch.sum(torch.softmax(logits, 1) ** 2, 1)
    return 1 - g

Please, place the following lines in your .env file if you want to modify any of the default folders.

#.env
export DATA_DIR=""
export IMAGENET_ROOT=""
export CHECKPOINTS_DIR="checkpoints/"

Where:

• DATA_DIR is the directory where the datasets will be downloaded.
• IMAGENET_ROOT is the directory where the ImageNet dataset is located.
• CHECKPOINTS_DIR is the directory where the pre-trained models will be placed.