This repository is the PyTorch implementation of PAC Prediction Sets for Meta-Learning (NeurIPS22). This code generates a prediction set that satisfies the probably approximately correct (PAC) guarantee for meta learning.

Download the Mini-ImageNet dataset from the original repo. In particular, download the postprocessed dataset from this link, and put it under data/miniimagenet (i.e., 'data/miniimagenet/mini-imagenet'). The following script takes care of the rest postprocessing.

cd data/miniimagenet
./process.sh

Train a Prototypical network as follows:

./scripts/train_miniimagenet_protonet.sh 

Construct and evaluate a meta PAC prediction set along with baselines as follows:

./scripts/cal_miniimagenet_protonet.sh

To reproduce evaluation results, run the following script to generate plots:

./scripts/plot_miniimagenet.sh

We use FewRel 1.0 and a related toolkit. The required part of the toolkit and dataset are included in this repository. the following script initializes the toolkit and the dataset for you.

cd data/fewrel
./process.sh

Train a Prototypical network as follows:

./scripts/train_fewrel_protonet.sh 

Construct and evaluate a meta PAC prediction set along with baselines as follows:

./scripts/cal_fewrel_protonet.sh

To reproduce evaluation results, run the following script to generate plots:

./scripts/plot_fewrel.sh

Download the Heart dataset as follows:

cd data/heart
./download.sh

Train a Prototypical network as follows:

./scripts/train_heart_protonet.sh

Construct and evaluate a meta PAC prediction set along with baselines as follows:

./scripts/cal_heart_protonet.sh

To reproduce evaluation results, run the following script to generate plots:

./scripts/plot_heart.sh