Toolbox for assessing generative models working with PyTorch

• Generate feature vectors from image dataset
  • With classifiers trained on ImageNet dataset
    • Inception v3 (Paper (arXiv))
      • pytorch-fid by mseitzer is used internally and its model has the same weights as the original implemention of FID (Paper (arXiv), GitHub)
    • VGG-16 (Paper (arXiv))
      • Use of this model is proposed in Improved Precision and Recall Metric for Assessing Generative Models by Kynkäänniemi et al. (Paper (arXiv), GitHub)
  • With classifiers trained on danbooru dataset (may be better for anime-like images)
    • danbooru-pretrained (pretrained ResNet50 (Paper (arXiv)))
      • Use of this model as a classifier is proposed as DaFID by birdManIkioiShota
  • Save the vectors as .npz files
• FID-like score
  • As you know, FID is one of the most common method to evaluate similarity between two datasets which is proposed in GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium by Heusel et al. (Paper (arXiv), GitHub)
  • Evaluate the similarity between two image datasets by using the method similar to FID
    • "similar" because it may not always use Inception v3
    • Approximate the distribution of feature vectors as multidimensional Gaussian and compute the Fréchet distance between two Gaussians.
  • Using pytorch-fid
• Improved Precision and Recall Metric
  • Evaluate the similarity between two image datasets with Improved Precision and Recall Metric
    • Approximate the distribution of feature vectors as a set of hyperspheres with radius equals to distance of their kth nearest neighbor.
  • The code is based on the official TensorFlow implemention and the PyTorch implemention by youngjung

All required libraries are listed in requirements.txt
Python 3 (developed with 3.10.7)
PyTorch (developed with 1.13.1+cu117, CUDA is needed if you want to use danbooru-pretrained)

I recommend you to use venv to separate environment.

python -m venv venv
./venv/Scripts/activate

or

python -m venv --system-site-packages venv
./venv/Scripts/activate

for minimal install.
After setup above, run following command.

pip install -r requirements.txt

1. Use preprocess.py to generate feature vectors from dataset
2. Use frechet_distance.py or precision_recall.py to evaluate their similality
3. See the output csv
   • Use simple_visualizer.py if needed

It is said that at least 10,000 images are needed to get meaningful result.
For reference, 50,000 images are used for assessing in the papers of FID or IPR.
This is because distribution of the feature vectors is extremely sparse since their dimensions are so high (like 512, 2048).