Official code for Revisiting precision recall definition for generative modeling by Loic Simmon, Ryan Webster, and Julien Rabin, presented at ICML 2019. The poster can be downloaded here and the highlight video can be watched here.

Note that this code was adapted from this github project corresponding to Assessing Generative Models via Precision and Recall. Besides the new method, the original method was also reimplemented in pytorch (in order to remove tensorflow dependency). Doing so we obtained slightly different curves compared to the orginal tensorflow implementation.

A list of required packages is provided in requirements_minimal.txt and may be installed by running:

pip install -r requirements_minimal.txt

Alternatively, you may find a yaml conda environment file conda_env.yaml that can be used as follows:

# If you want to complete your current environment
conda env update -f conda_env.yaml
# Or if you want a fresh new environment (named here prdenv)
conda env create -n prdenv -f conda_env.yaml
conda activate prdenv

Example: you have a folder of images from your true distribution (e.g., ~/real_images/) and any number of folders of generated images (e.g., ~/generated_images_1/ and ~/generated_images_2/). Note that the number of images in each folder needs to be the same.

In a shell, cd to the repository directory and run

python prd_from_image_folders.py --classif --reference_dir ~/real_images/ --eval_dirs ~/generated_images_1/ ~/generated_images_2/ --eval_labels model_1 model_2 # ICML'19 paper version
python prd_from_image_folders.py --reference_dir ~/real_images/ --eval_dirs ~/generated_images_1/ ~/generated_images_2/ --eval_labels model_1 model_2           # Original NeurIPS'18 paper version

Besides a dataset folder was provided along with a script runCifarModes.sh to reproduce the Cifar modes experiment from the paper.

For further customization, run ./prd_from_image_folders.py -h to see the list of available options.

@InProceedings{pmlr-v97-simon19a,
  title = 	 {Revisiting precision recall definition for generative modeling},
  author = 	 {Simon, Loic and Webster, Ryan and Rabin, Julien},
  booktitle = 	 {Proceedings of the 36th International Conference on Machine Learning},
  year = 	 {2019}
  }

External copyright for: prd_score.py and prd_from_image_folders.py goes to Mehdi S. M. Sajjadi Copyright for remaining files: Loic Simon

License for all files: Apache License 2.0

For any questions, comments or help to get it to run, please don't hesitate to mail us: loic.simon@ensicaen.fr