Note: A new version of this paper based on semantic segmentation and image classification results will be released soon. The previous version described only the image classification results. This repo will soon be updated with details of the new experiments (code is already uploaded).

This repository presents the code implementation for Stagewise Knowledge Distillation, a technique for improving knowledge transfer between a teacher model and student model.

• PyTorch 1.3.0
• fastai v1

Following ResNet architectures are used:

• ResNet - {10, 14, 18, 20, 26, 34}

Note: ResNet34 is used as a teacher (being a standard architecture), while others are used as student models.

• CIFAR10
• Imagenette
• Imagewoof

Note: Imagenette and Imagewoof are subsets of ImageNet.

• root/code/models/ - Contains code for models.
• root/code/create_dataset.py - Code for modifying dataset to get smaller sized dataset.
• root/code/medium.py - Code for simultaneous training of all stages on complete dataset
• root/code/stagewise_medium.py - Code for stagewise training on complete dataset.
• root/code/less_data_stagewise_medium.py - Code for stagewise training on smaller sized dataset.
• root/code/utils.py - Some utility code.
• root/notebooks/ - Contains notebooks with similar code as above.

If you use this code or method in your work, please cite using

@misc{kulkarni2019stagewise,
    title={Stagewise Knowledge Distillation},
    author={Akshay Kulkarni and Navid Panchi and Shital Chiddarwar},
    year={2019},
    eprint={1911.06786},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}