Realistic evaluation of recent state-of-the-art self-supervised learning and semi-supervised learning algorithms

This is a repository for my bachelor’s thesis on evaluation of semi-supervised and self-supervised learning algorithms. The purpose of this project is to somewhat recreate Realistic Evaluation of Deep Semi-Supervised Learning Algorithms paper, but with recent (as of 2020) state-of-the-art self-supervised and semi-supervised algorithms. I believe this is relevant and interesting, because we might be at the break point of achieving or surpasing supervised learning methods with self-supervised learning or semi-supervised learning (or both). So it is important to evaluate emerging algorithms in different settings (as described in aforementioned article about semi-supervised learning) along with improving them.

To get familiar with self-supervised learning and semi-supervised learning algorithms, which will be evaluated in this project I recommend the following articles:

• A survey on Semi-, Self- and Unsupervised Techniques in Image Classification – comprehensive survey about SOTA algorithms including consistency regularization, pseudolabeling, contrastive learning (good survey paper)
• UDA, Meta Pseudo Labels – two semi-supervised algorithms I plan to use (SOTA in semi-supervised learning today)
• SimCLR, MoCo – two self-supervised algorithms I plan to use.
• Realistic Evaluaiton of Deep Semi-Supervised Learning Algorithms – paper which describes problems with commonly used semi-supervised learning benchmarks. Most papers mentioned above (UDA, SimCLR, MoCo) use those semi-supervised learning benchmark (CIFAR-10 for example).
• Contrastive Representation Distilation distilation technique based on contrastive learning. My idea is that it could also work in semi-supervised setting (add cross entropy loss to teacher + distill it's representation on all images)

If you prefer to watch videos (great explanations):

• Meta Pseudo Labels, UDA
• SimCLR
• Lecture on self-supervised learning (UC Berkeley CS294-158 Deep Unsupervised Learning)

To install clone this repository and run:

pip install -e .

Reproducing UDA results in pytorch