The code for the Recombinator Networks paper.

If you use this code please consider citing our paper:

• Sina Honari, Jason Yosinski, Pascal Vincent, Christopher Pal. Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

Bibtex:

@inproceedings{honari2016RCN,
title={Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation},
author={Honari, Sina and Yosinski, Jason and Vincent, Pascal and Pal, Christopher},
booktitle={Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on},
year={2016},
organization={IEEE}
}

• The code is written in python 2.7
• OpenCV library for python should be installed to use this repo.

1. Install theano by following the instruction given here: http://deeplearning.net/software/theano/install.html

2. Clone RCN repo

3. In ~/.bashrc add the parent directory of the clonded RCN repo to PYTHONPATH:
   export PYTHONPATH=/path/to/parent/dir/of/RCN:$PYTHONPATH

There are two datasets: 5-keypoint (MTFL) and 68-keypoint (300W) datasets

Run create_raw_afw.py and create_raw_MTFL.py modules in preprocessing directory.
create_raw_MTFL.py creates the train set and the AFLW test set.
create_raw_afw.py creates the AFW test set.
Here is how to run these modules:

1. Download the images from: http://mmlab.ie.cuhk.edu.hk/projects/TCDCN/data/MTFL.zip

2. Unzip the folder and pass the complete path to it to --src_dir when calling create_raw_MTFL.py module.

3. Call create_raw_MTFL.py by passing complete path to --src_dir and --dest_dir arguments:
   

python create_raw_MTFL.py  --src_dir=/complete/path/to/MTFL/unzipped/folder --dest_dir=/complete/path/to/RCN/datasets

Note: dest_dir is the location where the dataset will be created. It should be finally put in RCN/datasets directory of the repo when training the models.

This module will create MTFL_test_160by160.pickle and MTFL_train_160by160.pickle in the given dest_dir path.

1. Download the images from: https://www.ics.uci.edu/~xzhu/face/AFW.zip

2. Unzip the folder and pass the complete path to it to --src_dir when calling create_raw_MTFL.py module.

3. Call create_raw_MTFL.py module by passing complete path to --src_dir and --dest_dir arguments:
   

python create_raw_afw.py  --src_dir=/complete/path/to/AFW/unzipped/folder --dest_dir=/complete/path/to/RCN/datasets

This module will create AFW_test_160by160.pickle in the given dest_dir path.

Run create_raw_300W.py module in preprocessing directory as follows:

1. Download Helen, LFPW, AFW and IBUG datasets from: http://ibug.doc.ic.ac.uk/resources/facial-point-annotations/

2. Once unzipped, the helen and lfpw have two subdirectories, 'testset' and 'trainset'. Rename them to 'X_testset' and 'X_trainset', for each dataset X.

3. Create one directory named 'Train_set' and put unzipped 'afw', 'helen_trainset' and 'lfpw_trainset' directories into it (as three sub-directories).

4. Create another directory named 'Test_set' and put unzipped 'ibug', 'helen_testset' and 'lfpw_testset' into it (as three sub-directories).

5. Put 'Train_set' and 'Test_set' directories into one direcotory (i.e. 300W) and pass the complete path to it to --src_dir when calling this module.

6. Call create_raw_300W.py module by passing complete path to --src_dir and --dest_dir arguments:
   

python create_raw_300W.py --src_dir=/complete/path/to/300W/folder --dest_dir=/complete/path/to/RCN/datasets

This module will create 300W_test_160by160.pickle and 300W_train_160by160.pickle files in the given dest_dir path.

create_procs.py module in RCN/models should be called with the right arguments to train each model.
The Theano flag THEANO_FLAGS=floatX=float32,device=gpu,force_device=True should be used right before the python command to train the model on gpu and set the type of shared varibles to float32:

THEANO_FLAGS=floatX=float32,device=gpu,force_device=True python create_procs.py <'flags'>

Here is the set of flags to be used for training each model:

python create_procs.py --L2_coef=1e-10 --L2_coef_ful=1e-14 --L2_coef_out=1e-14 --file_suffix=SumNet_MTFL_test --num_epochs=4000  --paral_conv=1.0 --use_lcn --block_img --rotation=20 --use_res_2

optional flags: --weight_per_pixel

###########

python create_procs.py --L2_coef=1e-14 --L2_coef_ful=1e-06 --file_suffix=SumNet_300W_test --num_epochs=3000 --paral_conv=2.0 --use_lcn --block_img 

optional flags: --use_res_2, --weight_per_pixel

###########

python create_procs.py --L2_coef=1e-06 --L2_coef_ful=1e-12 --file_suffix=RCN_MTFL_test --num_epochs=4000 --paral_conv=3.0 --use_lcn --block_img --rotation=20 --use_res_2

optional flags: --use_res_1

###########

python create_procs.py --L2_coef=1e-04 --L2_coef_ful=1e-04 --file_suffix=RCN_MTFL_skip_test --num_epochs=3000  --paral_conv=4.0 --use_lcn --block_img --rotation=20 --use_res_2

optional flags: --use_res_1

###########

python create_procs.py --L2_coef=1e-12 --L2_coef_ful=1e-08 --file_suffix=RCN_300W_test --num_epochs=3000 --paral_conv=5.0 --use_lcn --block_img

optional flags: --use_res_2, --use_res_1

###########

python create_procs.py --L2_coef=0.1 --L2_coef_ful=0.1 --file_suffix=RCN_300W_skip_test  --num_epochs=3000 --paral_conv=6.0 --use_lcn --block_img

###########

python create_procs.py --L2_coef=1e-06 --file_suffix=Denoising_300W_test --num_epochs=100  --denoise_conv=1.0

optional flags: --nMaps_shuffled=35, --conv_size=45

--use_res_2: indicates to go to resolution 2*2 as the coarsest resolution in the model. Default is 5*5.
--use_res_1: indicates to go to resolution 1*1 as the coarsest resolution in the model. Default is 5*5.
--weight_per_pixel: indicates in the SumNet model a weight per pixel to be used when summing the upsampled feature maps of different branches (default is one weight per feature map).
--batch_size: the batch size for training model in each iteration. Default is 64. If the model is too big to be set on gpu, use smaller values.
--file_suffix: gives a name to the model being trained (You can differentiate multiple trained models using this flag).
--nMaps_shuffled: the number of keypoints to be jittered when training the denoising model. Default is 35.
--conv_size: the size of the convolutional kernel when training the denoising model. Default is 45.

###########

Running one of the above models, e.g. with --file_suffix=test trained on MTFL dataset for 100 epochs, generates the following 5 outputs in RCN/models/exp_shared_conv directory:
adadelta_params_test.pickle -> keeps parameters of adadelta training algorithm
shared_conv_params_epoch_100_test_MTFL.pickle -> keeps the parameters of the model in the final epoch
shared_conv_params_test_MTFL.pickle -> keeps the parameters of the model based on best validation set performance
shared_conv_setting_test.pickle -> keeps the flags (hyper-parameters) used to run this model
epochs_log_test.pickle -> keeps the logs of each epoch while training the model

Note: If you want to train a model in multiple sessions (rather than continuously), copy the output files to a directory and then use the same command as above for training the model with the addition of the following flag:
--param_path=path/to/shared_conv_params_epoch_100_test_MTFL.pickle

It then loads the model's params in the final epoch (here 100) and also loads the adadelta parameres from adadelta_params_test.pickle and will continue training the model.

In order to draw keypoints using a trained SumNet or RCN model, run module draw_points_coarse_fine_conv.py in RCN/plotting directory. Assuming the model's name is 'test' and it is trained on 300W dataset, then the path to shared_conv_params_test_300W.pickle should be used as:

THEANO_FLAGS=floatX=float32,device=gpu,force_device=True python draw_points_coarse_fine_conv.py --path=/path/to/shared_conv_params_test_300W.pickle

The images will be saved in 'detected_kpts' directory, inside the directory where shared_conv_params_test_300W.pickle file exists.

In order to draw keypoints using a trained Denoising model on 300W dataset, you need to have a trained RCN_300W model. The following command uses the RCN_300W model (named 'RCN_test') to get one_hot_predictions and passes that to the Denoising_300W model (named 'Denoising_test') for its predictions. Note that this is not the joint model prediction, only the Denoising prediction:

THEANO_FLAGS=floatX=float32,device=gpu,force_device=True python draw_points_coarse_fine_conv.py --path=/path/to/shared_conv_params_Denoising_test_300W.pickle --cfNet_path=/path/to/shared_conv_params_RCN_test_300W.pickle

Run the same command as in 'Only Denoising model' case and pass --mult_probs flag to draw_points_coarse_fine_conv.py module

Run eval_test_set_coarse_fine_conv.py in RCN/utils directory. Assuming the model's name is 'test' and it is trained on 300W dataset, then the path to shared_conv_params_test_300W.pickle should be used as:

THEANO_FLAGS=floatX=float32,device=gpu,force_device=True python eval_test_set_coarse_fine_conv.py --pat
h=/path/to/shared_conv_params_test_300W.pickle

The error results will be saved in 'error_on_sets/test_set_results.txt', inside the directory where shared_conv_params_test_300W.pickle file exists.

In order to get the prediction error using a trained Denoising model on 300W dataset, you need to have a trained RCN_300W model. The following command uses the RCN_300W model (named 'RCN_test') to get one_hot_predictions of the RCN model and passes that as the input to the Denoising_300W model (named 'Denoising_test') for its predictions. Note that this is not the joint model prediction, only the Denoising model prediction:

THEANO_FLAGS=floatX=float32,device=gpu,force_device=True python eval_test_set_coarse_fine_conv.py --pat
h=/path/to/shared_conv_params_Denoising_test_300W.pickle --cfNet_path=/path/to/shared_conv_params_RCN_test_300W.pickle 

Run the same command as in 'Only Denoising model' case and pass --mult_probs flag to eval_test_set_coarse_fine_conv.py module