All the codes and models have been released! We'll post a blog to discuss the details and observations in OFF.

This repo holds the implementation code of the paper:

Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition, Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, Wei Zhang, CVPR 2018.

• OpenCV 2.4.12
• OpenMPI 1.8.5 (enable-thread-multiple when install)
• CUDA 7.5
• CUDNN 5
• Caffe Dependencies

You may refer to the project TSN to install these libs and prepare the data.

For training use, first modify the file make_train.sh with your own lib path filled in. Simply run sh make_train.sh, the script will automatically build the caffe for you.

For testing, you can simply run make pycaffe to make all stuff well prepared.

You need to make two folders before you launch your training. The one is logs under the root of this project, and the other is the model under the folder models/DATASET/METHOD/SPLIT/. For instance, if you want to train RGB_OFF on the dataset UCF101 split 1, then your model directory should be made under the path models/ucf101/RGB_OFF/1/. The models for initialization and reference will be available soon.

The network structure for training is defined in train.prototxt, and the hyperparameters are defined in solver.prototxt. For detailed training strategies and observations not included in the paper, please refer to our training recipes.

You need to create another directory proto_splits under the same folder of model. Our test code use pycaffe to call the functions defined in C++, therefore, we need to write some temporary files for synchronization. Remember to clean the temporary files everytime you launch a new test. Run the script test.sh with your METHOD, MODEL_NAME, SPLIT and NUM_GPU specified.

The deploy_tpl.prototxt defines the network for reference. To transfer your network defined in train.prototxt into deploy_tpl.prototxt, you may need to copy all the layers except the data layer and layers after each fully connected layer. As there are dynamic parameters defined in the deploy_tpl.prototxt, e.g. $SOURCE $OVERSAMPLE_ID_PATH, the format of the deploy_tpl.prototxt is a little bit different to the normal prototxt file.

Due to the unexpected server migration, our original models trained on all 3 splits of UCF101 and HMDB51 were all lost. Therefore, we re-train the models on the first split of UCF101:

Models on Google Drive will be available soon.

• Paper Link
• Network Prototxt
• Testing Code
• Training Code
• Pretrained Models
• Training Recipes
• Building Docs

If you find our research useful, please cite the paper:

@InProceedings{Sun_2018_CVPR,
author = {Sun, Shuyang and Kuang, Zhanghui and Sheng, Lu and Ouyang, Wanli and Zhang, Wei},
title = {Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2018}
}

Temporal Segment Networks

You can contact Mr.Shuyang Sun (Please do NOT call me with the title Prof., Dr., or any other kinds of weird prefixes. I'm still a master student....) by sending email to shuyang.sun@sydney.edu.au