Implement Natural Language Object Retrieval in Tensorflow. The original code is release here, written in Caffe.
note that some of the codes are from natural-language-object-retrieval

In real AI, you may refer something with its location, color, or other characteristic. The AI robot should know where the corresponding object is. Here's the illustration:

The blue box represents the ground truth, while the yellow stands for positive recall, red for negative recall.

• Tensorflow
• Caffe

Here, all the experiments are done on ReferIt dataset. You can also train on other dataset in the same way.

1. Download caffe: ./external/download_caffe.sh
2. modify Makefile.config
3. make -j
4. make pycaffe

1. Download the pretrained models: ./models/download_trained_models.sh
2. Download the ReferIt dataset: ./datasets/ReferIt/ReferitData/download_referit_dataset.sh and ./datasets/ReferIt/ImageCLEF/download_data.sh
3. Download pre-extracted EdgeBox proposals: ./data/download_edgebox_proposals.sh
4. Preprocess the ReferIt dataset to generate metadata needed for training and evaluation: python ./exp-referit/preprocess_dataset.py
5. Cache the scene-level contextual features to disk: python ./exp-referit/cache_referit_context_features.py
6. Cache the bbox features to disk: python ./exp-referit/cache_referit_local_features.py
7. Build training batches: python ./exp-referit/cache_referit_training_batches.py

Once you prepare the data, you can run the training code python train.py
If you want to use pretrained model, you should convert the pretrained model into .npy and load the pretrained model when training.

python initialize.py

Before testing, I cache the proposal feature to disk due to the RAM contraints.

1. cache the proposal feature: python ./exp-referit/cache_edgebox_feature.py
2. choose which model you want to test by modifying this line.
3. python test.py

I use R@10 to evaluate the performance and all the test is on ReferIt dataset.
the performance through different epoch [without pretrained]:

the performance through different epoch [with pretrained]:

And here is the loss thourgh every epoch:

• Natural language object retrieval, Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, Trevor Darrell, cvpr2016 oral
• natural-language-object-retrieval

Feel free to contact me whenever you encounter unexpected errors in this repo.