Code for paper Wandering Within a World: Online Contextualized Few-Shot Learning [arxiv]

Authors: Mengye Ren, Michael L. Iuzzolino, Michael C. Mozer, Richard S. Zemel

Although our code base is MIT licensed, the RoamingRooms dataset is not since it is derived from the Matterport3D dataset.

To download the RoamingRooms dataset, you first need to sign the agreement for a non-commercial license here.

Then, you need to submit a request here. We will manually approve your request afterwards.

For inquiries, please email: roaming-rooms@googlegroups.com

The whole dataset is around 60 GB. It has 1.2M video frames with 7k unique object instance classes. Please refer to our paper for more statistics of the dataset.

If you have downloaded the full Matterport3D dataset already, check here for a script to generate RoamingRooms.

Our code is tested on Ubuntu 18.04 with GPU capability. We provide docker files for reproducible environments. We recommend at least 20GB CPU memory and 11GB GPU memory. 2-4 GPUs are required for multi-GPU experiments. Our code is based on TensorFlow 2.

1. Install protoc from here.

2. Run make to build proto buffer configuration files.

3. Install docker and nvidia-docker.

4. Build the docker container using ./build_docker.sh.

5. Modify the environment paths. You need to change DATA_DIR and OURPUT_DIR in setup_environ.sh. DATA_DIR is the main folder where datasets are placed and OUTPUT_DIR is the main folder where training models are saved.

1. Install protoc from here.

2. Run make to build proto buffer configuration files.

3. Modify the environment paths. You need to change DATA_DIR and OURPUT_DIR in setup_environ.sh. DATA_DIR is the main folder where datasets are placed and OUTPUT_DIR is the main folder where training models are saved.

4. Create a conda environment:

conda create -n oc-fewshot python=3.6
conda activate oc-fewshot
conda install pip

5. Install CUDA 10.1

6. Install OpenMPI 4.0.0

7. Install NCCL 2.6.4 for CUDA 10.1

8. Modify installation paths in install.sh

9. Run install.sh

1. To set up the Omniglot dataset, run script/download_omniglot.sh. This script will download the Omniglot dataset to DATA_DIR.

2. To set up the Uppsala texture dataset (for spatiotemporal cue experiments), run script/download_uppsala.sh. This script will download the Uppsala texture dataset to DATA_DIR.

To run training on your own, use the following command.

./run_docker.sh {GPU_ID} python -m fewshot.experiments.oc_fewshot \
  --config {MODEL_CONFIG_PROTOTXT} \
  --data {EPISODE_CONFIG_PROTOTXT} \
  --env configs/environ/roaming-omniglot-docker.prototxt \
  --tag {TAG} \
  [--eval]

• MODEL_CONFIG_PROTOTXT can be found in configs/models.
• EPISODE_CONIFG_PROTOTXT can be found in configs/episodes.
• TAG is the name of the saved checkpoint folder.
• When the model finishes training, add the --eval flag to evaluate.

For example, to train CPM on the semisupervised benchmark:

./run_docker.sh 0 python -m fewshot.experiments.oc_fewshot \
  --config configs/models/roaming-omniglot/cpm.prototxt \
  --data configs/episodes/roaming-omniglot/roaming-omniglot-150-ssl.prototxt \
  --env configs/environ/roaming-omniglot-docker.prototxt \
  --tag roaming-omniglot-ssl-cpm

All of our code is tested using GTX 1080 Ti with 11GB GPU memory. Note that the above command uses a single GPU. Our original experiments in the paper is performed using two GPUs, with twice the batch size and doubled learning rate. To run that setting, use the following command:

./run_docker_hvd_01.sh python -m fewshot.experiments.oc_fewshot_hvd \
  --config {MODEL_CONFIG_PROTOTXT} \
  --data {EPISODE_CONFIG_PROTOTXT} \
  --env configs/environ/roaming-omniglot-docker.prototxt \
  --tag {TAG}

Below we include command to run experiments on RoamingRooms. Our original experiments in the paper is performed using four GPUs, with batch size to be 8. To run that setting, use the following command:

./run_docker_hvd_0123.sh python -m fewshot.experiments.oc_fewshot_hvd \
  --config {MODEL_CONFIG_PROTOTXT} \
  --data {EPISODE_CONFIG_PROTOTXT} \
  --env configs/environ/roaming-rooms-docker.prototxt \
  --tag {TAG}

When evaluate, use --eval --usebest to pick the checkpoint with the highest validation performance.

Table 1: RoamingOmniglot Results (Supervised)

Table 2: RoamingOmniglot Results (Semi-supervised)

Table 3: RoamingRooms Results (Supervised)

Table 4: RoamingRooms Results (Semi-supervised)

• Add a data iterator based on PyTorch (contribution welcome).

If you use our code, please consider cite the following:

• Mengye Ren, Michael L. Iuzzolino, Michael C. Mozer and Richard S. Zemel. Wandering Within a World: Online Contextualized Few-Shot Learning. CoRR, abs/2007.04546, 2020.

@article{ren20ocfewshot,
  author   = {Mengye Ren and
              Michael L. Iuzzolino and
              Michael C. Mozer and
              Richard S. Zemel},
  title    = {Wandering Within a World: Online Contextualized Few-Shot Learning},
  journal  = {CoRR},
  volume   = {abs/2007.04546},
  year     = {2020},
}