Joint work with Jianjin Xu and Zhaoyang Wang
This is the code base for reproducing the experiment.
This is demo video.
-
Install maskrcnn-benchmark according to the instructions
-
Prepare the NYUv2 dataset. Under the project root dir execute:
export DATA_HOME=./maskrcnn-benchmark/datasets/NYUv2/
mkdir $DATA_HOME
cp COCOLikeNYUV2.zip $DATA_HOME
cd $DATA_HOME
unzip COCOLikeNYUV2.zip
wget http://www.doc.ic.ac.uk/~ahanda/nyu_train_rgb.tgz
wget http://www.doc.ic.ac.uk/~ahanda/nyu_test_rgb.tgz
tar -xvf nyu_train_rgb.tgz
tar -xvf nyu_test_rgb.tgzUnder $DATA_HOME download the depth file from google drive.
- To train ED-MaskRCNN, you need to download all the estimated depth data from google drive and also place under
$DATA_HOME.
- RGB-MaskRCNN
python tools/train_net.py --config-file "configs/NYUBaselineFT.yaml"
- RGBD-MaskRCNN
python tools/train_net.py --config-file "configs/NYUDepthFT.yaml"
- ZD-MaskRCNN
python tools/train_net.py --config-file "configs/NYUProbDepthFT.yaml
- ED-MaskRCNN
Make sure preparation step 3 is completed, then run
python tools/train_net.py --config-file "configs/NYUProbDenseDepthPretrained.yaml"- SPADE-MaskRCNN
First make sure RGB-MaskRCNN is trained. If you follow the standard instruction, the RGB-MaskRCNN should be stored in expr/nyuv2_baseline_26_maskrcnn. Then run the following command:
# extract the weight from checkpoint file
python tools/extract_weights.py expr/nyuv2_baseline_26_maskrcnn/model_final.pth pretrained/rgb_baseline_26_imagenet.pth
# option III
python tools/train_net.py --config-file "configs/NYUSPADEFinetune_onrgb.yaml"
# extract the weight from checkpoint file
python tools/extract_weights.py expr/nyuv2_spade_ft_1_imagenet/model_final.pth pretrained/spade_ft1_26_imagenet.pth
# option IV
python tools/train_net.py --config-file "configs/NYUSPADEFinetune_onspade.yaml"Enter maskrcnn_benchmark folder. The script will test all the .pth file under <path of expr dir> with depth and without depth.
python tools/testall.py <path of expr> <path of config>
python collect_result.py- RGB-MaskRCNN
python tools/testall.py expr/nyuv2_baseline_26_maskrcnn configs/NYUBaselineFT.yaml-
RGBD-MaskRCNN
-
ZD-MaskRCNN
-
ED-MaskRCNN
-
SPADE-MaskRCNN
# testing model train by option I+II+III
python tools/testall.py expr/nyuv2_spade_ft_1_imagenet configs/NYUSPADE_26.yaml
# testing model trained by option I+II+III+IV
python tools/testall.py expr/nyuv2_spade_ft_2_imagenet configs/NYUSPADE_26.yamlThis script will visualize all the results collected by collect_result.py.
python tools/visualize.py