Image Harmonization is to harmonize a composite image by adjusting its foreground appearances consistent with the background region. A real composite image is generated by a foreground region of one image combined with the background of another image. Though it's easy to create real composite images, the harmonized outputs are too time-consuming and skill-demanding to generate. So there is no high-quality publicly available dataset for image harmonization.

Our dataset iHarmony4 is a synthesized dataset for Image Harmonization. It contains 4 sub-datasets: HCOCO,HAdobe5k, HFlickr, and Hday2night, each of which contains synthesized composite images, foreground masks of composite images and corresponding real images. The iHarmony4 dataset is provided in Baidu Cloud (Alternative_address)

HCOCO, containing 42k synthesized composite images, is generated based on Microsoft COCO dataset. The foreground region is corresponding object segmentation mask provided from COCO. Within the foreground region, the appearance of COCO image is edited using various color transfer methods. The HCOCO sub-dataset and training/testing split are provided in Baidu Cloud (Alternative_address).

HAdobe5k is generated based on MIT-Adobe FiveK dataset. Provided with 6 editions of the same image, we manually segment the foreground region and exchange foregrounds between 2 versions. The HAdobe5k sub-dataset and training/testing split are provided in Baidu Cloud (Alternative_address).

We collected 4833 images from Flickr. After manually segmenting the foreground region, we use the same method as HCOCO to generate HFlickr sub-dataset. The HFlickr sub-dataset and training/testing split are provided in Baidu Cloud (Alternative_address).

Hday2night is generated based on day2night dataset. We manually segment the foreground region, which is cropped and overlaid on another image captured on a different time. The Hday2night sub-dataset and training/testing split are provided in Baidu Cloud (Alternative_address).

To generate synthesized composite images, color transfer methods are adopted to transfer color information from reference images to real images. Considering that color transfer methods can be categorized into four groups based on parametric/non-parametric and correlated/decorrelated color space, we select one representative method from each group. Thanks to Wei Xu's efforts for releasing the code of color transfer method 1, 2 and 3 in their survey paper, we could implement color transfer methods specialized for foreground based on their implementation. And the source code of IDT regrain color transfer is downloaded from the author's GitHub

--Parametric method in decorrelated color space. Implementation of paper "Color transfer between images" [pdf].

--Parametric method in correlated color space. Implementation of paper "Color transfer in correlated color space" [pdf].

--Non-parametric method in decorrelated color space. Implementation of paper ""Histogram-based prefiltering for luminance and chrominance compensation of multiview video" [pdf].

--Non-parametric method in correlated color space. Implementation of paper "Automated colour grading using colour distribution transfer" [pdf].

Here, we provide the code of baselines used in our paper "DoveNet: Deep Image Harmonization via Domain Verification", which is accepted by CVPR2020. Refer to Bibtex for more details.

J.-F. Lalonde et al. provides their implementation of paper "Using color compatibility for assessing image realism" (ICCV2017) in their GitHub.

And we have arranged the code to a "click-and-run" way. demo.m is available in /lalonde/colorStatistics/mycode/demo/. Don't forget to specify the path of the code and results in your computer in getPathName.m, and run setPath.m before run demo.mto get everything ready.

This is Xue's implementation of their paper in 2012 ACM Transactions on Graphics "Understanding and improving the realism of image composites".

demo.m is available in /xue/demo/.

Notice to add the path of all dependent files using addpath(genpath('../dependency')).

Jun-Yan Zhu released the code of their paper "Learning a discriminative model for the perception of realism in composite images" (ICCV2015) in their GitHub.

Notice that it requires matcaffe interface. We make some changes corresponds to our dataset including how to preprocess data and how to save the harmonized results. Don't forget to specify DATA_DIR,MODEL_DIR and RST_DIR before running demo.m.

The pre-trained models of Zhu's work can also be found in BaiduCloud and remember to put it under MODEL_DIR.

Tsai released their pre-trained caffe model of their paper "Deep Image Harmonization" (CVPR2017) in their GitHub. This is a Tensorflow implementation based on the released caffe network.

Besides, we discard one inner-most convolutional layer and one inner-most deconvolutional layer to make it suitable for input of 256*256 size. In DIH, they proposed to use segmentation branch to help propogate semantics to harmonization branch and it contributes considerable improvments. So here we inplement this two versions, DIH without segmentation branch and DIH with segmentation branch, corresponding to DIH(w/o semantics) and DIH in their paper.

• without segmentation branch

We discard the scene parsing branch and preserve the remaining encoder-decoder structure and skip links. And this is the version used as one of the baselines in our paper.

To train DIH(w/o semantics) , under the folder wo_semantics/, run:

python train.py --data_dir <Your Path to Dataset> --init_lr 0.0001 --batch_size 32

Don't forget to specify the directory of Image Harmonization Dataset after data_dir.

Our trained model can be found in BaiduCloud. To test and re-produce the results, remember to put the model under /dih/wo_semantics/model/ and run:

python test.py --batch_size 1

• with segmentation branch

The structure is implemented the same as the Caffe network. In DIH, to pre-train the joint network, they constructed a synthesized composite dataset based on ADE20K, which provides images segmentations. While we use HCOCO instead, leveraging existing segmentaitons of COCO images from COCO-Stuff dataset GitHub. In our experiment, we leverage the object segmentations to pretrain the network. After downloading the dataset, preprocess the PNG segmentation to filter out stuff labels. Remember to rename the corresponding segmentation with the same name as real images and put them under <Your Path to Dataset>/HCOCO/object_segmentations/. Then, we freeze the segmentation branch and finetune harmonization branch using the whole dataset.

To pre-train this model, under the folder with_semantics/, run:

python train_seg.py --data_dir <Your Path to Dataset> --init_lr 0.0001 --batch_size 32

After that, freeze the segmentation branch and finetune harmonization branch. Run:

python finetune.py --data_dir <Your Path to Dataset> --init_lr 0.0001 --batch_size 32 --pretrain False

Specify the directory of Image Harmonization Dataset after data_dir.

Our trained model can be found in BaiduCloud. To test and re-produce the results, remember to put the model under /dih/with_semantics/model/ and run:

python test_seg.py --batch_size 1

The code is implemented based on the work of CVPR 2017: Image-to-Image Translation with Conditional Adversarial Networks, which is released by Jun-Yan Zhu in their GitHub

Since our dataset is not organized like a normally aligned dataset, we have to implement image loading and processing part according to our dataset. For more details, you could refer to data/ihd_dataset.py. We implement the U-net backbone based on Zhu's implementation of unet_256 and leverage attention blocks to enhance U-Net, which could be found in our Supplementary material. In total, we insert three attention blocks into U-Net. Refer to unetatt_model.py for more details.

To train U-net+attention:

python train.py --dataroot ./datasets/ihd/ --name unetatt --model unetatt --gpu_ids 1 --dataset_mode ihd --is_train 1 --no_flip --preprocess none --norm instance

Our trained model can be found in BaiduCloud. Download and put it under ./unetatt/checkpoint/unetatt/ To test and re-produce the results of U-net+att , run:

python test.py --dataroot ./datasets/ihd/ --name unetatt --model unetatt --gpu_ids 1 --dataset_mode ihd --is_train 0 --no_flip --preprocess none --norm instance

When conducting experiments, we merge training sets of four sub-datasets as a whole training set to train the model, and evaluate it on the test set of each sub-dataset and the whole test set. Here we show some example results of different baselines on our dataset. More examples can be found in our main paper.

Besides, to evaluate the effectiveness of different methods in real scenarios, we also conduct user study on 99 real composite images, of which 48 images from Xue and 51 images from Tsai. Below we present several results of different baselines on real composite images. The 99 real composite images could be found in BaiduCloud. And to visualize the comparison, we have put the results of different methods on all 99 real composite images in Supplementary.

When using images from our dataset, please cite our paper using the following BibTeX [preprint]:

@inproceedings{DoveNet2020,
title={DoveNet: Deep Image Harmonization via Domain Verification},
author={Wenyan Cong and Jianfu Zhang and Li Niu and Liu Liu and Zhixin Ling and Weiyuan Li and Liqing Zhang},
booktitle={CVPR},
year={2020}}