Before jumping in our code implementation based on Tensorflow, please refer to our paper Learning to Sketch with Shortcut Cycle Consistency for the basic idea.

In this paper, we present a novel approach for translating an object photo to a sketch, mimicking the human sketching process. Teaching a machine to generate a sketch from a photo just like humans do is not easy. This requires not only developing an abstract concept of a visual object instance, but also knowing what, where and when to sketch the next line stroke. Figure \ref{fig:highlight} shows that the developed photo-to-sketch synthesizer takes a photo as input and mimics the human sketching process by sequentially drawing one stroke at a time. The resulting synthesized sketches provide an abstract and semantically meaningful depiction of the given object, just like human sketches do.

Examples of our model mimicking to sketch stroke by stroke.

We aim to learn a mapping function between the photo domain X and sketch domain Y, where we denote the empirical data distribution as x ~ p_data(x) and y ~ p_data(y) and represent each vector sketch segment as (s_xi, s_yi), a two-dimensional offset vector. Our model includes four mapping functions, learned using four subnets namely a photo encoder, a sketch encoder, a photo decoder, a sketch decoder. The illustration of our model architecture is as shown as below.

Our deep photo-to-sketch (p2s) synthesis model is trained on the dataset of ShoeV2 and ChairV2.

Usage:

python sketchrnn_cnn_dual_train.py --dataset shoesv2

As mentioned in the paper, before you train a photo-to-sketch (p2s) synthesis model, you need pretrain your model on the QuickDraw Data from the corresponding categories.

We have tested this model on TensorFlow 1.4 for Python 2.7.

Example:

The datasets for our photo-to-sketch synthesis task are ShoeV2 and ChairV2 datasets, which can be dowloaded from the homepage of our group SketchX.

The pretraining dataset can be download from QuickDraw.

The original data can be converted to hdf5 format using prepare_data.py. Or you can download it from the GoogleDrive.

If you find this project useful for academic purposes, please cite it as:

@Inproceedings{song2018learning,
  title             = {Learning to Sketch with Shortcut Cycle Consistency},
  author            = {Song, Jifei and Pang, Kaiyue and Song, Yi-Zhe and Xiang, Tao and Hospedales, Timothy M},
  booktitle         = {CVPR},
  year              = {2018}
}