Upload
others
View
14
Download
0
Embed Size (px)
Citation preview
Face Sketch Synthesis
using Stagewise GAN
Introduction
• Photo and sketch
• The same content
• Different in geometry and in texture
• Different domain
Introduction
• Photo to sketch
• Enhance the artistry of pictures
• Criminal investigation
• can only search using sketch
• Solved the problem of large work-load of
manual creation sketch
Related work
Sketch Synthesis
• Face Photo-Sketch Synthesis and Recognition(2009)
• state-of-the-art approach using a multi-scale Markov random field (MRF)
• Markov Weight Fields for Face Sketch Synthesis(2012)
• the target sketch patch is represented by a linear combination of some candidate sketch
patches(MWF)
• Real-Time Exemplar-Based Face Sketch Synthesis(2014)
• casted the face sketch synthesis problem into a spatial sketch denoising (SSD) problem,local
fusion the patches after divided.
Related work
Synthesis Model
• Deep Belief Net-work(DBN)
• Generative Adversarial Networks(GANs)
• Variational Autoencoder(VAE )
Image to Image
• GANs
• cGANs
• cycleGANs
• Pix2pix
Our Model
• Use GAN model as main model
• Using Encoder-Decoder as Generator
• Using patch-Decoder
• 5 stage training
• End-to-end net
• Input data :photo and 5 different stage sketches
photos
Generator stage1
Generated stage1
Groundtruth stage1
Generator stage2
Discriminator stage1
Generated stage1
concat
patch 0,1
concat
Generator stage3
concat
Generator stage4
concat
Generator stage5
concat
Generated stage2
Groundtruth stage2
Discriminator stage1
concat
patch 0,1
Discriminator stage1
concat
patch 0,1
Discriminator stage1
concat
patch 0,1
Discriminator stage1
concat
patch 0,1
Generated stage2
Generated stage3
Generated stage4
Groundtruth stage3
Groundtruth stage4
Groundtruth stage5
Generated stage3
Generated stage4
Generated stage5
(final)
Full model
Conv1
Conv2
Conv3Conv8
Deconv7
Deconv6
Conv5
Conv4
Conv9
Pooling2
Poling3
Photo
or
Photo +last
250x200
16
32
64
128
128
outputoutput
128
64
32
1
Sigmoid
Euclidean
Loss
Generator
d_Conv1
d_Conv2
d_Conv3
dropout1
d_Pooling1
dropout2
dropout3
d_Pooling3
d_fc4
dropout4
d_fc4
SigmoidLoss
ConcatConv9
SketchPatch
output
64
128
256
128
1
Discriminator
16 Patches in every
discriminator
Experiments
• Use faces in CUFS data sketches are painted by our artists
• Use 62 faces for training and 32 for testing
• The size of every image is 200x250
Database
• CUHK Face Sketch database (CUFS) is for research on face sketch synthesis and face
sketch recognition.
• 188 faces from the Chinese University of Hong Kong (CUHK) student database(done)
• 123 faces from the AR database . (painting)
Database
因为数据没有画完,现在用的训练数据由62张,测试数据31张。
Data
Synthesis
SSD MWF MRF RSLCR GAN
Experiments
Experiments
(a) (b) (c) (d) (e) (f) (g) (h) (i)
(a) face photo;(b) sketch by artist from CUFS; (c) SSD; (d) MEF ;(e) MRF ;(f) RSLCR ;(g) GAN ; (h) our model ; (i)
sketch by our artist;