Upload
mark-chang
View
2.183
Download
0
Embed Size (px)
Citation preview
Neural Art by Mark Chang
A Neural Algorithm of Ar6s6c Style
• Author: – Leon A. Gatys. – Alexander S. Ecker. – Ma@hias Bethge
• Organiza6on: – Werner Reichardt Centre for Integra6ve Neuroscience and Ins6tute of Theore6cal Physics, University of Tubingen, Germany.
– Bernstein Center for Computa6onal Neuroscience, Tubingen, Germany.
The Mechanism of Pain6ng
Brain Ar6st
Scene Style ArtWork
Computer Neural Networks
Overview
• Visual Percep6on • Computer Vision • Neural Art • Demo
Visual Percep6on
• Neuron • Visual Pathway • Misconcep6on
Neuron
• Neuron • Ac6on Poten6al
Dendrite
Axon
Cell Body
Time
Voltage
Threshold
Visual Pathway
Re6na
Visual Area V1
Visual Area V4 Inferior Temporal Gyrus (IT)
Visual Pathway
Visual Area V1
Inferior Temporal Gyrus (IT)
Recep6ve Fields
Visual Area V4
Misconcep6on
Computer Vision
• Neural Networks • Convolu6onal Neural Networks • VGG 19
Neural Networks
n W1
W2
x1
x2
b Wb
y
nin = w1x1 + w2x2 + wb
nout
=1
1 + e�nin
Sigmoid
Rec6fied Linear
nout
=
⇢nin
if nin
> 0
0 otherwise
Neural Networks
x
y
n11
n12
n21
n22
b b
z1
z2
Input Layer
Hidden Layer
Output Layer
W12,y
W12,x
W11,y
W11,b W12,b
W11,x W21,11
W22,12
W21,12
W22,11
W21,b W22,b
Convolu6onal Neural Networks
• Convolu6onal Layer
depth
width width depth
weights weights
height
shared weight
Convolu6onal Neural Networks
• Stride • Padding
Stride = 1
Stride = 2
Padding = 0
Padding = 1
Convolu6onal Neural Networks
• Pooling Layer 1 3 2 4
5 7 6 8
0 0 4 4
6 6 0 0
4 5
3 2 no overlap
no padding no weights
depth = 1
7 8
6 4
Maximum Pooling
Average Pooling
Convolu6onal Neural Networks
Convolu6onal Layer
Convolu6onal Layer Pooling
Layer
Pooling Layer
Recep6ve Fields Recep6ve Fields
Input Layer
Convolu6onal Neural Networks
Input Layer
Convolu6onal Layer with
Recep6ve Fields:
Max-‐pooling Layer with
Width =3, Height = 3
Filter Responses
Filter Responses
Input Image
VGG 19
• Karen Simonyan & Andrew Zisserman. Very Deep Convolu6onal Networks for Large-‐scale Image Recogni6on.
• ImageNet Challenge 2014 • 19 (+5) layers – 16 Convolu6onal layers (width=3, height=3) – 5 Max-‐pooling layers (width=2, height=2) – 3 Fully-‐connected layers
VGG 19
depth=64 conv1_1 conv1_2
maxpool
depth=128 conv2_1 conv2_2
maxpool
depth=256 conv3_1 conv3_2 conv3_3 conv3_4
depth=512 conv4_1 conv4_2 conv4_3 conv4_4
depth=512 conv5_1 conv5_2 conv5_3 conv5_4
maxpool maxpool maxpool
size=4096 FC1 FC2
size=1000 sogmax
Neural Art
• Content Genera6on • Style Genera6on • Artwork Genera6on
Content Genera6on
Brain Ar6st Content
Canvas Minimize
the difference
Neural S6mula6on
Draw
Content Genera6on Filter
Responses VGG19
Update the color of the pixels
Content
Canvas
Result
Width*Height Depth
Minimize the
difference
Content Genera6on
Layer l’s Filter l Responses:
Lcontent
(P l, Cl) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2
Layer l’s Filter Responses: Lcontent
(P l, Cl) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2Input Photo: L
content
(p, c, l) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2
Lcontent
(p,x, l) =1
2
X
i,j
(X l
i,j
� P l
i,j
)2
@Lcontent
(p,x, l)
@X l
i,j
= X l
i,j
� P l
i,j
X l
X li,j
Input Canvas: x
Width*Height (j)
Depth (i)
Width*Height (j)
Depth (i)
Content Genera6on
• Backward Propaga6on Layer l’s Filter l Responses: X l
Input Canvas:
x
VGG19
@Lcontent
@x=
@Lcontent
@X l
@X l
x
x x� ⌘@L
content
@xUpdate Canvas
Learning Rate
Content Genera6on
Content Genera6on
VGG19
conv1_2 conv2_2 conv3_4 conv4_4 conv5_2 conv5_1
Style Genera6on
• ”Style” is posi6on-‐independent
style extrac6on
Style Genera6on
VGG19 Artwork
G
G
Filter Responses Gram Matrix
Width*Height
Depth
Depth
Depth
Posi6on-‐ dependent
Posi6on-‐ independent
Style Genera6on
1. .5
.5
.5
1.
1. .5 .25 1.
.5 .25 .5
.25 .25
1. .5 1.
Width*Height
Depth
k1 k2
k1
k2
Depth
Depth
Layer l’s Filter Responses Gram Matrix
F l1
F l2
F l3
F l4
F l1
F l2
F l3
F l4
G
Gli,j = F l
i · F lj
Gl4,1 = F l
4 · F l1
= 1⇥ 1 + 0⇥ 0.5 + 0⇥ 0 + ...
= 1
Style Genera6on
VGG19 Filter
Responses Gram Matrix
Minimize the
difference
G
G
Style
Canvas
Update the color of the pixels Result
Style Genera6on
Lstyle(a,x, l) =1
2
X
i,j
(X li,j �Al
i,j)2
@Lstyle(a,x, l)
@F li,j
= ((F l)T (X l �Al))j,i
Layer l’s Filter Responses
Layer l’s Gram Matrix
Layer l’s Gram Matrix
F li,j
Ali,j X l
i,j
Input Artwork:
Input Canvas: a
x
Style Genera6on
Style Genera6on
VGG19
Conv1_1 Conv1_1 Conv2_1
Conv1_1 Conv2_1 Conv3_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1
Artwork Genera6on
Ltotal
= ↵Lcontent
+ �Lstyle
a
Lcontent
(p, c, l) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2
x
x x� ⌘@L
total
@xx
Filter Responses VGG19
Lcontent
(p,x)
Lstyle(a,x)
Gram Matrix
Artwork Genera6on
VGG19 VGG19
Lcontent
(p,x) Lstyle(a,x)
Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1
Conv4_2
Ltotal
= ↵Lcontent
+ �Lstyle
Artwork Genera6on
Demo
• Content v.s. Style • Different Ini6al State • Different VGG Layers • Sketch & Watercolor • Pain6ng & Poem
Content v.s. Style
0.15 0.05
0.02 0.007
↵
�
Different Ini6al State noise 0.9 *noise + 0.1*photo photo
Different VGG Layers
Conv1_1 Conv2_1
Conv1_1 Conv2_1 Conv3_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1
↵
�= 0.002
Sketch & Watercolor
Pain6ng & Poem
Further Reading • A Neural Algorithm of Ar6s6c Style – h@p://arxiv.org/abs/1508.06576
• Texture Synthesis Using Convolu6onal Neural Networks – h@p://arxiv.org/abs/1505.07376
• Convolu6onal Neural Network – h@p://cs231n.github.io/convolu6onal-‐networks/
• Neural Network Back Propaga6on – h@p://cpmarkchang.logdown.com/posts/277349-‐neural-‐network-‐backward-‐propaga6on
• Computa6onal Poetry: – h@p://www.slideshare.net/ckmarkohchang/computa6onal-‐poetry
Code
• Python Tensorflow – h@ps://github.com/ckmarkoh/neuralart_tensorflow
• Python Theano – h@ps://github.com/woonketwong/ar6fy
• Python Theano (ipython notebook) – h@ps://github.com/Lasagne/Recipes/blob/master/examples/styletransfer/Art%20Style%20Transfer.ipynb
• Python deeppy – h@ps://github.com/andersbll/neural_ar6s6c_style
Image URL
• h@p://www.taipei-‐101.com.tw/upload/news/201502/2015021711505431705145.JPG
• h@ps://github.com/andersbll/neural_ar6s6c_style/blob/master/images/starry_night.jpg?raw=true
Acknowledgement
• NTU imlab
About the Speaker
• Email: ckmarkoh at gmail dot com • Blog: h@p://cpmarkchang.logdown.com • Github: h@ps://github.com/ckmarkoh
Mark Chang
• Facebook: h@ps://www.facebook.com/ckmarkoh.chang • Slideshare: h@p://www.slideshare.net/ckmarkohchang • Linkedin: h@ps://www.linkedin.com/pub/mark-‐chang/85/25b/847