Embed Size (px)
Citation preview
Multimodal Unsupervised Image-to-Image Translation
Ming-Yu Liu
NVIDIA
Supervised/Paired/Aligned/Registered Unsupervised/Unpaired/Unaligned/Unregistered
Supervised vs UnsupervisedPaired vs UnpairedAligned vs UnalignedSupervised vs Unsupervised
Image Domain Transfer
Example Applications
Generator
……
Discriminator real/fake
Real data
Goodfellow et al. 2014
Generative Adversarial Networks (GANs)
Sample
TranslationNetwork𝑭𝟏→𝟐
……
Discriminator real/fake
Domain 2
……Domain 1
Plain GAN for Unsupervised Image-to-Image Translation
CycleGAN and UNIT
• CycleGAN (cycle consistency) [Zhu et al. 2017]
• UNIT (shared latent space) [Liu et al. 2017]
shared latent space
cycleconsistency
shared latent space ⟹ cycle consistency
Unimodality
Towards Multimodality
…
Cycle consistency does not allow multimodality
Domain 𝒳1 Domain 𝒳2
𝑥1
𝑥2′
𝑥2′′
Cycle consistency:𝒳1 → 𝒳2 →𝒳1
Cycle consistency:𝒳2 → 𝒳1 →𝒳2
𝑥2′ = 𝑥2
′′
𝑝(𝑥2|𝑥1) = 𝛿 𝑥2 − 𝑥2′
𝑝(𝑥2|𝑥1) = 𝛿(𝑥2 − 𝑥2′′)
Shared latent space does not allow multimodality
shared latent space
cycleconsistency
Disentangling the Latent Space
•UNIT • A single shared, domain-invariant latent space 𝒵
Disentangling the Latent Space
•Multimodal UNIT (MUNIT)• A content space 𝒞 that is shared, domain-invariant• Two style spaces 𝒮1, 𝒮2 that are unshared, domain-specific
Training
•Notations:• 𝑥: images• 𝑐: content• 𝑠: style
• Loss:• Bidirectional reconstruction loss
• Image reconstruction loss
• Latent reconstruction loss
• GAN lossWithin-domain reconstructionCross-domain translation
Bidirectional Reconstruction Loss:Image Reconstruction
Notations:•𝑥: images• 𝑐: content• 𝑠: style
Bidirectional Reconstruction Loss:Latent Reconstruction
Notations:•𝑥: images• 𝑐: content• 𝑠: style
GAN Loss
Notations:•𝑥: images• 𝑐: content• 𝑠: style
remove style
Background: Instance Normalization (IN)
Content feature: 𝑐normalization affine
transformation
“Instance Normalization: The Missing Ingredient for Fast Stylization”, Ulyanov et al. 2017
apply style
Feedforward transfer of a single style Content Output
Adaptive Instance Normalization (AdaIN)
Style feature: 𝑠
Content feature: 𝑐
Feedforward transfer of arbitrary styles
VGG
Content
Style
AdaIN Decoder Output
remove style
apply style
AdaIN in a Generative Network
En
co
de
rE
nco
de
r
Ad
aIN
ML
PE
nco
de
r
Ad
aIN
De
co
de
r
De
co
de
r
AdaIN in style transfer AdaIN in a generative network
AdaIN in a Generative Network
En
co
de
rE
nco
de
r
Ad
aIN
De
co
de
r
ML
PE
nco
de
r
Ad
aIN
De
co
de
rAd
aIN
Ad
aIN
Co
nv
Co
nv
AdaIN in style transfer AdaIN in a generative network
Architectural Implementation
Sketches <-> Photo
Input Outputs
Cats ↔ Dogs
Input Outputs
Synthetic ↔ Real
Input Outputs
Summer ↔ Winter
Input Outputs
Example-guided Translation
Example-guided Translation
Conclusion
• Translate one input image to multiple corresponding images in the target domain.
• Content and style decomposition via the AdaIN design
• ECCV 2018
• MUNIT code: https://github.com/nvlabs/munit/
• Paper: https://arxiv.org/abs/1804.04732
Xun HuangNVIDIA, Cornell
Serge BelongieCornell
Jan KautzNVIDIA
Ming-Yu LiuNVIDIA
![Microsoft€¦ · percentage a method is preferred over MUNIT. For both metrics, the higher the better. UNIT [15] CycleGAN [8] CycleGAN* [8] with noise MUNIT w/o cc recon MUNIT w/o](https://img.pdfslide.us/doc/110x75/5f3db9c5dd13901e3c4989f2/microsoft-percentage-a-method-is-preferred-over-munit-for-both-metrics-the-higher.jpg)
