Addressing GAN limitations: resolution, lack of novelty and control on generations

Camille Couprie

Facebook AI Research

Joint works with O. Sbai, M. Aubry, A, Bordes, M. Elhoseiny, M. Riviere, Y.

LeCun, M. Mathieu, P. Luc, N. Neverova, J. Verbeek.

2019

1

Why do we care about generative models

2

• Scene Understanding can be assessed by checking the ability to

generate plausible new scenes.

• Generative models are interesting if they can be used to go beyond

training data: data of higher resolution, data augmentation to help

train better classifiers, use the learned representations in other

tasks, or make prediction about uncertain events...

• 1/ Design inspiration from adversarial generative networks

• 2/ High resolution, decoupled generation

• 3/ Vector image generation by learning parametric layer

decomposition

• 4/ Future frame prediction

Outline

3

4

Sbai, Elhoseiny, Bordes, LeCun, Couprie, ECCV workshop 17

Design inspiration from generative networks

N o v e l t y

H e d o n i c

Va l u e

1 2 30 4

RANDOM NUMBERS

0 . 3

0 . 7

0 . 1

0 . 8

AdVERSARIAL

NETWORKGenerator

FakeGENERATED

IMAGE

Generative Adversarial Networks

Goodfellow et al, 2014

RANDOM NUMBERS

Real INPUT

Generator

Real

AdVERSARIAL

NETWORK

GENERATED

IMAGE

0 . 3

0 . 7

0 . 1

0 . 8

Generative Adversarial Networks

Goodfellow et al, 2014

Deep convolutional GANsRADFORD ET AL : ICLR 2015

Training with pictures of about 2000 Clothing items

Floral StripedTiled Uniform Dotted Animal Print Graphical

Texture and shape labels

DressSkirt JacketPullover T-Shirt Coat Top

RANDOM NUMBERS

Real INPUT

GeneratorAdVERSARIAL

NETWORK

GENERATED

IMAGE

0 . 3

0 . 7

0 . 1

0 . 8

S h a p e C L A S S

0 / 1

( R E A L / F A K E )

T E X T U R E C L A S S

Class conditioned GAN

• Generator’s

loss

• Discriminator’s

loss

• Auxiliary

classifier

discriminator:

• Additional loss

for the generator:

GAN Optimization objectives

Without conditioning With class conditioning

RANDOM

NUMBERSGenerator

0 . 3

0 . 7

0 . 1

0 . 8

AdVERSARIAL

NETWORK

Generated

Image

Dotted

Floral

graphical

uniform

tiled

striped

Animal print

1 0 0 %

Introduction of a Style Deviation criterion

8 %

6 %

7 %

5 7 %

6 %

1 1 %

9 %

RANDOM

NUMBERSGenerator

0 . 3

0 . 7

0 . 1

0 . 8

AdVERSARIAL

NETWORK

Generated

Image

Dotted

Floral

graphical

uniform

tiled

striped

Animal print

Introduction of a Style Deviation criterion

RANDOM

NUMBERSGenerator

0 . 3

0 . 7

0 . 1

0 . 8

AdVERSARIAL

NETWORKGenerated Image

D o t t e d

F l o r a l

g r a p h i c a l

u n i f o r m

t i l e d

s t r i p e d

A n i m a l p r i n t

With the Style

Deviation criterion

(CAN H)

Multi-class cross entropy loss:

Binary cross entropy loss :

Tested deviation objectives

O v e r a l L L i k a b i l i t y ( % )

6 5 7 0 7 56 08

0

CAN: GAN with Creativity loss, (H) stands for the use of a holistic loss.

CAN

texTUREGAN

Style

CAN(h)

texTURE

CAN(h)

texTUREr e a l i s t i c

A p p e a r a n c e

4 8

5 3 . 5

5 9

6 4 . 5

7

0

Human Evaluation Study

Models with texture deviation are Most Popular

N o v e l t y0 1

L i k e a b i l i t y

6 2

6 6

7 0

7 4

7

8

judged by humans and measured as a distance to similar training images

Can

texture

Can (H)

Shape Can (H)

texture

style

can (h)

GAN

2

0

M. Riviere, C. Couprie, Y. LeCun

Decoupled adversarial image generation

Motivation:

- Take advantage of white background clothing datasets

- Potentially avoid defaults in generated shapes

- Better enforce shape conditioning of generations

1024x1024 generations on the RTW dataset

2

1

Using Morgane’s pytorch “progressive growing of GANs” available online,

Karras et al., ICLR’18

Decoupled architecture

2

2

Shape

Generator

Texture

Generator

Generation

Real image

Shape and

pose classes

Texture, color, shape and pose classes

Discriminator

Real / Fake

Real image

Discriminator

Real / Fake

Texture, color,

shape and

pose classes

×

DgDs

Gs

Gt

z

Random generations

2

3

Progressive growingProgressive growing with decoupled

architecture

Better class conditioning

2

4

Accuracy of classifiers trained on FashionGen

Clothing and FashionGen Shoes on our different

models results (GAN-test metric)

=

+12%

+14%

-12%

Overall average improvement: 4.7%

2

5

Sbai, Couprie, Aubry, arxiv dec18

Vector Image Generation

by learning parametric layer decomposition

Current deep generative models

are great but…

… are limited in resolution, and

control in generations

Kanan et al: Layered

GANs (LR-GANs), ICLR’17

GANIN et al. SPIRAL,

ICML’18

Related work

2

6

Our approach

2

7

Spoiler alert: yes, we can

generate sharper images,

this is just an example.

Iterative generation : !" = $(!, !"'()

Vectorized mask generation *"(+, ,) = $(+, ,, -")

Alpha-blending !"(+, ,) = !"'( +, , . (1-*"(+, ,))+ /"*"(+, ,)

Our iterative pipeline for image reconstruction

2

8

x x

×<latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit>

×<latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit>

+<latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit><latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit><latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit><latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit>

ct<latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit><latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit><latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit><latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit>

ctMt<latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit><latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit><latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit><latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit>

2

9

Our iterative pipeline for image generation

3

0

Training criteria

Adversarial net criterion: Wasserstein loss with Gradient Penalty (WGAN-GP),

Gulrajani et al. NIPS’17

Our generator loss in the GAN setting:

Our generator loss in the image reconstruction setting:

Results using a l1 reconstruction loss

3

1

Results using a l1 reconstruction loss

3

2

Target Our iterative reconstruction

Editing the original image using

extracted masks, by performing

local modifications of

luminosity (top), or color

modification using a blending

of masks (bottom).

Applications

3

3

Image editing using masks:

Using chosen extracted mask(s)

from image reconstruction, we

apply object removal (top)

and color modifications with

object removal (bottom).

Applications

3

4

Image vectorization:

Reconstruction results on

MNIST images. Our model

learns a vectorized mask

representation of digits that

can be generated at any

resolution without

interpolation artifacts.

Applications

3

5

RGB image

Baselines

3

6

RGB imagez

1/ MLP-baseline

MLPResnet

RGB image

Baselines

3

8

RGB imagez

1/ MLP-baseline

2/ ResNet baseline

ResnetResnet

RGB image

Baselines

3

9

RGB imagez

1/ MLP-baseline

2/ ResNet baseline

3/ MLP-xy baseline

MLP

(x,y)

Resnet

Comparative results

4

0

- Using a l1 reconstruction loss

- Parameters for our approach:

20 masks, p of size 10, c of size 3:

260 parameters

- Baselines: size of the latent

code z = 20x13=260

CelebA generations trained on 64x64

images, sampled at 256x256

GAN results

4

1

CIFAR10 generations trained on

32x32 images, sampled at 256x256

GAN

Results on

ImageNet

4

2

trained on

64x64

images,

sampled at

1024x1024

b

Results

4

4

Result

with

perceptual

loss

Perceptual

lossTarget

Conclusion and Future work

4

5

Faster training

Use class conditioning

Texture image generation

Predicting next frames in videos

Michael Mathieu, Camille Couprie, Yann LeCun, ICLR16

4 input images Our 2 predictions

Deep multiscale video prediction beyond Mean square error

• Result with a simple convolutional network trained minimizing an l2 loss

• Our result using

• A multiscale architecture

• an image gradient different loss

• Use adversarial training

4

8

P. Luc, N. Neverova, C. Couprie, J. Verbeek, Y. LeCun ECCV18

Predicting deeper into the future of semantic

segmentations

4 input images Our 2 predictions

• Predictions in the RGB space quickly become blurry despites previous attempts

• Idea: predict in the space of semantic segmentation

Approach – predicting deeper into the future

Single time-step

Autoregressive model

Batch model

St−3 St−2 St−1 St St+1

Lt+1

St−3 St−2 St−1 St St+1 St+2 St+3

Lt+1 Lt+2 Lt+3

St−3 St−2 St−1 St St+1 St+2 St+3

Lt+1 Lt+2 Lt+3

Autoregressive model is either :

• used for inference without

additional training (w.r.t. to single

time step model) AR

• Fine-tuned using BPTT

AR fine-tune

Autoregressive mode is only possible

for X2X, S2S, XS2XS

Same color = shared weights

4

9

Some results

50

Baselines :• Copy the last input frame to the

output

• Estimate flow between the two last inputs, and project the last input forward using the flow

Performance measure (mean IoU) of our approach and baselines on the Cityscapes dataset

Flow baseline

Our AutoRegressive fine-tune result

Instance level segmentation: Mask RCNN

• Extends Faster RCNN [Ren et al.’15] by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition

K. He G. Gkioxari P. Dollar R. Girshick’17

1

0

5

2

P. Luc, C. Couprie, Y. LeCun, J. Verbeek, ECCV18

Predicting future segmentation instances by

forecasting convolutional features

car 1.00

bicycle 0.92bicycle 0.97

person 0.99person 0.97person 0.98 person 0.93person 0.72

person 0.97

bicycle 1.00

person 0.76person 0.97

person 0.99person 0.98person 0.99person 0.94

person 0.91 person 1.00 person 0.56person 0.84

Luc, Neverova et al. ICCV17 F2F predictions

Conclusions

5

3

Some open problems:

- Automatic metrics to evaluate generative models performances

- Non deterministic training losses for future prediction

Torch code online :

- For future video prediction:

- Vector image generation: available soon on Othman Sbai’s github

- of RGBs : on Michael Mathieu’s github

- of semantic segmentations : on Pauline’s Luc github

- of instance segmentation : on Pauline’s Luc github