Towards Weakly- and Semi- Supervised Object Localization and...

1

Towards Weakly- and Semi- Supervised Object Localization and Semantic

Segmentation

Lecturer: Yunchao WeiImage Formation and Processing (IFP)

Grouphttps://weiyc.github.io

University of Illinois at Urbana-Champaign

2

Fully Supervised Semantic Segmentation

Masks

…

Hard to Collect

3

4

100/C 5min/I

5

Weakly Supervised Annotations

horse personhorse

person

scribbles points image-level labelsbounding boxes

The simplest and the most efficient one

6

person

horsetable

images

annotations

…

…

Loc

Seg

Object Localization Maps

Our Targets

7

CVPR 2017PR 2016 PAMI 2017

CVPR 2018 CVPR 2018 AAAI 2018

Proposal-based Localization Simple to Complex Adversarial Erasing

Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network

Achievements

8

CVPR 2017PR 2016 PAMI 2017

CVPR 2018 CVPR 2018 AAAI 2018

Proposal-based Localization Simple to Complex Adversarial Erasing

Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network

Achievements

9

Proposal-based Localization

Learning to Segment with Image-level Annotations. PR 2016

10

Proposal-based Localization

Hypotheses-CNN-Pooling Localization Map Generation

o Exhaustedly examine each proposal to generate localization

o Introducing false negative pixels(background)

Weakness

Pascal VOC

43.2%

Learning to Segment with Image-level Annotations. PR 2016

11

CVPR 2017PR 2016 PAMI 2017

CVPR 2018 CVPR 2018 AAAI 2018

Proposal-based Localization Simple to Complex Adversarial Erasing

Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network

Achievements

12

Simple to Complex

Simple Images Complex Images

STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation. PAMI 2017

13

Simple images with the corresponding saliency maps

Simple to Complex

STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation. PAMI 2017

14

o Initial-DCNN

o Enhanced-DCNN

o Powerful-DCNN

Simple to Complex

STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation. PAMI 2017

15

Flickr-Clean (40K)

Simple to Complex

STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation. PAMI 2017

16

Networks Training Set mIoU

I-DCNN Flickr-Clean 44.1

E-DCNN Flickr-Clean 46.8

P-DCNN Flickr-Clean+VOC 49.8

Ablation Analysis on Pascal VOC12 val

o Collecting a large number of simple images

o Time consuming for training

Weakness

Simple to ComplexPascal VOC

43.2%

51.2%

STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation. PAMI 2017

17

CVPR 2017PR 2016 PAMI 2017

CVPR 2018 CVPR 2018 AAAI 2018

Proposal-based Localization Simple to Complex Adversarial Erasing

Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network

Achievements

18

Adversarial Erasing

Small and sparse object localization maps Dense and integral object localization maps

dog bird cow

Previous works Our target

Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. CVPR 2017 (oral)

19

Adversarial Erasing

headconf: 1.0

body

conf: 0.8

foot

conf: 0.5

Motivation

Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. CVPR 2017 (oral)

20

Adversarial Erasing

Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. CVPR 2017 (oral)

Our Solution Visualizations

21

Adversarial Erasing

Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. CVPR 2017 (oral)

The pipeline of weakly semantic segmentation based on AE

22

Weakness of Adversarial Erasing

o Time consuming to learn several classification networks.

o Hard to determine how many AE steps should be conducted.

0 150

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Epoch

Loss

AE-step4

AE-step3

AE-step2

AE-step1

Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. CVPR 2017 (oral)

Pascal VOC

43.2%

51.2%

55.7%

23

CVPR 2017PR 2016 PAMI 2017

CVPR 2018 CVPR 2018 AAAI 2018

Proposal-based Localization Simple to Complex Adversarial Erasing

Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network

Achievements

24

Adversarial Complementary Learning

Imag

esC

AM

Ou

rs

…

Classification

…

GAP FC

Conv 1x1 GAP

CA

MO

urs

Adversarial Complementary Learning for Weakly Supervised Object Localization. CVPR 2018

Revisiting CAM

25

Adversarial Complementary Learning

Detailed Framework

Adversarial Complementary Learning for Weakly Supervised Object Localization. CVPR 2018

Erasing

Thresholding

26

Adversarial Complementary Learning

Adversarial Complementary Learning for Weakly Supervised Object Localization. CVPR 2018

27

Adversarial Complementary Learning

Localization Comparison

Adversarial Complementary Learning for Weakly Supervised Object Localization. CVPR 2018

28

Methods Top-1 err. Top-5 err.

Backprop on GoogleLeNet 61.31 50.55

GoogLeNet-GAP (CVPR 2016) 56.40 43.00

GoogLeNet-HaS-32 (ICCV 2017) 54.53 -

GoogLeNet-ACoL(Ours) 53.28 42.58

GoogLeNet-ACoL*(Ours) 53.28 35.22

Backprop on VGGnet 61.12 51.46

VGG-GAP (CVPR 2016) 57.20 45.14

VGGnet-ACoL(Ours) 54.17 40.57

VGGnet-ACoL*(Ours) 54.17 36.66

Adversarial Complementary Learning

Adversarial Complementary Learning for Weakly Supervised Object Localization. CVPR 2018

Localization error on ILSVRC validation set

Pascal VOC

43.2%

51.2%

55.7%

58.8%

29

CVPR 2017PR 2016 PAMI 2017

CVPR 2018 CVPR 2018 AAAI 2018

Proposal-based Localization Simple to Complex Adversarial Erasing

Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network

Achievements

30

Multi-dilated Convolution

Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation. CVPR 2018 (spotlight)

3x3

In

Out

Motivation

kernel

31

o Multi-dilated Convolutional Network for Object Localization

Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation. CVPR 2018 (spotlight)

Multi-dilated Convolution

32

Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation. CVPR 2018 (spotlight)

Multi-dilated Convolution

33

o Weakly- and Semi- Supervised Semantic Segmentation

Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation. CVPR 2018 (spotlight)

Multi-dilated Convolution

34

Multi-dilated Convolution

Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation. CVPR 2018 (spotlight)

Pascal VOC

43.2%

51.2%

55.7%

58.8%

60.8%68.5%

35

CVPR 2017PR 2016 PAMI 2017

CVPR 2018 CVPR 2018 AAAI 2018

Proposal-based Localization Simple to Complex Adversarial Erasing

Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network

Achievements

36

Transferable Semi-supervised Network

In-category Semi-supervised Semantic Segmentation (I3S)

Transferable Semi-supervised Semantic Segmentation. AAAI 2018

37

100/C 5min/I

38

Transferable Semi-supervised Network

In-category Semi-supervised Semantic Segmentation (I3S)

Cross-category Semi-supervised Semantic Segmentation (C3S)

Transferable Semi-supervised Semantic Segmentation. AAAI 2018

39

Label Transfer Network(L-Net)

Prediction Transfer Network(P-Net)

Transferable Semi-supervised Network

Transferable Semi-supervised Semantic Segmentation. AAAI 2018

40

denotes the standard element-wise binary cross-entropy loss

Transferable Semi-supervised Network

Transferable Semi-supervised Semantic Segmentation. AAAI 2018

41

Classification Activation Map

Transferable Semi-supervised Network

Transferable Semi-supervised Semantic Segmentation. AAAI 2018

42

Random Walk based self-diffusion algorithm

Transferable Semi-supervised Network

Transferable Semi-supervised Semantic Segmentation. AAAI 2018

43

Transferable Semi-supervised Network

Transferable Semi-supervised Semantic Segmentation. AAAI 2018

44

Transferable Semi-supervised Network

Transferable Semi-supervised Semantic Segmentation. AAAI 2018

45

Transferable Semi-supervised NetworkPascal VOC

43.2%

51.2%

55.7%

58.8%

60.8%68.5%

64.6%

46

Transferable Semi-supervised Semantic Segmentation. AAAI 2018

47

CVPR 2017PR 2016 PAMI 2017

CVPR 2018 CVPR 2018 AAAI 2018

Proposal-based Localization Simple to Complex Adversarial Erasing

Adversarial Complementary Learning Multi-dilated Convolution Transferable Semi-supervised Network

Summary

person

horsetable

images

annotations

…

…

Loc

Seg

Object Localization Maps

48

Thanks