View
2
Download
0
Category
Preview:
Citation preview
ECE 6554: Topic PresentationXiaolong Li
Semantic Segmentation
UC
LA:h
ttps:
//goo
.gl/i
mag
es/I0
VTi2
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille. ICLR 2015
Fully Convolutional Networks for Semantic Segmentation. J. Long, E. Shelhamer, and T. Darrell, CVPR 2015
OUTLINE
Paper to talk about:
Semantic Segmentation
Why?
ECE 6554: Topic PresentationXiaolong Li
What is Semantic Segmentation
ECE 6554: Topic PresentationXiaolong Li
‘Lena’ lena mirror
What is Semantic Segmentation
ECE 6554: Topic PresentationXiaolong Li
Goal: Partition the image into semantically meaningfulparts, and classify each part ——>Patch-wiseRecognizing and delineating objects in an image Classifying each pixel in the image ——>Pixel-wise
Why Semantic Segmentation?
ECE 6554: Topic PresentationXiaolong Li
To let robots segment objects so that they can graspthem
http
s://g
oo.g
l/im
ages
/6xA
QAM
ECE 6554: Topic PresentationXiaolong Li
Why Semantic Segmentation?
Useful tool for editing images, visual effects
CVF
X Le
ctur
e1:
http
s://w
ww.
yout
ube.
com
/wat
ch?v
=rE-
hVty
tT-I
ECE 6554: Topic PresentationXiaolong Li
Why Semantic Segmentation?
Autonomous Driving, to differentiate pedestrian and background
Citydataset
Fully Convolutional Networks for Semantic Segmentation. J. Long, E. Shelhamer, and T. Darrell, CVPR 2015
ECE 6554: Topic PresentationXiaolong Li
Fully Convolutional Networks for Semantic Segmentation. J. Long, E. Shelhamer, and T. Darrell, CVPR 2015
Usual convolutional networks
Fully convolutional networks
ECE 6554: Topic PresentationXiaolong Li
ECE 6554: Topic PresentationXiaolong Li
2015
, Ber
kele
y Vi
sion
: http
://tu
toria
l.caf
fe.b
erke
leyv
isio
n.or
g/
To understand “Fully Convolutional”
ECE 6554: Topic PresentationXiaolong Li
2015
, Ber
kele
y Vi
sion
: http
://tu
toria
l.caf
fe.b
erke
leyv
isio
n.or
g/
To understand “Fully Convolutional”
A typical CNN
ECE 6554: Topic PresentationXiaolong Li
To understand “Fully Convolutional”A classification CNN
A FCN
2015
, Ber
kele
y Vi
sion
: http
://tu
toria
l.caf
fe.b
erke
leyv
isio
n.or
g/
FCN: segmentation that combines layers of hierarchy and refines the spatial precision of the output.
ECE 6554: Topic PresentationXiaolong Li
Segmentation Architecture
1. ILSVRC classifiers, in-network up sampling and a pixel-wise loss.
2. Add skips between layers to fuse coarse, semantic and local, appearance
3. Dense predictions, pixel-wise prediction
ECE 6554: Topic PresentationXiaolong Li
Some Tricks
ECE 6554: Topic PresentationXiaolong Li
skip layers
Some Tricks
ECE 6554: Topic PresentationXiaolong Li
skip layers refinement
Interpolation
1. Up-sampling is performed in-network for end-to-end learning by
back-propagation from the pixel wise loss.
2. The deconvolution filter in such a layer can be learned.
ECE 6554: Topic PresentationXiaolong Li
Some Tricks
Some results:
PASCAL VOC
NYUDv2
ECE 6554: Topic PresentationXiaolong Li
Conclusion
1. Fine-tuning from classification to segmentation gives reasonable
predictions for each net.
2. Learning through up-sampling combined with
the skip layer fusion to be more effective and efficient
ECE 6554: Topic PresentationXiaolong Li
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille. ICLR 2015
ECE 6554: Topic PresentationXiaolong Li
Paper’s main idea
1.Use CNN to generate a rough prediction of segmentation (smooth, blurry heat map)
2.Refine this prediction with a conditional random field (CRF)
ECE 6554: Topic PresentationXiaolong Li
Why are CNNs insufficient?
Good for high-level vision tasks like classification, bad for low level tasks like segmentation.
• Problem: subsampling• Problem: spatial invariance (shared kernel weights)
Solution: fully connected CRF
ECE 6554: Topic PresentationXiaolong Li
Holes’ algorithms
Solution: fully connected CRF
Solution: fully connected CRF
ECE 6554: Topic PresentationXiaolong Li
Solution: fully connected CRF
CRF
Randomly choose points and give initial label
ECE 6554: Topic PresentationXiaolong Li
CRF Energy Function
ECE 6554: Topic PresentationXiaolong Li
Global Map
ECE 6554: Topic PresentationXiaolong Li
Comparison to state-of-the-art
ECE 6554: Topic PresentationXiaolong Li
Comparison to state-of-the-art
ECE 6554: Topic PresentationXiaolong Li
Comparison to state-of-the-art
ECE 6554: Topic PresentationXiaolong Li
Successful Cases
ECE 6554: Topic PresentationXiaolong Li
Failure Cases
ECE 6554: Topic PresentationXiaolong Li
Conclusion
• Modify the CNN architecture to become less spatially invariant.
• Use the CNN to compute a rough score map.
• Use a fully connected CRF to sharpen the score
ECE 6554: Topic PresentationXiaolong Li
Experiments
Intel Xeon E5-2670
NVIDIA GPU
Caffe
VOC_FCN_32s
Python
Cuda8.0
ECE 6554: Topic PresentationXiaolong Li
load image, switch to BGR, subtract mean, and make dims C x H x W for Caffe
Data_preparation
ECE 6554: Topic PresentationXiaolong Li
Experiment
ECE 6554: Topic PresentationXiaolong Li
26.862607 1.238836
Experiment
ECE 6554: Topic PresentationXiaolong Li
39.570141 1.738234
Experiment
ECE 6554: Topic PresentationXiaolong Li
32.238836 1.238836
Experiment
ECE 6554: Topic PresentationXiaolong Li
39.570141 1.5334832
Experiment
ECE 6554: Topic PresentationXiaolong Li
27.895173 1.239234
Conclusion
ECE 6554: Topic PresentationXiaolong Li
1.Their network is very fast even when dealing with high resolutionimage, and GPU is at least 20 times faster than CPU.
2. The algorithms show good performance towards images when the objects are either well-separated or overlapped with each other 3. The background of image like sky, grass has a big influence on the segmentation.
Better performance could be expected with their FCN_8s, and detailed performance on validation dataset needs to be checked.
ECE 6554: Topic PresentationXiaolong Li
Thanks
Recommended