Download pdf - Supervisors: Sergio Escalera - UBsergio/linked/tonitesisslides.pdf · Supervisors: Sergio Escalera Petia Radeva. Title: Microsoft PowerPoint - Presentation_CVPR_FINAL.pptx Author:

20 – 07 ‐ 2010

Author: Antonio HernándezVela

Supervisors: Sergio EscaleraPetia Radeva

2

IntroductionM h d l Methodology Spatio‐Temporal GrabCut Human segmentaton Active Appearance models Face fitting Conditional Random Field Limb recovery

V lid i Validation Existent public data sets New Human Body Limb data set

Conclusions and future work

3




4

Video oriented

Human segmentation and pose recovery

5

Human segmentation for human pose [1] and face human pose [1] and face recovery [2] assistance

[1] Ramanan, D. "Learning to Parse Images of Articulated Bodies ", In NIPS, 2006.[2] T. Cootes, J.Edwards and C. Taylor, "Active Appearance models.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998. 6




7

Person facefittingfitting

Person poseInitialization STGrabCut

Person poseestimation

8

HOG‐based [4] person detectorSeedinitialization

STGrabCut

PPerson poseestimation

Backgroundseeds

Unknownregion

Person facefitting

9

[4] Navneet Dalal and Bill Triggs, "Histograms of Oriented Gradients for Human Detection" , In CVPR, 2005.

d Face detection [5] Foreground

seedsSeedinitialization

STGrabCut


Person facefitting

10

[5] Paul Viola and Michael Jones, "Robust real‐time face detection" , International Journal of Computer Vision, 2004.

GrabCutSeed

initialization

Pixel‐levelsegmentation

STGrabCut

GrabCut segmentation

segmentationrefinement

P GrabCut segmentationGrabCut


Seed init fornext frame

Person facefitting

11

Sl S Toy example Log‐likelihood of

GMM models(BGD,FGD) over

S RGB

T

Minimum cut == Pi l diffTminimum energy

Ci

iC ||

Pixel differencesbased on RGB

Euclidean distance

12[3] C Rother, V Kolmogorov, A Blake. "Grabcut: Interactive foreground extraction using iterated graph cuts" , ACM Transactions on Graphics, 2004.

Ci

Iterative procedure

Until convergence

GMM Assign pixelstoGMM Segmentation

modelling toGMM components

g(min‐cut)

[3] C Rother, V Kolmogorov, A Blake. "Grabcut: Interactive foreground extraction using iterated graph cuts" , ACM Transactions on Graphics, 2004.

13

GrabCutSeed

initialization


STGrabCut

GrabCut segmentation


P GrabCut segmentation Mean‐shift GMM initialization

GrabCutPerson poseestimation

(spatial coherence)Seed init fornext frame

Person facefitting

14

Seedinitialization GrabCut

STGrabCut Pixel‐levelsegmentation

P


Biased to convex regions!Person poseestimation GrabCut

Biased to convex regions!

B k d d f i lPerson face

fittingSeed init fornext frame

Background seeds for pixels:

15

GrabCutSeed

initialization


STGrabCutsegmentationrefinement

PGrabCut

GrabCut re‐iterationI f t ti ith



Infer segmentation with new background seeds

Person facefitting

16

GrabCutSeed

initialization


STGrabCutsegmentationrefinement

PGrabCut

Initialization for next frame



Person facefitting

17

CRF‐based [1] pose recoverySeedinitialization

STGrabCut


Person facefitting

18

[1] Ramanan, D. "Learning to Parse Images of Articulated Bodies ", In NIPS, 2006.

AAM‐based [2] face recoverySeedinitialization

STGrabCut

PPerson poseestimation 3 meshes: 1 frontal, 2 lateral

Person facefitting

19

[2] T. Cootes, J.Edwards and C. Taylor, "Active Appearance models.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998.


V lid ti Validation Existent public data sets New Human Body Limb data set


20

CVSG data set [6]Vid f Video sequence: 307 frames Ground truth

Undergraduate thesis defenseg 4 video sequences: 720 frames each

21

[6] F. Tiburzi, M. Escudero, J. Bescos, and J. Martinez.”A ground‐truth for motion‐based video‐object segmentation” IEEE International Conference on Image Processing (Workshop on Multimedia Information Retrieval, 2008

New human body limb data set 227 images 25 different people5 p p Ground truth

22http://www.maia.ub.es/~sergio/Code.html

Quantitative results (CVSG data set)

0,8747

0,80,91

0,53560,54240,6229

0,50,60,7

GrabCut

Mean‐shift

pping%

0 20,30,4,5

Morphology

ST‐GrabCut

Ove

rlap

00,10,2

23

Qualitative results (CVSG data set)

24

l l d Qualitative results (UB data set)

25

d Body pose recovery

nmen

tatio

nNo segm

utT‐GrabC

u

26

S

F Face recovery

27

Temporal joint body and face recovery

28

Face and body pose recovery (CVSG)

0 96360,896

0,7919

0,96360,876

0,8

1

0,6

No segmentationpping%

0,4No segmentation

STGrabCut

Ove

rlap

0

0,2

29

Face Body pose

Body pose recovery

200 Trunk

Up‐arms

#overlapping wins

100

150Up arms

Up‐legs

Low‐arms

50Low‐legs

Head

0

STGrabCut*

30

No segmentation* No temporal information is used


V lid i Validation Public data sets Human Body Limb data set


31

Extension of GrabCut for human segmentationF ll i h d Fully automatic method Spatio‐temporal coherence Segmentation convexity problem

Face recovery with temporal coherence New human body limb database New human body limb database Human segmentation helps to retrieve face and b dbody pose

32

Oral paper at AMFG Workshop (CVPR Oral paper at AMFG Workshop (CVPR 2010) Antonio Hernández Miguel Reyes Sergio Antonio Hernández, Miguel Reyes, Sergio Escalera and Petia Radeva, “Spatio‐temporal GrabCut HumanSegmentationtemporal GrabCut HumanSegmentationfor Face and Pose Recovery”

Submission to Machine Vision and ApplicationsApplications

33

Include spatio‐temporal coherence inside graphc ts frame orkcuts framework Extended graph for image volumes New spatio‐temporal potential

Improve segmentation using face and pose recovery feedbackrecovery feedback

34

20 – 07 ‐ 2010

Author: Antonio HernándezVela

Supervisors: Sergio EscaleraPetia Radeva