Supervisors: Sergio Escalera - UB sergio/linked/ ¢  Supervisors: Sergio Escalera Petia Radeva. Title:

  • View
    1

  • Download
    0

Embed Size (px)

Text of Supervisors: Sergio Escalera - UB sergio/linked/ ¢  Supervisors: Sergio Escalera Petia...

  • 20 – 07 ‐ 2010

    Author:  Antonio HernándezVela

    Supervisors:  Sergio Escalera Petia Radeva

  • 2

  •  Introduction M h d l Methodology  Spatio‐Temporal GrabCut Human segmentaton  Active Appearance models   Face fitting  Conditional Random Field   Limb recovery

    V lid i Validation  Existent public data sets  New Human Body Limb data set

     Conclusions and future work

    3

  •  Introduction M h d l Methodology  Spatio‐Temporal GrabCut Human segmentaton  Active Appearance models   Face fitting  Conditional Random Field   Limb recovery

    V lid i Validation  Existent public data sets  New Human Body Limb data set

     Conclusions and future work

    4

  •  Video oriented

     Human segmentation and pose recovery

    5

  •  Human segmentation for  human pose [1] and face human pose [1] and face  recovery [2] assistance

    [1] Ramanan, D. "Learning to Parse Images of Articulated Bodies ", In NIPS, 2006. [2] T. Cootes, J.Edwards and C. Taylor, "Active Appearance models.", IEEE Transactions on Pattern  Analysis and Machine Intelligence, 1998. 6

  •  Introduction M h d l Methodology  Spatio‐Temporal GrabCut Human segmentaton  Active Appearance models   Face fitting  Conditional Random Field   Limb recovery

    V lid i Validation  Existent public data sets  New Human Body Limb data set

     Conclusions and future work

    7

  • Person face fittingfitting

    Person pose Initialization STGrabCut

    Person pose estimation

    8

  •  HOG‐based [4] person detectorSeed initialization

    STGrabCut

    PPerson pose estimation

    Background seeds

    Unknown region

    Person face fitting

    9

    [4] Navneet Dalal and Bill Triggs, "Histograms of Oriented Gradients for Human Detection" , In  CVPR, 2005.

  • d  Face detection [5] ForegroundseedsSeed

    initialization

    STGrabCut

    PPerson pose estimation

    Person face fitting

    10

    [5] Paul Viola and Michael Jones, "Robust real‐time face detection" , International Journal of  Computer Vision, 2004.

  • GrabCut Seed

    initialization

    Pixel‐level segmentation

    STGrabCut

     GrabCut segmentation

    segmentation refinement

    P  GrabCut segmentation GrabCut

    Person pose estimation

    Seed init for next frame

    Person face fitting

    11

  • Sl S Toy example Log‐likelihood of  GMM models

    (BGD,FGD) over S RGB

    T

    Minimum cut ==  Pi l diffTminimum energy

     Ci

    iC ||

    Pixel differences based on RGB 

    Euclidean distance

    12 [3] C Rother, V Kolmogorov, A Blake. "Grabcut: Interactive foreground extraction using iterated  graph cuts" , ACM Transactions on Graphics, 2004.

    Ci

  •  Iterative procedure

    Until convergence

    GMM  Assign pixelstoGMM  Segmentationmodelling toGMM components

    g (min‐cut)

    [3] C Rother, V Kolmogorov, A Blake. "Grabcut: Interactive foreground extraction using iterated  graph cuts" , ACM Transactions on Graphics, 2004.

    13

  • GrabCut Seed

    initialization

    Pixel‐level segmentation

    STGrabCut

     GrabCut segmentation

    segmentation refinement

    P  GrabCut segmentation  Mean‐shift GMM initialization 

    GrabCut Person pose estimation

    (spatial coherence) Seed init for next frame

    Person face fitting

    14

  • Seed initialization GrabCut

    STGrabCut Pixel‐level segmentation

    P

    segmentation refinement

     Biased to convex regions!Person pose estimation GrabCut

     Biased to convex regions!

    B k d  d  f   i l Person face

    fitting Seed init for next frame

     Background seeds for pixels:

    15

  • GrabCut Seed

    initialization

    Pixel‐level segmentation

    STGrabCut segmentation refinement

    P GrabCut

     GrabCut re‐iteration I f   t ti   ith   

    Person pose estimation

    Seed init for next frame

     Infer segmentation with new  background seeds

    Person face fitting

    16

  • GrabCut Seed

    initialization

    Pixel‐level segmentation

    STGrabCut segmentation refinement

    P GrabCut

     Initialization for next frame

    Person pose estimation

    Seed init for next frame

    Person face fitting

    17

  •  CRF‐based [1] pose recoverySeed initialization

    STGrabCut

    PPerson pose estimation

    Person face fitting

    18

    [1] Ramanan, D. "Learning to Parse Images of Articulated Bodies ", In NIPS, 2006.

  •  AAM‐based [2] face recoverySeed initialization

    STGrabCut

    PPerson pose estimation  3 meshes: 1 frontal, 2 lateral

    Person face fitting

    19

    [2] T. Cootes, J.Edwards and C. Taylor, "Active Appearance models.", IEEE Transactions on Pattern  Analysis and Machine Intelligence, 1998.

  •  Introduction M h d l Methodology  Spatio‐Temporal GrabCut Human segmentaton  Active Appearance models   Face fitting  Conditional Random Field   Limb recovery

    V lid ti Validation  Existent public data sets  New Human Body Limb data set

     Conclusions and future work

    20

  •  CVSG data set [6] Vid      f Video sequence: 307 frames  Ground truth

     Undergraduate thesis defenseg  4 video sequences: 720 frames each

    21

    [6] F. Tiburzi, M. Escudero, J. Bescos, and J. Martinez.”A ground‐truth for motion‐based video‐ object segmentation” IEEE International Conference on Image Processing (Workshop on Multimedia  Information Retrieval, 2008

  •  New human body limb data set  227 images  25 different people5 p p  Ground truth

    22http://www.maia.ub.es/~sergio/Code.html

  •  Quantitative results (CVSG data set)

    0,8747

    0,8 0,9 1

    0,53560,5424 0,6229

    0,5 0,6 0,7

    GrabCut

    Mean‐shift

    pp in g %

    0 2 0,3 0,4 ,5

    Morphology

    ST‐GrabCut

    O ve

    rl ap

    0 0,1 0,2

    23

  •  Qualitative results (CVSG data set)

    24

  • l l d Qualitative results (UB data set)

    25

  • d Body pose recovery

    n m en

    ta tio

    n N o  se gm

    ut T‐ G ra bC

    u

    26

    S

  • F Face recovery

    27

  •  Temporal joint body and face recovery

    28

  •  Face and body pose recovery (CVSG)

    0 9636 0,896

    0,7919

    0,9636 0,876

    0,8

    1

    0,6

    No segmentationpp in g %

    0,4 No segmentation

    STGrabCut

    O ve

    rl ap

    0

    0,2

    29

    Face Body pose

  •  Body pose recovery

    200 Trunk

    Up‐arms

    #overlapping wins

    100

    150 Up arms

    Up‐legs

    Low‐arms

    50 Low‐legs

    Head

    0

    STGrabCut*

    30

    No segmentation * No temporal information is used

  •  Introduction M h d l Methodology  Spatio‐Temporal GrabCut Human segmentaton  Active Appearance models   Face fitting  Conditional Random Field   Limb recovery

    V lid i Validation  Public data sets  Human Body Limb data set

     Conclusions and future work

    31

  •  Extension of GrabCut for human segmentation F ll i h d Fully automatic method  Spatio‐temporal coherence  Segmentation convexity problem

     Face recovery with temporal coherence  New human body limb database New human body limb database  Human segmentation helps to retrieve face and  b dbody pose

    32

  •  Oral paper at AMFG Workshop (CVPR  Oral paper at AMFG Workshop (CVPR  2010)  Antonio Hernández  Miguel Reyes  Sergio  Antonio Hernández, Miguel Reyes, Sergio  Escalera and Petia Radeva, “Spatio‐ temporal GrabCut HumanSegmentationtemporal GrabCut HumanSegmentation for Face and Pose Recovery”

     Submission to Machine Vision and  ApplicationsApplications

    33

  •  Include spatio‐temporal coherence inside graph c ts frame orkcuts framework  Extended graph for image volumes  New spatio‐temporal potential

     Improve segmentation using face and pose  recovery feedbackrecovery feedback

    34

  • 20 – 07 ‐ 2010

    Author:  Antonio HernándezVela

    Supervisors:  Sergio Escalera Petia Radeva