Articulated Pose Estimation with flexible mixtures of parts

Pictorial Structures for Articulated Pose EstimationAnkit GuptaCSE 590V: Vision Seminar

1GoalArticulated pose estimation ( )recovers the pose of an articulated object which consists of joints and rigid parts

Slide taken from authors, Yang et al.2Classic ApproachPart RepresentationHead, Torso, Arm, LegLocation, Rotation, Scale

Marr & Nishihara 1978

Slide taken from authors, Yang et al.3Classic ApproachPart RepresentationHead, Torso, Arm, LegLocation, Rotation, Scale

Fischler & Elschlager 1973 Felzenszwalb & Huttenlocher 2005

Marr & Nishihara 1978

Pictorial StructureUnary TemplatesPairwise SpringsSlide taken from authors, Yang et al.4Classic ApproachPart RepresentationHead, Torso, Arm, LegLocation, Rotation, Scale

Fischler & Elschlager 1973 Felzenszwalb & Huttenlocher 2005

Marr & Nishihara 1978

Andriluka etc. 2009Eichner etc. 2009Johnson & Everingham 2010Sapp etc. 2010Singh etc. 2010Tran & Forsyth 2010Epshteian & Ullman 2007Ferrari etc. 2008Lan & Huttenlocher 2005Ramanan 2007Sigal & Black 2006Wang & Mori 2008Pictorial StructureUnary TemplatesPairwise Springs

Slide taken from authors, Yang et al.5Pictorial Structures for Object RecognitionPedro F. Felzenszwalb, Daniel P. Huttenlocher IJCV, 2005

Pictorial structure for Face

Pictorial Structure Model

Slide taken from authors, Yang et al.8Pictorial Structure Model

Slide taken from authors, Yang et al.9Pictorial Structure Model

Slide taken from authors, Yang et al.10Using this ModelTest phaseTrain phaseGiven: - Images (I) - Known locations of the parts (L)

Need to learn

- Unary templates

- Spatial featuresTest phaseTrain phase

Given: - Images (I) - Known locations of the parts (L)

Need to learn

- Unary templates

- Spatial featuresTest phaseTrain phase

Standard Structural SVM formulation - Standard solvers available (SVMStruct)

Given: - Images (I) - Known locations of the parts (L)

Need to learn

- Unary templates

- Spatial featuresTest phaseTrain phase

Standard Structural SVM formulation - Standard solvers available (SVMStruct)

Given: - Image (I)

Need to compute

- Part locations (L)

Algorithm - L* = arg max (S(I,L))

Given: - Images (I) - Known locations of the parts (L)

Need to learn

- Unary templates

- Spatial featuresTest phaseTrain phase

Standard Structural SVM formulation - Standard solvers available (SVMStruct)

Given: - Image (I)

Need to compute

- Part locations (L)

Algorithm - L* = arg max (S(I,L))

Standard inference problem - For tree graphs, can be exactly computed using belief propagationArticulated Pose Estimation with Flexible Mixtures of Parts Yi Yang & Deva RamananUniversity of California, Irvine17Problems with previous methods: Wide VariationsIn-plane rotationForeshorteningScaling

Out-of-plane rotationIntra-category variationAspect ratio

Slide taken from authors, Yang et al.18Problems with previous methods: Wide VariationsIn-plane rotationForeshorteningScaling

Out-of-plane rotationIntra-category variationAspect ratio

Nave brute-force evaluation is expensive

Slide taken from authors, Yang et al.19Our Method Mini-Parts

Key idea:

mini part model can approximate deformationsSlide taken from authors, Yang et al.20Example: Arm Approximation

Slide taken from authors, Yang et al.21Pictorial Structure Model

Slide taken from authors, Yang et al.22The Flexible Mixture Model

Slide taken from authors, Yang et al.23Our Flexible Mixture Model

Slide taken from authors, Yang et al.24Our Flexible Mixture Model

Slide taken from authors, Yang et al.25Our Flexible Mixture Model

Slide taken from authors, Yang et al.26Co-occurrence Bias

Slide taken from authors, Yang et al.27Co-occurrence Bias: ExampleLet part i : eyes, mixture mi = {open, closed}part j : mouth,mixture mj = {smile, frown}We visualize the co-occurrence model for each pair as a 4 by 4 matrix. For example, if two parts are on the same rigid limb, they should have consistent orientations. Then this matrix will be diagonal .28Co-occurrence Bias: Example b (closed eyes, smiling mouth) b (open eyes, smiling mouth)

vsLet part i : eyes, mixture mi = {open, closed}part j : mouth,mixture mj = {smile, frown}29Co-occurrence Bias: Example b (closed eyes, smiling mouth) b (open eyes, smiling mouth)