View
221
Download
4
Tags:
Embed Size (px)
Citation preview
Simultaneous Segmentation and 3D Pose Estimation of Humans
Philip H.S. TorrPawan Kumar, Pushmeet Kohli, Matt Bray
Oxford Brookes University
Arasanathan Thayananthan, Bjorn Stenger, Roberto CipollaCambridge University
Algebra
Unifying Conjecture
Tracking = Detection = Recognition Detection = Segmentation
• therefore Tracking (pose estimation)=Segmentation?
Objective
Image Segmentation Pose Estimate??
Aim to get a clean segmentation of a human…
Developments
ICCV 2003, pose estimation as fast nearest neighbour plus dynamics (inspired by Gavrilla and Toyoma & Blake)
BMVC 2004, parts based chamfer to make space of templates more flexible (a la pictorial structures of Huttenlocher)
CVPR 2005, ObjCut combining segmentation and detection.
ICCV 2005 Dynamic Graph Cuts ECCV 2006, interpolation of poses using the MVRVM
(Agarwal and Triggs) ECCV 2006 combination of pose estimation and
segmentation using graph cuts.
Tracking as Detection (Stenger et al ICCV 2003)
Detection has become very efficient,e.g. real-time face detection, pedestrian detection
Example: Pedestrian detection [Gavrila & Philomin, 1999]: Find match among large number of exemplar templates
Issues: Number of templates needed Efficient search Robust cost function
Cascaded Classifiers
First filter : 19.8 % patches remaining
1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7
Filter 10 : 0.74 % patches remaining
1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7
Filter 20 : 0.06 % patches remaining
1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7
Filter 30 : 0.01 % patches remaining
1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7
Filter 70 : 0.007 % patches remaining
1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7
Hierarchical Detection Efficient template matching (Huttenlocher & Olson,
Gavrila) Idea: When matching similar objects, speed-up by
forming template hierarchy found by clustering Match prototypes first, sub-tree only if cost below
threshold
Trees
These search trees are the same as used for efficient nearest neighbour.
Add dynamic model and • Detection = Tracking = Recognition
Evaluation at Multiple Resolutions
One traversal of tree per time step
Evaluation at Multiple Resolutions
Tree: 9000 templates of hand pointing, rigid
Templates at Level 1
Templates at Level 2
Templates at Level 3
Comparison with Particle Filters
This method is grid based,• No need to render the model on line• Like efficient search• Can always use this as a proposal process for
a particle filter if need be.
Interpolation, MVRVM, ECCV 2006
Code available.
Energy being Optimized, link to graph cuts
Combination of• Edge term (quickly evaluated using chamfer)• Interior term (quickly evaluated using integral
images)
Note that possible templates are a bit like cuts that we put down, one could think of this whole process as a constrained search for the best graph cut.
Likelihood : Edges
Edge Detection Projected Contours
Robust EdgeMatching
Input Image 3D Model
Chamfer MatchingInput image Canny edges
Distance transform Projected Contours
Likelihood : Colour
Skin Colour ModelProjected Silhouette
Input Image 3D Model
Template Matching
Template Matching =
Template Matching = constrained search for a cut/segmentation?
Detection = Segmentation?
Objective
Image Segmentation Pose Estimate??
Aim to get a clean segmentation of a human…
MRF for Interactive Image Segmentation, Boykov and Jolly [ICCV 2001]
EnergyMR
F
Pair-wise Terms MAP SolutionUnary likelihoodData (D)
Unary likelihood Contrast Term Uniform Prior(Potts Model)
Maximum-a-posteriori (MAP) solution x* = arg min E(x)x
=
However…
This energy formulation rarely provides realistic (target-
like) results.
ObjCut (yesterday)
Unary potential
Pairwise potential
Pose parameters
Labels
Pixels
Prior Potts model
Pose-specific MRF
Layer 2
Layer 1
Transformations
Θ1
P(Θ1) = 0.9
Cow Instance
Do we really need accurate models?
Do we really need accurate models?
Segmentation boundary can be extracted from edges
Rough 3D Shape-prior enough for region disambiguation
Energy of the Pose-specific MRFEnergy to be
minimizedUnary term
Shape prior
Pairwise potential
Potts model
But what should be the value of θ?
The different terms of the MRF
Original image
Likelihood of being foreground given a
foreground histogram
Grimson-Stauffer
segmentation
Shape prior model
Shape prior (distance transform)
Likelihood of being foreground
given all the terms
Resulting Graph-Cuts
segmentation
Can segment multiple views simultaneously
Solve via gradient descent
Comparable to level set methods
Could use other approaches (e.g. Objcut)
Need a graph cut per function evaluation
Formulating the Pose Inference Problem
But…But…
… to compute the MAP of E(x) w.r.t the pose, it means that the unary terms will be changed at EACHEACH iteration and the maxflow recomputed!
However…However… Kohli and Torr showed how dynamic graph cuts can
be used to efficiently find MAP solutions for MRFs that change minimally from one time instant to the next: Dynamic Graph Cuts (ICCV05).
Dynamic Graph Cuts
PB SB
cheaperoperation
computationally
expensive operation
Simplerproblem
PB*
differencesbetweenA and B
A and Bsimilar
PA SA
solve
Dynamic Image Segmentation
Image
Flows in n-edges Segmentation Obtained
First segmentation problem MAP solution
Ga
Our Algorithm
Gb
second segmentation problem
Maximum flow
residual graph (Gr)
G`
differencebetween
Ga and Gbupdated residual
graph
Energy Minimization using Graph cuts
EMRF(a1,a2)
Sink (1)
Source (0)
a1 a2
Graph Construction for Binary Random Variables
Energy Minimization using Graph cuts
Sink (1)
Source (0)
a1 a2
EMRF(a1,a2) = 2a1
2t-edges
(unary terms)
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1
Sink (1)
Source (0)
a1 a2
2
5
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2
Sink (1)
Source (0)
a1 a2
2
5
9
4
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2
Sink (1)
Source (0)
a1 a2
2
5
9
4
2
n-edges(pair-wise term)
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
Sink (1)
Source (0)
a1 a2
2
5
9
4
2
1
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
Sink (1)
Source (0)
a1 a2
2
5
9
4
2
1
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
Sink (1)
Source (0)
a1 a2
2
5
9
4
2
1
a1 = 1 a2 = 1
EMRF(1,1) = 11
Cost of st-cut = 11
Energy Minimization using Graph cuts
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
Sink (1)
Source (0)
a1 a2
2
5
9
4
2
1
a1 = 1 a2 = 0
EMRF(1,0) = 8
Cost of st-cut = 8
Energy Minimization using Graph cuts
• Most probable (MAP) configuration ≡ minimum cost st-cut.
• st-mincut is in general a NP-hard problem - negative edge weights
• Solvable in polynomial time- non-negative edge weights- corresponds to sub-modular (regular) energy functions
• The Max-flow Problem- Edge capacity and flow balance constraints
Computing the st-mincut from Max-flow algorithms
• Notation- Residual capacity (edge capacity – current flow)- Augmenting path
• Simple Augmenting Path based Algorithms- Repeatedly find augmenting paths and push flow.- Saturated edges constitute the st-mincut. [Ford-Fulkerson Theorem]
Sink (1)
Source (0)
a1 a2
2
5
9
42
1
9 + α
4 + α
Adding a constant to both thet-edges of a node does not change the edges constituting the st-mincut.
Key Observation
Sink (1)
Source (0)
a1 a2
2
5
2
1
E (a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
E*(a1,a2 ) = E(a1,a2) + α(a2+ā2)
= E(a1,a2) + α [a2+ā2 =1]
Reparametrization
9 + α
4
All reparametrizations of the graph are sums of these two types.
Other type of reparametrization
Sink (1)
Source (0)
a1 a2
2
5 + α
2 + α
1 - α
Reparametrization, second type
Both maintain the solution and add a constant α to the energy.
s
Gt
original graph
0/9
0/7
0/5
0/2 0/4
0/1
xi xj
flow/residual capacity
Graph Re-parameterization
s
Gt
original graph
0/9
0/7
0/5
0/2 0/4
0/1
xi xj
flow/residual capacity
Graph Re-parameterization
t residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0st-mincut
ComputeMaxflow
Gr
Edges cut
Update t-edge Capacities
s
Gr
t residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0
Update t-edge Capacities
s
Gr
t residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0
capacitychanges from
7 to 4
Update t-edge Capacities
s
G`t
updated residual graph
xi xj0/12
5/-1
3/2
1/0
2/0 4/0
capacitychanges from
7 to 4
edge capacityconstraint violated!(flow > capacity)
= 5 – 4 = 1
excess flow (e) = flow – new capacity
add e to both t-edges connected to node i
Update t-edge Capacities
s
G`t
updated residual graph
xi xj0/12
3/2
1/0
2/0 4/0
capacitychanges from
7 to 4
edge capacityconstraint violated!(flow > capacity)
= 5 – 4 = 1
excess flow (e) = flow – new capacity
5/-1
Update t-edge Capacities
s
G`t
updated residual graph
xi xj0/12
3/2
1/0
4/0
capacitychanges from
7 to 4
excess flow (e) = flow – new capacity
add e to both t-edges connected to node i
= 5 – 4 = 1
5/0
2/1
edge capacityconstraint violated!(flow > capacity)
Update n-edge Capacities
s
Gr
t
residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0
• Capacity changes from 5 to 2
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/12
5/2
3/-1
1/0
2/0 4/0
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/12
5/2
3/-1
1/0
2/0 4/0
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/11
5/2
2/0
1/0
2/0 4/0
excess
deficiency
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/11
5/2
2/0
1/0
2/0 4/0
deficiency
excess
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
• Push excess flow to/from the terminals
• Create capacity by adding α = excess to both t-edges.
Update n-edge Capacities
Updated residual graph
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
• Push excess flow to the terminals
• Create capacity by adding α = excess to both t-edges.
G`
xi xj0/11
5/3
2/0
2/0
3/0 4/1
s
t
Update n-edge Capacities
Updated residual graph
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
• Push excess flow to the terminals
• Create capacity by adding α = excess to both t-edges.
G`
xi xj0/11
5/3
2/0
2/0
3/0 4/1
s
t
Complexity analysis of MRF Update Operations
MRF Energy Operation
Graph Operation Complexity
modifying a unary term
modifying a pair-wise term
adding a latent variable
delete a latent variable
Updating a t-edge capacity
Updating a n-edge capacity
adding a node
set the capacities of all edges of a node zero
O(1)
O(1)
O(1)
O(k)*
*requires k edge update operations where k is degree of the node
• Finding augmenting paths is time consuming.
• Dual-tree maxflow algorithm [Boykov & Kolmogorov PAMI 2004]- Reuses search trees after each augmentation.- Empirically shown to be substantially faster.
• Our Idea • Reuse search trees from previous graph cut computation• Saves us search tree creation tree time [O(#edges)]• Search trees have to be modified to make them consistent with
new graphs• Constrain the search of augmenting paths
– New paths must contain at least one updated edge
Improving the Algorithm
Reusing Search Trees
c’ = measure of change in the energy
• Running time– Dynamic algorithm (c’ + re-create search tree )– Improved dynamic algorithm (c’)– Video Segmentation Example
- Duplicate image frames (No time is needed)
Dynamic Graph Cut vs Active Cuts
Our method flow recycling
AC cut recycling
Both methods: Tree recycling
• Compared results with the best static algorithm.- Dual-tree algorithm [Boykov & Kolmogorov PAMI 2004]
• Applications- Interactive Image Segmentation- Image Segmentation in Videos
Experimental Analysis
Experimental Analysis
additional segmentation
cues
user segmentation cuesstatic: 175 msecdynamic : 80 msecdynamic (optimized): 15 msec
static : 175 msec
Interactive Image segmentation (update unary terms)
EnergyMRF =
Experimental Analysis
Image resolution: 720x576 static: 220 msec dynamic (optimized): 50 msec
Image segmentation in videos (unary & pairwise terms)
Graph CutsDynamic Graph Cuts
EnergyMRF =
Experimental Analysis
Image resolution: 720x576 static: 177 msec dynamic (optimized): 60 msec
Image segmentation in videos (unary & pairwise terms)
Graph CutsDynamic Graph Cuts
EnergyMRF =
Experimental Analysis
MRF consisting of 2x105 latent variables connected in a 4-neighborhood.
Running time of the dynamic algorithm
Other uses
Can be used to compute uncertainty in graph cuts via max marginals.
Max marginals can be used for parameter learning in MRF’s.
Inference in Graphical Models
Graphical Model
Topology
Tree Graph with cycles
Belief Propagation and variants
Exact solution
True Marginals/ min-marginals
Approximate solution
Approximate Marginals/ min-marginals
Graph Cuts
No Marginals/
Min-Marginals
Class 1: Max-flow Computation, Exact
Class 2: Alpha expansions, Approximate Solution with bounds
Class 3: Local Minima (with no bounds)
Inference in Graphical Models
Min-Marginals Energies(ψ)- Minimize joint energy over all other variables.
- Related to max-marginals as:
- Can be used to compute confidence as:
σj = µj / Σa µa = exp(-ψi) / Σa exp(-ψa)
µj = (1/z)*exp(-ψj)
Energy Projections and Graph Construction
EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 + Kā2
a1
a2
2
5
9
4
2
1
Sink (0)
Source (1)
∞
A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection.
Alternative Construction
K
Segmentation Comparison
Gri
mson
-G
rim
son
-S
tau
ffer
Sta
uff
er
Bath
ia0
Bath
ia0
44O
ur
Ou
r m
eth
od
meth
od
Face Detector and ObjCut
Segmentation
Segmentation
Conclusion
Combining pose inference and segmentation worth investigating.
Tracking = Detection Detection = Segmentation Tracking = Segmentation. Segmentation = SFM ??