Upload
arline-stafford
View
220
Download
3
Tags:
Embed Size (px)
Citation preview
Stephen J. Guy
1
PhotomontagePhotomontage PhotomontagePhotomontage
GrabCut – Interactive Foreground Extraction 1
Background Segmentation Intro Graph Cut for Segmentation (Boykov & Jolly
‘01) Copying
Lazy Snapping (Li et al. ’04) GrabCut (Rother et al. ’04)
Other Texture & Video Synthesis (Kwatra et al. ’04)
3
Goal Splitting Image into similar, meaningful(?)
regions E.g. Clustering, Edge Detection, *Graph Cuts*
X. Ren and J. Malik. Learning a classification model for segmentation.ICCV 2003.
“SuperPixels” GrabCutRother et al. 2004Rother et al. 2004
4
A – Proposed Segmentation
E(A) – Overall Energy
R(A) – Degree to which pixels fits model
B(A) – Degree to which the cuts breaks up similar pixels
- Balance A() and B()
• Goal: Find Segmentation, A, which minimizes E(A) 5
Image As Graph Each pixel is a node Create links between each pixel More nodes/links possible (e.g. foreground node)
Graph Cuts Assign energy for each link based on similarity Break the links with least energy, but Require each pixel is still connected to exactly one
foreground or background node Benefits
No Topological constraints on regions (can be concave, unconnected, etc.)
Fast (polynomial time) solution – Ford-Fulkerson Algorithm
6
7
Pixel links based on color/intensity similarities
Source/Target links based on histogram models of fore/background
8
Effect of Recall:
9
10
Main Steps:1. Mark strokes as foreground & background2. Perform Boykov & Jolly style graphcut3. *Edit Boundary*
Familiar Formulation Minimize Gibbs Energy:
E1(X) – Cluster (K-means) known foreground/background colors to find representative colors
E2(X) – Similar to Boykov & Jolly 11
Too Slow to allow for real-time refinement Solution?
Superpixels!
12
13
Graph Cut is only step 1 Allow user to edit boundary directly Formulate as Graph Cut Allow user to guide boundary regions w/
brush
14
15
Compared Lazy Snapping to Photoshop
16
Results Easy of Use – Lazy Snapping 20% less mistakes Speed – Lazy Snapping 60% less time Accuracy – Lazy Snapping 60% less pixels
wrong Feedback
“Almost Magic” “Much Easier”
Suggestions Get rid of 2 step process Make it easy to switch between graph cut
strokes and boundary editing17
18
GrabCutGrabCut Interactive Foreground Extraction Interactive Foreground Extraction
using Iterated Graph Cutsusing Iterated Graph Cuts
Carsten RotherCarsten RotherVladimir Kolmogorov Vladimir Kolmogorov
Andrew BlakeAndrew Blake
Microsoft Research Cambridge-UKMicrosoft Research Cambridge-UK
GrabCutGrabCut Interactive Foreground Extraction Interactive Foreground Extraction
using Iterated Graph Cutsusing Iterated Graph Cuts
Carsten RotherCarsten RotherVladimir Kolmogorov Vladimir Kolmogorov
Andrew BlakeAndrew Blake
Microsoft Research Cambridge-UKMicrosoft Research Cambridge-UK
New model of foreground & background pixels:GMM vs. K-mean colors vs.
Intensity Histogram More automation:
No need to specify foregroundIterative Graph Cuts
20
What GrabCut doesWhat GrabCut does What GrabCut doesWhat GrabCut does
User Input
Result
Magic Wand (198?)
Intelligent ScissorsMortensen and Barrett (1995)
GrabCut
Regions Boundary Regions & Boundary
GrabCut – Interactive Foreground Extraction 3
1. Assign Selection box or Lasso as background
2. Assign other pixels as unknown3. Learn Model of Fore/Background4. Assign Link Energy, GraphCut (Boykov &
Jolly)5. Use GraphCut results to assign
Fore/Background6. Goto Step 37. Allow user to add cleanup strokes
23
Graph CutsGraph Cuts Boykov and Jolly (2001)Boykov and Jolly (2001)
Graph CutsGraph Cuts Boykov and Jolly (2001)Boykov and Jolly (2001)
GrabCut – Interactive Foreground Extraction 5
ImageImage Min CutMin Cut
Cut: separating source and sink; Energy: collection of edges
Min Cut: Global minimal enegry in polynomial time
Foreground Foreground (source)(source)
BackgroundBackground(sink)(sink)
Iterated Graph CutIterated Graph Cut Iterated Graph CutIterated Graph Cut
User Initialisation
K-means for learning
colour distributions
Graph cuts to infer the
segmentation
?
GrabCut – Interactive Foreground Extraction 6
1 2 3 4
Iterated Graph CutsIterated Graph Cuts Iterated Graph CutsIterated Graph Cuts
GrabCut – Interactive Foreground Extraction 7
Energy after each IterationResult
Guaranteed to
converge
Colour ModelColour Model Colour ModelColour Model
Gaussian Mixture Model (typically 5-8 components)
Foreground &Background
Background
Foreground
BackgroundG
GrabCut – Interactive Foreground Extraction 8
R
G
RIterated graph cut
Moderately straightforward Moderately straightforward
examples examples
Moderately straightforward Moderately straightforward
examples examples
… GrabCut completes automatically
GrabCut – Interactive Foreground Extraction 10
Difficult ExamplesDifficult Examples Difficult ExamplesDifficult Examples
Camouflage &
Low ContrastNo telepathyFine structure
Initial Rectangle
InitialResult
GrabCut – Interactive Foreground Extraction 11
Comparison Comparison Comparison Comparison
GrabCutBoykov and Jolly (2001)
Error Rate: 0.72%Error Rate: 1.87%Error Rate: 1.81%Error Rate: 1.32%Error Rate: 1.25%Error Rate: 0.72%
GrabCut – Interactive Foreground Extraction 13
User Input
Result
Border Matting Border Matting Border Matting Border Matting
Hard Segmentation Automatic Trimap Soft Segmentation
GrabCut – Interactive Foreground Extraction 16
Solve
Mean Colour Foreground
Mean ColourBackground
GrabCut – Interactive Foreground Extraction 17
Natural Image MattingNatural Image MattingNatural Image MattingNatural Image Matting
Ruzon and Tomasi (2000): Alpha estimation in natural images
Noisy alpha-profile
Border MattingBorder MattingBorder MattingBorder Matting
GrabCut – Interactive Foreground Extraction 19
1
0
Foreground
Mix
Back-ground
Foreground Mix Background
Fit a smooth alpha-profile with parameters
ResultsResultsResultsResults
GrabCut – Interactive Foreground Extraction 21
SummarySummary SummarySummary
Magic Wand (198?)
Intelligent Scissors Mortensen and Barrett (1995)
GrabCutRother et al. (2004)
Graph Cuts Boykov and Jolly (2001)
LazySnappingLi et al. (2004)
GrabCut – Interactive Foreground Extraction 22
Boykov&Jolly Lazy Snapping GrabCut
Foreground & Background Model
Histogram of Intensities
K-means clustering, 64 representative colors
K-mean
Background Source
User Strokes User Strokes Initial Box (Lasso) + User Strokes
Foreground Source
User Strokes User Strokes None or User Strokes
Boundary Term
Similarity Penalty
Similarity Penalty
Similarity Penalty
Segmentation GraphCut Superpixels + GraphCut
Iterative GraphCut
Refinement User Strokes Boundary Editing
User Strokes
36
Texture Synthesis Revisited
37
Copy overlapping patches one at a times Find the seamwhich minimize change in
neighboring pixels Formulate as Graph Cut
38
Extends well into 3D
39