Qualifying Exam:Qualifying Exam:

Contour GroupingContour GroupingVida MovahediVida Movahedi

Supervisor: James ElderSupervisor: James Elder

Supervisory Committee:Supervisory Committee:

Minas Spetsakis, Jeff EdmondsMinas Spetsakis, Jeff Edmonds

York UniversityYork University

Summer 2009Summer 2009

ContentsContents• Introduction

• Preliminary Concepts– Pre-processing

– Gestalt cues

• Methods– Local & Heuristic

– Local & Probabilistic

– Global Saliency

• Evaluation

• Conclusion & open problems

ContentsContents• Introduction

• Preliminary Concepts– Pre-processing

– Gestalt cues

• Methods– Local & Heuristic

– Local & Probabilistic

– Global Saliency

• Evaluation

• Conclusion & open problems

IntroductionIntroduction

• SegmentationPartition an image into regions, each corresponding to

an object or entity

• Figure-Ground segmentation

Segmentation MethodsSegmentation Methods• Regional Segmentation

– Use regional info, optimize labelling of regional tokens, e.g. clustering

– Depending on uniformity in object region

• Active Contour Models– Use regional (external) & boundary (internal) info,

optimize deformation of model

– Sensitivity to initialization, too smooth

• Contour Grouping– Use boundary info (& regional info), optimize grouping of

contour fragments

Problem DefinitionProblem Definition• Input: Color image

• Goal: Figure-ground segmentation

• Method: Contour Grouping

• Other available info: None

- No motion, stereo or video information

- No user interactions

- No assumptions on object types, shapes, color, etc.

- No assumptions on background or lighting conditions

ChallengesChallenges• High-dimensional data space, lots of information,

many cues

• Unknown cue integration

• Global optimization in a non-convex multidimensional space

• Camera, imaging, quantization noise

• Clutter in natural scenes

• Occluded or overlapping objects

ContentsContents• Introduction

• Preliminary Concepts– Pre-processing

– Gestalt cues

• Methods– Local & Heuristic

– Local & Probabilistic

– Global Saliency

• Evaluation

• Conclusion & open problems

StepsStepsImage

Grouping Algorithm

Saliency

Computations

Optimization

Algorithm

Figure/Ground Segmentation

Pre-processing

Edge

Detection

Line /Curve

Approximation

Learned Parameters or Distributions

Pre-processingPre-processing

Image Edge Map Line Map Contour

Gestalt CuesGestalt Cues

How is grouping done in human vision?

• Proximity

• Similarity– Brightness– Contrast

• Good continuation – Parallelism– Co-circularity

ContentsContents• Introduction

• Preliminary Concepts– Pre-processing

– Gestalt cues

• Methods– Local & Heuristic

– Local & Probabilistic

– Global Saliency

• Evaluation

• Conclusion & open problems

Grouping MethodsGrouping Methods• Local Heuristic methods

– Defining a heuristic cost for contour hypotheses, find the optimal one

• Local Probabilistic methods– Find posterior probability of contour

hypotheses given cues, find the optimal one

• Global methods– An extra step of calculating global saliencies

based on local measures

ContentsContents• Introduction

• Preliminary Concepts– Pre-processing

– Gestalt cues

• Methods– Local & Heuristic

– Local & Probabilistic

– Global Saliency

• Evaluation

• Conclusion & open problems

Local & HeuristicLocal & HeuristicExample: Ratio Contour Method Example: Ratio Contour Method

(Wang et. al, PAMI’05)(Wang et. al, PAMI’05)

• Detected/ virtual fragments

• Contour cost= curvature & gap per unit length

• Graph model

• Alternate cycle

Local & HeuristicLocal & HeuristicExample: Ratio Contour Method Example: Ratio Contour Method

(Wang et. al, PAMI’05)(Wang et. al, PAMI’05)

• Edge/ Link costs

• Ratio Contour Algorithm

)(

2 )]()([)(eB

dtttew

)(

)(eB

dtel

Ce

Ce

el

ewC

)(

)()(

fragment real aon is )( if0

fragment virtualaon is )( f1)(

tv

tvit

Sample Results for RC methodSample Results for RC method

Image RC RRC

RC

Image

(from Stahl & Wang, TIP’07)

(from Wang et al., PAMI’05)

ContentsContents• Introduction

• Preliminary Concepts– Pre-processing

– Gestalt cues

• Methods– Local & Heuristic

– Local & Probabilistic

– Global Saliency

• Evaluation

• Conclusion & open problems

Local & ProbabilisticLocal & Probabilistic(Elder et al., PAMI’03)(Elder et al., PAMI’03)

• Bayesian Rule:

• Contour saliency= posterior probability of contour

• Assumptions:– Markov Chain Assumption– Independence of evidence from cues– Comparing contours of same length

11

1

)(

)().|()|(

LPDp

HpHDpDHp

)(

)(,

)|(

)|(

Hp

HpP

HDp

HDpL

)|(maxarg* DCcpcc

}...1{},...1{,),...,(11

niNCttc in

ctt

cij

ct

oi

jii

ppDCcp

),(

)|(

Local & Probabilistic Local & Probabilistic (Elder et al., PAMI’03)(Elder et al., PAMI’03)

• Graph Model

• Node weight

• Link weight

• Shortest path/cycle

• Approximate search

Nipvw oii

o ..1 ),log()( iv

ije jv

Njipew cijij

c ..1, ),log()(

cjicijii Pvvij

c

Pvi

o

ctt

cij

ct

oi ewvwppDCcp

),(),(

)()()log()log()|(log

Sample Results for Probabilistic MethodsSample Results for Probabilistic Methods

(from Estrada & Elder- CVPRW’06)

ContentsContents• Introduction

• Preliminary Concepts– Pre-processing

– Gestalt cues

• Methods– Local & Heuristic

– Local & Probabilistic

– Global Saliency

• Evaluation

• Conclusion & open problems

Global ModelGlobal Model

Local weights Global weights

Global SaliencyGlobal Saliency• Edge/Link Affinity

Based on collinearity, proximity, etc.

• Edge/ Link Saliency

Relative number of closed random walks which visit that edge/link (Mahamud et al., PAMI’03)

• Shown to be relevant to the eigenvalues and eigenvectors of the affinity matrix

• Grouping based on global saliency

Some Results of the Untangling methodSome Results of the Untangling method

(from Zhu; Song; Shi- ICCV’07)

ContentsContents• Introduction

• Preliminary Concepts– Pre-processing

– Gestalt cues

• Methods– Local & Heuristic

– Local & Probabilistic

– Global Saliency

• Evaluation

• Conclusion & open problems

EvaluationEvaluation

• Empirical discrepancy methods

The output of algorithms is compared with a reference segmentation or ground truth

• Requirements– A ground truth dataset– An error measure

SOD: Salient Object DatasetSOD: Salient Object Dataset

• Based on Berkeley Segmentation Dataset (BSD)

• 300 images, randomly showing 818 segmentations (half of BSD) to each of 7 subjects

• 12,110 object boundaries obtained

1

1

1

1

1

Region-based Error MeasuresRegion-based Error Measures• Example

• Not sensitive to some large shape features (e.g., spikes, wiggles)

BA

BAB

BA

BAA

BA

BA

RR

RRR

RR

RRRRR

RRBARIM

||||

1),(

Boundary-based Error Boundary-based Error MeasuresMeasures

• Not sensitive to object topology and some large shape features (e.g., loop-backs, wiggles)

),(minmax)),(max(),( badBASDBAhBbAa

B

}),({),( AaadBASD BB

),(),,(max),( ABhBAhBAH

AabadadBb

B

,),(min)(

Mixed Error MeasuresMixed Error Measures

fpfn N

kkB

fp

N

jjA

fn

qdN

pdN

BAMM11

)(1

)(1

2

1),(

• Example

• Not sensitive to some large shape features. Does not respect ordering along contours.

pj, j=1..Nfn are pixels in the false negative region (RB-RA)

qk, k=1..Nfp are pixels in the false positive region (RA-RB)

Contour Mapping MeasureContour Mapping Measure

• Based upon a matching between all points on the two boundaries

• Monotonically non-decreasing

• Allowing one-to-one, many-to-one, and one-to-many matching

• Error= average distance between matched pairs

• Dynamic Programming

Contour Mapping Distance=7.73

ContentsContents• Introduction

• Preliminary Concepts– Pre-processing

– Gestalt cues

• Methods– Local & Heuristic

– Local & Probabilistic

– Global Saliency

• Evaluation

• Conclusion & open problems

Conclusion & Open ProblemsConclusion & Open Problems• Cue selection and combination

• Grouping Model– Global saliency

– Probabilistic models

• Optimization Algorithms

• Hierarchical and multi-scale algorithms

• Quantitative evaluation