Graph-based Graph-based SegmentationSegmentation

Main IdeasMain Ideas

Convert image into a graphConvert image into a graph Vertices for the pixelsVertices for the pixels Edges between the pixelsEdges between the pixels Additional vertices and edges to encode other Additional vertices and edges to encode other

constraintsconstraints

Manipulate the graph to segment the Manipulate the graph to segment the imageimage

PapersPapers

Interactive graph cuts for optimal boundary Interactive graph cuts for optimal boundary & region segmentation of& region segmentation ofobjects in N-D images objects in N-D images Boykov and JollyBoykov and Jolly Minimize an energy functionMinimize an energy function

Efficient Graph-based SegmentationEfficient Graph-based Segmentation Felzenszwalb and HuttenlocherFelzenszwalb and Huttenlocher Cluster the vertices based on edge weightCluster the vertices based on edge weight

Boykov and JollyBoykov and Jolly

Binary image segmentationBinary image segmentation Classify pixels as object or backgroundClassify pixels as object or background Their contribution is adding interactivityTheir contribution is adding interactivity

Minimise an energy functionMinimise an energy function E(A) = B(A) + E(A) = B(A) + λλR(A)R(A)

A: Segmentation (assign each pixels to object A: Segmentation (assign each pixels to object or background)or background)B(A): The cost of all the edges between object B(A): The cost of all the edges between object pixels and background pixelspixels and background pixelsR(A): The cost of deciding a pixel to be object or R(A): The cost of deciding a pixel to be object or backgroundbackground

Creating the GraphCreating the Graph

Each pixel has a corresponding vertexEach pixel has a corresponding vertex

Additionally, a source (“object”) and a sink Additionally, a source (“object”) and a sink (“background”)(“background”)

Each pixel vertex has an edge to its Each pixel vertex has an edge to its neighbours (e.g. 8 adjacent neighbours in neighbours (e.g. 8 adjacent neighbours in 2D), an edge to the source, an edge to 2D), an edge to the source, an edge to the sinkthe sink

Edge Weights between pixelsEdge Weights between pixels

Weight of edges between pixel vertices Weight of edges between pixel vertices are determined by the B() functionare determined by the B() function

Low score when boundary is likely to pass Low score when boundary is likely to pass between the verticesbetween the vertices

high score when vertices are probably part high score when vertices are probably part of the same elementof the same element

E.g. the difference in pixel intensities, the E.g. the difference in pixel intensities, the gradientgradient

Edges to Source/SinkEdges to Source/Sink

If pixel is known to be an object, use a high If pixel is known to be an object, use a high weight (K) to the source, zero weight to the sinkweight (K) to the source, zero weight to the sink K is chosen so that it will never be cutK is chosen so that it will never be cut

Conversely, if pixel is background, use weight K Conversely, if pixel is background, use weight K to the sink, zero weight to the sourceto the sink, zero weight to the source

Otherwise, weigh edges to source and sink Otherwise, weigh edges to source and sink appropriately using the R() functionappropriately using the R() function Note that the edge to the source is the “likelihood” for Note that the edge to the source is the “likelihood” for

the pixel being the the pixel being the backgroundbackground – – we break this edge we break this edge when the pixel is assigned to the backgroundwhen the pixel is assigned to the background

ApplicationsApplications

Handles arbitrary number of dimensionsHandles arbitrary number of dimensions

Finds global minimum energyFinds global minimum energy

Needs “good” user input to work effectivelyNeeds “good” user input to work effectively

Need intelligent functionsNeed intelligent functions

Need to select Need to select λλ

Felzenszwalb and HuttenlocherFelzenszwalb and Huttenlocher

Download the program from the webpage:Download the program from the webpage: http://people.cs.uchicago.edu/~pff/segment/http://people.cs.uchicago.edu/~pff/segment/

Minimal documentationMinimal documentation Short README fileShort README file PaperPaper

The programThe program

Comes as a tar.gz archive and .zip archiveComes as a tar.gz archive and .zip archive

ProcessProcess Extract archiveExtract archive ‘‘make’ (Makefile supplied)make’ (Makefile supplied)

Program consists ofProgram consists of A .cpp “wrapper” file (only calls the functions)A .cpp “wrapper” file (only calls the functions) Actual algorithm functions are in .h filesActual algorithm functions are in .h files Basic portable C++ codeBasic portable C++ code

Program TestingProgram Testing

Built on Mac OS X, Linux, Windows cygwinBuilt on Mac OS X, Linux, Windows cygwin Gcc toolchain, but any C++ compiler should workGcc toolchain, but any C++ compiler should work Supplied basic MakefileSupplied basic Makefile

Results were basically the same between Results were basically the same between platformsplatforms Colors are chosen randomlyColors are chosen randomly

Results obtained are not the same as posted on Results obtained are not the same as posted on the websitethe website Image files on the website may be modified Image files on the website may be modified

(scaled/compressed/downsampled)(scaled/compressed/downsampled)

AlgorithmAlgorithm

Create a graphCreate a graph Each vertex corresponds to a vertexEach vertex corresponds to a vertex

Edges are between “neighbouring” verticesEdges are between “neighbouring” vertices Choose a small neighbourhood to reduce Choose a small neighbourhood to reduce

computation time (otherwise we have a complete computation time (otherwise we have a complete graph)graph)

Weight on the edge is the 5D distance between the Weight on the edge is the 5D distance between the points (for a 2D image)points (for a 2D image)

5D vector = x position, y position and 3 color components5D vector = x position, y position and 3 color components

ParametersParameters

σσ: Use this value and do Gaussian : Use this value and do Gaussian smoothing (preprocessing the image to smoothing (preprocessing the image to reduce noise)reduce noise)

k: threshold value for doing the clusteringk: threshold value for doing the clustering

min: “hack” parametermin: “hack” parameter the smallest cluster size must contain at least the smallest cluster size must contain at least

this many vertices – clusters that are too this many vertices – clusters that are too small will be merged with other clusters until small will be merged with other clusters until sufficiently largesufficiently large

ClusteringClustering

Put each vertex in a component Put each vertex in a component

Sort edges by weightSort edges by weight

Take each edge in turnTake each edge in turn If the edge is between vertices in two different If the edge is between vertices in two different

components A and B, we can merge if the edge components A and B, we can merge if the edge weight is lower enough than the threshold weight is lower enough than the threshold

Threshold is the minimum of the following value, Threshold is the minimum of the following value, computed on A and Bcomputed on A and B

(Lowest weight edge in minimum spanning tree of the (Lowest weight edge in minimum spanning tree of the component) + (k / size of component)component) + (k / size of component)

NotesNotes

Low Low edge weights between vertices that edge weights between vertices that are likely to be in the same clusterare likely to be in the same clusterAs a cluster gets larger, it becomes As a cluster gets larger, it becomes harder to add vertices to itharder to add vertices to itHeuristic – not really minimising a Heuristic – not really minimising a particular energy functionparticular energy function More similar to “region growing”More similar to “region growing”

User has select “good” parameters to get User has select “good” parameters to get good resultsgood results

Effect of Effect of σσ

Increased smoothing results in removal of Increased smoothing results in removal of noisenoise

Can cause “bleeding” – the algorithm has Can cause “bleeding” – the algorithm has difficulty separating background from the difficulty separating background from the object if the boundaries are too smoothobject if the boundaries are too smooth

Reference ImagesReference Images

Increasing Increasing σσ

Clouds are Clouds are recognised as recognised as one objectone object

Palm tree Palm tree gets confused gets confused with oceanwith ocean

GrainGrain

Increasing Increasing σσ introduces more blurring introduces more blurring (reduces the edge weight between pixels)(reduces the edge weight between pixels)

Vertebrae MRIVertebrae MRI

Gets rid of noise (bottom left, right hand side), Gets rid of noise (bottom left, right hand side), but purple vertebrate piece bleeds outbut purple vertebrate piece bleeds out

Increasing ThresholdIncreasing Threshold

Clusters Clusters more more aggresivelyaggresively

Palm tree is Palm tree is confused confused with ocean with ocean and cloudsand clouds

GrainGrain

Non-grain pixels are almost all clustered Non-grain pixels are almost all clustered togethertogetherMeasure of how “similar” all the pixels of an Measure of how “similar” all the pixels of an object should beobject should be

MRIMRI

Vertebrae and region next to vertebrae are Vertebrae and region next to vertebrae are very similar shade so are easily confusedvery similar shade so are easily confused

Increasing ‘min’ valueIncreasing ‘min’ value

Segmentation Segmentation is the same, is the same, but small but small components components are merged are merged with with neighboring neighboring onesones

GrainGrain

Easy to control change, gets rid of small Easy to control change, gets rid of small artifactsartifacts

MRIMRI

Not Not much much effect if effect if regions regions are are already already largelarge

Parameter TweakingParameter Tweaking

Need to manually Need to manually tune parameters to tune parameters to get a good imageget a good image

Image of MRI after Image of MRI after selectingselecting σσ = 0.6 = 0.6 k = 200k = 200 min = 60min = 60

PerformancePerformance

Need to tune parameters by handNeed to tune parameters by handVery fastVery fast Program usually takes a couple seconds on Program usually takes a couple seconds on

the test images providedthe test images provided

Only takes ppm images in RAW data Only takes ppm images in RAW data formatformatTheoretically, algorithm generalises to Theoretically, algorithm generalises to arbitrary number of dimensions and arbitrary number of dimensions and arbitrary number of features per pixelarbitrary number of features per pixel

Command-line toolsCommand-line toolsWorking with images involves opening up the results Working with images involves opening up the results in an image viewer – can get messyin an image viewer – can get messy