Segmentation Lucia Ballerini Digital Image Processing Lecture 8 Course book reading: GW 10

SegmentationSegmentation

Lucia Ballerini

Digital Image Processing

Lecture 8

Course book reading: GW 10Course book reading: GW 10

ImageImage

"What are the objects to be analyzed?"

Pre-processing, image enhancement

Segmentation

Binary operations

Morphological operations and feature extraction

Classification and matching

Image analysisDataData

SegmentationSegmentation

► Full segmentation: Individual objects are separated from the background and given individual ID numbers (labels).

► Partial segmentation: The amount of data is reduced (usually by separating objects from background) to speed up the further processing.

► Segmentation is often the most difficult problem to solve in the process; there is no universal solution!

► The problem can be made much easier if solved in cooperation with the constructor of the imaging system (choice of sensors, illumination, background etc) .

Three types of segmentationThree types of segmentation

► Classification – Based on some similarity measure between pixel values. The simplest form is thresholding.

► Edge-based – Search for edges in the image. They are then used as borders between regions

► Region-based – Region growing, merge & split

Common idea: search for discontinuities or/and similitudes in the image

ThresholdingThresholdingglobalglobal or or locallocal

► global: based on some kind of histogram: grey-level, edge, feature etc. Lighting conditions are extremely important, and it will only

work under very controlled circumstances.

► Fixed thresholds: the same value is used in the whole image

► local (or dynamic thresholding): depends on the position in the image. The image is divided into overlapping sections which are thresholded one by one.

Classical automatic Classical automatic thresholding algorithmthresholding algorithm

1.1. Select an initial estimate for TSelect an initial estimate for T

2.2. Segment the image using T. This produces 2 groups: Segment the image using T. This produces 2 groups: GG11 , pixels with value >T and G , pixels with value >T and G2 2 , with value <T, with value <T

3.3. Compute µCompute µ11 and µ and µ22, average pixel value of G, average pixel value of G11 and G and G22

4.4. New threshold: T=1/2(µNew threshold: T=1/2(µ11+µ+µ22))

5.5. Repeat steps 2 to 4 until T stabilizes.Repeat steps 2 to 4 until T stabilizes.

► Very easy + very fastVery easy + very fast► Assumptions: normal dist. + low noiseAssumptions: normal dist. + low noise

Optimal ThresholdingOptimal Thresholding

► based on the shape of the current image histogram. Search for valleys, Gaussian distributions etc.

Background

Real histogram

Optimalthreshold ?

Both

Foreground

HistogramsHistograms

To love… …and to hate

Thresholding and illuminationThresholding and illumination

► Solutions:Solutions: Calibration of the Calibration of the

imaging systemimaging system percentile filter with percentile filter with

very large maskvery large mask Morphological Morphological

operatorsoperators

MR non-uniformityMR non-uniformity

median filtering thresholding

-

More thresholdingMore thresholding► Can also be used on other kinds of histogram: grey-

level, edge, feature etc. Multivariate data ( see next lectures)

► Problems: Only considers the graylevel pixel value, so it can leave “holes”

in segmented objects.► Solution: post-processing with morphological operators

Requires strong assumptions to be efficient Local thresholding is better see region growing techniques

Edge-based SegmentationEdge-based Segmentation

Based on finding discontinuities (local variations of image intensity)

1. Apply an edge detector ex gradient operator (Sobel)

second derivative (Laplace)

2. Threshold the edge image to get a binary image

3. Depending on the type of edge detector: Link edges together to close shapes (using edge direction for ex) Remove spurrious edges

Gradient based procedureGradient based procedure

Sobel

Sobel

Zero-crossing based procedureZero-crossing based procedure

LoG

Laplacian of GaussianLaplacian of Gaussian

Edge-based Segmentation: examplesEdge-based Segmentation: examples

Prewitt: needs edge linking Canny: needs “cleaning”

Region based segmentationRegion based segmentation

►Work by extending some region based on local similarities between pixels region growing (bottom-up method) region splitting and merging (top-down

method)

Bottom-up: from data to representation Top-down: from model to data

Region growing Region growing (bottom-up method)(bottom-up method)

1. Find starting points

2. Include neighbouring pixels with similar features (grey-level, texture, color).

3. Continue untill all pixels have been included with one of the starting points.

► Problems: Not trivial to find good strating points, difficult to

automate Need good criteria for similarity.

WatershedWatershed(a kind of region growing)(a kind of region growing)

► Think of the grey-level image as a landscape. Let water rise from the bottom of each valley (the water from each valley is given its own label). As soon as the water from two valleys meet, build a dam, or watershed. These watersheds will then define the borders between different regions.

Example of watershed directly on a gray-level image

Example of Watershed on a binary image

Watershed: problems and solutionsWatershed: problems and solutions

► Oversegmentation Oversegmentation watershed from markerswatershed from markers

► ComputationComputation new algorithm for fast watershednew algorithm for fast watershed

► Graylevel might not be the optimal choice as the Graylevel might not be the optimal choice as the local similarity measurelocal similarity measure bigger neighborhood when growingbigger neighborhood when growing other local features (statistical, other local features (statistical, edge enhanced

image, distance transformed image…)

Split & MergeSplit & Merge(top-down metod)(top-down metod)

1. Set up som criteria for what is a uniform area (ex mean, variance, bimodality of histogram, texture, etc…)

2. Start with the full image and split it in to 4 sub-images.

3. Check each sub-image. If not uniform, divide into 4 new sub-images.

4. After each iteration, compare touching regions with neighboring regions end merge if uniform.

The method is also called "quadtree" division(and is also used for compression)

Split & MergeSplit & Merge

The Hough transformThe Hough transform

► A method for finding global relationships between pixels.Example: We want to find straight lines in an image 1. Apply edge enhancing filter (ex Laplace) 2. Set a threshold for what filter response is considered a true

”edge pixel” 3. Extract the pixels that are on a straight line using the Hough

transform

original image edge enhanced image

thresholded edge image

The Hough transformThe Hough transformFinding straight lines:

► 1. consider a pixel in position (xi,yi)► 2. equation of a straight line yi=axi+b ► 3. set b=-axi+ yi and draw this (single) line in ”ab-space”► 4. consider the next pixel with position (xj,yj) and draw

the line b=-axj+ yj ”ab-space” (also called parameter space). The poins (a’,b’) where the two lines intersect represent the line y=a’x+b’ in ”xy-space” which will go through both (xi,yi) and (xj,yj).

► 5. draw the line in ab-space corresponding to each pixel in xy-space.

► 6. divide ab-space into accumulator cells and find most common (a’, b’) which will give the line connecting the largest number of pixels

y

The Hough transformThe Hough transform

xy-space

x

ab- or parameter space

b

a

The Hough transformThe Hough transform► In reality we have a problem with y=ax+b because a

reaches infinity for vertical lines.Use instead.

► It is common to use ”filters” for finding the intersection: ”butterfly filters”

► Different variations of the Hough transform can also be used for finding other shapes of the form g(v,c)=0, v is a vector of coordinates, c is a vector of coefficients.

► Possible to find any kind of simple shapeex. circle: (3D parameter space)

sincos yx

( ) ( )x c y c c 12

22

32

The Hough transformThe Hough transform

ConclusionsConclusions

► The segmentation procedureThe segmentation procedure1.1. Pre-processingPre-processing

2.2. SegmentationSegmentation

3.3. Post-processingPost-processing

Like any IP procedureLike any IP procedure

► There exists NO universal segmentation There exists NO universal segmentation methodmethod

► Evaluation of segmentation performance is Evaluation of segmentation performance is importantimportant

SnakesSnakes►Example: segmentation of the brain in MRIExample: segmentation of the brain in MRI

Snake after initialization Snake at equilibrium

User interaction

Snakes (active contours)Snakes (active contours)

► A A snake snake is an active contour parametrically is an active contour parametrically represented by its position v(s)=(x(s), y(s))represented by its position v(s)=(x(s), y(s))

► Each position is associated to an energy:Each position is associated to an energy:

► The final position corresponds to the minimum The final position corresponds to the minimum of the energyof the energy

E E s ds E s dssnake int ext 0

1

0

1

[ ( )] [ ( )]v v

Internal EnergyInternal Energy

The internal energy of the snake is due to The internal energy of the snake is due to bending and it is associated with bending and it is associated with a prioria priori constraints:constraints:

► (s) controls the tension of the contour(s) controls the tension of the contour► (s) controls its rigidity(s) controls its rigidity

E s ss

ss

s

sint[ ( )] ( )

( )( )

( )v

v v

1

2

2 2

2

2

External EnergyExternal Energy

► The external energy depends on the image and The external energy depends on the image and accounts for accounts for a posterioria posteriori information information

► Several energy forms have been proposed Several energy forms have been proposed based on features of interest in the image based on features of interest in the image

► An energy commonly used to attract snakes An energy commonly used to attract snakes towards edges is:towards edges is:

E s G I x yext[ ( )] ( , )v 2

ApplicationsApplications

ApplicationsApplications

(by Terzopoulos)

ConsiderationsConsiderations

► The number of nodes is an important factor for the The number of nodes is an important factor for the behavior of the snake. Ability to resample the contour may behavior of the snake. Ability to resample the contour may be necessary.be necessary.

► If we want a closed contour, we set the first and the last If we want a closed contour, we set the first and the last point equal.point equal.

► Anchor points are necessary to keep the snake in position if Anchor points are necessary to keep the snake in position if the image forces are not enough.the image forces are not enough.

► It may be necessary to allow a snake contour to divide into It may be necessary to allow a snake contour to divide into two contours, or two contours to merge into one contour.two contours, or two contours to merge into one contour.

► Different applications may need different potential Different applications may need different potential functions and different settings of the control parameters functions and different settings of the control parameters (damping, tension and rigidity).(damping, tension and rigidity).

ApplicationsApplications► Tracking of a moving objectTracking of a moving object

An initial estimate for the contour (e.g. interactively defined) is An initial estimate for the contour (e.g. interactively defined) is used in the first frame. used in the first frame.

The contour at equilibrium is used as the starting contour for the The contour at equilibrium is used as the starting contour for the next frame. The snake locks on to the object.next frame. The snake locks on to the object.

► Reconstruction from serial sectionsReconstruction from serial sections The user draws an approximate contour in the first slice. The user draws an approximate contour in the first slice. The contour at equilibrium is used as the starting contour in the The contour at equilibrium is used as the starting contour in the

next slice.next slice. The 3D object is reconstructed from the contours using The 3D object is reconstructed from the contours using

triangulation.triangulation.

► ……....

More segmentationMore segmentation

Important in Image Processing in general:Important in Image Processing in general:

““If you can use expert knowledge (user If you can use expert knowledge (user interaction, modelling,…) at relatively low interaction, modelling,…) at relatively low cost (development, computational,…)”cost (development, computational,…)”

JUST DO IT!!JUST DO IT!!