tum ipc 14 00.ppt [Kompatibilitätsmodus] · PDF filePraxisbuch. Aachen: Elektor ... Haralick R, Shapiro L (1992) Computer and Robot Vision. Vol. 1, ... Image Processing [tum_ipc_14_01]

Technische Universität München

Photogrammetry & Remote SensingProf. Dr.-Ing. U. Stilla

Image Processing

2014 WS

Uwe Stilla

[email protected]

Ludwig [email protected]

STILLA & HOEGNER (2014-10-14) Image Processing [tum_ipc_14_00] 2

0.1 Information

Lecture ( Wednesday, 08:00 – 09:30):

Participation necessary

Basis: Printed slides

Completion and notes necessary

Questions welcome

Exercises:

Partially during lecture

Some additional exercise courses for programming

Exams:

Answering questions

Calculations, working on images

Question block – true / false statements

Prof. Stilla

L. Hoegner


0.2 Literature (1)

Literature (German):

Azad P, Gockel T, Dillmann R (2007) Computer Vision – Das Praxisbuch. Aachen: Elektor

Bässmann H, Besslich PW (1989) Konturorientierte Verfahren der digitalen Bildverarbeitung.Berlin: Springer

Burger W, Burge M J (2005) Digitale Bildverarbeitung, eXamen.press: Springer

Haberäcker P (1987)Digitale Bildverarbeitung:Grundlagen und Anwendungen.München: Hanser

Kopp H (1997) Bildverarbeitung interaktiv: Teubner

Wahl F (1984)Digitale Bildverarbeitung.Berlin: Springer

Voss K, Süße H (1991)Praktische Bildverarbeitung.München: Hanser

Liedtke CE, Ender M (1989)WissensbasierteBildverarbeitung.Berlin: Springer

Bunke H (1985)Modellbasierte Bildanalyse.Stuttgart: Teubner

Jähne B (1993)Digitale Bildverarbeitung.Berlin: Springer


0.2 Literature (2)

Literature English:

Castleman KR (1995)Digital Image Processing.New Jersey, Englewood Cliff: Prentice Hall

Gonzalez RC, Woods RE (2002) Digital image processing. Prentice Hall

Haralick R, Shapiro L (1992) Computer and Robot Vision. Vol. 1, New York: Addison-Wesley

Jain AK (1988)Fundamentals of digital image processing. Prentice Hall

Rosenfeld A, Kak A () Digital picture processing. Academic Press


0.3 Software

Viewer: IrfanView free

http://www.irfanview.com/

MATLAB Mathworks & Image Processing Toolbox kommerziell

Scilab & SIVP Scilab Image and Video Processing Toolbox frei

http://www.scilab.org/

http://sivp.sourceforge.net/

HALCON MVTec commercial

GNU Octave free

http://www.octave.com/


0.4 Image Processing - Contents

Image processing

Introduction

Digital image characterization

Image transformation

Segmentation

Binary image processing

Vectorization and geometric primitives

Feature extraction



Image Processing 1 Introduction

2014 WS

Uwe Stilla

[email protected]



Image Processing - Contents

Image Processing

Introduction



Segmentation



Feature extraction


1 Introduction

Introduction

Motivation

Applications

Application: Tomography

Application: Character recognition

Application: Interpretation of Earth surface

Imaging

Separation of image processing

Image analysis

Image- and scene description

From knowledge to model

Hierarchical object modeling

Processing layers


1.1 Motivation

Digital Image processing

operates upon pictorial information of a digit form

uses digital circuit, computer processors and software to carry out an operation

Aim

better or more illustrative representation

information extraction (as automatic as possible)

Information extraction requires

knowledge about imaging process (real world → image) and

interpretation process of humans (image → objects)

therefore, image processing involves methods of

signal processing

pattern recognition and object recognition

Shining example

Human / eye, visual perception

2 images → 3 D description of the visible scene in object space

Dream of a “visual equipped" robot (very old idea)


1.2 Applications

Archeology: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Astronomy: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Biology: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Remote sensing: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Radio and television: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Geodesy: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Criminology: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Medicine: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Metallurgy: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Metrology: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Physics: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Photogrammetry: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Production: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Quality control: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Robotics: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Military: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


1.2.1 Application: Tomography, NMR - 3D-Plot


1.2.1 Application: Tomography, NMR

spin echo image (34ms)


1.2.1 Application: Tomography, NMR


1.2.2 Application: character recognition, metal

Milled and punched characters in metal workpieces illuminated in different ways


1.2.2 Application: character recognition, metal

Number five (5) illuminated in different ways


1.2.2 Application: character recognition - 3D Plot

(Inverted brightness)


1.2.2 Application: character recognition, gray value profile

255 255


1.2.3 Image interpretation (aerial image)


1.2.3 Image interpreation (map TK 1:25 000)


1.3 Imaging: Image brightness

Geometric and radiometric (physical radiation) projection:

Scene (3D) → Image (2D)

geometry: position of the 3D point in the image e.g. central perspective

radiometry: quantitative measure of the brightness

optics are included

Information content: electromagnetic radiation (EMR) for a certain wavelength

reflected radiation (e.g. sunlight, radar pulse)

particular radiation (e.g. thermal radiation)

Recording: Energy of the EMR or another additional proportional measurement

Device (sensor):

photographic Film (is simultaneously storing the images)

electrical sensors e.g. CCD sensor (additional image storage is required)


1.3 Imaging: general images-radar image (SAR)


1.3 Imaging: general images

Beside brightness, an image can represent other values which are distributed in space, e.g.

elevations related to a reference plane

distances to a sensor

concentrations of chemical substances

distributions of temperature

distributions of pressure

Etc.

In principle any mathematical function having the following form

z = z(x,y)

can be seen as an image.

Interpretation of those images requires in general information about:

way of projection

type of sensor

transmitted information

information transmitter


1.3 Imaging: general images–Digital Elevation Model


1.4 Separation: Digital image processing

Image analysis (also):

Image understanding

Machine vision

Image interpretation

Computer Vision

General aim of image analysis:Generation of an image or scene description

Image Description

Image

Descript.

inout

ImageProcessing

generativeComputerGraphics

Everythingelse

ImageAnalysis


1.5.1 Image and scene description

Steps of generation Items or topics to describe

Objects of the image or scene

Features of the objects

Relations between objects

Interpretation of objects

Segmented image

sampled image

primitives

objecs

scene

Mapping

Preprocessing

Feature extraction

Image interpretation


1.5.2 From knowledge to model

Knowledge Model

model–based image analysis

model-driven image analysis

...

Description of image formation through:

Illumination modeling

Sensor modeling

Scene or object modeling

Object modeling

attributive description specification of the object through its global properties (e.g. building: attributes list such as height, surface area, building volume, ...)

structural description specification of the general object layout through its components (e.g. building = walls + roofs+…)


1.5.3 Hierarchical object modeling

Hierarchical object modelingModels that include an hierarchical ordering of the knowledge

Scalar hierarchy

Ordering of the knowledge according to a desired level of details

The levels could be nested through statements such as „is_composed_of“or „is_part_from“

Example:

A settlement is_composed_of buildings and roads. A building is_composed_of walls and a roof. A wall is_composed_of masonry, doors and windows. Or: A building is_part_from a settlement. A road is_part_from…

Specification hierarchy

Ordering of the knowledge according to a certain level of specification

The specification levels could be linked through statements such as „is_a_kind_of“

Example:

A family house is_a_kind_of residential houses. A residential house is_a_kind_of buildings. A building is_a_kind_of constructions.


1.5.4 Processing layers

processinglayer

processeddata operations

high level objects symbolicrecognition

medium level features featureextraction

low level pixel imagemanipulation



Image Processing2 Digital image characterization

2014 WS

Uwe Stilla

[email protected]


STILLA & HOEGNER (2014-10-14) Image Processing [tum_ip2_14_02] 2

Image Processing - Topics

Introduction



Segmentation



Feature extraction


2 Digital image characterization

Image model

Sampling

Pixels

Geometries and neighborhoods

Coordinate systems

Quantization

Examples

Storage of digital images

Memory space

Coding

Statistical image characterization

Mean, variance and standard deviation

Histogram

Entropy

Co-occurrence matrix

Covariance

Correlation coefficient


2.1.1.1 Raster - different geometries

neighborhoodsgeometries connections


2.1.1.2 Sampling – pixels

Grey value

Gray value (Intensity value ) in the position (r,c)

Pixel

row

column

black

gray

white


2.1.1.2 Sampling - coordinate systems

Pixel b: picture element

c

r

x

y

x

yx

y

B = b(r,c) B = b(x,y)

B = b(x,y)B = b(x,y)

Image matrix B


2.1.1.2 Sampling - coordinate systems - Exercise

From a given image B=(b(x,y)) with nx=768 columns and ny=512 rows a subimage S=(s(u,v)) with nu=384 columns and nv=256 rows is extracted. The upper left image point of B is blu=b(1,1) and the y-axis points down. The upper left image point of S is slu = s(1,1) and is identical with b(37,25).

a) Which x,y -coordinates correspond to the lower right image point srl of S ?

b) The same image content of B is available in a different image processing system given by B’=(b’(x’,y’)). However, in this system the y-axis points up and the lower left image point of B’ is b’(0,0). You want to cut out a subimageS’=(s’(u’,v’)) showing the same content like S. Which x’,y’-coordinates correspond to the lower left image point s’ll=s’(0,0) and the upper right image point s’ru=s’(383,255) ?

for a) sru: b ( , )for b) s‘lu: b‘ ( , ); s‘ro: b‘ ( , )


2.1.2 Quantization (1)

The continuous gray- or multi-channel spectral function is quantized by an A/D converter into a number of possible gray values.

gray value

b(r,c) G = {g0, g1, g2, ..., gk-1}

black white

k=2L

L=number of the converter bits

The 8-bit converter is the mostly used one:

After sampling and quantization, the digital image is generated.

In order to save memory, the data is usually compressed.

black

gray

white


2.1.2 Quantization (2)

G={0,1} b scalarbinary image

for storage and display of the digital image 1 bit is required for each pixel

G={0,255}b scalar 2-value image

e.g. g0=0, g1=255

G={0,1, … 255} b scalar

gray value image (monochrome)for storing often

8 bit (=1 byte) are used

G={0,1, … 255}b vector

(n elements)

multi spectral imagefor storing often8xn bits are used

G={0,1, … 255}b vector

(3 elements)

(true-) color image

3 Bands: RGB



2.1.3 Example: Different samplings

Resolution:

1x1, 2x2, 4x4

8x8, 16x16, 32x32

64x64, 128x128, 256x256


2.1.3 Example – image sequences (in space and time)

Spatially extended objects

A spatially extended object is split into a sequence of sub-images Bi (x, y, zi). This sequence could be used to get a snapshot of the 3D object.

Dynamic 2D-process

If a 2-dimensional object shows changes over time, a sequence Bi (x, y, ti) could be created to describe the temporal changing of the object.

x

y

tz

x

y

Bi(x,y,zi) Bi(x,y,ti)

Bi+1(x,y,ti+1)

Bi+2(x,y,ti+2)

Bi+1(x,y,zi+1)

Bi+2(x,y,zi+2)


Memory space

Image matrix nx=512; ny=512 nxy=262144

Binary image 1 bit = KB

Gray value image 8 bit = KB

RGB- image 3x8 bit = KB

Storage of multi-band images

Band-Interleaved-by-Line (BIL) or Row-Interleaved

Band-Interleaved-by-Pixel or Pixel-Interleaved (BIP)

Band-Sequential or Band-Interleaved (BSQ)

2.2 Storage of digital images

Data compression

Lossy/lossless compression

Binary images

Elimination of gray levels

Storage of the differences between the gray values

Coding

Run-length encoding

Quad-trees structures

Chain codes

File formats

STILLA & HOEGNER (2014-10-14) Image Processing [tum_ip2_14_02]

2.2.1 Storage of digital images - memory space

Physical nomenclature

1 Megabyte (MByte) = 1000 Kilobyte (kByte)

1 Kilobyte (kByte) = 1000 Byte

1 Megabyte (MByte) = 1.000.000 Byte

Computer science

1 Megabyte (MB) = 1024 Kilobyte (KB)

1 Kilobyte (KB) = 1024 Byte

1 Megabyte (MB) = 1.048.576 Byte

Example

Grey value image 8 bit per pixel

Image size 23x23 cm²

Pixel size 15x15 μm²

23 cm / 15 μm = 15300 pixel / row

15300 pixel² = 234.090.000 pixel

234.090.000 Byte = 234 MByte = 223 MB

18


2.2.1 Storage of digital images - memory space

Assumption:1 bit is used per pixel and band (channel)

image type pixel size approx.image size given in

Grey value image

monochrome

color image


Example of binary row: 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 0 0 0

Coding 1:

Coding 2:

Background value g=0

Origin and length of g=1

Efficiency of the RLE?

Error-free reconstruction?

Extension to 2D binary images: row neighborhood graph (4th neighbor)

2.2.2.1 Run-length encoding:RLE (1)

20

1 2 3 4 5 6 70

1:

2:

3:

4:

5:

6:

7:

0:

STILLA & HOEGNER (2014-10-14) Image Processing [tum_ip2_14_02]

2.2.2.1 Run-length encoding:RLE (2)

Extension to gray-value images

Description: (Origin, gray value, Length)

meaningful: ?

22


2.2.2.2 Quad-Trees (1)

24

Partition into quadrants

Homogeneity analysis

Tree structure

nodes

edges

Example: Binary image, nx=ny=2k, k=3

Types of nodes

homogeneous black areas

homogeneous white areas

inhomogeneous areas

Q1 (k-1)

Q3 (k-1)

Q0(k-1)

Q2 (k-1)

102 3

3

1

2

0

step

root



Node indexing

26

3

1

2

0

step


Compact storage

Without node indexing and edges

Definition, for example:

0: black

1: white

2: hybrid nodes

For example level 2: 2 0 2 1

Recursive storage of sub-trees


Compression versus analysis flexibility

28

1 0 0 0

2 0 2 11 2 2 0

1 0 1 02 2 1 11 0 1 00 0 0 1


2.2.2.3 Chain codes

30

Description of objects through the boundary

Binary image: Object (g=0) on a background (g=1)

Definition of neighborhood codes

(e.g. 8th)

Boundary description through

Origin

Chain direction (code direction)

Example:

0 1, 0 1 1,-12 0,-13 -1,-14 -1, 05 -1, 16 0, 17 1, 1

αi (aix, aiy) code vector

0 1 2 3 4 5 6 7

0

1

2

3

4

5

6

7

* 0

123

45 6 7

x

y

x

y

Origin:

Chain code:


2.2.2.3 Chain code- in practice

Integration of additional information through special codes (04++)

End of the chain (e.g. 0400)

Generalization for gray value images through storage of the gray values g (e.g. 0422 g)

Identification of areas or one-pixel-thick lines (contour lines)

Design of the chain through masks comparison

Line-by-line digital scanning

Raster directed chains (directions 0,5,6,7)-> general directed chains

32


Further processing steps

Convex elementary musters(splitting any area into convex elementary musters)directed codes have special structure -> monotone directed codes

Chain code:

Coding: (8 parameters)

2.2.2.3 Chain codes

33

Lenght of direction chain

ng: number of even k

nu: number of uneven k

Translation

Roation of k x 45°, k=1,2,3

Object dimensionschain code ai -> vector (aix,aiy)

p.e. ai=5 (aix,aiy) = (-1, 1)chain code: a0 a1 … an-1

Width and height:

i

i jx 0j 1

x a x

i

i jy 0j 1

y a y


2.2.3 Data formats RAW

Raw data with 8, 10, 12 or 14 bpp (bit per pixel), lossless, camera specific

TIF or TIFF (Tagged Image File Format)

Several images in one file possible (Multipage-TIFF)

Many color represenations and compressions possible, common 8,16,24 bpp, uncompressed or run length coding, lossless

„tags“ for additional information like GeoTIFF: Tag 33922 (ModelTiepointTag) tie point on rasterized coordinated with pixelvalue (i,j,k) and model coordinates(x,y,z)

Not applicable for web due to complexity and interal pointers

Often used for archiving

GIF (Graphics Interchange Format)

max 8 bpp, colors with LUT (Look Up Table)

Color index in LUT can be marked as transparent

lossless LZW compression (Lempel-Ziv-Welch)

Introduced by CompuServe, algorithm under patent

Often used for websites


2.2.3 Data formats PNG

max 48 bpp, prefiltering (difference to left and /or upper neighboring pixel)

lossless LZ77 compression (Lempl-Ziv) plus Huffmann coding

Uses an „alpha chanal" (transparency mask)

PNG is licence free in contrast to GIF

JPG or JPEG

Color space transform RGB -> YCbCr,

The YCbCr model seperates color information in luminance Y and two colorcomponents Cb (/Blue-Yellow Chrominance/) and Cr (/Red-Green Chrominance/).

Structured in 8×8 blocks with discrete cosine transform of these blocks, theoretically lossless or with defined specific loss

BMP

Standard format for Microsoft Windows

No compression

Often as 24bpp 3 channel true color (3x8bpp) or as 8 bpp color with look uptable


2.3 Statistical image characterization

Mean grey value

Variance and standard deviation

Relative histogram

Cumulative histogram

Histogram of several channels

Entropy

Co-occurrence matrix

Covariance

Correlation coefficient

Spatial frequency

37


2.3.1 Mean, variance and standard deviation

38

Mean value μThe mean is the average gray values of an image B or a certain image region

Variance δB2

The variance is the gray values‘ mean-squared deviation from the mean

Standard deviation δB


2.3.2 Mean square deviation - variance - calculation

40

Note: μ and δB2 can be calculated

in one step and thus lesscalculations are necessary.

cr

cr

c c cr r r

cr

cr

nn22

Br 1 c 1rc

nn2 2

r 1 c 1rc

n n nn n n2 2

r 1 c 1 r 1 c 1 r 1 c 1rc

nn2 2

rc rcr 1 c 1rc

nn2 2

rcr 1 c 1rc

1b r,c

n

1b r,c 2b(r,c)

n

1b r,c 2 b(r,c)

n

1b r,c 2 n n

n

1b r,c n

n

cr nn

2 2 2B

r 1 c 1rc

1b r,c

n


2.3.3 Histogram

43

The histogram PB(g) is a distribution function, which provides for each gray value the number of pixels having this same gray value.

Often the histogram is scaled by the number of pixels and called: Histogram of the relative frequencies p(g).

max

i min

g

B i rcg g

P g n

B Brc

1p g P g

n

max

i min

g

B ig g

p g 1

B i0 p g 1

The histogram of an image gives reliable information about brightness and contrast, although the entire spatial information get lost.

Graphical representations of histograms= bar charts


2.3.3.1 Computation of mean and variance from histogram

44

A simplified calculation is given by using the histogram

mean

variance

max

i min

max

i min

g

i B ig grc

g

i B ig g

1g P g

n

g p g

max

i min

max

i min

g22

B i B ig grc

g22

B i B ig g

1g P g

n

g p g


2.3.4 Cumulative histogram

46

Cumulative histogram S(g)

Cumulative realtive histogram s(g)

i min

g

B B ig g

S g P g

B max rcS g n

B Brc

1s g S g

n

B maxs g 1


2.3.4 Example histogram and cumulative histogram

47

50 100 150 200 2500

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

50 100 150 200 2500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


2.3.5 Entropy

49

Entropy: average information content of the image

Computation using the histogram PB(g)

Example

homogeneous image B=(b(x,y))=g

Dual image (pB (g1)=0,5; pB (g2)=0,5)

PB (g) uniformly distributed

max

i min

max

i min

g

B i 2g g B i

g

B i 2 B ig g

1H p g log

p g

H p g log p g


2.3.6 Co-occurrence matrix

The co-occurrence matrix is useful for the description of certain relations in the image

φ: relation between pairs of image point positions (r1,c1) and (r2,c2)

Example:

right neighbor

neighbor at Δr, Δc

ag1,g2 : frequency of gray value combinations (g1,g2)=(b(r1,c1), b(r2,c2))

The co-occurrence matrix for the relation φ is

51

B, 1 2 g1,g2W (g ,g ) a


2.3.6 Co-occurrence matrix - example

52

Example

Relation φ=φ1 (right neighbor)

Grey values G= {0, 1, 2, 3}

image nr=6 , nc=6

WB,φ(g1,g2) =

Meaning of the elements in the main diagonal :

Meaning of the side elements g1,g2

Low contrast

High contrast

0

0

0

0

0

0

0

0

1 1

1

1

1

1

2 2

2

22

1

2

2

2

3

33

3

3

33 3 3

33

33


2.3.7 Spatial frequency (1)

54

Analogue to temporal periodic changes I(t), images can be described as spatial periodic changes I(x) by frequencies. To distinguish temporal and spatial changes, you then use the term spatial frequencies.

A spatial frequency is defined as the inverse of the length of a spatial period. The spatial frequency gives the number of periods within a spatial interval und thus has the dimension 1/length (e.g. line pairs per millimeter).

Sinus grid low

spatial frequency

(long periods)

Sinus grid high

spatial frequency

(short periods)

x in [mm] or [pixel]

Period length (line pair bright/dark)


2.3.7 Spatial frequency (2)

Draw the grey value changes of a line and determine the spatial frequency in x direction!


2.4.1 Covariance

58

The covariance of two images B1 and B2 is written as

cr

1 2

cr

1 2

nn

B B 1 1 2 2r 1 c 1rc

nn

B B 1 2 1 2r 1 c 1rc

1b r,c b r,c

n

1b r,c b r,c

n

If B1=B2 covariance = variance


2.4.2 Correlation coefficient

59

The correlation coefficient ρ of two images B1 und B2 is described by

cr

1 2

1 2c cr r

1 2

nn

1 1 2 2B B r 1 c 1

B B 2 2 n nn n2 2B B

1 1 2 2r 1 c 1 r 1 c 1

b r,c b r,c

b r,c b r,c

δB1B2 and ρB1B2 decribe the similarity of two images

ρB1B2 = 0 B1 and B2 are uncorrelated

ρB1B2 = 1 B1 and B2 are correlated to 100% positiveb2=b1+ ; > 0

ρB1B2 = -1 B1 and B2 are correlated to 100% negativeb2= b1+; < 0

1 2 1 2B B 1 2 B B 1 2( b , b ) sgn( , ) (b , b ) The correlation coefficient ρ is invariant to linear tranformations!


2.4.2 Correlation coefficient - examples

60

not defined


2.4.2 Correlation coefficient – example character recognition

Example of use of the correlation coefficient for character recognition

Correlation coefficient = measure of similarity between a reference or „template“ and a sliding window in the test image

Template B2 is moved all over the test image so that the corresponding window B1 is found

The correlation coefficient is computed between the template B2 and the window B1

The correlation coefficients are then stored in the correlation image(correlation image: black = -1 ; white = +1)

Identification of the maxima (high positive correlation) and minima (high negative correlation) of the correlation image

61

B2 B1

Test image Correlation image

Template



62

search image correlation imagearrow: high positive correlation

search image correlation imagearrow: high negative correlation


2.4.2 Correlation coefficient – masked correlation

Special case: the reference (or template) has only 2 gray values. The computation of the corresponding correlation coefficient could get easier.

Not all the pixel pairs b1(r,c) (from the sliding window B1 in the test image), b2(r,c) (from the template B2) are used in the computation of the correlation. Only certain areas that remain after applying a mask to the template or after zeroing some values from the template image, are investigated. This is called the masked correlation.

Example of a masked template B2

Black area: gray value g1

Gray area : gray value g2

white area: the corresponding pixels arenot used in the computation of the correlation.


2.4.2 Correlation coefficient – masked correlation - example

Masked correlation

reference

Correlattion(white: 0 black: 1)

Test image

Test image

Correlation image



Mask-based correlation

Query with one template in order to find the position in the test image where the template and the character in the test image fit at best

65

referenceTest image

Correlation image (white: 0, black: 1)



66

Correlation image (white: 0, black: 1)

referencesTest image

Mask-based correlation

Query with different templates for the template that matches the best with the test image



Image Processing 3 Image transformation

2014 WS

Uwe Stilla

[email protected]




Image Processing

Introduction



Segmentation



Feature extraction


3 Image transformation

Geometrical transformation

A geometrical image transformation converts an input digital image into an output digital image where the geometrical position of the pixels have changed.

Spatial transformation

Resampling

Modification of the gray value distribution

Point processing

Local image operators

Global image operators


3.1 Geometrical transformation

Geometrical transformation are often used for

Co-registration between two images

Image geocoding

Image rectification

The geometrical transformation is a 2-step process

Spatial transformation

ResamplingDifferent methods:

Nearest neighbor

g gets the gray value of its closest pixel

Bilinear interpolation

g is a linear weighted average of its 4 neighbors gray values

Higher order interpolation (e.g. cubic spline)

g is a weighted average of the 16 closest neighbors gray values

Quality versus computational effort


3.1.1.1 Spatial image transformation (Examples)

Translation

Mirroring (according to the mirror line (x=c))

Rotation (rotation angle α)

Zooming

Affine transformation

Projective transformation


3.1.1.2 Direct spatial transformation

Direct method

ProcedureEach pixel in the input image is projected in the output one, where the gray values are interpolated.

Disadvantage When few pixels are available in the input image, the corresponding pixels in the output image will be assigned with “NO VALUE” (holes in the output image).

Input image Output image


3.1.1.3 Indirect spatial transformation

Indirect method

ProcedureEach pixel is replaced in its position in the input image and then assigned with the interpolated gray values from the output image

AdvantageNo uncovered pixels

Often used in the digital photogrammety

Input image Output image


3.1.2 Resampling - Bilinear interpolation

Computation of the gray values from the 4 closest neighboring pixelsg(0,0), g(0,1), g(1,0) und g(1,1)

(1,1)

(0,1)

(0,0)

(1,0) x

y

x‘‘

y‘‘

g(0,0)

g(0,1)g(1,1)

g(1,0)


3.1.2 Resampling: bilinear interpolation - Exercise

An image B is subjected to a geometrical transformation.

A point (x‘1,y‘1) in the transformed image B‘ has the following coordinates in the input image B

(x1,y1)= ( 36.75 , 24.25 ).

The gray values of its 4 neighbors are:

b ( 36, 24 ) = 64

b ( 36, 25 ) = 192

b ( 37, 24 ) = 192

b ( 37, 25 ) = 64

Draw a 3D sketch and determine the new gray value b‘ ( x‘1, y‘1 ) by a bilinear interpolation!

b‘ ( x‘1, y‘1 ) =


3.2 Modification of the gray value distribution

The semantics of an image are stored in its gray values, as well as the local and global arrangements of its pixels. When an image transformation is based on the modification of its gray value distribution, different procedures are possible.

Point processing A pixel is assigned with a new value according to a certain rule that involves only its original value and possibly its position (in the input image)

Local processing A pixel, located in a specific area (e.g. central pixel) will be assigned with a new gray value, according to certain rule that involves the gray values of the surrounding pixels

Global processing A pixel is assigned with a new gray value according to a certain rule that involves the entire image


3.2.1 Point processing

Point processing

Homogeneous point processingThe point operation is independent of the position of the pixel. It involves only the original gray value of the pixel.

Inhomogeneous point processingThis point operation is dependent of the position and gray value of the pixel.

Examples of homogeneous point processing

Characteristic curves

Histogram scaling and linearization

Color transformation

The different transformations based on the above described point processing will be illustrated and discussed in terms of histograms.


3.2.1.1 Characteristic curves: inverting

]255),([),(' yxbyxb

b

b‘

0 2550

255


3.2.1.2 Characteristic curves: linear scaling

12)]),([),(' kkyxbyxb

b

b‘

0 2550

255

11 k

11 k

02 k

02 k


3.2.1.2 Characteristic curves: linear scaling

Areas with steep curves (slope>1) are characterized by a wide range of gray values (higher contrast).

Areas with smooth curves (slope<1) are characterized by few gray values (lower contrast).

The general layout of the histogram does not change after a linear scaling

Mean and variance of b‘ are:

(Proof by inserting the equations of μ und δ2 in the linear equation)

The images could thus be normalized for a given mean and standard deviation.


3.2.1.3 Characteristic curves: piecewise linear scaling

2 1

2 1

2 1

0 [ ( , ) ] 0

'( , ) [ ( , ) ]

255 [ ( , ) ] 255

if b x y k k

b x y b x y k k

if b x y k k

b

b‘

0 2550

255

min2 bk

minmax1

255

bbk

minb maxb


3.2.1.4 Characteristic curves: general scaling

maxmax bff

minmin bff

f(b): general function

255]),([255

255

]),([255

0]),([255

0

),('

minminmax

minminmax

minminmax

fyxbfff

falls

fyxbfff

fyxbfff

falls

yxb


3.2.1.5 Characteristic curves: cut off

suppress non-relevant information (e.g. background), or emphasize certain Information

min

min max

max

0

'( , ) ( , )

0

if b b

b x y b x y if b b b

if b b

b

b‘

0 2550

255

minb maxb


3.2.1.6 Non-linear characteristic curve transformation

Contrast enhancement through non-linear characteristic curve transformations, for e.g.

Elimination of the non-linearities caused by the sensors

Compensation of image display non-linearities

Through, e.g.

Exponential transformation

Logarithmic transformation

Sigmoid transformation

Tangent transformation


3.2.1.6 Non-linear characteristic curves: Exponential

stretching a small bright gray-value interval

b

b‘

0 2550

255

]1[),(' 1),( kyxbeyxb

985,451 k


3.2.1.6 Non-linear characteristic curves: Logarithmic

stretching a small dark gray-value interval

b

b‘

0 2550

255

]1),([log),(' 2 yxbkyxb

09,1062 k


3.2.1.6 Non-linear characteristic curves: Sigmoid

Stretch the mid-luminance range: increase contrast at the central part of the histogram (slope>1) and

Flatten the very dark and light regions of the image: decrease contrast at the histogram borders (slope <1)

max

max

b 1 b(x, y) 1b'(x, y) 1 sin 0 1

2 b 2sin

2

b

b‘

0 2550

255

α = 0.9


3.2.1.6 Non-linear characteristic curves: Tangent

α = 0.9

b

b‘

0 2550

255

max

max

b 1 b(x, y) 1b'(x, y) 1 tan 0 1

2 b 2tan

2

Flatten the mid-luminance areas: decrease contrast at the central part of the histogram (slope<1) and

Stretch the very dark and very light areas: increase contrast at the histogram borders (slope >1)


3.2.1.7 Characteristic curves: static threshold

Segmentation by a static threshold

Gray value image: 2-level image

(e.g. images of scanned drawings or text)

b

b‘

0 2550

255

sb

255 ( , )'( , )

0sif b x y b

b x yelse

128sb


3.2.1.8 Characteristic curves: two static thresholds

Segmentation by 2 static thresholds

Gray value image: 2-level image

b

b‘

0 2550

255

minb maxb

min

min max

max

0 ( , )

'( , ) 255 ( , )

0 ( , )

if b x y b

b x y if b b x y b

if b x y b

128min b 160max b


3.2.1.9 Characteristic curves: iso densities

Piecewise constant scaling function

Gray value image: multi-level image

b

b‘

0 2550

255

1b 2b

1),(für),(' kkk byxbbgyxb

4n

1...,2,1,0 nk

3b


3.2.1.10 Modification of grey value distribution

Histogram scaling

Histogram linearization or equalization

Linearization function using relative cumulative frequencies

input histogram output histogram

i min

g

B B ig g

s g p g

B0 s g 1

nb` r,c f (b(r,c))transformation with fn

so that B'max

1s g g

g

Advantage: simple implementation, since the scaling function can be directly derived from the gray-value distribution

pB(g)

g

pB‘(g)

gg

g

pB(g)pB‘(g)

input histogram output histogram


3.2.1.10 Example of histogram scaling

50 100 150 200 2500

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

50 100 150 200 2500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

50 100 150 200 2500

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

50 100 150 200 2500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


3.2.1.10 Example of histogram linearization

50 100 150 200 2500

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

50 100 150 200 2500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

50 100 150 200 2500

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

50 100 150 200 2500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


3.2.1.11 Color transformation

From „Look-Up-Table“ (LUT) to rgb (red-green-blue)

LUT r g b

0 r0 g0 b01 r1 g1 b1. . . .5 000 101 189 .. . . .

255 r255 g255 b255

5


3.2.2 Local image operators

Digital filter

Convolution

Mean operator

Gradient operator

Laplace operator

LOG and DOG

Effects of linear convolution operators

Computation effort for convolution

Morphological operators


3.2.2.1 Digitale Filter

Applications

reduction of noise

elimination of distortions

balancing of brightness

restoration of burred edges

emphasizing or suppressing details

detection of edges and objects

segmentation of ROIs(ROI: Region Of Interest)

Characterization of filters

effect

image point

local

global

operation

linear

non-linear

processing

parallel

sequential

recursive

analysis

Space invariant (homogeneous)

Space variant (inhomogeneous)

b‘(r,c)b(r,c)

Input image Output imageDiscretesystem


3.2.2.2 Convolution

Input image B with b(r,c)

Filter matrix h(k,l) with operators K,L: (uneven)(Filter mask, Filter kernel, Impule response)

Convolution

K l L l

2 2

K l L lk l

2 2

h(k, l) 1

b ' b h

K l L l

2 2

K l L lk l

2 2

b ' r, c b(r k,c l) h(k, l) r,c

K 1 K 1 L 1 L 1k ; ; l ;

2 2 2 2

Often the sum of weights (= elements of the filter matrix)

* is the convolution operator


3.2.2.2 Convolution - properties

b‘ is calculated as weighted arithmetic average of the corresponding section of the input image and the filter matrix

The convolution is a linear operation: a gray value b‘(r,c) is a linear function of gray values b(r-k,c-l)

Since all gray values b‘(r,c) of the output image are calculated in the same way convolution is shift invariant

Properties of convolution (results from linearity)

commutative

associative

distributive

scalar multiplication


3.2.2.2 Convolution - graphical representation

In graphical representation a convolution can by displayed by successively moving the mask over the image and calculating the output value by muliplying superposed values and summing up the results

Note:this will only be valid, if the filter mask is mirrored at the center in advance (due to the definition of indices k und l)

input image b(r,c)

output image b‘(r,c)

operator mask h(k,l)


3.2.2.2 Convolution - borders

Problem: convolution is not defined for pixels of the border.

Solutions:

Border effects can be neglected in case of a single convolution. In case of multiple convolutions the propagation of border effects can not be neglected.

In complex image processing systems, an appropriate handling of borders is important.

?


3.2.2.3 Mean operator (smoothing, low-pass)

Application: Reduction of noise by smoothing (low-pass filtering) (pre-processing)

Procedure: Computation of the mean for a given region nk x nl surrounding a pixel at the position (r,c)

(region often square: e.g. 3x3, 5x5, 7x7, …)

i i

k i l ikl

1b ' r,c b r k,c l

n

1 1 1

1h k, l 1 1 1

91 1 1

b r 1,c 1 b r 1,c b r 1,c 1

b r,c 1 b r, c b r,c 1

b r 1,c 1 b r 1,c b r 1,c 1

1 1

k 1 l 1

1b ' r,c b r k,c l

9

with nk= nl ; nk= 3, 5, 7, …

i = (nk-1) / 2nkl = nknl

with nk= 3

b ' b h

considered region


3.2.2.3 Effect of mean operators - edge

8020 802020 80 8020 20 20 20 2020 80 80 80 80 8080

1 1 1

BR

HR3

BR*HR3

HR5 1 1 1 1 1

BR*HR5

31

51


3.2.2.3 Effect of mean operators - distortions

BR*HR

8020 802020 80 8020 20 20 26 2020 80 80 50 80 8050

1 1 1

BR

HR 31


3.2.2.3 Gaussian smoothing operator (1)

Smoothing filter with a Gaussian distribution

2

22

222

1),(

mm llkk

elkh

g

0.0001 0.0015 0.0067 0.0111 0.0067 0.0015 0.0001

0.0015 0.0183 0.0821 0.1353 0.0821 0.0183 0.0015

0.0067 0.0821 0.3679 0.6065 0.3679 0.0821 0.00671

h (k, l) 0.0111 0.1353 0.6065 1.0000 0.6065 0.1353 0.01112

0.0067 0.0821 0.3679 0.

6065 0.3679 0.0821 0.0067

0.0015 0.0183 0.0821 0.1353 0.0821 0.0183 0.0015

0.0001 0.0015 0.0067 0.0111 0.0067 0.0015 0.0001

with 1

-3 -2 -1 0 1 2 3

-3

-2

-1

0

1

2

3

where km,lm= coordinate of the central pixel

gg

k,l

1h(k, l) h (k,l)

h (k, l)



Properties in comparison to the mean operator

A higher weighting is given to the central pixel

The weighting of the neighboring pixels is dependent of the distance

Better and more adaptative filter properties

Size of the filter mask

Gauss function is infinite

Filter mask finite

Typical window/kernel size: 2x3+1

Only elements with high weights are kept

Elements with insignificant weighting are eliminated no needless multiplications

3-area covers >99%



Example of Gaussian filters with different sigma and a same filter size (31x31)

-15 -10 -5 0 5 10 15

-15

-10

-5

0

5

10

15-15 -10 -5 0 5 10 15

-15

-10

-5

0

5

10

15


3.2.2.4 Gradient operator (1)

For sake of simplicity

1D case.

Continuous smooth distribution.

1D function g(x) shows an edge (dark/light transition).

First derivative g'(x)

The maximum of g'(x) indicates where the strongest jump of g(x)(highest variation of the gray values) occurs.

The location x0 of the maximum corresponds to the position of the edge.

Second derivative g''(x)

The zero-crossing indicates the inflexion point of g(x).

The location x0 of the zero-crossing represents the position of the edge.

x

x0

gg(x)

g´(x)

g´´(x)

x

dark bright

edge


3.2.2.4 Gradient operator (2)

Application: emphasis on local differences of gray values

The gradient of a continuous function b(x,y) is defined as a vector

x y

b(x, y) b(x, y)grad{b(x, y)} e e

x y

Using partial derivations, the strength and orientation can be calculated

magnitude:

direction:

22b(x, y) b(x, y)

grad{b(x, y)}x y

b(x, y) b(x, y)grad{b(x, y)} arctan

y x


3.2.2.4 Gradient operator - effect along a row

5032 682120 79 8020 20 20 20 2020 80 80 80 80 8080

-1 0 1

Bx

Hx

B‘x


3.2.2.4 PREWITT Operator

Only few pixels are taken into consideration when the discrete differential gradient operator is used. Therefore, this operator is very sensitive to noise. To cope with this problem, low pass filters are generally applied before the derivation is computed.

Alternatively, a larger neighborhood can be taken into consideration for the computation of the derivates, which then includes an average operation. An example is shown by the PREWITT operator, where the original image is convolved with the following two 3x3 kernels to calculate approximations of the derivatives - one for horizontal changes, and one for vertical ones:

101

101

101

x_PREWITTH

111

000

111

y_PREWITTH


3.2.2.4 SOBEL Operator

A better effect can be achieved by weighting the elements in a different way as done in the SOBEL operator whose kernels are:

101

202

101

x_SOBELH

121

000

121

y_SOBELH


3.2.2.4 Magnitude & direction of gradient by SOBEL

SOBEL _ MAG

2 2SOBEL _ x SOBEL _ y

b

b b

SOBEL _ DIR

SOBEL _ y

SOBEL _ x

b

barctan

b

SOBEL _ xB

SOBEL _ yB

SOBEL _ MAGB

SOBEL _ DIRB


3.2.2.4 Gradient operator - exercise

Draw the steps of the calculation of the gradient using the sobel operator!


3.2.2.4 Approximations for magnitude of the gradient

Often approximations for the gradient magnitude are calculated from the direction dependent differences of intensity by a sum e.g.:

ySOBELxSOBEL bb

),yb(x),yb(x

)b(x,y)b(x,y

),yb(x),yb(x

),yb(x),y(x

,y)b(x,y)b(x

),yb(x),yb(x

b

__

SOBEL

''

1111

1212

1111

1111

1212

1111

'

),1()1,(,),()1,1('ROBERTS yxbyxbyxbyxbMaxb

SWNWSONOWSON bbbbbbbbMaxb ',',',',',',',''KOMPAß

With convolution with compass gradient masks


3.2.2.4 Kompaßgradienten, Masken

Direction dependent difference operators

111

121

111

NH

111

121

111

NOH

111

121

111

OH

111

121

111

SOH

111

121

111

SH

111

121

111

SWH

111

121

111

WH

111

121

111

NWH

EastWest

North NortheastNorthwest

SoutheastSouthwest South


3.2.2.5 Laplace operator

The Laplace operator calculates the sum of the second partial derivations of a continuous function b(x,y) for the variables x and y.

2

2

2

2 ),(),(),(

y

yxb

x

yxbyxb

2

22

2

2

b(x 1, y) b(x, y) (b(x, y) b(x 1, y)b(x, y)

x (x 1) x

b(x 1, y) 2b(x, y) b(x 1, y)

b(x, y)b(x, y 1) 2b(x, y) b(x, y 1)

y

The second derivation can be implemented by the difference of the difference of neighboring pixels along the row and column


3.2.2.5 Laplace operator, masks

The differences can be calculated by vectors or a matrix H.

010

141

010

010

020

010

000

121

000

yx HHH

010

141

010

1H

111

181

111

2H

121

242

121

3H

121

2122

121

4H

Often different masks are used, which use algebraic sign vice versa or other binomial representations.


3.2.2.5 Laplace operator, effect along a row

The values of the resulting image can not immediately displayed, because of positive and negative values -> scaling between 0 and 255

The Laplace-Operator ist very sensitive to noise.

5032 682120 79 8020 20 20 20 2020 80 80 80 80 8080

1 -2 1

Bx

Hx

B‘x


3.2.2.5 Laplace operator, examples

010

141

010

1H

111

181

111

2H


3.2.2.6 LoG Operator (Laplace of Gaussian)

To reduce the noise, the image can be first smoothed by a Gaussian filter (G*Image). Then, the result is convoluted by the Laplace operator (D(G*Image)). Alternatively, ((DG)*Image) can by computed.

2

22

222

1),(

yx

eyxG

2

2

2

2

)(

),(

)(

),(:

y

yxG

x

yxGGLoG

2 2 2 2

2 2

2 2 2 2

2 2

2 2

2

2 2

2

x y x y

2 22 2 4

x y x y22 2

2 4 2 4

2x y

24

2x y22

2 4

G(x, y) 1 x 1e e ( x)

x 2 2

G(x, y) 1 x 1e ( x) e ( 1)

( x) 2 2

1 xe 1

2

G(x, y) 1 ye 1

( y) 2



22

22

kyx

2 2

2

x y 2 22

4 2

1 x yG e 2

2

2k

224

1LoG : G e k 2

2



Due to the similarity between the shape of LoG and a Sombrero, the operator is called Mexican-Hat-Operator.

The effect of the LoG-Operators for edge detection depends on the

standard deviation .

=4 =5

An edge is detected by finding the zero-crossings


3.2.2.6 DoG-Operator (Difference of Gaussians)

The LoG operator can be approximated by the difference of two Gaussian (DoG).

The relation of the standard deviation should be 1/2=1.6


3.2.2.7 Effect of linear operators (convolution)

smoothing edges

reduction of distortions by small objects

reducing noise

deleting small details

weighted summation of neighboring pixel values

weighted differences of neighboring pixel values

decreasing of high space frequencies

(lowpass filter)

decreasing of low space frequencies

(highpass filter)

space domain frequency domain

emphasizing edges

detecting contours

suppression of big homogeneous areas

emphasizing details


3.2.2.8 Computational effort for convolution

Direct computation

For a mask size nhx x nhx, for each pixel

nhx2 multiplications

nhx2-1 additions

Example: mask with the size 5x5, for an 512x512 pixels image

Computational effort:

Decomposition of the filters

Objective: reduction of the computational effort

Decomposition of the filter mask H into two vectors H1,H2, so that H=H1*H2

B * H = B * (H1*H2) = (B * H1) * H2

Procedure: first line-by-line then column-by-column convolution

Example: smoothing filter

before decomposition:

after decomposition:

1 1 1 0 0 0 0 1 0 11 1 1

h(k, l) 1 1 1 1 1 1 0 1 0 1 1 1 19 9 9

1 1 1 0 0 0 0 1 0 1


3.2.2.8 Filter decomposition, Example: smoothing filter

Example

Line-by-line convolution b2 = b1 * h1

(2 additions each) b2(14.93) = 113+116+104 = 333

b2(15.93) = 099+101+125 = 325

b2(16.93) = 000+107+105 = 212

Convolution of the result column-by column b3 = b2 * h2

(2 additions + 1 division)b3(15.93) =1/9 · (333+325+212) = 96.7 ≈ 97

Note: Perform the division as last step to minimize errors from rounding and quantization

1 1 1 1

1 1h(k, l) 1 1 1 1 1 1 1

9 91 1 1 1


3.2.2.8 Computational effort: Binomial filter decomposition (1)

Binomial filter properties

Discrete approximation of Gaussian filters

Used as low pass filter

Deducible from Pascal´schem Dreieck with the binomial coefficients

All binomial masks are separable: (h(k,l))=(h‘(k))(h‘(k))

Linear increase of the computation effort with nhx

2nhx multiplications

nhx-2 additions

1 4 6 4 1 1

4 16 24 16 4 41 1 1

h(k, l) 1 4 6 4 16 24 36 24 6 6256 16 16

4 16 24 16 4 4

1 4 6 4 1 1

n n!

m m! n m !

Further decomposition:[1 1]*[1 1] =[1 2 1][1 2 1] *[1 1]=[1 3 3 1]usw.

STILLA & HOEGNER (2014-10-14) Image Processing [tum_ipc_14_03]

Binomial filter Example

Computation of the convolution without decomposition

Computation after decomposition

Before the separation: 5 multiplications + 8 additions + 1 division

After the Separation: 4 additions + 1 division

91

3.2.2.8 Computational effort: Binomial filter decomposition (2)

1b (15,93) (1 113 2 116 1 104 2 99 4 101 2 125 1 0 2 107 1 105) 101.3 101

16

1 2 11

h(k, l) 2 4 216

1 2 1

1

113 116 116 104 229 220

b 1 1 99 101 101 125 200 226

0 107 107 105 107 212

1

229 220 449

b 1 1 1 1 200 226 426

107 212 319

1

1 449 426 875b 1 1 1 1

1 426 319 745

2 1

1 11 1b b 1 1 1 1 875 745 101.25 101

1 116 16


3.2.2.9 Morphology - Rank operators

The set of pixels within an operator window (structuring element, mask M) will be sorted according to their gray values. The value of a defined pixel position (often central pixel) will be replace by the value of the pixel with the rank which is characteristic for the operator.

13

133639

31

5796

22

92

98 91 19

88

42

123233

1819

16

69

10

17

15

11

35

Rank sequence : {13, 16, 22, 31, 36, 39, 57, 69, 96}

36 96

niedrigsterRang

mittlerRang

höchsterRang

Erosion: Median: Dilatation:

M


3.2.2.9 Median Filter

Median value med{f(k)} of a sequence f(k)

Example: f1(k)= 11, 12, 11, 9, 13, 15, 14

med{f1(k)} = 1212

Example: f2(k)= 13, 12, 13, 14, 13, 13, 13

med{f2(k)} = 1313

Effect

smoothing

elimination of small sporadic image distortions

Rules

med{c+f(k)}=c+med{f(k)}

med{c f(k)}=c med{f(k)}


3.3 Algebraic transformations

An algebraic transformation combines two input images to an output image:

b3(r,c) = f (b1(r,c), b2(r,c))

Addition:

b1(r,c) b2(r,c) b3(r,c)=b1(r,c)+b2(r,c)

Subtraction:

b1(r,c) b2(r,c) b3(r,c)=b1(r,c)-b2(r,c)


3.3 Algebraic transformations - applications

Applications

Noise reduction by multiple image capturing and averaging

Capturing object and background (b1) and background alone (b2) and subtraction

Movement detection by subtraction of two images captured at different times

Color composite and color transformation

In special cases algebraic transformations can be replaced by point transformations, e.g.

b1(r,c) any image and b2(r,c) = const. r,c

b1(r,c) = b2(r,c) r,c

n

kk 1

1b r,c b r,c r,c

n


3.3 Algebraic transformations - Example

red green blue

gray

Generation of a monochromatic image from a true color imageggray = 0,299 gred + 0,587 ggreen + 0,144 gblue


3.3 Algebraic transformations - Example

Generation of a monochromatic image from a true color image

ggray = f (gred, ggreen, gblue)


3.3 Algebraic transformations - Example NDVI Normalized Difference Vegetations Index (NDVI)

bNDVI = (binfrared - bred) / (binfrared + bred); [-1;+1]

„rule of thumb“: as higher NDVI, the more vital is

vegetation

Bred

Binfrared

BNDVI

Pseudo color

NDVI


3.3 Algebraic transformations – Example: motion tracking

B2

B = B1 - B2 +128

B1


3.4 Image pyramid representation (1)

Storage of the image information in different resolution levels Multi-scale sub-sampling of the image (often performed with a factor of 2)

Problem: Aliasing

Procedure for the first level E1:

Definition of the scale factor

Smoothing the image E0 with a low-pass filter, so that frequencies over a certain threshold are eliminated

Sub-sampling the smoothed image to generate E1

Further levels are obtained by replacing E0 by E1, smoothing and then sub-sampling E0 …


3.4 Image pyramid representation (2)

Applications Memory saving: the higher resolution levels are often not needed.

Hierarchical algorithms:

Approximations in lower resolution levels

More accurate computation/results in higher resolution!

Efficiency increase

Remarks For its interesting frequency

properties, the Gaussian filter is often used!

Gaussian pyramid

Instead of discrete levels, a continuous scale factor could be used

scale-space representation

The total memory needed to store an image pyramid, generated with a sub-sampling factor of 2, is increased only with the 1/3 of the memory required for the original image.



Image Processing4 Segmentation

2014 WS

Uwe Stilla

[email protected]




Image Processing

Introduction



Segmentation



Feature extraction


4 Segmentation

Pixel based operators

Simple threshold

Histogram based threshold

Dynamic threshold

Hysteretic threshold

Region based operators

Difference to start point

Difference to mean value

Difference to neighborhood

Low-level model based operators

Local operators

1d case

2d case

Global operators (Hough-Transformation)


4 Segmentation (1)

Segmentation

Definition (Haralick und Shapiro, 1992): ”An image segmentation is the partition of an image into a set of nonoverlapping regions whose union is the entire image. The purpose of segmentation is to decompose the image into parts that are meaningful with respect to a particular application“

In image analysis often certain objects or relevant areas has to be quantitatively described. For this purpose the objects have to be extracted from the hole image. Formally spoken: pixels of region R1 have to be composed using a uniformity condition p and to distinguished from region R2.

Goal of segmentation:Splitting of images into meaningful image parts

R1: ObjectR2: Background

p: Uniformity condition

R1/p

R2


4 Segmentation (2)

Example

Input: image, if necessary preprocessed, e.g. by image point operation, filtering

Output: different image areas (”regions“, ”equivalence classes“) which fulfill certain conditions (e.g. foreground = dark image areas, background= bright image areas)

Segmentation is a decision process about (important and unimportant) image contents


4 Segmentation (3)

Approaches:

Point based approach: Search all points or point sets that apply to criterion p.

Check each pixel, if it satisfies a specified (homogeneity) criterion and assign the pixel to the according region.

For the check, the attribute of the pixel is necessary. Often the grey value or color of the pixel is used, but others are possible too like: local neighborhood or global mathematical models.

Region based approach: Determine the crossover – border of the area of validity of p.

Segmentation operations can combine several criterions to check the pixel’s attributes (i.e. gradient strenght and gradient direction). Dependent on the rules for combining criterions pixel can be assigned to different regions. Resulting overlapping regions should be seperated afterwards.


4 Segmentation (4)

Coding of the results of the segmentation

binary image for two equivalence classes (foreground and background)

”Label“-image for more equivalence classes (every grey value corresponds to a class), or several binary images (allows to describe overlapping equivalence classes)

Targets and applications

Data reduction(less grey values)

Simple description of image content

pre-stage of vectorization and feature extraction (points, lines, regions)

Focus on interesting image parts

pre-stage to object recognition / image interpretation

In this chapter mainly grey values are used as examples. The methods are usable for other features without any limitations.


4.1 Pixel based procedures

considering a pixel as a isolated unity (almost) without taking into account the local neighborhood of a pixels

Segmentation is based on featured related to the pixels: grayvalue, colorvalue, edge magnitude, edge direction, etc.

Simple Threshold

Histogram based Threshold

Dynamic Threshold

Hysteretic Threshold


4.1.1 Simple threshold

Assumption: a searched object has in the (pre-processed) input image a homogenous grayvalue, which do not occur in the hole background exact knowledge about brightness distribution of the image required

Segmentation of image B(r, c) by using one or more thresholds. Often a lower (su) and upper threshold (so) is used

absolute, global thresholds

segmented region R is defined as: R = {(r,c) | su ≤ b(r,c) ≤ so}

Bild Segmentierung: su = 130, so = 180


4.1.1 Simple threshold - example

so=176 so=190 so=197so=162so=148


4.1.2 Histogram based threshold

same as simple threshold, but the value is automatically taken from the histogram of the image

assumption: objects and background have homogeneous, but different grayvalues, which are dominant (= great occurrence, local maxima in the histogram)

method:

calculation of the histogram

determination of local minima in the histogram ( = thresholds)

segmentation by thresholds

advantage compared to simple threshold:

automatic determination of thresholds

relations in brightness or contrast can change as long as grayvalues of objects and background have different areas in the histogram.

disadvantage: possibly many minima are not significant over segmentation by many thresholds

(remedy: smoothing of histograms)


4.1.2 Histogram based threshold - example

Histogram bases segmentation: Coding of equivalence classes in grey scale

thresholds from original histogram55 equivalence classes

thresholds from smoothed histogram 6 equivalence classes


4.1.3 Dynamic threshold

assumption: searched object has a homogeneous grayvalue which differers locally between object and background relativ, global threshold (difference between object and background)

brightness of background must be locally estimated in the image

method:

estimating the background by intense smoothing(rule of thumb: filter mask twice as large as the searched object)

Segmentation based on the difference per pixel between original and smoothed image

let:

B (r,c) original imge

G (r,c) smoothed image

s relative threshold (minimal difference between object and background)R segmented region

then:

R = {(r,c) | B (r,c) − G (r,c) s} for segmentation of bright objects

R = {(r,c) | G (r, c) − B (r,c) s} for segmentation of dark objects


4.1.3 Dynamic threshold - example profile

s

b(r,c)


4.1.3 Dynamic threshold – example image

original image

smoothed image

segmentation with dynamic

threshold

segmentation with global threshold


4.1.3 Dynamic threshold – pros and cons

Advantages:

Compensation of brightness differences

Knowledge of relative brightness differences of object an background sufficient

Usage of local neighborhood for background estimation

Disadvantages:

More computations, critical only for real-time applications

Object size has to be known

Objects have to be positioned isolated. Otherwise the background estimation fails

Possible wrong segmentation on edges

In general

Better solution compared to global threshold

Universal usability


4.1.4 Hysteretic threshold

A generel problem of threshold operations: seldom a correct threshold can be adjusted ”ad hoc“

oversegementation, if threshold to generous,

undersegmentation, if threshold to conservative

s = 175 (oversegmented)

Example: thresholds for road marking

s = 220 (undersegmented)


4.1.4 Hysteretic threshold (2)

Often grayvalue intervals are known, which cover in any case the grayvalues of the searched object or never cover the grayvalues of the searched object pre-segmentation into ”reliable foreground“ (s1) and ” reliable

background“ (s2)

afterwards the remaining ”unreliable“ grayvalues will be assigned to one of both pre-segmented regions according to a pre-defined condition.

Most commonly used condition for assignment to the foreground:

unreliable grayvalues must be neighbored to at least one reliable grayvalue

starting from a reliable grayvalue all unreliable grayvalues will be iterativly tracked. Possible additional condition: shortest path from a unreliable to a reliable grayvalue not have to exceed maximal length (lmax).

All remaining unreliable grayvalues will be assigned to the background.


4.1.4 Hysteretic threshold - pros and cons

Advantage:

fixed threshold will be avoided

minor variations of brightness in the image can be compensated

local neighbourhood will be considered

Disadvantage:

depending on (hysteresis) condition -> longer calculation time

global thresholds, major variations of brightness in the image cannot be compensated

Common applications:

segmentaion of homogeneous regions

determination of prominent edges from image of the gradient magnitudes


4.1.4 Hysteretic threshold - example

Results for foreground with hysteresis and

lmax=30 (black)

Reliable background

g≥s2=175 (black)

Unreliable greyvalules

s1≥g≥s2 (black)

Foreground: assigned unreliable greyvalues

Background: assigned unrealiable greyvalues

Reliable foreground

g≥s1=220 (black)


4.2 Region based procedures

basic idea: most objects in the image are represented by larger connected regions ( = connected pixels) considering of pixel neighborhood during segmentation

used approach : region growing

Initialization (t0): selection of start point first pixel of the region

examination of neighbored pixels concerning a pre-defined criterion of homogeneity related to the features of the start point adding of ”positiv“ tested pixels to region

iterativ growing of region (ti i=1,2, . . .): examination of currently neighboring pixels

abort iteration, if no neighbouring pixels fulfill the criterion or all pixels of the image are processed


Difference to average value

Difference to neighboring values


4.2 Region based procedures (2)

Remark: Only region growing algorithms are introduced here. There are many other – more complex – methods of region based segmentation.

segmentation

examined region

growinginitialization

current region


4.2.1 Difference to start point


Criterion for region growing: magnitude of grayvalue difference ΔgS,(r,c)

between start point S and checked pixel (r, c) segmentation of homogeneous regions

Advantage:

fast calculation

stops at prominent edges

Disadvantage:

sensitive to noise

grayvalue of start point possibly not representative for whole region ”holey“ or ”frayed“ regions as segmentation result

For time-critical applications using simple images


4.2.1 Difference to start point - example

Segmented region

(black)

Initialisation – maximum difference ΔgS,(r,c)max=18

Segmented region

(black)

Initialisation – maximum difference ΔgS,(r,c)max=18


4.2.2 Difference to mean value

difference to mean value

Criterion: magnitude of grayvalue difference Δgμi,(r,c) between checked pixel (r,c) and average grayvalue μi of segmented region at iteration i.Update average value, if all examined pixels of one iteration are processed. segmentation of to a large extend homogeneous regions

Advantage:

less sensitive to noise compared to difference to start point

Disdvantage:

possibly cross the edges

calculation time is longer

segmented region (black)for lane border maximal difference ΔgS,(r,c)max=18

segmented region (black)for road

maximal difference ΔgS,(r,c)max=18


4.2.3 Difference to neighboring points

Difference to neighboring points

Criterion: magnitude of smallest grayvalue difference Δg min,(r,c)

between checked pixel (r c) and neighbored (maximal 8) pixels, which belongs already to the region. segmentation of smooth areas, which must not be homogeneous,

e.g. grayvalue wedge

Advantage:

fast calculation

stops at prominent edges

result is independent on position of start point in area which has to be segmented, because of considering only differences of neighbored pixels.

Disadvantage:

tends to cross weak edge areas (see example)

sensitive to noise (but less than difference to start point)

for more application useful, more flexible


4.2.3 Difference to neighbored points (2)

Problem with difference to next neighbour (maximal difference Δg min,(r,c)max =1):

two grayvalue wedges in opposite direction and a black disconected bar

Original resulting label image with 3 regions

segmented region (black) for lane border maximal difference Δg min,(r,c)max = 3

segmented region (black) für Straße (rechts)maximal difference Δg min,(r,c)max = 3


4.3 Low-level model-based approaches

Goal: Description of discrete image information with simple mathematical functions derived from curves and areas

Approaches:

Local: Parameters of the function are calculated for every pixel from ist direct neighborhood

Global: Parameters are calculated taking into account the complete image

Calculation of features from the functions (extrema, inlection, etc.)

Segmentation = Determination of areas, where the image functions fits the calcualted features

Local approaches

Determination of local function properties: Example with local maximum, but transferable to other properties like minimum, inflecioni point, etc.

Global approachesDetermination of global function propertiesEsample with Hough transformation


4.3.1.1 1d case

Feature from image function

Test, if investigated pixel is local extremum in its neighborhood

Segmented region R defined as:R = {x | g(x) = max(g(x + n)), n = −N, . . . ,N} (meist N = 1 oder N = 2)

Main application:Postprocessing of edge filtered images

Edge strenght and gradient direction per pixel given

Test, if investigated pixel x0 is local maximum in gradient direction (where necessary interpolation of neighboring grey values for x−1 und x1, graphic).

Pixel that are no maxima are supprest Thninnung pf blurred edges =

Non-Maximum-Suppression

g(x)


4.3.1.1 1d case – non-maximum-suppression


4.3.1.1 1d case (2)

Attributes from approximated function (here: parable)

Adaptation of second order polynom (parable) in each pixel. Local maximum, if apex of parable (=maximum) falls into pixel

Determination of maximum with sub-pixel accuracy

y=g(x)

Formula parable:

y = a + bx + cx2 y‘ = b + 2cx y‘‘ = 2c

Determine coefficients a, b, c with grey values y−1, y0, y1 at x−1, x0, x1 (mit x−1 = −1, x0 = 0, x1 = 1)

20 0 0 0

2 '1 1 1 1 1 0

2 ''1 1 1 1 0 1 0

y a bx cx a a y

1y a bx cx a b c b y y y

21 1

y a bx cx a b c c y 2y y y2 2


4.3.1.1 1d case (3)

coeffizients b and c can be determined with the 1. and 2. derivative at position y0

Determination of apex xS of the parable (from coefficients at position x0):

xS is local maximum, if y‘‘ = 2c < 0 and xS within the borders of the investigated pixel.

Applicattions: like before, but with higher accuracy

Camera calibation, object measurements in industrial IP, extraction of roads from airborne images.

'0

S S ''0

yby ' b 2cx 0 x

2c y


4.3.1.1 1d case (4)

original

subpixel accurate edge positionEdge strength


4.3.1.2 2d case

Features from image functions

Test, if investigated pixel is local maximum (minimum) for a predifined neighborhood (2D Non-Maximum-Suppression):

For the segmentation of linear structures: Grey value mit describe a local maximum in horizontal and/or vertical direction

Test of 4 pixels

For the segmentation of local maxima: Grey value must be local maximum within a 3 × 3 or bigger neighborhhood

Test of 8 pixels

Applications: Non-Maximum-Suppression, if no information about the direction is available

4 neighborhood 8 neighborhood


4.3.1.2 2d case (2)

Non-Maximum-Suppression without information about the direction

Grey value must be maximum in vertical and/or horizontal direction


4.3.1.2 2d case (3)

Features from approximated functions

Image function is seen as „grey value mountains“: Adaptation of planes of second order or higher in3×3 or bigger neighborhood of each pixel. Normally formalized as adjustment:

observations: grey values

unknown: coefficients of the plane function

Determination of differential geometric values from calculated planes (Extrema, like Maxima, Minima, flat points, saddle points): ”Topographic Primal Sketch“ Determination of extrema with sub-pixel accuracy


4.3.1.2 2d case (4)

Main applications:

Like before but with higher accuracy (subpixel)

Determination of subpixel center from prominent points determined with point operations

Detection of conspicious image structures like gauge marks, nucleus…)

Topographic Primal Sketch: Struktural complete description of complete image function

Input for different applications in image interpretation: Analysis of digital terrain models (DTM), analysis of cellular structures …

Example: valley line and ridge line from DTM with differential geometry, i.e. watersheds of glaciers and creeks.


4.3.1.2 2d case (5)

DTM-Analysis Vernagtferner: valley and ridge lines: white and black

DTM-Analysis Vernagtferner: 3D view of valley and ridge lines


4.3.2 Global approaches – Hough Transformation (1)

Determination of global attributes of the image function normally for linear structures (after edge filtering)

Example: Segementation of staight lines (Hough-Transformation).

Useable for other structures like curves and circles (generalised Hough-Transformation)

Hough-Transformation:

Parameter description of a line with d = r · sin(α) + c · cos(α)

Determination of a and d per pixel:

a from edges detection,

d from line parameterisation



Calcualtion of 2d histogram p(a,d) with pixel coordinates (r,c) saved. Sometimes weighting of histogram entries with the edge strenght→ Strong maxima in the hinstogram correspond to strong straight lines in the image

Determination of strong histogram maxima; the pixel coordinates saved at the found maxima are the segmented edges

If only points on edges are given with no edge direction, the (distance-) curve of the point is added to the histogram



Determination of noticable histogram maxima. The pixel coordinates stored there (and in their direct neighborhood) are the segemented noticable straight image edges.



Advantages

Fast calculation. Usable for real-time applications

Lines may be discontinuous (important for road mark extraction and detection of partial occluded objects)

Disadvantage

For functions with many parameters the maxima search can be very complex (ellipsoid: 5 parameters 5D histogram).


4.3.2 Example – segmentation of straight edges

Original

Edge strenght (Sobel filter)

Edge direction

Non-Maximum-Suppression to edge direction

Hysteresis thresold

Hough-Transformation



Image Processing5 Binary image processing

2014 WS

Uwe Stilla

[email protected]




Image Processing

Introduction



Segmentation



Feature extraction


5 Binary image processing

Neighborhood

Euler number and holes

Morphologic operations

Basic operations

Dilatation

Erosion

Opening

Closing

Mean axis

Distance transformation


5 Binary image processing (2)

Intensity values of pixels are limited to 0 and 1 for binary images (quatization).

Example:

Pixel with value 1

relevant information: set pixels

Total quantity of these pixels: image foreground

Pixel with value 0

Total quantity of these pixels: image background

Binary image result from:

Segmentation (without subpixel accuracy)

manual digitizing

Scanning of documents / maps

Reading of a GIS layer

Binary image often result in very efficient algorithms


5.1 Neighborhood and connected component

In image defferent types of neighborhood can be defined, i.e:

4 neighborhood

8 neighborhood

Definition connected component

A path from position (i0, j0) to position (in, jn) is called a pixel sequence based on the k neighborhood, if (ih−1, jh−1) is connected to (ih, jh) by such a k neighborhood. (h = 1, . . . , n)

A pixel set S is called a pixel set based on the k neighborhood, if

Two random pixels of S can be connected by a pixel sequence based on the k neighborhood, which lies completetly in S, or if

S has only one pixel.

A pixel set S is called connected component based on the k neighborhood, if:

S is a pixel set based on the k neighborhood,

The value of the image elements of pixel set is equal (often: image foreground), and

No other pixel set S0 exists, containing S

A connected component contains the maximum pixel set based on the k neighborhood (of the image foreground).


5.1 Connected component - example

Binary image with set (1) and unset (0) pixels

Two connected components based on the 8 neighborhood.

Four connected components based on the 4 neighborhood


5.2 Euler number and holes

Euler number eis the number of connected components of an image B minus the number of holes in B.

Holeis a conncted component of hte background which is not conneted to the image border.

The calculation of the Euler number uses either the 4 neighborhood for the foreground and the 8 neighborhood for the background or vice versa.

e4(B1) =

e8(B1) =

e4(B2) =

e8(B2) =

B1 B2


5.3 Morphologic operations

Mathematic morphology is based on

Image processing based on shape information

Operations with set theory

Simplify images with

Isolation or amplification of relevant shape information

Elimination of disturbances

Structuring element is the base element of mathematic morphology

It consists of a set of pixels

Rerefence point of a structuring element is the point of reference of the morphologic operation

Structuring element is in general much smaller than the binary image

Examples:


5.3.1 Erosion and dilatation on binary images

Erosion

Bright objects (g=1) on dark background (g=0) will decrease

For binary images bb‘(x,y) small objects (g=1) disappear, e.g. 3x3

Dilatation

Bright objects (g=1) on dark background (g=0) will increase

For binary images bb‘(x,y) small objects (g=0) disappear, e.g. 3x3

)1,1(),1()1,1(

)1,(),()1,(

)1,1(),1()1,1(),('

yxbyxbyxb

yxbyxbyxb

yxbyxbyxbyxb

bbb

bbb

bbbb

)1,1(),1()1,1(

)1,(),()1,(

)1,1(),1()1,1(),('

yxbyxbyxb

yxbyxbyxb

yxbyxbyxbyxb

bbb

bbb

bbbb


5.3.1 Erosion, Dilatation and Median

:0

:1

a) section of image B c) Dilatation: B M

b) Erosion: B M d) Median

M

duality:

B M = B M

B M = B M


5.3.2 Opening and Closing

M

B B

:0 :1

B M = (B M) M B M = (B M) M

DilatationB M

ErosionB M

ClosingB M

OpeningB M


5.3.2 n times Opening or Closing

M

B B

:0 :1

B MB M

(BM)M (BM)M


5.3.3 Morphology - binary images, contour

M

:0 :1

Erosion: B M

B

contour: B = B / B M

/ set difference

B = B / B M

= B B M

= B B M


5.3.4 Morphology - binary images, skeleton

M B nM

B

n=0 n=2n=1

(B nM) M

(B nM) / ((B nM) M )

Simple skeleton: S = [(B nM) / ((B nM) M )]n=0

nmax

S

disadvantage:

nmax: maximal size after B is eroded to an empty set


5.4 Centerline (I)

Elongated regions in binary images often are reduced to topological (connected) lines with one pixel width (also called skeleton)

This is also done for binary edges extracted from grey value images using thresholds. But for edges non-maximum-suppression is preferred because the center of an edge segmented using a threshold is not necessarily the maximum of the gradient.


5.4 Centerline (II)

Centerline: Set of all points within an object satisfying the condition that at least two points exist on the object‘s border with the same shortest distance to the centerline.

Calculation of the centerline:

Raster based (here without subpixel accuracy)

Vector based (here not discussed)


5.4.1 Centerline calculation – raster based

„steppe-file - algorithm“

Fire is sparked at the border of an “ homogeneous“ object, which is burning inwards with constant speed. The centerline corresponds to the extinction points of the flame fronts.

Rasterized image is removed in layers around the object until only the topological connected line remains ( = centerline).

Iterative execution of the erosion with different structuring elements.


5.4.1 Centerline calculation - vector based

Computational geometry: Calculation of the centerline directly from the surrounding polygon(more precise, computational effort)

Remark:If the border of the objects consists of straight line pieces (polygon), then the centerline only constists of straight line and parable pieces.


5.4.2 Centerline - problems (1)

Attention: Results of centerline transformation do mostly not correspond to the optimal geometric model:

Node shift: Line intersections are shifted

Node bridges: Line intersections are falsified – two ore more nodes instead of one

This is not an error of the method but inherent to the model of the centerline.

Node bridge Node shift


5.4.2 Centerline - example

c) Closing and Opening

f) Centerlined) Selection of big connected components e) Dilatation

b) Segentation with histogram threshold

a) Original image



The segmented image (b) is processed with thinning algorithms of Hilditch (c), Tsuruoka (d), Deutsch (e) and Tamura (f).

The results show the different influence of disturbances of the contour on the algorithms and the similar influence of inclusions.

One possible solution to reduce the disturbances of the segmented image are morphologic filling operations like dilatation and erosion

a b fc ed



Segementation with simple threshold so

so=176 so=190 so=197so=162so=148

Erosion and Dilatation

Centerline


5.5 Distance transformation (1)

Goal: Image containing the shortest distance of every pixel to a line segment. The line segment can be the border of the given object respectively the border of the connected component.

Centerline is generated by chaining of the realtive maxima.

Attention: All pixels of the centerline are maxima of the distance transformation, but not vice versa! That‘s why the centerline is often calculated using thinning and only the distance values are calculated from the distance transformation.

Distance metric determines how accurate a circle is approximated (how isotrop the metric is: Metric non isotrop different distances in different directions).

In general, filters are the faster and less isotrop, the more local they operate and the less they weight.



Generation of distance map

Pixel are given the distance to the next edge pixel as greyvalue in the distance map

Edge pixels are given the grey value 0

Efficient calculation with non-iterative algorithm

Distance map is generated for model image and search image

Object recognition by matching of distance maps (z.B. correlation)

=> Advantage to grey value matching

homogeneous regions (without edge pixels) get additional information (distance to nearest edge)!



8 neighborhood (Chessboard distance)

3 3 3 3 3 3 3 3 3 3 3 33 2 2 2 2 2 2 2 2 2 2 33 2 1 1 1 1 1 1 1 1 2 3 1 1 13 2 1 * * * * * * 1 2 3 1 * 13 2 1 1 1 1 1 1 1 1 2 3 1 1 13 2 2 2 2 2 2 2 2 2 2 33 3 3 3 3 3 3 3 3 3 3 3

4 neighborhood(Cityblock distance)

6 5 4 3 3 3 3 3 3 4 5 65 4 3 2 2 2 2 2 2 3 4 54 3 2 1 1 1 1 1 1 2 3 4 13 2 1 * * * * * * 1 2 3 1 * 14 3 2 1 1 1 1 1 1 2 3 4 15 4 3 2 2 2 2 2 2 3 4 56 5 4 3 3 3 3 3 3 4 5 6

Kampfer(3,4)-distance: 4/3≈√2,i.e. Euklidean distance over diagonal is approximated

2 11 10 9 9 9 9 9 9 10 11 1211 8 7 6 6 6 6 6 6 7 8 11

10 7 4 3 3 3 3 3 3 4 7 10 4 3 49 6 3 * * * * * * 3 6 3 3 * 3

10 7 4 3 3 3 3 3 3 4 7 10 4 3 411 8 7 6 6 6 6 6 6 7 8 1112 11 10 9 9 9 9 9 9 10 11 12


5.5 Distance transformation - example

Distance matrices of the line picture of the number five. The distance values are coded as grey values (Euklidian distance). The initial point set of the lines is set to black; the higher the distance value, the brighter the grey value.

Image of reference line segements Image of line segments with noise



Image Processing6 Vectorization and geometric primitives

2014 WS

Uwe Stilla

[email protected]




Image Processing

Introduction



Segmentation



Feature extraction


6 Vectorization and geometric primitives (1)

Input:

Plane or line structures as binary image, for example region as segmentation result.

Eventually additional information like edge direction or edge magnitude of a pixel

Output:

Vector data like polygons, straight lines, circles and ellipses-> so called geometric „primitives“

Goal:

Data reduction with focussing on relevant features

Extraction of more significant structural and topological information like start point and end point of a polygon, neighborhood relation of regions, et al.

-> Symbolic description of important image content with simple geoemtric elements.



Applications:

Precise measurements in industrial image processing

calibration

Prestep for methods in automatic image interpretation like grouping and graphs

Skeleton as binary image

Approximation with parable line

Region as binary image

Approximation with circle



Often two-step processing for geoemtric primitives:

Determination of topological explicit pixel chains, so called „contours“ „Vectorization“, „Raster-vector-conversation“

Simplification of pixel chains with fitting of polygons, lines, circle and ellipsoid elements Determination of geometric primitives


Approximation with polygons

Kontur 1 (r,c):0 01 11 22 33 3Kontur 2 (r,c):3 34 45 5Kontur 3 (r,c):3 33 24 15 1

Vectorization: contours


6.1 Vectorization of linear structures

Goal

Linking of (centers of) pixels of skeletons

Storage with topological explicit data structur

Results

Contours with end nodes, inner nodes and cross nodes

Approach

Linking

Simple Linking

Direction dependent Linking

Chain-Code (Freeman-Code)


6.1.1 Linking - simple Linking

Background: Skeleton as binary image

Approach:

Determination of start points: Search image for endpoints and cross points of the skeleton (e.g. with masks)

Linking: Follow the skeleton from the start point until the next start point is reached

Break, if all start point are processed

Advantage:

Explicite description of the skeleton, no loss of information

Disadvantage:

Needs much memory


6.1.1 Linking - simple Linking


Contour 1 (r,c):0 01 11 22 33 3Contour 2 (r,c):3 34 45 5Contour 3 (r,c):3 33 24 15 1

Result: Contours

Masks for cross points

Found start points

Start of Linking Linking (1st step)


6.1.1 Linking – direction dependent Linking

Background: Skeleton as binary image Direction information per pixel, p.e. from edge detection

Approach: Determination of start points: like simple linking Linking: Check if the difference of the directions of neighboring pixels is smaller

than a given threshold. smaller (equal): Linking greater: End of contour and definition of a new star point (p.e. in the break

between the two pixels) Continuation with new contour Break, if all start point are processed

Advantage: explicite desription of the skeleton, no loss of information Better adaptation to image information

Disadvantage:

Needs much memory Possible fragmentation at contours with uncertain definition of th direction, p.e.

in the direct neighborhood of cross points


6.1.1 Linking – direction dependent Linking

Contour 1 (r,c):0 01 11 21.5 2.5Contour 2 (r,c):1.5 2.52 33 3Contour 3 (r,c):3 34 45 5Contour 4 (r,c):3 33 24 15 1

Result: more contours

Skeleton with directional information and start point

Addintional start point for difference of direction

> 60°


6.1.2 Chain-Code (Freeman-Code)

Method related to simple linking but with modified representation of contours

Determination of start points (like simple linking)

Linking: Store coordinates of start point and direction zu next pixel

Coding of direction from 0, ... , 7

7 0 1

26

5 4 3

Contour 1:r,c: 0 0dir: 3

234

Contour 2:r,c: 3 3dir: 3

3Contour 3:r,c: 3 3dir: 6

54

Result: contoursStart pointsDirection code


6.1.2 Chain-Code (Freeman-Code)

Advantages:

Only few memory space necessary (coordinates of start points and 3 bit per additional contour point)

Simple identification of breaks in a contours directly out of the chain-code (alike for straight or parallel line segements, if their direction is 0°, 45°, 90°,…)

Disadvantage:

less explicite

Time consuming because the original coordinates have to be reconstructed for many algorithms and applications (p.e. Display contour).


6.2 Vectorization of plane structures

Goal Surrounding polygon with similar area than original region

Result Always closed contours

In principle similar approach like for linear structures Determination of border pixels of a region

Store pixel coordinates as contour

Problem Surrounding polygons of neighboring regions do not necessarily tangent

(see regions A and B) Changes in the topology

Solution Store coordinates of outer pixel edges instead of pixel coordinates

more memory space, but normally, the perpetuation of the topology is critical.



Two regions Surrounding polygons with stored pixel

coordinates

Surrounding polygons with stored pixel edges

AB



For more than one region often post processing

Splitting at cross points

Elimination of overlapping (redundant) contour segments

Additionally hybrid data structure

per contour: Store indices of neighboring regions

per region: Store indices of (maybe splitted) surrounding polygons

This is necessary p.e. for geo-information systems

Example border of estates

„Which regions are separated by the given contour?“

Example Landmanagment

„Which regions are neighbored to a given region?“

Example borders of a glacier

„Which borders enclose a given region?“



-Split contour at cross points

- Eliminate redundant segements (contour „2“)

Contour 1:- coordinates (r/c):.. , .. , ..

- neighboring regions:A

Contour 2:- coordinates (r/c):.. , .. , ..

- neighboring regions :

A , BContour 3:...

Region A:- coordinates (r/c):0/2, 0/3, 1/1, 1/2, ..

- contours:1 , 2 , 3

Region B:- coordinates (r/c):.. , .. , ..

- contours :5 , 4 , 2

...

Data structure for contours

Data structure for regions

13

45

A

B2


6.3 Determination of geometric primitives

Goal

Simplification from contours to geometric curves like polygons or circle and ellipsoid segments Replacement of many contour points with few curve parameters opt. With start unf end points (datareduction) Accentuation of fundamental geometric structures (focussing)

Contour approximation

With polygons

With line segments

With curve and ellipsoid segements

combined


6.3.1 Polygon approximation

here: „Ramer“ or „Douglas-Peucker“ algorithm

Conditions

Approximating polygon must not have a distance bigger than d to the contour at every contour point.

Recursive procedure:

Linking of start and end points of the contour or a contour segment with a line segment initial polygon

Determination of contour point ps with the biggest distance ds to the line segment

Test: ds > dmax

no: line segment approximates the contour adequate

yes: P is inserted as new polygon point Polygon constisting of two line segments

Continuation of the procedure for every line segment of the polygon

Break if no line segment has a distance bigger than dmax to the contour


6.3.1 Polygon approximation – splitt-algorithm of Ramer



p peb

ps

Ps

ps



Advantages

Fast and efficient

Resulting polygons are equal to a sub group of contour points no shift of the polygon

Disadvantages:

The surface area is in general getting smaller for closed contours

Polygon point are not always in the corners of a contour ( = points with maximum curvature)


6.3.1 Polygon approximation – sequential splitt-algorithm

Ramer algorithm

Sometimes oversegmentation sequential splitt-algorithm

Approximation of contour sequentially with line segments

Conditions

Minimal lenght nS of the line segements nkl of the contour

Break if quality criterion dappr is reached


6.3.1 Polygon approximation – sequential splitt-algorithmps=25

Pk=1 p

l=47

ps=17

pk=1

pl=25

pk=1

pl=17

pl=47

pk=17

ps=33

pl=33

pk=17

pb

pe

p17

p33

pl=47

pk=33

n step

p peb

kli Pp

ikl

appr pdn

d )(1 2

)(max)( ; ilik

skls pdpdPp

max min or appr kl sd d n n


6.3.2 Approximation of line segments

a) Grey value image b) Binary image (after threshold operation)

c) Contour image (Contour chains)

d) Approximation with line segments. Short contours are not approximated.


6.3.2 Approximation of line segments

Simple case: Regression line of a contour

Parametrization: Hesse-Normalform

a xi + b yi + c = 0 mit a2 + b2 = 1

Adjustment:

Observations: contour points xi , yi

Unknown: Parameter of the line a, b, c

Condition: a2 + b2 = 1

Approximation: deduce from contour points

Determination of start and end points of the line segment:

Perpendicular from start and end point of the contour to the line


6.3.3 Approximation of circle and ellipsoid segments

Similar methold like line segments:

Parameterization:

Circle: (xi – xm)2 + (yi – ym)2 – r2 = 0

Ellipsoid: axi2 + bxiyi + cyi

2 + dxi + eyi + f = 0

Adjustment:

Observations: Contour points xi , yi

Unknown: center point xm , ym and radius r

respectively params a,b,c,d,e,f

Conditions for ellipsoids: b2 – 4 a c < 0

in Implementation: b2 – 4 a c = –1

Approximation: deduce from contour points

→ generally a robust adjustment is essential

Determination of start and end points of circle and ellipsoid segments like for lines


6.3.3 Approximation of circle and ellipsoid segments

Segmented objects of a calibration field

Contour (red) and fitted circle (green)

Contour (red) and robust adjusted circle (green)


6.3.4 Combined Approximation

Usual problem:

No a priori knowledge about the best approximating geometric primitives for a contour. combined processing necessary

Approach:

Polygon approximation and splitting of contour at polygon points many short contour segments in curved sections

Approximation of a circle or ellipsoid segment to the aggregation of two neighboring contour segments If the error of the approximation gets smaller, replace the contour segtment with the circle or ellipsoid segment

Repeat until all contour segment are replaced and no replacement is successful

Post processing: Combination of end points of neighboring primitives (p.e. snapping)

Final remark:

Small disturbances can impede the approximation of contours with few geometric primitives because then the approximation error does not become smaller for circle and ellipsoid segments. Often grouping algorithms have to be performed after to handle these problems in a better way.


6.3.4 Combined Approximation



Image Processing 7 Feature extraction

2014 WS

Uwe Stilla

[email protected]




Image Processing

Introduction



Segmentation



Feature extraction


7.1 Introduction

Similar task as segmentation (classification of the image in important andnon-important regions, see chapter 5), exactly: Extraction of geometricprimitives Feature points Linear structures (lines and edges) Regions (homogeneous grey values or texture or color or…)

Feature extraction focusses less on separation of foreground andbackground (like segmentation) but more in specific geometric, radiometric or topologic properties of primitives

Main task is the description of important image content with points, linear structures and regions (in contrast to segmentation: binary or labelledimage)

necessary: „Model“ of a primitive: „Which criterions distinguish a specific primitive like p.e.

a line from other image content?“ Extraction scheme: point transformation, filtering, transfer from raster to vector

data or the like, mostly combination of such operations.

Goal and application: Focussing on important image content Data reduction Basic of many algorithms in digital photogrammetry, remote sensing and image

analysis / interpretation


7.2 Extraction of feature points (1)

Requirements for the model of feature points:Description of points, which are

conspicuous (rare)

invariant to geometric abd radiometric transformations (noise insensitive)

Easy to localize / indentify

Example: Moravec operator

Belongs to the class of so called statistic interest operators

Creates an interest value V(r,c) per pixel (r,c)

Calculation of the mean value of the four directional quadratic grey value difference V0, V45, V90, V135 in a windows of size K / L.



Interest value V is calculated from:

V = min{ V0, V45, V90, V135 }

A pixel is counted as feature point, if V is greater than an given threshold t

5



Determination of threshold t for V:

P.e. from relative cummulativehistogram s(V) of all values V,calculated for the complete image window(p.e. the best 30% of all points: α = 0.3 ⇒ t = s−1 (1−α )

If the points should be distributed homogeneous over the image: Kachelung des Bildes und Anwendung des Schwellwertes pro Kachel

Requirement for:

Assignment of homologous („equal“) points to determine the relative orientation of image pairs for 3d reconstruction from stereo images

Object localization

Determination of center of gravity of rings, discs, reticles …

Application example: Identification of homologous points in photogrammetry to determine the relative orientation of image pairs in images blocks.





7.3 Extraction of lines and edges (1)

Lines: elongated bright or dark image structures

Edges: linear image structures with high grey value changes across the edge direction

Image edges appear as bright line in the image of the gradient strenght

An algorithm for line extraction is also capable for edge extraction, if it is applied on the gradient image.

Requirements:

High completeness and correctness of the extraction

High geometric accuracy (Subpixel accuracy)

Few parameters; their influence on the extraction should be easy to understand.

Topologic correct data structure (Contours or polygons with unique identification of end and cross points)



Example: Line extraction by Busch

Model: several line points show big curvature across the line direction.

Method: local, differential-geometric explication of the image function (see chapter „low-level models “) for describing the direction and curvature of lines

Procedure:

Smoothing of the image function b(r,c) with Gauß- (Binomial-) filter h(k,l) to suppress noise (see chapter 3.2.2.3)

Calculation of the second derivate of the smoothed image (see chapter 3.2.2.5 )

b f h

rc

db dbb

dr dc

2

rr 2

d bb

dr

2

cc 2

d bb

dc



Determination of direction (r,c) across the line direction in every pixel (see differential geometry)

Detection of line points with adaptation of a 1d parable in direction and determination of the subpixel accurate position (see chapter 4.3.1.1). The curvature (2nd derivation) is a degree for the conspicuity of the line at this position.

rccc rr

rccc rr

b r,c1arctan für b r,c b r,c

2 t r,cr,c

b r,c1arctan für b r,c b r,c

2 t r,c 2

cc rr

1mit t r,c b r,c b r,c

2



Selection of conspicuous line points with threshold, related to (better: Hysteretic threshold, see chapter 4.1.4).

Generation of contours by connecting neighboring line points (see chapter 6.1)






7.4 Extraction of regions (1)

Search for homogeneous image regions:

quasi identical definition to segmentation: homogeneous grey value, texture or color or ...

That means, every segmentation algorithm whose result consists of regions, us usable (especially chapter „Pixel oriented procedures“ and „Regionbased procedures“)

Pre-processing steps to make image region homegeneous before segmentation.

Example: Texture based plane extraction

Goal: Algorithm should distinguish strong textures areas, that means that these areas should be homogeneous bright, the other areas dark

Model: strong textured areas show high standard deviation of the grey values in ther local neighborhood; weak textured region show a low standard deviation.


7.4 Extraction of regions (2)

Method:

Calculation of the standard deviation ( see chapter 3) in a pre-defined neighborhood of a pixel, p.e. within an window of specific size

Moving the mask over the hole image und calculation of the standard deviation for all pixels, that means a image of local standard deviations is generated

Smoothing o the image by median filtering to distinguish homogeneous regions

Segmentation of the filtered image with histogram based threshold, that means no additional parameter necessary

If applicable, post-processing of the segmented region with morphologic operations


7.4 Simultaneous extraction of forest and open areas

18

b) Image of local standard deviation ( 9x9 mask)

c) Median filtering (19x19 mask)a) image

d) Threshold t from histogram minima of the median image (here t = 26)

e) Segmented regions (from median image) overlayed with the original image

f) Post-processing: automatic detection of tree crowns in textured regions

Documents

tum ipc 14 00.ppt [Kompatibilitätsmodus] · PDF filePraxisbuch. Aachen: Elektor ... Haralick R, Shapiro L (1992) Computer and Robot Vision. Vol. 1, ... Image Processing [tum_ipc_14_01]