Colour Image Processing Allan Hanbury PRIP, Vienna University of Technology Favoritenstraße 9/1832...

Colour Image Processing

Allan Hanbury

PRIP, Vienna University of TechnologyFavoritenstraße 9/1832A-1040 Vienna, Austria

hanbury@prip.tuwien.ac.athttp://www.prip.tuwien.ac.at/~hanbury

Contents

• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images

Contents

• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images

Physical Background

• Visible light: a narrow band of electromagnetic radiation → 380nm (blue) 780nm (red)

• Wavelength: Each physically distinct colour corresponds to at least one wavelength in this band.

• Pure Colours: Pure or monochromatic colours do not exist in nature.

• Spectrum: Intensity asa function of wavelength.

• The colour of an object: is the product of the spectrum of the incident light with the light absorption and/or reflection properties of the object.

From http://fuse.pha.jhu.edu/~wpb/spectroscopy/basics.html

Human colour perception

• The human eye does not perceive individual light wavelengths.

• It contains three types of colour receptor (cones) which integrate over parts of the spectrum:

From http://math.ucr.edu/home/baez/physics/General/BlueSky/blue_sky.html

• It is therefore possible to characterise a psycho-visual colour by specifying the amounts of three primary colours: red, green and blue, mixed together.

• This leads to the standard RGB space used in television, computer monitors, etc.

• We specify the levels of R, G and B in the range [0, 1], but they can easily be extended to other ranges (8-bit integers for example).

(0,0,0)

(1,1,1)

RGB

Problems with Processing Colour Images

• When processing colour images, the following problems (amongst others) have to be dealt with:– The images are vectorial → 3 numbers are associated with

each pixel.

– The colours recorded by a camera are heavily dependent on the lighting conditions.

Lighting conditions

Image taken lit by a flash. Image taken lit by a tungsten lamp.

• The lighting conditions of the scene have a large effect on the colours recorded.

• The following four images of the same scene were acquired under different lighting conditions:

Dealing with Lighting Changes

• Knowing just the RGB values is not enough to know everything about the image.– The R, G and B primaries used by different devices are

usually different.

• For scientific work, the camera and lighting should be calibrated.

• For multimedia applications, this is more difficult to organise:– Algorithms exist for estimating the illumination colour.

Contents

• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images

Processing Vectorial Images• A vectorial image has a vector at each pixel. For

colour images, these vectors each have 3 components.• Vectorial images with larger numbers of components

also exist, e.g. in satellite imagery.• There are two ways one can process vectorial images:

– Marginal processing.

– Vectorial processing.

f (x, y) {0,1,…, N} f (x, y) [ {0,…,N}, {0,…,N}, {0,…,N} ]

Gre

ysca

le

Co

lou

r

Scalar process.

Red

Scalar process.

Green

Scalar process.

Blue

Red

Green

Blue

Marginal Processing

• Each channel is processed separately:

Red

Green

Blue

Vectorial process.

Vectorial Processing

Red

Green

Blue

• The colour triplets are processed as single units:

The Problem of False Colours

FalseColours !!

Marginal Median

VectorialMedian

The Problem of False Colours

0B

0V

100R

0B

100V

0R

100B

100V

100R

Marginalmedian

0B

100V

100R

Vectorialmedian

0B

100V

0R

Contents

• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images

Alternative Colour Spaces

• Various other colour representations can be calculated from the RGB representation.

• This can be done for:– Decorrelating the colour channels:

• principal components.

– Bringing colour information to the fore:• Hue, saturation and brightness.

– Perceptual uniformity:• CIELuv, CIELab, …

Red

Green

Blue

T-1

Pro

cess

ing

Processing Strategy

T

Red

Green

Blue

Colour spaces

• RGB (CIE), RnGnBn (TV - National Television Standard Comittee)• XYZ (CIE)• UVW (UCS de la CIE), U*V*W* (UCS modified by the CIE)• YUV, YIQ, YCbCr• YDbDr• DSH, HSV, HLS, IHS• Munsel colour space (cylindrical representation)• CIELuv• CIELab• SMPTE-C RGB• YES (Xerox)• Kodak Photo CD, YCC, YPbPr, ...

Contents

• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images

3D-polar Coordinate Colour Spaces• These spaces use a cylindrical (3D-polar) coordinate system to

encode the following three psycho-visual coordinates:– Hue (dominant colour seen)

• Wavelength of the pure colour observed in the signal.

• Distinguishes red, yellow, green, etc.

• More the 400 hues can be seen by the human eye.

– Saturation (degree of dilution)• Inverse of the quantity of “white” present in the signal. A pure colour has

100% saturation, the white and grey have 0% saturation.

• Distinguishes red from pink, marine blue from royal blue, etc.

• About 20 saturation levels are visible per hue.

– Brightness• Amount of light emitted.

• Distinguishes the greylevels.

• The human eye perceives about 100 levels.

3D-polar Coordinate Colour Spaces• The transformation of the RGB colour space to a hue, saturation

and brightness colour space is essentially a conversion from a set of rectangular coordinates to a set of (3D-polar) cylindrical coordinates.

H

HSV

IHS

HSI

HLS

triangle• Yet there are many such spaces described in books.

• How does one choose which one to use?

Idea Behind the Transformation• In the RGB space, the vectors [R, G, B] specify the amount of each

red, green and blue primary in the colour.– For convenience, we take R, G, B [0, 1]– The valid coordinates form the RGB cube [0, 1] [0, 1] [0, 1].

• The basic idea behind the transformation to the hue, saturation and brightness coordinate system is to place a new axis between [0, 0, 0] and [1, 1, 1], and to specify the colours in 3D-polar coordinates based on this axis.– This new axis passes through all the grey points (R = G = B), so we call it the

achromatic axis.

Achromatic axis

Brightness

Hue

H

Saturation

[0,0,0]

[1,1,1]

Achro

mati

c

axis

Basic Problem• Many of the spaces were originally developed for

easy numerical specification of colours in computer graphics applications.

• Due to its brightness function, the “natural” shape of the HSV colour gamut is a cone.

• However, with this shape, there are many coordinates which are not valid.

• In order to avoid costly verification of the validity of specified coordinates, these gamuts were often artificially expanded into cylinders.

• These cylindrical spaces have often been unwittingly carried over into image processing applications.

Con

ic H

SV

Cyl

indr

ical

HS

V

Achromatic axis

H=0°H=180°

Standard RGB → HSV Transform• max = sup(R, G, B) min = inf(R, G, B)• L = max

•

•

• If Httt

•

H = Ht

otherwise0

0 if maxmax

minmaxS

maxB

maxG

maxR

H

minmaxGR

minmaxRB

minmaxBG

t

if4

if2

if

• The same problem occurs for the HLS transformation.

• Here the double cone has been artificially expanded at both ends.

Con

ic H

LS

Cyl

indr

ical

HL

S

H=0°H=180°

Standard RGB → HLS Transform

• max = sup(R, G, B) min = inf(R, G, B)

•

•

•

• If Httt

•

H = Ht

2minmaxL

21

2

21

if

if

L

LS

minmaxminmax

minmaxminmax

c

maxB

maxG

maxR

H

minmaxGR

minmaxRB

minmaxBG

t

if4

if2

if

R, G and B are between 0 and 1.

Removal of the Brightness Dependence of the Saturation (1)

• HSV model

• But the HSV brightness LHSV = max(R, G, B)

• So the saturation with the brightness-dependence removed is

otherwise0

0,,max if,,max

,,min,,max

HSV

BGRBGR

BGRBGRS

BGRBGRS ,,min,,maxNCHSV

Removal of the Brightness Dependence of the Saturation (2)

• HLS model

• where the brightness

otherwise

,,min,,max2

,,min,,max

if,,min,,max

,,min,,max,,min,,max if0

21

HLSHLS

BGRBGR

BGRBGR

LBGRBGR

BGRBGRBGRBGR

S

2

,,min,,max BGRBGRLHLS

• The brightness dependence is removed by using

• to give

HLSLSS 21

HLSNCHLS 21

BGRBGRS ,,min,,maxNCHLS

Same as the non-cylindrical saturation obtained for HSV!

• Consider the colour image below. Not all the pixels which appear white have RGB coordinates of exactly [1, 1, 1] (similar for black pixels).

“Le chanteur”, Joan Mirò (bottom half inverted)

• These small variations are amplified in the saturation images due to the expansion of the cones into cylinders.

HSV cylindrical saturation

HLS cylindrical saturation

• The advantage of this measure of saturation is visible on the example image.

Le chanteur, Joan Mirò (bottom half inverted)

HSV cylindrical saturation

Compare to

Proposed Saturation

Advantages of the IHLS space

• This is the Improved HLS space.• It has the following advantages:

– The saturation is low for black and white pixels.

– The brightness is independent of the saturation (can be shown mathematically). This means that you are free to choose any brightness, luminance or lightness function.

– The saturation values can be easily compared. This is important for mathematical morphology operators.

Transformation into the IHLS space

• The full transformation is implemented efficiently as:

BGRY 0722.07152.02126.0 Or your own favourite brightness expression. BGRBGRS ,,min,,max

21

222

21

21

arccosBGRBRGBGR

BGRH

otherwise

if360

H

GBHH

Trigonometric version is more accurate.

Contents

• Introduction• Processing Vectorial Images• Alternative Colour Spaces• 3D-polar Coordinate Colour Spaces• Processing and Analysing Colour Images

Processing and Analysing Colour Images

• Some applications which take advantage of the good properties of the IHLS space are presented.

• The following are discussed:– Hue statistics.

– Mathematical morphology on colour images.

C

han

ge o

f re

pre

sen

tati

on

R

G

B

H 

L 

S Im

ag

e P

roce

ssin

g o

r A

naly

sis

Basic Idea

Chan

ge o

f re

pre

sen

tati

on

H

L

S

R

G

B

Processing the Hue Component

Colour image – “The virgin”, P. Serra (16th century)

The reds and violets are separated by a large discontinuity, even though they look similar.

Histogram of the hue

Hue component

Hue Statistics• For the brightness and the saturation, one can use standard

statistical methods for calculating the mean, standard deviation, etc.

• For data ai (i = 1, 2, …, n) distributed on the unit circle, the mean direction is that of the resultant vector obtained by adding unit vectors with directions ai.

• A measure of the variation in the directions of the data is given by the length of this vector divided by n (the mean length), which has the following characteristics:– range [0:1]– Values close to 1 the data is less spread out.

n = 3

Mean

• Given n values of the hue Hi

• The mean direction H is calculated as follows:

(1)

(2)

• The mean length R is:

(3)

n

ii

n

ii BARHBHA

1

222

1

,sin,cos

0,0if2arctan

0ifarctan

0,0ifarctan

AB

A

AB

H

AB

AB

AB

n

RR

Hue Mean which Takes the Saturation Into Account

• The previous formulation is standard in the texts on circular statistics, but it ignores the fact that not all hues have the same importance.

• We take this into account by weighting the length of each hue vector by the associated saturation value.

n = 3

• Let Si be the saturation associated with hue Hi.• We replace equation (1) by:

• To calculate the saturation-weighted mean direction HS, we replace A and B by AS and BS in equation (2).

• Equation (3) becomes:

• Note that RS remains a measure of the angular dispersion, and does not give information on the mean of the saturation.

n

iSSSii

n

iSiiS BARHSBHSA

1

222

1

,sin,cos

n

i i

SS

S

RR

1

ExampleC

olou

r Im

age

Hue

Satu

rati

on

Hue histogram

Saturation histogram

• HS = 20°• RS = 0.16W

eigh

ted

Hue threshold

0° - 40°

• H = 327°• R = 0.24

Hue threshold

307° - 347°

Mathematical Morphology in a 3D-Polar Coordinate Colour Space

• One can easily order the brightness and saturation values.

• On the other hand, the hue is defined on the circle, for which there is no obvious order (blue larger than red?, green smaller than red? ).

• Applying morphological operators to the brightness and saturation is therefore easy. – We just have to be sure that we order the vectors and not the

components (i.e. marginal order), so as to avoid introducing false colours.

– For this purpose we use the lexicographical order.

• Applying mathematical morphology to the hue is trickier, and won’t be discussed in this talk.

0 0

1 1

L S

H 0°

Lexicographical Order• This is the order in which words are arranged

in a dictionary.

Aardvark

Abacus

Abandon

.

.

.

Borough

Bough

.

.

Lexicographical Order for Vectors• For example, given two vectors x = (x1, x2, x3) and y = (y1, y2, y3):

• The lexicographical order is a total vector order. This means:– There are no pairs of vectors for which the order is unknown.

– The maximum and minimum of a set of vectors is always part of the set no false colours!

• The disadvantage is that one of the vector components has to play a dominant role.

332211

2211

11

andand

or

and

or

if

yxyxyx

yxyx

yx

yx

Angular Distance

• To enable us to work on the circle, we begin by defining an angular distance.

• Given a circle C with centre o

• We choose an arbitrary origin a0 on the circle. The points ai are then described by their curvilinear coordinate between 0 and 2 starting at a0.

• Given two points a and a, the size of the acute angle aoa is

a a

a a

aaaa

aaaaaaaaD

if2

if,

o

Lexicographical Order with Brightness at the Top Level (1)

• We define

• and

00andand

or

and

or

if

HHHHSSLL

SSLL

LL

jijiji

jiji

ji

ji cc

00andand

or

and

or

if

HHHHSSLL

SSLL

LL

jijiji

jiji

ji

ji cc

Lexicographical Order with Brightness at the Top Level (2)

• H0 is an arbitrary parameter. It does not have a major effect on the results, as, being at the third level, it is almost never taken into account.

• The erosion is defined in the standard way as follows:

f (x) = {f (y) : f (y) = inf [f (z)] , z Bx }

and the dilation:

f (x) = {f (y) : f (y) = sup [f (z)] , z Bx }

Original Image (293 418)

Ero

sion

Dila

tion

Ope

ning

Clo

sing

ExampleBrightness at the

top level

SE: Square of size 5 5

Lexicographical Order with Saturation at the Top Level

• We define

• and

00andand

or

and

or

if

HHHHLLSS

LLSS

SS

jijiji

jiji

ji

ji cc

00andand

or

and

or

if

HHHHLLSS

LLSS

SS

jijiji

jiji

ji

ji cc

Original image (293 418)

ExampleSaturation at the

top level

Ero

sion

Dila

tion

Ope

ning

Clo

sing

SE: Square of size 5 5

Colour Top-Hat

• We can define a top-hat operator for colour images similar to the one used for greyscale images.

• We simply calculate the Euclidean distance E between each corresponding pixel of the original image I and the result of one of the colour opening or closing operators:– Opening top-hat:

– Closing top-hat:

IIE ,

IIE ,

We wish to extract the greyish lines in between the mosaic tiles

Example

These lines are visible in the saturation image

Original Image

A colour morphological closing using a lexicographical order with saturation at the first level

Saturation

Top-hat operator. The pixel-by-pixel Euclidean distance between the two colour images to the left.

Summary

• A 3D-polar coordinate representation of the RGB colour space can be very useful in image processing and analysis.

• One should be careful with how the saturation is defined.

• Applications demonstrated are:– Colour statistics.

– Colour morphology.

• Further applications include:– Colour histograms.

– 2-dimensional brightness-saturation histograms (have been used for segmentation).

– …