Color Feature Detection

8/8/2019 Color Feature Detection

1/43

9

Color Feature Detection

Theo Gevers, Joost van de Weijer, and Harro Stokman

CONTENTS9.1 Introduction........................................................................................................................ 203

9.2 Color Invariance................................................................................................................ 205

9.2.1 Dichromatic Reflection Model............................................................................. 2059.2.2 ColorInvariants.....................................................................................................2069.2.3 Color Derivatives.................................................................................................. 207

9.3 Combining Derivatives..................................................................................................... 209

9.3.1 The Color Tensor................................................................................................... 2099.3.2 Color Tensor-Based Features............................................................................... 210

9.3.2.1 Eigenvalue-Based Features.................................................................. 2109.3.2.2 Color Canny Edge Detection............................................................... 2119.3.2.3 Circular Object Detection..................................................................... 212

9.4 Color Feature Detection: Fusion of Color Derivatives................................................. 213

9.4.1 ProblemFormulation............................................................................................ 2139.4.2 Feature Fusion....................................................................................................... 2149.4.3 Corner Detection................................................................................................... 215

9.5 Color Feature Detection: Boosting ColorSaliency........................................................ 2169.6 Color Feature Detection: Classification of ColorStructures........................................ 219

9.6.1 Combining Shape and Color............................................................................... 2199.6.2 Experimental Results............................................................................................ 2209.6.3 Detection ofHighlights........................................................................................ 221


2/43

9.6.4 Detection of Geometry/Shadow Edges............................................................. 2219.6.5 Detection ofCorners............................................................................................. 222

9.7 Conclusion..........................................................................................................................

223References..................................................................................................................................... 224

9.1

IntroductionThe detection and classification of local structures (i.e., edges, corners, and T-junctions) in color images is important for many applications, such asimage segmentation, image matching, object recognition, and visual trackingin the fields of image processing and computer vision [1], [2], [3]. In general,those local image structures are detected by differential operators that arecommonly restricted to luminance information. However, most of the imagesrecorded today are in color. Therefore, in this chapter, the focus is on the useof color information to detect and classify local image features.

203


3/43

Color Feature 20

The basic approach to compute color image derivatives is to calculateseparately the derivatives of the channels and add them to produce the finalcolor gradient. However, the derivatives of a color edge can be in opposingdirections for the separate color channels. Therefore, a summation of thederivatives per channel will discard the correlation between color channels

[4]. As a solution to the opposing vector problem, DiZenzo [4] proposes thecolor tensor, derived from the structure tensor, for the computation of thecolor gradient. Adaptations of the tensor lead to a variety of local imagefeatures,suchascircledetectors and curvature estimation [5], [6], [7], [8]. Inthis chapter, we study the methods and techniques to combine derivatives ofthe different color channels to compute local image structures.

To better understand the formation of color images, the dichromaticreflection model was introduced by Shafer [9]. The model describes howphotometric changes, such as shadows and specularities, influence the red,green, blue (RGB) values in an image. On the basis of this model, algorithmshave been proposed that are invariant to different photometric phenomenasuch as shadows, illumination, and specularities [10], [11], [12]. The extensionto differential photometric invariance was proposed by Geusebroek et al. [13].Van de Weijer et al. [14] proposed photometric quasi-invariants that havebetter noise and stability characteristics compared to existing photometricinvariants. Combining photometric quasi- invariants with derivative-basedfeature detectors leads to features that can identify various physical causes(e.g., shadow corners and object corners). In this chapter, the theory andpractice is reviewed to obtain color invariance such as shading/shadow andillumination invariance incorporated into the color feature detectors.

Two important criteria for color feature detectors are repeatability, meaning

that they should be invariant(stable) under varying viewing conditions, suchas illumination, shading, and highlights; and distinctiveness, meaning thatthey should have high discriminative power. It was shown that there exists atrade-off between color invariant models and their discriminative power [10].For example, color constant derivatives were proposed [11] that are invariantto all possible light sources, assuming a diagonal model for illuminationchanges. However, such a strong assumption will significantly reduce thediscriminative power. For a particular computer vision task that assumes onlya few different light sources, color models should be selected that areinvariant (only) to these few light sources, resulting in an augmentation of

the discriminative power of the algorithm. Therefore, in this chapter, weoutline an approach to the selection and weighting of color (invariant) modelsfor discriminatory and robust image feature detection.

Further, although color is important to express saliency [15], the explicitincorporation of color distinctiveness into the design of salient pointsdetectors has been largely ignored. To this end, in this chapter, we reviewhow color distinctiveness can be explicitly incorporated in the design of imagefeature detectors [16], [17]. The method is based upon the analysis of thestatistics of color derivatives. It will be shown that isosalient derivativesgenerate ellipsoids in the color derivative histograms. This fact is exploited toadapt derivatives in such a way that equal saliency implies equal impact on

the saliency map.Classifying image features (e.g., edges, corners, andT-junctions) is useful for

a large number of applications where corresponding feature types (e.g.,material edges) from distinct images are selected for image matching, whileother accidental feature types (e.g. shadow and highlight edges) arediscounted. Therefore, in this chapter, a classification framework is discussed


4/43

Color Feature 20

to combine the local differential structure (i.e., geometrical information suchas edges, corners, and T-junctions) and color invariance (i.e., photometricalinformation, such as shadows, shading, illumination, and highlights) in amultidimensional feature space [18]. This feature space is used to yield properrule-based and training-based classifiers to label salient image structures on

the basis of their physical nature [19].In summary, in this chapter, we will review methods and techniques solvingthe following important issues in the field of color feature detection: to obtaincolor invariance, such as


5/43

Color Feature 20

with shading and shadows, and illumination invariance; to combinederivatives of the different color channels to compute local image structures,such as edges, corners, circles, and so forth; to select and weight color(invariant) models for discriminatory and robust image feature detection; toimprove color saliency to arrive at color distinctiveness (focus-of- attention);

and to classify the physical nature of image structures, such as shadow,highlight, and material edges and corners.

This chapter is organized as follows. First, in Section 9.2, a brief review isgiven on the various color models and their invariant properties based on thedichromatic reflection model. Further, color derivatives are introduced. InSection 9.3, color feature detection is proposed based on the color tensor.Information on color feature detection and its appli- cation to color featurelearning, color boosting, and color feature classification is given in Sections9.4, 9.5, and 9.6.

9.2 Color

Invariance

In this section, the dichromatic reflection model [9] is explained. Thedichromatic reflection model explains the image formation process and,therefore, models the photometric changes, such as shadows andspecularities. On the basis of this model, methods are discussed containinginvariance. In Section 9.2.1, the dichromatic reflection model is introduced.

Then, in Sections 9.2.2 and 9.2.3, color invariants and color (invariant)derivatives will be explained.

9.2.1 Dichromatic Reflection Model

The dichromatic reflection model [9] divides the light that falls upon a surfaceinto two distinct components: specular reflection and body reflection. Specularreflection is when a ray of light hits a smooth surface at some angle. Thereflection of that ray will reflect at the same angle as the incident ray of light.

This kind of reflection causes highlights. Diffuse reflection is when a ray oflight hits the surface and is then reflected back in every direction.

Suppose we have an infinitesimally small surface patch of some object, andthree sensors are used for red, green, and blue (with spectral sensitivities fR(), fG() and fB()) to obtain an image of the surface patch. Then, the sensorvalues are [9]

C = mb(n,

s)

fC ()e()cb()d + ms(n,

s, v)

fC ()e()cs()d (9.1)

for C {R, G, B}, and where e() is the incident light. Further, cb() andcs () are the surface albedo and Fresnel reflectance, respectively. Thegeometric terms mb and ms are the geometric dependencies on the body andsurface reflection component, is the wavelength, n is the surface patchnormal, s is the direction of the illumination source, and v is the direction ofthe viewer. The first term in the equation is the diffuse reflection term. Thesecond term is the specular reflection term.

Let us assume that white illumination is when all wavelengths within thevisible spectrum have similar energy: e () = e . Further assume that theneutral interface reflection model holds, so that cs () has a constant valueindependent of the wavelength (cs() = cs). First, we construct a variable that


6/43

Color Feature 20

depends only on the sensors and the surface albedo:

kC =

fc()cb()d (9.2)


7/43

R

R

R

k

k

k

Finally, we assume that the following holds:

fR(A)dA =A A

fG(A)dA =A

fB(A)dA = f (9.3)

With these assumptions, we have the following equation for the sensorvalues from an object under white light [11]:

Cw = emb(n, s)kC + ems(n, s, v)cs f (9.4)

with Cw {Rw, Gw, Bw}.

9.2.2 Color Invariants

To derive that normalized color, given by,

Rr=+ G +

Gg =

+ G +B

b =+ G +

(9.5)B

(9.6)B

(9.7)B

is insensitive to surface orientation, illumination direction, and illuminationintensity, the diffuse reflection term

isused.

Cb = emb(n, s)kC (9.8)

By substituting Equation 9.8 in the equations ofr, g, and b, we obtain

kR

r(Rb, Gb, Bb) =R + kG + k

B

(9.9)

g(Rb, Gb, Bb) =R +

b(Rb, Gb, Bb) =R +

kG

kG + k

B

kB

kG + k

B

(9.10)

(9.11)

and hence, rgb is only dependent on the sensor characteristics and surfacealbedo. Note that rgb is dependent on highlights (i.e., dependent on thespecular reflection term of Equation 9.4).

The same argument holds for the c1c2c3 color space:

c1(Rb, Gb, Bb) =

arctan

c2(Rb, Gb, Bb) =

arctan

k

R

max{k

G

kG

maxk

N

. kB}N

, k


8/43

(9.12) (9.13){ R B}

c1(Rb, Gb, Bb) =arctan

Invariant properties for saturation

k

B

max{kG

N

. kR}

(9.14)

min(R, G,B)

S(R, G, B) = 1

R + G+

(9.15)B


9/43

(

TABLE9.1

Invariance for Different Color Spaces for Varying Image Properties

Syste Viewpoin Geometr Illumination Illumination Highligh

RG B

rgb + + +

Hue + + + +

S + + +

I

c1c2c3 + + +

Note: A + means that the color space is not sensitive to the property; a means that it is.

are obtained by substituting the diffuse reflection term into the equation of

saturation:

min{kR, kG, kB}S(Rb, Gb, Bb) = 1

R + kG +(9.16)

kB)

where S is only dependent on the sensors and the surface albedo.Further, hue

3(G B)N

H(R, G, B) =arctan

((R G)

+ (R

B))

(9.17)

is also invariant to surface orientation, illumination direction, and intensity:

3emb(n, s)(kG kB)

N

H(Rb, Gb, Bb) =arctan

emb(n, s)((k

R

kg)

+ (kR

N

kB))

3(kG kB)= arctan

(kR

kG)

+ (kR

kB)

(9.18)

In addition, hue is invariant to highlights:

3(Gw Bw)

N

H(Rw, Gw, Bw) =arctan

(Rw

Gw)

+ (Rw

Bw)

3emb(n, s)(kG kB)N

= arctanemb(n, s)((k

R

kG)

+ (kR

N

kB))

3(kG

kB)= arctan(kR

kG)

+ (kR

kB)

(9.19)

A taxonomy of color invariant models is shown in Table 9.1.


10/43

9.2.3 Color Derivatives

Here we describe three color coordinate transformations from whichderivatives are taken [20]. The transformations are derived from photometricinvariance theory, as discussed in the previous section.


11/43

x

x

x

x

i

R 2

i

2

+

r 2 i

i

6 i

For an image f= (R, G, B)T, the spherical color transformation is given by

arctan

(GN

R i

=

arcsin

R2 + G2

i (9.20)

r

2 + G + B2i

r=

R2 + G + B2

The spatial derivatives are transformed to the spherical coordinate system by

rsin x

GxR RxG

R2 G2

iRxRB + GxGB Bx(R

2 + G2)i

S (fx) = fs

=rx

=

i

(9.21)x

x R2 + G

2

i

R2 + G2 + Bii

RxR + GxG + BxB

R2 + G2 +

B2

The scale factors follow from the Jacobian of the transformation. They ensurethat the norm of the derivative remains constant under the transformation,hence |fx| = |f

s|. In the spherical coordinate system, the derivative vector is asummation of a shadowshadingvariant part, Sx = (0, 0, rx)

T and a shadowshading quasi-invariant part,given by Sc =(rsin x, rx, 0)

T[20].

The opponent color space is givenby

o1

R G

2 i

R + G 2B

i

o2

i=

i

(9.22)

o3

6

i

i R + G + B

i

3

For this, the following transformation of the derivatives is obtained:

o1x

1

(RxG

x) 2 i 1

i

O (fx) = f

o

=

o2

i

=

(R +G

2B )

i

(9.23)x x

x x xi

o3 i

1i

(Rx+ Gx+ Bx)

3

The opponent color space decorrelates the derivative with respect to specular


12/43

i

i +

changes. The derivative is divided into a specular variant part, Ox = (0, 0, o3x)

T, and a specular quasi-invariant part Oc = (o1x, o2x, 0)

T.

The huesaturationintensity is given by

h

arctan

(o1

N

o2s

i

=

i

(9.24) o12 o22

i

o3


13/43

X

X

The transformation of the spatial derivatives into the h si space decorrelatesthe derivative with respect to specular, shadow, and shading variations,

s h

X

(R (BXGX) + G (RX BX) + B (GX RX))

2(R2 + G2 + B2 RG RB GB) ii

R (2RXGX BX) + G (2GX RX BX) + B (2BX RXGX) i

H (fX) = fh =

s

i=

i

Xj X

iX

j

6(R2 + G2 + B2 RG RB

GB)

(RX+ GX+

BX)3

iiii

(9.25)

The shadowshadingspecular variant is given by HX = (0, 0, iX)T, and the

shadow

shadingspecular quasi-invariant is given by Hc = (shX, sX, 0)T.Because the length of a vector is not changed by orthonormal coordinate

transformations,

thenormofthederivativeremainsthesameinallthreerepresentations|fX| =|fc| =|fo| = X X|fh|. For both the opponent color space and the huesaturationintensity color

space, thephotometrically variant direction is givenby the

L1 norm of the intensity. For thespherical

coordinate system, the variant is equal to the L2 norm of the intensity.

9.3 Combining

Derivatives

In the previous section, color (invariant) derivatives were discussed. Thequestion is how to combine these derivatives into a single outcome. A defaultmethod to combine edges is to use equal weights for the different colorfeatures. This naive approach is used by many feature detectors. For example,to achieve color edge detection, intensity-based edge detection techniquesare extended by taking the sum or Euclidean distance from the individual

gradient maps [21], [22]. However, the summation of the derivativescomputed for the different color channels may result in the cancellation oflocal image structures [4]. A more principled way is to sum the orientationinformation (defined on [0, r)) of the channelsinstead of adding the direction information (defined on [0, 2r )). Tensormathematics pro-vide a convenient representation in which vectors in opposite directions willreinforce oneanother. Tensors describe the local orientation rather than the direction (i.e.,the tensor of a vector and its 180 rotated counterpart vector are equal).

Therefore, tensors are convenientfor describing color derivative vectors and will be used as a basis for colorfeature detection.

9.3.1 The Color Tensor

Given a luminance image f , the structure tensor is

given by [6]


14/43

y

f2G =

X

fXf

y

(9.26)

fXfy f2

where the subscripts indicate spatial derivatives, and the bar (.) indicatesconvolution with a Gaussian filter. The structure tensor describes the localdifferential structure of images and is suited to find features such as edgesand corners [4], [5], [7]. For a multichannel image f= ( f1, f2, ..., fn)T, thestructure tensor is given by

fXfX f

Xfy

(9.27)G =fyfX fyfy


15/43

2l0 Color Image Processing: Methodsand

=

w

w

y

w

x x y y

x x y y

2

y

In the case that f = ( R, G, B), Equation 9.27 is the color tensor. Forderivatives that are accompanied by a weighting function, wx and wy, whichappoint a weight to every measurement in fx and fy, the structure tensor isdefined by

xfx fx

x

G

wxwyfx fy

wxwyiii

(9.28)

wywxfy fx w2fy fy

ij

wywx 2

9.3.2 Color Tensor-Based Features

In this section, a number of detectors are discussed that can be derived fromthe weighted color tensor. In the previous section, we described how tocompute (quasi) invariant derivatives. In fact, dependent on the task at hand,either quasi-invariants are selected for detection or full invariants. For featuredetection tasks quasi-invariants have been shown to perform best, while forfeature description and extraction tasks, full invariants are required [20]. Thefeatures in this chapter will be derived for a general derivative gx . To obtainthe desired photometric invariance for the color feature detector, the innerproduct ofgx, see Equation 9.27, is replaced by one of the following:

fx fx if no invariance is required

c c c cgx gx =Sx Sx or Hx Hx for invariant feature

detection

(9.29)

c c c cSx Sx|f|

2or

Hx Hx

|s|2for invariant feature extraction

where s is the saturation.In Section 9.3.2.l, we describe features derived from the eigenvalues ofthetensor. Further,

more features are derived from an adapted version ofthe structure tensor, suchas the Cannyedge detection, in Section 9.3.2.2, and the detection of circular objects inSection 9.3.2.3.

9.3.2.1 Eigenvalue-BasedFeatures

Two eigenvalues are derived from the eigenvalue analysis defined by

lAl =

2(gxgx + gy

gy +

lA2 =

2(gxgx + gy

gy

(

g g g

g )2(

g g g

g )2

+(2gxgy)2

+(2gxgy)2

(9.30)

The direction ofAl points in the direction of the most prominent localorientation:

l(

2gxgyN

=2

arctangxgxgy

gy

(9.3l)


16/43

2l0 Color Image Processing: Methodsand

TheA terms can be combined to give the following localdescriptors:

Al +A2 describes the total local derivative energy.

Al is the derivative energy in the most prominent direction.

Al

A2 describes the line energy (see Reference [23]). The derivativeenergy in the prominent orientation is corrected for the energycontributed by the noiseA2. A2 describes the amount ofderivative energy perpendicular to the

prominent local orientation.


17/43

l 2G

=

=

H =i

Color Feature Detection 2ll

The Harris corner detector [24] is often used in the literature. In fact, thecolor Harris operator H can easily be written as a function of the eigenvaluesof the structure tensor:

Hf= gxgx gygygxgy2 k(gxgx +

gy

gy)2

=AlA2 k(Al +A2)2 . (9.32)

Further, the structure tensor of Equation 9.27 can also be seen as a localprojection of the derivative energy on two perpendicular axes [5], [7], [8],namely, ul = ( l 0 )

T and u2 = ( 0 l )T,

u ,u

i

(Gx,yul) (Gx,yul) (Gx,yul)

(Gx,yu2)

= (Gx,yul) (Gx,yu2) (Gx,yu2) (Gx,yu2)

(9.33)

in which Gx, y = ( gx gy). From the Lie group of transformation, several otherchoices of perpendicular projections can be derived [5], [7]. They includefeature extraction for circle, spiral, and star-like structures.

The star and circle detector is given as an example. It is basedon ul

l

x2+y2

(xy)T,

whichcoincides with the derivative pattern of a circular patterns andu2 l

x2+y2

( yx)T,which

denotes the perpendicular vector field that coincides with the derivativepattern of star-like patterns. These vectors can be used to compute the adapted structuretensor with

Equation 9.33. Only the elements on the diagonal have nonzero entries and areequal to

x2

x2 +y2

gxgx+

2xy

x2 +y2

gxgy+

x2

y2

x2 +y2

gygy 0

2xy y 2

i

(9.34)i

j

0x2 +y2

gygy

x2 +y2

gxgy

+x2 +y2

gxgx

whereAl describes the amount of derivative energy contributing to circular

structures andA2 the derivative energy that describes a star-like structure.

Curvature is another feature that can be derived from an adaption of thestructure tensor.

For vector data, the equation for the curvature is given by

w2gvgvw2

gwgw (w2 gwgww2gvgv)2 + 4w2

wgvgw2

=2w2 wgv

gw

(9.35)

in which gvand gw are the derivatives in gauge coordinates.

9.3.2.2 Color Canny EdgeDetection

We now introduce the Canny color edge detector based on eigenvalues. The


18/43

algorithm consists of the following steps:

l. Compute the spatial derivatives, fx, and combine them if desiredinto a quasi- invariant, as discussed in Section 9.2.3.

2. Compute the maximum eigenvalue using Equation 9.30 and itsorientation using

Equation 9.3l.

3. Apply nonmaximum suppression onAl in the prominent direction.

To illustrate the performance, the results of Canny color edge detection forseveral photometric quasi-invariants is shown in Figure 9.la to Figure 9.le.

The image is recorded in three RGB colors with the aid of the SONY XC-003PCCD color camera (three chips) and the


19/43

Color Feature 21

(a) (b) (c) (d) (e)

FIGURE9.1

(a) Input image with Canny edge detection based on, successively, (b) luminance derivative, (c)RGB derivatives, (d) the shadowshading quasi-invariant, and (e) the shadowshadingspecularquasi-invariant.

Matrox Magic Color frame grabber. Two light sources of average daylight color

are used to illuminate the objects in the scene. The digitization was done in 8bits per color. The results show that the luminance-based Canny (Figure 9.lb)misses several edges that are correctly found by the RGB-based method(Figure 9.lc). Also, the removal of spurious edges by photometric invarianceis demonstrated. In Figure 9.ld, the edge detection is robust to shadow andshading changes and only detects material and specular edges. In Figure 9.le,only the material edges are depicted.

9.3.2.3 CircularObject Detection

In this section, the combination of photometric invariant orientation and

curvature estimation is used to detect circles robust against varying imagingconditions such as shadows and illumination changes.The following algorithm is introduced for the invariant detection of color

circles [20]:

l. Compute the spatial derivatives, fx, and combine them if desiredinto a quasi- invariant as discussed in Section 9.2.3.

2. Compute the local orientation using Equation 9.3l andcurvature using

Equation 9.35.

3. Compute the Hough space [25], H(

R.x0. y0 , where R is the radius

of the circle,andx0 andy0 indicate the center of the circle. The computation of theorientationand curvature reduces the number of votes per pixel to one. Namely,for a pixel at position x = (xl.yl),

lR =

x0 =xl +

y

0

=y

l

+

lcos (9.36)

l sin Each pixel will vote by means of its derivative energy

fxfx.

4. Compute the maxima in the hough space. These maxima indicate thecircle centers and the radii of the circle.


20/43

Color Feature 21

To illustrate the performance, the results of the circle detection are given inFigure 9.2a to Figure 9.2c. Images have been recorded by the Nikon Coolpix950, a commercial digital camera of average quality. The images have size267 200 pixels with JPEG compression. The digitization was done in 8 bitsper color. It is shown that the luminance-based circle


21/43

(a) (b) (c)

FIGURE9.2

(a) Detected circles based on luminance, (b) detected circles based on shadowshadingspecularquasi-invariant, and (c) detected circles based on shadowshadingspecular quasi-invariant.

detection is sensitive to photometric variations, as nine circles are detectedbefore the five balls were extracted. For the circle detector based on the(shadowshadingspecular) quasi- invariant, the five most prominent peaks inthe hough space (not shown here) correspond to the radii and center points ofthe circles found. In Figure 9.2c, an outdoor example with a shadow partiallycovering the objects (tennis balls) is given. The detector finds the rightcircular objects and, hence, performs well, even under severe varying imagingconditions, such as shading and shadow, and geometrical changes of thetennis balls.

9.4 Color Feature Detection: Fusion of Color

Derivatives

In the previous section, various image feature detection methods to extractlocale image structures such as edges, corners, and circles were discussed. Asthere are many color invariant models available, the inherent difficulty is howto automatically select the weighted subset of color models producing thebest result for a particular task. In this section, we outline how to select andweight color (invariant) models for discriminatory and robust image featuredetection.

To achieve proper color model selection and fusion, we discuss a methodthat exploits nonperfect correlation between color models or feature detectionalgorithms derived from the principles ofdiversification. As a consequence, anoptimal balance is obtained between repeatability and distinctiveness. Theresult is a weighting scheme that yields maximal feature discrimination [l8],[l9].

9.4.1 Problem Formulation

The measuring of a quantity u can bestated as

u = E(u) u (9.37)

where E (u) is the best estimate for u (e.g., the average value), and urepresents the uncer- tainty or error in the measurement of u (e.g., thestandard deviation). Estimates of a quantity u, resulting from N differentmethods, may be constructed using the following weighting scheme:

N


22/43

E(u) =V

wi E(ui ) (9.38)i

where E(ui ) isthebestestimateofaparticularmethodi.Simplytakingtheweighted average of the different methods allows features from very differentdomains to be combined.


23/43

+

For a function u(ul. u2. . uN) depending on N correlated variables, thepropagated error u is N N

uu

u (ul. u2. . uN) =V V

u

u

cov(ui . uj ) (9.39)

i=l j=l i j

where cov(ui . uj ) denotes the covariance between two variables. From thisequation, it can be seen that if the function u is nonlinear, the resulting error,u , depends strongly on the values of the variables ul . u2 . . uN . BecauseEquation 9.38 involves a linear combination of estimates, the error of thecombined estimate is only dependent on the covariances of the individualestimates. So, through Equation 9.39, we established that the proposedweighting scheme guarantees robustness, in contrast to possible, morecomplex, combination schemes.

Now we are left with the problem of determining the weights wi in aprincipled way. In the next section, we will propose such an algorithm.

9.4.2 Feature FusionWhen using Equation 9.38, the variance of the combined color models can befound throughEquation9.39:

N N2

V V

u = wi wj cov(ui . uj ) (9.40)

or,equivalently, N

2

V

i=l

iu

j=l

NV V

u =j=l

w22ii=l j

=

i

wi wj cov(ui . uj ) (9.4l)

where wi denotes the weight assigned to color channel i , ui denotes theaverage output for channel i, u denotes the standard deviation of quantity uin channel i, and cov(ui . uj ) corresponds to the covariance between channelsi and j.

From Equation 9.4l, it can be seen how diversification over various channelscan reduce the overall variance due to the covariance that may exist betweenchannels. The Markowitz selection model [26] is a mathematical method forfinding weights that achieve an optimal diversification. The model willminimize the variance for a given expected estimate for quantity u or will

maximize the expected estimate for a given variance u. The model definesa set of optimal u and u pairs. The constraints of this selection model aregiven as follows:

minimize u (9.42)

for the formula described in Equation 9.38. The weights are constrained bythe following conditions:

NV

wi = l

(9.43a)i=l

l


24/43

space for wi .This model is quadratic with linear constraints and can be solved by linear

program-

ming [27]. When u is varied parametrically, the solutions for this system willresult in meanvariance pairs representing different weightings of the featurechannels. The pairs

that maximize the expected u versus u or minimize the u versus expectedu, define the optimal frontier. They form a curve in the meanvariance plane,and the corresponding weights are optimal.


25/43

u

(a) (b) (c) (d)

FIGURE9.3

(a) Lab image and (b) ground-truth for learning edges. Input image for the edge and cornerdetection: on the left, the edge is indicated for the learning algorithm. (c)The -squared error ofthe transformed image and the predicted expected value: here the edges have a very lowintensity. (d) The local signal-to-noise ratio for the transformed image. The edges have a higherratio.

A point ofparticular interest on this curve is the point that has the maximalratio between the expected combined output E(u) and the expected variance2 . This point has the weights for which the combined feature space offersthe best trade-off between repeatability and distinctiveness.

In summary, the discussed selection model is used to arrive at a set ofweights to combine different color models into one feature. The expectedvalue of this feature E (u) is the weighted average of its componentexpected values. The standard deviation of this combined feature will be lessthan or equal to the weighted average of the component standard deviations.When the component colors or features are not perfectly correlated, theweighted average of the features will have a better variance-to-output ratiothan the individual components on their own. New features or colors canalways be safely added, and the ratio will never deteriorate, because zero

weights can be assigned to components that will not improve the ratio.

9.4.3 Corner Detection

The purpose is to detect corners by learning. Our aim is to arrive at anoptimal balance between color invariance (repeatability) and discriminativepower (distinctiveness). In the context of combining feature detectors, inparticular, in the case of color (invariant) edge detection, a default method tocombine edges is to use equal weights for the different color features. Thisnaive approach is used by many feature detectors. Instead of experi- mentingwith different ways to combine algorithms, in this section, we use theprincipled method, outlined in the previous section, on the basis of thebenefits of diversification. Because our method is based on learning, we needa set of training examples. The problem of corners is, however, that there arealways a few pixels at a corner, making it hard to create a training set. Wecircumvent this problem by training on the edges, the first-order derivatives,where many more pixels are located.

Because the structure tensor (Equation 9.28) and the derived Harrisoperator (Equation 9.32) are based on spatial derivatives, we will train theweighting vector w on edges. This will allow for a much simpler collection oftraining points. So, the weights are trained with the spatial derivatives of the

color channels as input. The resulting weights are then put in the w weightsvector of the Harris operator.

To illustrate the performance of the corner detector based on learning, thefirst experiment was done on the image ofSection 9.3, recorded in three RGBcolors with the aid of the SONY XC-003P CCD color camera. The weights weretrained on the edges of the green cube see Figure 9.3a and Figure 9.3b.


26/43

The edges were trained on the first-order derivatives in


27/43

(a) (b)

FIGURE9.4

(a) Results of the Harris corner detector. (b) Corners projected on the input image. The results ofthe Harris corner detector, trained on the lower right cube.

all color spaces. The results of applying these weights to the same image areshown in Figure 9.3c and Figure 9.3d. The edge is especially visible in the

signal-to-noise image. Using the weights learned on the edges with the Harrisoperator, according to Equation 9.32, the corners of the green cubeparticularly stand out (see Figure 9.4a and Figure 9.4b).

Another experiment is done on images taken from an outdoor object atraffic sign (see Figure 9.5a and Figure 9.5b). The weights were trained on oneimage and tested on images of the same object while varying the viewpoint.Again, the edges were defined by the first-order derivative in gaugecoordinates. The results of the Harris operator are shown in Figure 9.6. Thecorner detector performs well even under varying viewpoints and illuminationchanges. Note that the learning method results in an optimal balance between

repeatability and distinctiveness.

9.5 Color Feature Detection: Boosting

Color Saliency

So far, we have outlined how to obtain color invariant derivatives for imagefeature detection. Further, we discussed how to learn a proper set of weightsto yield proper color model selection and fusion of feature detection

algorithms.In addition, it is known that color is important to express saliency [l5],[l7]. To this end, in this section, we review how color distinctiveness can beexplicitly incorporated in the design of image feature detectors [l6], [l7]. Themethod is based upon the analysis of the statistics of color derivatives. Whenstudying the statistics of color image derivatives, points of equal frequencyform regular structures [l7]. Van de Weijer et al. [l7] propose a color saliencyboosting function based on transforming the color coordinates and using


28/43

(a) (b)

FIGURE9.5

The input image for the edge and corner detection: (a) the training image and (b) the trained

edges.


29/43

G

FIGURE9.6

Original images (top) and output (bottom) of the Harris corner detector trained on

redblue edges.

the statistical properties of the image derivatives. The RGB color derivativesare correlated. By transforming the RGB color coordinates to other systems,photometric events in images can be ignored as discussed in Section 9.2,where it was shown that the spatial derivatives are separated intophotometrical variant and invariant parts. For the purpose of color saliency,the three different color spaces are evaluated the spherical color space in

Equation 9.2l, the opponent color space in Equation 9.23, and the h si colorspace in Equation 9.25. In these decorrelated color spaces, only thephotometric axes are influenced by these common photometric variations.

The statistics of color images are shown for the Corel database [28], whichconsists of

40,000 images (black and white images were excluded). In Figure 9.7a toFigure 9.7c, thedistributions (histograms) of the first-order derivatives, fx , are given for thevarious color coordinate systems.

When the distributions of the transformed image derivatives are observedfromFigure 9.7, regular structures are generated by points of equal frequency

(i.e., isosalient surfaces).These surfaces are formed by connecting the pointsin the histogram that occur the same number

100

Bx

50

0

50

150

o3x

75

0

75

150

rx

75

0

75

100100

500

050

10050 150

10050

00

50

10050 150

10050

0

50

10050

50 100 100 Rxx

(a)

o1x

50 100 100

(b)

o2x 0 50 100 100r

x

(c)

rsinx

FIGURE9.7

The histograms of the distribution of the transformed derivatives of the Corel image database in,


30/43

respectively, (a) the RGB coordinates, (b) the opponent coordinates, and (c) the sphericalcoordinates. The three planes correspond with the isosalient surfaces that contain (from dark tolight), respectively, 90%, 99%, and 99.9% of the total number of pixels.


31/43

2l8 Color Image Processing: MetAodsand

S f O f H

X X X

2

X

TABLE9.2

The Diagonal Entries ofI for the Corel Data SetComputed for Gaussian Derivatives With = lParameter fX *fX*l f

s

c

o c A cX X X X

Ill 0.577 l 0.85l 0.85 0.85 0.85l 0.858 lI22 0.577 0.5l5 0.5l8 0.52 0.52 0.509 0I33 0.577 0.09 0 0.06 0 0.066 0

of times. The shapes of the isosalient surfaces correspond to ellipses. Themajor axis of the ellipsoid coincides with the axis of maximum variation in thehistogram (i.e., the intensity axes). Based on the observed statistics, asaliency measure can be derived in which vectors with an equal informationcontent have an equal effect on the saliency. This is called the color saliency

boosting function. It is obtained by deriving a function that describes theisosalient surfaces.

More precisely, the ellipsoids are equal to

(A

l 2+

(A

2 2

+

(A

3 = *IA(fX)*2 (9.44)

and then the following holds:

p(fX =p(fX) *IA(fX)* =

II

AA(f )

I(9.45)I

XI

where I is a 3 3 diagonal matrix with Ill = , I22 =, and I33 = . I is

restricted to

I2 2 2ll + I22 + I33 = l. The desired saliency boosting function is obtained by

g (fX) = IA(fX)

(9.46) By a rotation of the color axes followed by a rescaling of the axis, the

oriented isosalientellipsoids are transformed into spheres, and thus, vectors of equal saliencyare transformedinto vectors of equallength.

Before color saliency boosting can be applied, the I-parameters have to beinitialized by

fitting ellipses to the histogram of the data set. The results for the varioustransformations

are summarized in Table 9.2. The relation between the axes in the variouscolor spaces clearly confirms the dominance of the luminance axis in the RGB


32/43

2l8 Color Image Processing: MetAodsand

cube, because I33 , the multiplication factor of the luminance axis, is muchsmaller than the color-axes multiplication factors, Ill and I22 . After colorsaliency boosting, there is an increase in information context, see Reference[l7] for more details.

To illustrate the performance of the color-boosting method, Figure 9.8a toFigure 9.8d show the results before and after saliency boosting. Althoughfocus point detection is already an extension from luminance to color, black-and-white transition still dominates the result. Only after boosting the colorsaliency are the less interesting black-and-white structures in the imageignored and most of the red Chinese signs are found.


33/43

(a)(b) (c) (d)

FIGURE9.8 (See color insert.)

In columns, respectively, (a) input image, (b) RGB-gradient-based saliency map, (c) color-boostedsaliency map, and (d) the results with red dots (lines) for gradient-based method and yellow dots(lines) for salient points after color saliency boosting.

9.6 Color Feature Detection: Classificationof Color

Structures

In Section 9.2, we showed that color models may contain a certain amount ofinvariance to the imaging process. From the taxonomy of color invariantmodels, shown in Table 9.l, we now discuss a classification framework to detectand classify local image structures based on photometrical and geometricalinformation [l8]. The classification of local image structures (e.g., shadowversus material edges) is important for many image processing and computervision tasks (e.g., object recognition, stereo vision, three-dimensionalreconstruction).

By combining the differential structure and reflectance invariance of colorimages, local image structures are extracted and classified into one of thefollowing types: shadow geometry edges, corners, and T-junctions; highlightedges, corners, and T-junctions; and material edges, corners, and T-junctions.First, for detection, the differential nature of the local image structures isderived. Then, color invariant properties are taken into account to determinethe reflectance characteristics of the detected local image structures. Thegeo- metrical and photometrical properties of these salient points (structures)are represented as vectors in a multidimensional feature space. Finally, aclassifier is built to learn the specific characteristics of each class of imagestructures.

9.6.1 Combining Shape and Color

By combining geometrical and photometrical information, we are able tospecify the physical nature of salient points. For example, to detecthighlights, we need to use both a highlight invariant color space, and one ormore highlight variant spaces. It was already shown in Table 9.l that hue isinvariant to highlights. Intensity I and saturation S are not invariant.


34/43

Further, a highlight will yield a certain image shape: a local maximum (I) anda local minimum (S). These local structures are detected by differentialoperators as discussed in Section 9.2. Therefore, in the brightness image, weare looking for a local maximum.


35/43

Color Feature 22

xy

xy

The saturation at a highlight is lower than its surroundings, yielding a localminimum. Finally, for hue, the values will be near zero at that location. In thisway, a five-dimensional feature vector is formed by combining the color spaceHSI and the differential information on each location in an image.

The same procedure holds for the detection ofshadow

geometry/highlight/material edges, corners, and T-junctions. The featuresused to detect shadowgeometry edges are first-order derivate applied onboth the RGB and the c1 c2 c3 color channels. Further, the second-orderderivative is only applied on the RGB color image. To be precise, in thissection, we use the curvature gauge to characterize local structures thatare only characterized by their second-order structure. It is a coordinatesystem on which the Hessian becomes diagonal, yielding the ( p. q )-coordinate system. The two eigenvectors of the Hessian are K1 and K2 and aredefined by

K1 = fxx + fyy ( fxx+ fyy)2

+ 4 f2

K2 = fxx + fyy+ ( fxx + fyy)2

+ 4 f2

(9.47)

(9.48)

To obtain the appropriate density distribution of feature values in featurespace, classifiers are built to learn the density functions for shadowgeometry/highlight/material edges, corners, and T-junctions.

In this section, the learning-based classification approach is taken as

proposed by Gevers and Aldershoff [19]. This approach is adaptive, as theunderlying characteristics of image feature classes are determined bytraining. In fact, the probability density functions of the local image structuresare learned by determining the probability that an image patch underconsideration is of a particular class (e.g., edge, corner, or T-junctions). If twoimage patches share similar characteristics (not only the same color, but alsothe same gradient size, and curvature), both patches are represented by thesame point in feature space. These points are represented by a (n d )-matrix, in which ddepends on the number of feature dimensions and n on thenumber of training samples.

Then, the density function p(x*) is computed, where x represents the dataof the pixel under consideration, and is the class to be determined. From thedata and training points, the parameters of the density function p(x*) areestimated. We use a single Gaussian and multiple Gaussians (mixture ofGaussians [MoG]). Besides this, the k-nearest neighbor method is used.

9.6.2 ExperimentalResults

In this section, the results are given to classify the physical nature of salientpoints by learning. A separate set of tests is computed for each classifier (i.e.,Gaussian, mixture of Gaussians, and k-nearest neighbor).

The images are recorded in three RGB colors with the aid of the SONY XC-003P CCD color camera (three chips) and the Matrox Magic Color framegrabber. Two light sources of average daylight color are used to illuminate theobjects in the scene. The digitization was done in 8 bits per color. Threeexamples of the five images are shown in Figure 9.9a to Figure 9.9c. For allexperiments, = 2 is used for the Gaussian smoothing parameter of the

differential operators.The classifiers are trained using all but one image. This last image is used asa test image. In this way, the test image is not used in the training set. In allexperiments, a total of three Gaussian components is used for the MoGclassifier, and k= 3 for the k-NN classifier.


36/43

(a) (b) (c)

FIGURE9.9

(a) Example image 1 (b) example image 2, and (c) example image 3. The images are recorded inthree RGB-colors with the aid of the SONY XC-003P CCD color camera (three chips) and theMatrox Magic Color frame grabber. Two light sources of average daylight color are used toilluminate the objects in the scene.

9.6.3 Detection of Highlights

The features used for highlight detection are K1 and K2 applied on the HSBcolor channels yielding a five-dimensional space for each image point:

Gaussian: The Gaussian method performs well to detecthighlights, see Figure 9.10b. Most of the highlights are detected.However, only a few false posi- tives are found (e.g., bar-shapedstructures). This is because the reflectances at these structures arecomposed of a portion of specular reflection.

Mixture of Gaussians:The MoG method gives slightly better resultsthan the Gaussian method (see Figure 9.10c). For this method, the

highlighted bars, found by the Gaussian method, are discarded. k-Nearest neighbor: This method performs slightly worse as

opposed to the detection method based on a single Gaussian (seeFigure 9.10d). The problem with the highlighted bars is still present.

Summary:The detection methods based on a single Gaussian as wellas on the

MoG are well suited for highlight detection.

9.6.4 Detection of Geometry/Shadow Edges

The features that are used to detect geometry/shadow edges are the first-order derivatives applied on both the RGB and the c1 c2 c3 color models.Further, the second-order derivative is applied on the RGB color images:

(a)(b) (c) (d)

FIGURE9.10

(a) Test image, (b) Gaussian classifier, (c) mixture of Gaussians, and (d) k-nearest neighbor. Based


37/43

on the (training)efficiency and accuracy of the results, the Gaussian or MoG are most appropriate for highlightdetection.


38/43


39/43

Recall

FIGURE9.12

Precision/recall graph for the classifiers of corners.


40/43

TABLE9.3

Classifiers and Their Performance forCorner Detection

Classifier Precisio Recall

Gaussian 34.9% 56.6%Mixture of 77.8% 52.8%k-Nearest neighbor 83.9% 58.5%

various settings have been examined. The results are shown in Figure 9.12.The threshold providing the highest accuracy, and subsequently used in ourexperiments, is 0.75.

From Table 9.3, it is observed that the k-nearest neighbor classifier provides

the highest performance. Examining the precision/recall graphs for the threeclassifiers reveals that this method provides good performance. Further, theMoG performs slightly better than the single Gaussian method.

The k-nearest neighbor classifier provides the best performance to detectcorners. Al- though the recall of all three methods is similar, the precision ofthe k-NN classifier is higher.

9.7

Conclusion

In this chapter, we discussed methods and techniques in the field of colorfeature detection. In particular, the focus was on the following importantissues: color invariance, combining derivatives, fusion of color models, colorsaliency boosting, and classifying image structures.

To this end, the dichromatic reflection model was outlined first. Thedichromatic reflection model explains the RGB values variations due to theimage formation process. From the model, various color models are obtainedshowing a certain amount of invariance. Then, color (invariant) derivatives

were discussed. These derivatives include quasi-invariants that have propernoise and stability characteristics. To combine color derivatives into a singleoutcome, the color tensor was used instead of taking the sum or Euclideandistance. Tensors are convenient to describe color derivative vectors. Basedon the color tensor, various image feature detection methods wereintroduced to extract locale image structures such as edges, corners, andcircles. The experimental results of Canny color edge detection for severalphotometric quasi-invariants showed stable and accurate edge detection.Further, a proper model was discussed to select and weight color (invariant)models for discriminatory and robust image feature detection. The use of thefusion model is important, as there are many color invariant models available.In addition, we used color to express saliency. It was shown that after colorsaliency boosting, (less interesting) black-and-white structures in the imageare ignored and more interesting color structures are detected. Finally, aclassification framework was outlined to detect and classify local image struc-tures based on photometrical and geometrical information. High classification


41/43

accuracy is obtained by simple learning strategies.In conclusion, this chapter provides a survey on methods solving important

issues in the field of color feature detection. We hope that these solutions onlow-level image feature detection will aid the continuing challenging task ofhandling higher-level computer vision tasks, such as object recognition and

tracking.


42/43

References

[1] R. Haralick and L. Shapiro, Computer and Robot Vision, Vol. II, Addison-Wesley,Reading, MA,

1992.[2] C. Schmid, R. Mohr, and C. Bauckhage, Evaluation of interest point detectors, Int.J. Comput.

Vision, 37, 2, 151172, 2000.[3] J. Shi and C. Tomasi, Good features to track, in Proceedings of IEEE Conference onComputerVision

and Pattern Recognition, Seattle, pp. 819825, 1994.[4] S. Di Zenzo, Note: A note on the gradient of a multi-image, Comput. Vision.GrapAics.and Image

Process., 33, 1, 116125, 1986.[5] J. Bigun, Pattern recognition in images by symmetry and coordinatetransformations, Comput.

Vision and Image Understanding, 68, 3, 290307, 1997.[6] J. Bigun, G. Granlund, and J. Wiklund, Multidimensional orientation estimation withapplica-

tions to texture analysis and optical flow, IEEEArans. on Patt. Anal. andMacAineIntelligence, 13,8, 775790, 1991.

[7] O. Hansen and J. Bigun, Local symmetry modeling in multi-dimensional images,Patt. Recog-

nition Lett., 13, 253262, 1992.[8] J. van de Weijer, L. van Vliet, P. Verbeek, and M. van Ginkel, Curvature estimationin oriented

patterns using curvilinear models applied to gradient vector fields, IEEEArans.Patt. Anal. and

MacAineIntelligence, 23, 9, 10351042, 2001.[9] S. Shafer, Using color to seperate reflection components, COLOR Res. Appl., 10,210218, Winter

1985.[10] T. Gevers and H. Stokman, Robust histogram construction from colorinvariants for ob-

ject recognition, IEEE Arans. on Patt. Anal. and MacAine Intelligence (PAMI), 26,1, 113118,2004.

[11] T. Gevers and A. W. M. Smeulders, Color based object recognition, Patt.Recognition, 32, 453464,

March 1999.[12] G. Klinker and S. Shafer, A physical approach to color image understanding, Int.

J. Comput.Vision, 4, 738, 1990.

[13] J. Geusebroek, R. van den Boomgaard, A. Smeulders, and H. Geerts, Colorinvariance, IEEE

Arans. Patt. Anal. MacAineIntelligence, 23, 12, 13381350, 2001.[14] J. van de Weijer, T. Gevers, and J. Geusebroek, Edge and corner detection byphotometric quasi-

invariants, IEEEArans. Patt. Anal. andMacAineIntelligence, 27, 4, 625630, 2005.

[15] L. Itti, C. Koch, and E. Niebur, Computation modeling of visual attention, Nat. Rev.Neuroscience,

2, 194203, March 2001, 365372.[16] J. van de Weijer andT. Gevers, Boosting color saliency in image feature detection,in International

Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA, pp.


43/43

365372, 2005.[17] J. van de Weijer, T. Gevers, and A. Bagdanov, Boosting color saliency in imagefeature detection,

IEEEArans. Patt. Anal. andMacAineIntelligence, 28, 1, 150156, 2006.[18] T. Gevers and H. Stokman, Classification of color edges in video into shadow-

geometry, high-light, or material transitions, IEEEArans. on Multimedia, 5, 2, 237243, 2003.

[19] T. Gevers and F. Aldershoff, Color feature detection and classification by learning,in Proceedings

IEEE International Conference on Image Processing (ICIP), Vol. II, pp. 714717,2005.

[20] J. van de Weijer, T. Gevers, and A. Smeulders, Robust photometric invariantfeatures from the

color tensor, IEEEArans. Image Process., 15, 1, 118127, 2006.[21] S.D. Zenzo, A note on the gradient of a multi-image, Comput. Vision. GrapAics.and Image Process.,

33, 116125, 1986.[22] G. Sapiro and D.L. Ringach, Anisotropic diffusion of multivalued imageswith app-

lications to color filtering, IEEE Arans. Patt. Anal. and MacAine Intelligence, 5,15821586,1996.

[23] G. Sapiro and D. Ringach, Anisotropic diffusion of multivalued images withapplications to

color filtering, IEEEArans. Image Process., 5, 15821586, October 1996.

[24] C. Harris and M. Stephens, A combined corner and Manchester, UK edge detector,in Proceedings oftAe FourtAAlvey Vision Conference, Manchester, UK, Vol. 15,1988, pp. 147151.

[25] D.H. Ballard, Generalizing the Hough transform to detect arbitrary shapes, Patt.Recognition,

12, 111122, 1981.[26] H. Markowitz, Portfolio selection,J. Finance, 7, 1952.[27] P. Wolfe, The simplex method for quadratic programming, Econometrica, 27,1959.[28] C. Gallery, www.corel.com.
http://www.corel.com/http://www.corel.com/

Documents

Color Feature Detection