Pattern Recognition Letters 12 (1991) 519-523 September 1991 North-Holland

A Lie group theoretic approach to the invariance problem in feature extraction and object recognition

James B. Cole*, Hiroshi Murase and Seiichiro Naito NTT Basic Research Laboratory, Musashino-Shi, Tokyo 180, Japan

Received 5 April 1991

Abstract

Cole, J.B., H. Murase and S. Naito, A Lie group theoretic approach to the invariance problem in feature extraction and object recognition, Pattern Recognition Letters 12 (1991) 519-523.

We derive a formal solution to the invariance problem and construct it using Lie group generators. Representations of these generators with respect to image data are discussed. Group theoretical obstacles to three-dimensional invariant recognition and possible solutions are considered.

Keywords. Invariance problem, Lie group generators.

1. Introduction

A major goal of image analysis is to recognize objects and to extract features from their images independently of the positions, orientations and sizes of the objects.

Let x be a set of image data from some object, and let f be a function which operates on x to yield the information f ( x ) . If the image data change from x to Sx when the object 'moves' in some way (e.g., translation or rotation), then in general f ( S x ) Cf(x). S is an operator that acts on x, the 'old' data to give the new image data, Sx. Since the

*Now at Naval Research Laboratory Washington, DC 20375-5000, USA.

original object remains physically unchanged, one would like to find an 'invariant' function g such that

g(Sx) = g(x). (1)

This is a very difficult problem to solve in general. If S is an element of a Lie group, however, it is

possible to construct a formal solution of (1) (see for example (Belinfante and Kolman, 1972) or (Schultz, 1990) for a review of Lie groul~s). We seek an operator, T, that acts on f such that

Tf(Sx) = f ( x ) . (2)

T changes f in such a way as to 'compensate' for the change from x to Sx. It is straightforward, as we will show, to find T.

0167-8655/91/$03.50 © 1991 -- Elsevier Science Publishers B.V. (North-Holland) 519

Volume 12. Number 9 PATTERN RECOGNITION LETTERS September 1991

2. Formal solution

We assume that the image data can be represented in the form x = ( x l , x 2, .. .I, where x i ( i= 1,2,3 . . . . ) are 'coordinates ' that could represent the pixel brightnesses in the scene or which could be the coefficients in some kind of 'expansion ' such as that described by (Roseborough and Murase, 19901. For simplicity, we t ake f (x ) to be a real scalar func- tion. Let S(O) be a t ransformation characterized by the parameter set O=(Ot,02,03 . . . . ). S can be any operat ion such as translation, rotation, or dilation that forms a Lie group. An infinitesimal transfor- mat ion is given by

S(d0) = I + dOiGi (3)

where the Gi are the group generators and dO i are infinitesimal. Throughout this paper, unless other- wise specified, we sum on upper and lower repeated indices in accord with the Einstein summa- tion convention. A group generator is defined as

G i = lim (S(0) - 1)/0 i 0-0

where S(0)=1 by definition. Applying an in- finitesimal t ransformation, S(dO), to x, we obtain

S(d0) x = x+ dOiGi x. (5)

Now making the substitutions x ~ S(dO)-Ix and S--, S(d0) in (2), where S(O) -t denotes the inverse o f S(O), we find

Tf(x) = f ( S ( d 0 ) - t x). (6)

For Lie groups

S(d0) -! = S ( - d 0 ) = 1 - dOiGi,

thus

S (d0) - Ix = X-- dOiGi x.

Substituting into (6) and expanding in a Taylor series up to the first order in d0, we obtain

Tf(x) = f ( x ) - dO i(Gi x) j aj f ( x ) , (71

where (Gix) j denotes the j - th component o f (Gix), and Ojf(x) denotes the partial derivative of

f with respect to x j. T thus depends on 0 and its infinitesimal form, T(d0), is given by

T(d0) = 1 - dOi(Gi x) j aj. (81

The generators of 7", H i , are thus

Hi = -(Gi x) j aj. (9)

From the theory of Lie groups, a finite Lie group transformation can be expressed in terms of its generators by

T(O) = e °'n'. (10)

See (Lenz, 1990) or (Kanatani, 1990) for more details o f this derivation.

Definingf*(x, 0) = T(O)f(x) and expanding (10) we obtain

f* (x , 0) = T(O) f ( x ) = e - ° ' ~ ' x~, a ,f(x)

= f ( x ) - Oi(G i x)i Oj f ( x )

l oioJ(Gix)l al((Gjx) kak f ( x ) ) +2-~ + - . . . ( l l )

f* (x ,O) satisfies (1) in the sense that

f* (x , O) = f*(S(O)-Ix, 0).

Although f is arbitrary, in order to actually con- struct f * (x , 0), its derivatives must be sufficiently 'well behaved' that the series in (l 1) converges. Although f* (x , 0) is an invariant function, it con- tains 0, whose value is usually not accessible f rom raw image data.

But knowing T, another approach is now pos- sible. We can attempt to solve the differential equation

T(O) f = f (121

for f . This is usually a difficult problem but it is relatively easy to find a class of ' f i rst-order ' solu- tions, as we shall later see.

3. Constructing an invariant image function

We now discuss how to actually constructf*(x, 0) with respect to some transformation. For simplicity, we take S(0) to be a one-parameter Lie group.

520

Volume 12. Number 9 PATTERN RECOGNITION LETTERS September 1991

Making the replacements G i --+ G and oi--* O, (11) becomes

f * (x, 0 ) = f (x) - O(Gx) i 8i f (x)

~. 02(Gx)i Oj((Gx) i 8if(s)) +

1 03(Gx) t Ot((Gx)J Oj((Gx)i 8if(x))) 3[

+ - - - . 0 3 )

Calculating the derivatives, we obtain

f*(x, O) = f (x) - O(Gx) i Oif(x)

1 02{(Gx)i(Gx) i aj i)if(x) + (G2x) i aif(x) } +

! 03{(Gx)i(Gx)J(Gx)k Ok Oj Oif(x) 3!

+ 2(G2x)i(Gx) j Oi Ojf(x) + (G3x) i 8if(x)}

+ . . . . (14)

Now if we choose f = f t o where fo~ satisfies

(Gx) i 8 i f O~ = O, (15)

the second term on the right in (14) vanishes along with parts of the higher order terms. By postu- lating a form f o r f 0~, a class o f 'first-order' solu- tions to (12) can be determined. Such solutions may be useful in practical problems.

Now, to proceed further, we must determine the group generator, G.

4. Lie group generators for image data

Let us represent x as an n x I column vector, and let {el} ( i=1,2 . . . . . n) be any complete set of orthonormal basis vectors that could be used to represent the image data, such as the Walsh pat- terns. Expanding x in the form X= ~,i xiei , a rep- resentation for G in terms of an n x n square matrix in terms o f the {el} can be found. Re- arranging (5), making the substitutions x ~ e j , Gi -* G and multiplying on the left by el , we find, using the definition of G,

eT (S(dO)ey - ej) = dO e/r Ge~

= dOG]. (16)

To determine G, we apply an infinitesimal trans- formation, S(d0), to ej and expand S(dO)e~-ej in the form

S(dO)ej - ej = ~ (dej)ie~, i

where the (dej) i are the expansion coefficients. The elements of G are thus

Gj = (defli/dO.

Defining S(O)ej =ej(O), we can write

i e T d Gj = i -~ej(O), (17)

where (d/dO)ej(O) denotes the derivative at 0 = 0 . G is thus determined in the form of an n x n matrix. This procedure can be carried out either 'theoretically' or 'empirically' using (16) or (17).

From (16), Gj=eTGej. Inserting a basis vector expansion of x into (Gx)', it is easy to show that (Gx)i=xJGj, where repeated upper and lower in- dices are summed as usual. T, as given in (14), can then be expressed as

f*(x, O) = f (x) - OxJ Gj Oi f (x)

+102{xkx tG i k G[ Oj aif(x)+xJ(G2)~Oif(x)}

. . . . . (18)

The foregoing developments have been carried out without reference to any specific set o f basis vectors. To actually construct f * , however, a 'suit- able' set must be selected, and their transformation properties defined. These choices then determine the generator representations. These choices are not completely arbitrary. Generators o f such physi- cal operations as translation and rotation must reflect their corresponding physical properties. Translation generators in x- and y-directions, Gx and Gy, respectively, must obey [Gx, Gr]=O (where [a, b] = a b - ba), since translating first in the x-direction and then in the y-directiod should give the same result as performing these operations in the reverse order.

In a future paper we shah discuss representa- tions of image data that are suitable for the analysis outline above, but we illustrate our ap- proach with a simple example.

521

Volume 12, Number 9 PATTERN RECOGNITION LETTERS September 1991

6. Example

Let o be an 'ordinary' 2-dimensional column vector, given by

o T = x e I +Ye2,

where

eT = (1 O) and e~ = (0 1)

are basis vectors. Let e)(O)=S(O)ey be a basis vec- tor that has been rotated by angle 0. We can deter- mine 'experimentally' that

el(O) = (cos O)el + (sin 0)e2

and

e2(0) = ( -s in 0 ) e I + (COS 0 ) e 2.

i Using (17) we calculate the Gj=ei.(d/dO)ej(O) and find that

G ~ = - I , G 2 = 1 , and G l , = G ~ = 0 .

Now let f = f ( x , y ) be some arbitrary function. Subst i tut ingfand G into (18), we construct a func- tion, f * , that is invariant under rotations accor- ding to

f * (x, y, O) = f (x , y) - Ox Oy f (x , y) + Oy Ox f(x , y)

. 1 1 . . . . o

A first-order invariant function with respect to G must satisfy

x ayf-y a x f = O.

It is not difficult to show that one such function is

f (x , y) = x 2 + y2.

brightness data alone, PQ can be guaranteed to form a Lie group only in certain special cases, such as when Q is restricted to rotations about an axis perpendicular to the plane of projection or when the object has special symmetries. The requirement that PQ form a group thus constitutes a severe constraint on Q with respect to pixel brightness data.

If we use a set of basis vectors that contain 'extra' information about an object (for example, the appearance of occluded surfaces or texture in- formation), such as the eigenvectors described by (Roseborough and Murase, 1990), S may have Lie group properties even though it is not a Lie group element with respect to a pixel brightness represen- tation. This topic will be explored in a later paper.

The human perception systsm is able to solve the invariance problem in a wide variety of situations even when S is not a Lie group element with respect to pixel brightness data. The perception system utilizes many cues such as shading and curvature as well as prior knowledge along with a priori assump- tions to extract information from an image. We might say that this process amounts to a modifica- tion of P to a new operator P* such that S = P * Q is an element of a Lie group for a wide class of operations, Q. An interesting topic for future work is to study P*.

We wander from a firm mathematical founda- tion, but even if S = P Q only 'approximately' satisfies the properties of a Lie group, f* (x , 0) as given by (11) may be 'approximately' invariant and 'reasonable' first-order solutions of (12) may still be possible. Such questions can only be addressed empirically.

7. Discussion

Most image data are obtained by gathering the light reflected from the surface of a three- dimensional object onto a two-dimensional plane. In this case, S can be expressed in the form S = PQ, where P is a 'projection' operator, and Q is some operation (such as rotation) on a physical object of which we have a two-dimensional image. Unfor- tunately S = P Q is not necessarily an element of a Lie group even when Q is. With respect to pixel

8. Summary

If S(O) is a Lie group transformation on a set of image data, x, an invariant funct ionf*(x , 0) can be constructed according to (1 I) that formally satis- fies (1). To find an invariant function that is in- dependent of 0, one must solve (12). General solutions are difficult, but it is relatively easy to determine a set of 'first-order' solutions by solving 05) . Although it is difficult to actually solve (12), eqs. (14) and (18) yield criteria for constructing im-

522

Volume 12, Number 9 PATTERN RECOGNITION LETTERS September 1991

age invariants. I f we want invariance with respect to more than one image transformation, the general development of Section 2 must be applied.

The requirement that S be a member of a Lie group constitutes a severe constraint with respect to pixel brightness data alone, but may be less of a constraint with respect to other image represen-

tations.

Kanatani, K. (1990). Group Theoretical Methods in Image Understanding. Springer, Berlin.

Lenz, R. (1990). Group Theoretical Methods in Image Proces- sing. Springer, Berlin.

Roseborough, J. and H. Murase (1990). Partial eigenvalue de- composition for very large matrices using run-length en- coding. NTT Basic Research Laboratory, Internal Report, Tokyo. Submitted to IEEE Trans. Pattern Anal. Machine Intell.

Schutz, B.F. (1985). Geometrical Methods of Mathematical Physics. Cambridge Univ. Press, London.

References

Belinfante, J.G.F. and B. Kolman (1972). A Survey of Lie Groups and Lie Algebras with Applications to Computa- tional Methods. SIAM, Philadelphia, PA.

523