[Marc Moonen] SVD and Signal Processing III Algor(BookFi.org)

SVD AND SIGNAL PROCESSING, III

Algorithms, Architectures and Applications

This Page Intentionally Left Blank

SVD AND SIGNAI~ PROCESSING, III

Algorithms, Architectures and Applications

Edited by

MARC MOONEN BART DE MOOR

ESAT-SISTA

Department of Electrical Engineering Katholieke Universiteit Leuven

Leuven, Belgium

1995

ELSEVIER

Amsterdam Lausanne New York Oxford Shannon - Tokyo

ELSEVIER SCIENCE B.V. Sara Burgerhartstraat 25

P.O. Box 211, 1000 AE Amsterdam, The Netherlands

Library of Congress Cata ]oglng- ln -Publ lcat ton Data

SVD and s igna l p rocess ing I I I : a l g o r i t h m s , a r c h i t e c t u r e s , and a p p l i c a t i o n s / ed i ted by Marc Moonen, Bar t De Moor.

p. cm. Includes blbl lographlcal references and index. ISBN 0-444-82107-4 (a]k. paper) I. Signal processing--Dlgita] techniques--Congresses.

2. Decomposition (Mathematlcs)--Congresses. I . Moonen, Marc S., 1963- . I I . Moor, Bart L. R. de., 1960- . I I I . T i t l e : SVD and signal processing three. IV. T i t l e : SVD and signal processing 3. TK5102.9.$83 1995 621 .382 '2 '015194- -dc20 95-3193

CIP

ISBN: 0 444 82107 4

�9 1995 Elsevier Science B.V. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the publisher, Elsevier Science B.V., Copyright & Permissions Department, P.O. Box 521, 1000 AM Amsterdam,

The Netherlands.

Special regulations for readers in the U.S.A.- This publication has been registered with the Copyright Clearance Center Inc. (CCC), 222 Rosewood Drive, Danvers, MA, 01923. Information can be obtained from the CCC about conditions under which photocopies of parts of this publication may be made in the U.S.A. All other copyright questions, including photocopying outside of the U.S.A., should be referred to the copyright owner,

Elsevier Science B.V., unless otherwise specified.

No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or

ideas contained in the material herein.

pp. 375-382, 415-422, 467-474: Copyright not transferred.

This book is printed on acid-free paper.

Printed in The Netherlands.

P R E F A C E

This book is a compilation of papers dealing with matrix singular value decomposition and its application to problems in signal processing. Algorithms and implementation architectures for computing the SVD are discussed, as well as a variety of applications such as systems and signal modeling and detection. The book opens with a translation of a historical paper by Eugenio Beltrami, containing one of the earliest published discussions of the singular value decomposition. It also contains a number of keynote papers, highlighting recent developments in the field.

The material in the book is based on the author contributions presented at the 3rd International Workshop on SVD and Signal Processing, held in Leuven, August 22-25, 1994. This workshop was partly sponsored by EURASIP and the Belgian NFWO (National Fund for Scientific Research), and organized in co-operation with the IEEE Benelux Signal Processing Chapter, the IEEE Benelux Circuits and Systems Chapter. It was a continuation of two previous workshops of the same name which were held at the Les Houches Summer School for Physics, Les Houches, France, September 1987, and the University of Rhode Island, Kingston, Rhode Island, U.S.A., June 1990. The results of these workshops were also published by Elsevier Science Publishers B.V. (SVD and Signal Processing, Algorithms, Applications and Architectures, edited by E.F. Deprettere, and SVD and Signal Processing, Algorithms, Analysis and Applications, edited by R. Vaccaro).

It has been a pleasure for us to organize the workshop and to work together with the authors to assemble this book. We feel amply rewarded with the result of this co-operation, and we want to thank all the authors here for their effort. We would be remiss not to thank the other members of the workshop organizing committee, Prof. Ed Deprettere, Delft Uni- versity of Technology, Delft, The Netherlands, Prof. Gene Golub, Stanford University, Stanford CA, U.S.A., Dr. Sven Hammarling, The Numerical Algorithms Group Ltd, Ox- ford, U.K., Prof. Franklin Luk, Rensselaer Polytechnic Institute, Troy NY, U.S.A., and Prof. Paul Van Dooren, Universit@ Catholique de Louvain, Louvain-la-Neuve, Belgium. We also gratefully acknowledge the support of L. Delathauwer, who managed all of the workshop administration.

Marc Moonen Bart De Moor Leuven, November 199~


vii

C O N T E N T S

A short introduction to Beltrami's paper

On bilinear functions E. Beltrami, 1873 English Translation" D. Boley

PART 1. K E Y N O T E P A P E R S

1. Implicitly restarted Arnoldi/Lanczos methods and large scale SVD applications D. C. Sorensen 21

2. Isospectral matrix flows for numerical analysis U. Helmke 33

3. The Riemannian singular value decomposition B.L.R. De Moor 61

4. Consistent signal reconstruction and convex coding N. T. Thao, M. VetterIi 79

PART 2. A L G O R I T H M S AND T H E O R E T I C A L C O N C E P T S

5. The orthogonal qd-algorithm U. von Matt 99

6. Accurate singular value computation with the Jacobi method Z. Drmad 107

viii Contents

7. Note on the accuracy of the eigensolution of matrices generated by finite elements Z. DrmaS, K. Veselid 115

8. Transpose-free Arnoldi iterations for approximating extremal singular values and vectors M.W. Berry, S. Varadhan 123

9. A Lanczos algorithm for computing the largest quotient singular values in regularization problems P.C. Hansen, M. Hanke 131

10. A QR-like SVD algorithm for a product/quotient of several matrices G.H. Golub, K. SoMa, P. Van Dooren 139

11. Approximating the PSVD and QSVD S. Qiao 149

12. Bounds on singular values revealed by QR factorizations C.- T. Pan, P. T.P. Tang 157

13. A stable algorithm for downdating the ULV decomposition J.L. Barlow, H. Zha, P.A. Yoon 167

14. The importance of a good condition estimator in the URV and ULV algorithms R.D. Fierro, J.R. Bunch 175

15. L-ULV(A), a low-rank revealing ULV algorithm R.D. Fierro, P.C. Hansen 183

16. Fast algorithms for signal subspace fitting with Toeplitz matrices and applications to exponential data modeling S. Van Huffel, P. Lemmerling, L. Vanhamme 191

17. A block Toeplitz look-ahead Schur algorithm K. Gallivan, S. Thirumalai, P. Van Dooren 199

Contents ix

18. The set of 2-by-3 matrix pencils - Kronecker structures and their transitions under perturbations - And versal deformations of matrix pencils B. KdgstrSm 207

19. J-Unitary matrices for algebraic approximation and interpolation -The singular case P. Dewilde 209

PART 3. A R C H I T E C T U R E S AND REAL T I M E I M P L E M E N T A T I O N

20. Sphericalized SVD Updating for Subspace Tracking E.M. Dowling, R.D. DeGroat, D.A. Linebarger, H. Ye 227

21. Real-time architectures for sphericalized SVD updating E.M. Dowling, R.D. DeGroat, D.A. Linebarger, Z. Fu 235

22. Systolic Arrays for SVD Downdating F. Lorenzelli, K. Yao 243

23. Subspace separation by discretizations of double bracket flows K. Hiiper, J. GStze, S. Paul 251

24. A continuous time approach to the analysis and design of parallel algorithms for subspace tracking J. Dehaene, M. Moonen, J. Vandewalle 259

25. Stable Jacobi SVD updating by factorization of the orthogonal matrix F. Vanpoucke, M. Moonen, E.F. Deprettere 267

26. Transformational reasoning on time-adaptive Jacobi type algorithms H.W. van Dijk, E.F. Deprettere 277

27. Adaptive direction-of-arrival estimation based on rank and subspace tracking B. Yang, F. Gersemsky 287

28. Multiple subspace ULV algorithm and LMS tracking S. Hosur, A.H. Tewfik, D. Boley 295

Conmn~

P A R T 4. A P P L I C A T I O N S

29. SVD-based analysis of image boundary distortion F.T. L uk, D. Vandevoorde 305

30. The SVD in image restoration D.P. O'Leary 315

31. Two dimensional zero error modeling for image compression J. Skowronski, L Dologlou 323

32. Robust image processing for remote sensing data L.P. Ammann 333

33. SVD for linear inverse problems M. Bertero, C. De Mol 341

34. Fitting of circles and ellipses, least squares solution W. Gander, R. Strebel, G.H. Golub 349

35. The use of SVD for the study of multivariate noise and vibration problems D. Otte 357

36. On applications of SVD and SEVD for NURBS identification W. Ma, J.P. Kruth 367

37. A tetradic decomposition of 4th-order tensors : Application to the source separation problem J.-F. Cardoso 375

38. The application of higher order singular value decomposition to independent component analysis L. De Lathauwer, B. De Moor, J. Vandewalle 383

39. Bandpass filtering for the HTLS estimation algorithm: design, evaluation and SVD analysis H. Chen, S. Van Huffel, J. Vandewalle 391

Con~n~

40. Structure preserving total least squares method and its application to parameter estimation H. Park, J. B. Rosen, S. Van HuffeI 399

41. Parameter estimation and order determination in the low-rank linear statistical model R.J. Vaccaro, D.W. Tufts, A.A. Shah 407

42. Adaptive detection using low rank approximation to a data matrix L P. Kirsteins, D.W. Tufts 415

43. Realization of discrete-time periodic systems from input-output data E.L Verriest, J.A. Kullstam 423

44. Canonical correlation analysis of the deterministic realization problem J.A. Ramos, E.L Verriest 433

45. An updating algorithm for on-line MIMO system identification M. Stewart, P. Van Dooren 441

46. Subspace techniques in blind mobile radio channel identification and equalization using fractional spacing and/or multiple antennas D.T.M. Slock 449

47. Reduction of general broad-band noise in speech by truncated QSVD: implementation aspects S.H. Jensen, P.C. Hansen, S.D. Hansen, J.A. Sorensen 459

48. SVD-based modelling of medical NMR signals R. de Beer, D. van Ormondt, F.T.A.W. Wayer, S. Cavassila, D. Graveron-Demilly, S. Van Huffel 467

49. Inversion of bremsstrahlung spectra emitted by solar plasma M. Piana 475

Authors index 485


SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) �9 1995 Elsevier Science B.V. All rights reserved.

A S H O R T I N T R O D U C T I O N T O B E L T R A M I ~ S P A P E R

The last 30 years the singular value decomposition has become a popular numerical tool in statistical data analysis, signal processing, system identification and control system analysis and design. This book certainly is one of the convincing illustrations of this claim. It may come as a surprise that its existence was already established more than 120 years ago, in 1873, by the Italian geometer Beltrami. This is only 20 years after the conception of a matrix as a multiple quantity by Cayley in his 'Memoir on Matrices' of 1858. Says Lanczos in [8, p.100]: "The concept of a matrix has become so universal in the meantime that we often forget its great philosophical significance. To call an array of letters by the letter A was much more than a matter of notation. It had the significance that we are no longer interested in the numerical values of the coe.O~cients aij. The matrix A was thus divested of its arithmetic significance and became an algebraic operator." And Sylvester himself says: "The idea of subjecting matrices to the additive process and of their consequent amenability to the laws of functional operation was not taken from it (Cayley's 'Memoir ' ) but occured to me independently before I had seen the memoir or was acquainted with its contents." Approximately twenty years thereafter, there was a sequence of papers on the singular value decomposition, starting with the one by Beltrami [3] that is translated here. Others were by Jordan [7], Sylvester [11] [12] [13] (who wrote some of his papers in French and called the singular values, les multiplicateurs canoniques.), and by Autonne [1] [2]. After that, the singular value decomposition was rediscovered a couple of times and received more and more attention. For the further history and a more detailed explanation of these historical papers, we refer to the paper by Stewart [9] and a collection of historical references in the book by Horn and Johnson [6, Chapter 3].

We would like to thank Prof. Dan Boley of the University of Minnesota, Minneapolis, U.S.A., for his translation of the Beltrami paper.

E. Beltrami

References

[1] Autonne L. Sur les matrices hypohermitiennes et les unitaires. Comptes Rendus de l'Acadhmie des Sciences, Paris, Vo1.156, 1913, pp.858-860.

[2] Autonne L. Sur les matrices hypohermitiennes et sur les matrices unitaires. Ann. Univ. Lyon, Nouvelle S6rie I, Fasc. 38, 1915, pp.1- 77.

[3] Beltrami E. Sutle Funzioni Bilineari, Giornale di Mathematiche, Battagline G., Fergola E. (editors), Vol. 11, pp. 98-106, 1873.

[4] Cayley A. Trans. London Phil. Soc., Vo1.148, pp.17-37, 1858. Collected works: Vol. II, pp.475-496.

[5] Dieudonn4 J. Oeuvres de Camille Jordan. Gauthier-Villars & Cie, Editeur-Imprimeur-Libraire, Paris, 1961, Tome I-II-III.

[6] Horn R.A., Johnson C.R. Topics in matrix analysis. Cambridge University Press, 1991.

[7] Jordan C. Mdmoire sur les formes bilindaires. J. Math. Pures Appl. II, Vol.19, pp.35-54, 1874. (see reprint in [5, Vol.III, pp.23-42]).

[8] Lanczos C. Linear Differential Operators. D. Van Nostrand Com- pany Limited London, 1961.

[9] Stewart G.W. On the early history of the singular value decomposition. SIAM Review, December 1993, Vol.35, no.4, pp.551-566.

[10] Sylvester J.J. C.R. Acad. Sci. Paris Vo1.108, 1889 pp.651-653- Mess. of Math. Vol.19, 1890, pp.42-46.

[ i i ] Sylvester J.J. Sur la rdduction biorthogonale d'une forme lindo- lin'eaire h sa forme canonique. Comptes Rendus, CVIII., pp.651- 653, 1889 (see also in [10, p.638]).

[12] Sylvester J.J. A new proof that a general quadric may be reduced to its canonical form (that is, a linear function of squares) by means of a real orthogonal substitution. Messenger of Mathematics, XIX, pp.l-5, 1890 (see also [10, p.650])

A Short Introduction to Beltrami's Paper

[13] Sylvester J.J. On the reduction of a bilinear quantic of the n T H

order to the form of a sum of n products by a double orthogonal substitution. Messenger of Mathematics, XIX, pp.42-46, 1890 (see also [10, p.654]).


SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 1995 Elsevier Science B.V.

ON BILINEAR FUNCTIONS

(SULLE FUNZIONI BILINEARI)

by

E. B E L T R A M I

English Translation- Copyright @1990 Daniel Boley

A Translation from the Original Italian of one of the Earliest Published Discussions of the Singular Value Decomposition.

Eugenio Beltrami (Cremona, 1835 - Rome, 1900) was professor of Physics and Mathemat- ics at the University of Pavia from 1876 and was named a senator in 1899. His writings were on the geometry of bilinear forms, on the foundations of geometry, on the theory of elasticity, electricity, and hydrodynamics, on the kinematics of fluids, on the attractions of ellipsoids, on potential functions, and even some in experimental physics.

E. Beltrami

GIORNALE

DI MATEMATICHE AD USO DEGLI STUDENTI

DELLE UNIVERSITA ITALIANE

PUBBLICATO PER CURA DEI PROFESSORI

G. B A T T A G L I N I , E. F E R G O L A

IN UNIONE DEI PROFESSORI

E. D ' O V I D I O , G. T O R E L L I e C. S A R D I

V o l u m e XI . -1873

NAPOLI

BENEDETTO PELLERANO EDITORE LIBRERIA SCIENTIFICA E INDUSTRIALE

Strada di Chiaia, 60

On Bilinear Functions

JOURNAL

OF MATHEMATICS FOR USE OF THE STUDENTS

OF THE ITALIAN UNIVERSITIES

PUBLISHED UNDER THE DIRECTION OF THE PROFESSORS

G. B A T T A G L I N I , E. F E R G O L A

TOGETHER WITH THE PROFESSORS

E. D 'OVIDIO, G. TORELLI and C. SARDI

V o l u m e XI.-1873

NAPLES

BENEDETTO PELLERANO PUBLISHER SCIENTIFIC AND INDUSTRIAL BOOKSTORE

50 Chiaia Road

English Translation- Copyright @1990 Daniel Boley



O N B I L I N E A R F U N C T I O N S

by

E. B E L T R A M I

The theory of bilinear functions, already the subject of subtle and advanced research on ~he part of the eminent geometers KttONECKER and CHttISTOFFEL (Journal of BOttCHAttDT t. 68), gives occasion for elegant and simple problems, if one removes from it the restriction, almost always assumed until now, that the two series of variables be subjected to identical substitutions, or to inverse substitutions. I think it not entirely unuseful to treat briefly a few of these problems, in order to encourage the young readers of this Journal to become familiar ever more with these algebraic processes that form the fundamental subject mat ter of the new analytic geometry, and without which this most beautiful branch of mathematical science would remain confined within a symbolic geometry, which for a long time the perspicuity and the power of pure synthesis has dominated.

Let f - ~rsCrsXrYs

be a bilinear function formed with the two groups of independent variables

Xl~ x2, ...Xn;

Yl , Y2 , .. . Yn.

Transforming these variables simultaneously with two d i s t i n c t linear substitutions

(1) xr = ~rarp~p , Ys = F~sbsq~?q,

(whose determinants one supposes to be always different from zero) one obtains a transformed form

= 2pqTpq~p?Tq,

whose coefficients 7pq axe related to the coefficients crs of the original function by the n 2 equations that have the following form:

(2) ~ / p q - ~rsCrsarpbsq.

Setting for brevity 2 m c , ~ r a m s = hrs , E m c ~ m b , ~ = k ~ ,

10 E. Be l t rami

this typical equation can be writ ten in the two equivalent forms

(3) Eshspbsq - 7pq, ~rkrqarp - 7pq.

Indicating with A, B, H, K, I" the determinants formed respectively with the elements a,

b, c, h, k, 7, one has, from these last equations, I' = HB = KA. But, by the definition of the quantities h, k, one has as well H = CA, K = CB; hence

F = ABC,

that is, the determinant of the transformed function is equal to that of the original one multiplied by the products of the moduli of the two substitutions.

Let us suppose initially that the linear substitutions (1) are both orthogonal. In such a case, their 2n 2 coefficients depend, as is known, on n 9 - n independent parameters , and on the other hand, the transformed function ~ can be, generally speaking, subjected to as many conditions. Now the coefficients 7pq whose indices p, q are mutual ly unequal are exactly n 2 - n in number: one can therefore seek if it is possible to annihilate all these coefficients, and to reduce the bilinear function f to the canonical form

To resolve this question, it suffices to observe that if, after having set in equations (3)

> 7vq = 0 for p < q and 7pp = 7p,

one multiplies the first by brq and one carries out on the result the summation Eq; then one multiplies the second by asp and one carries out the summation Ep, one obtains

h~p = 7vbrp, ksq - %asq.

These two typical equations are mutual ly equivalent to the corresponding ones of equations (3), and, as a consequence of these lat ter ones, one could in this way recover the equations (3) in the process. In them are contained the entire resolution of the problem posed, tha t one obtains in this way: Writing out the last two equations in the following way

c l ra l s + c2ra2s + ... + Cnrans = 7sbrs,

(4) crlbls + cr2b2s + ... + crnbns = 7sars ,

then setting, for brevity,

CrlCsl + Cr2Cs2 + ... + CrnCsn • ~rs ,

ClrCls ~- C2rC2s ~" ... -~ Cnrgns -- Vrs ,

(so tha t #rs = #st, vrs = r'sr), the substitution into the second equation (4) of the values of the quantities b obtained from the first one yields

(5)1 #rqals + Pr2a2s + ...-4- ~rnans = 72ars;

On Bilinear Functions 11

likewise, the subst i tut ion onto the first equation (4) of the values of the quantities a recovered from the second one yields

(5)2 Vrlbls + v r 2 b 2 s + . . . + vrnbns = " / ' 2b r s .

The elimination of the quantities a from the n equations tha t one deduces from equations (5)x setting r = 1, 2, ... n in succession, leads one to the equation

#11 - 7 2 #12 ... #1,~

A 1 = # 2 1 # 2 2 - - ,,[2 . . . # 2 n " - 0 .oo ~ . . . . . . . . . . . . . . . . . . . . ~176 .~

2 # n l # n 2 . . . # n n - - "~

which the n values 3,12, 72, ... 72 of 3, 2 must satisfy. Likewise, the elimination of the quantities b from the n equations tha t one deduces from equation (5)2 sett ing r = 1, 2, ...n in succession, leads one to the equation

V l l - - ,,/2 /212 . . . V l n

A 2 _. /221 /222 _ ,)12 .. /'/2n �9 - - 0~

�9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12nl 12n2 . . . V n n - - 7 2

possessing the same properties as the preceding equation. It follows tha t the two determinants A1, and A 2 are mutual ly identical for any value of gamma (*). In fact, they are entire functions of degree n with respect to 9 ,2, which become identical for n -t- 1 values of 7 2, that is for the values 72, 722, ... 72 that simultaneously make both determinants zero, and for the value h' = 0 tha t makes them both equal to C 2.

The n roots 7 2, 72, ... 72 of the equation A = 0 (likewise indicating indifferently A1 = 0 or A2 = 0) are all real, by virtue of a very well known theorem; to convince oneself tha t they are also posit ive, it suffices to observe that the coefficients of

7 0 , - -7 2 , 7 4 , --7 6 , etc.

are sums of squares. But one can also consider tha t , by the elementary theory of ordinary quadratic forms and by virtue of the preceding equations, one has

2 2 2 2 2 2 F = Z~r~#~,z~z, = ")'I ~I + ... + 2 2 2 2 2 2 .

G - - ~ r , V r s Y r Y s ---- ~'17"]1 ~- "/2T]2 "~ "'" ~- ~ 'nT]n,

on the other hand, one has as well

f - - ~ m ( g l m X 1 -4- C 2 m Z 2 J I - . . . ~- C n r n X n ) 2 '

G = Zm(cmlyl + c,~2Y2 + ... + c,~,y~)2;

hence the two quadrat ic functions F, and G are essentially positive, and the coefficients 72, 72, ... 7 2 of the t ransformed expressions, i.e. the roots of the equation A = 0, are necessarily all positive.

(*) This theorem one finds demonstrated in a different way in w of Determinants by BRIOSCHI

12 E. Beltrami

The proposed problem is therefore susceptible of a real solution, and here is the procedure: Find first the roots 7 2, 7 2, ... 7 2 of the equation A = 0, (which is equivalent to reducing one or the other of the quadratic functions F, G to the canonical form); then with the help of the equations of the form (5)1 and of those of the form

2 = i , a~, + a~, + ... + a , ,

one determines the coefficients a of the first substitution (coefficients that admit an ambiguity of sign common to all those items in the same column). This done, the equations that have the form (4) supply the values of the coefficients b of the second substitution (coeffi- dents that also admit an ambiguity of sign common to all those items in the same column, so that each of the quantities 7, are determined only by its square 72). Having done all these operations, one has two orthogonal substitutions that yield, exactly as desired by the problem, the identity

ErsCrsZrYs = EmTmfm~m,

in which everyone of the coefficients 7m must be taken with the same sign that it is assigned in the calculation of the coefficients b.

It is worth observing that the quadratic functions denominated F and G can be derived from the bilinear function f setting in the latter on the one hand

ys : C l s X l "4" C2s;r,2 "Jr .. . "~ CnsZn~

and on the other

Zr = CrlYl + cr2Y2 + ... + C r n y n .

Now if in these two relations one applies the substitutions (1), one then sees immediately that they are respectively converted into the following relations in the new variables ~ and ~?:

~,~ = 7,~f,~, f,~ = %~-~,

which transform the canonical bilinear function

into the respective quadratic functions

2 2 2 2 2 2 72~2 + + 7 ~ , 7:f: + ...

2 2 2 2 "~: ~1 + "y22~ 2 + ... + "Y.~..

And, in fact, we have already noted that these two last functions are equivalent to the quadratics F, G.

We ask of what form must be the bilinear function f so that the two orthogonal substitutions that reduce it to the canonical form turn out substantially mutually identical. To this end, we observe that setting

b:s = =t=als, b2, - -4-a2,, ... bns = =t=a,~s

On B i l i n e a r Func t ions 13

the equations (4) are converted to the following:

Clrals "4- c2ra2s + ... ~- cnra,~s = 4"Tsars,

c r l a l s "4" cr2a2s -}-... + Crnans -- +7oa,.~,

which, since they must hold for every value of r and s, give crs = Csr. Reciprocally this hypothesis implies the equivalence of the two linear substitutions. So every bilinear form of the desired form is a s s o c i a t e d h a r m o n i c a l l y with an ordinary quadratic form: that is to say that designating this quadratic form with

the bilinear function is 1 de

f = r ~ , g ~ y , .

In this case, the equation zX = 0 can be decomposed in this way:

(:11 - - '~ s . . . C l n (:11 "~- "~ C12 . . . C l n

A - - c2 1 (:22 - - "}I . . . C 2 n C21 C22 "~- "[ . . . C 2 n .

Cnl On2 . . . Cnn -- "~ Cnl On2 . . . Cnn "4-'~

The first factor of the second member, set to zero, gives the well known equation that serves to reduce the function r to its canonical form. If the two substitutions are absolutely mutually identical, then coefficients 7 are the same for the quadratic function and for the bilinear one. But if, as we have already supposed, one concedes the possibility of an opposition of sign between the coefficients a and b belonging to two columns of equal index, the coefficient 7s of the corresponding index in the bilinear function can have sign opposite to that in the quadratic. From this, the presence in the equation A = 0 of a factor having for roots the quantities 7 taken negatively.

In the particular case just now considered, the quadratic function denoted F is

and, on the other hand, one finds that there always exists an orthogonal substitution which makes simultaneously identical the two equations

r 2, 2 ~ =2~2~ 2.

This is a consequence of the fact that, as is well known,

is a symbol invariant with respect to any orthogonal substitution.

14 E. Beltrami

We translate into geometrical language the results of the preceding analysis, assuming (as is generally useful to do) that the signs of the coefficients 7 are chosen to make AB = 1 and hence F = C.

Let Sn, S~ be two spaces of n dimensions with null curvature, referred to, respectively, by the two systems of orthogonal linear coordinates z and y, for which we will call O and O ~ the origins. To a straight line $1 drawn through the origin 0 in the space Sn, there corresponds a specific set of ratios zz : x2 : ... : x,~; and on the other hand, the equation f = 0, homogeneous and of first degree in the xl , x2, ..., zn and in the Yl, Y2, ..., yn defines a correlation of figures in which to each line through the point O in the space Sn corresponds a locus of first order in n - 1 dimensions that we call S ~ within the space n--1 S~; and vice-versa.

By virtue of the demonstrated theorem, it is always possible to substi tute for the original coordinate axes in the z's and y's new axes in the ~'s and r/'s, respectively, with the same origins O and O ~, so that the correlation rule assumes the simpler form

~1~IT}1 "~" ~2~2~2 "~" ... "~- "~n~nT}r~ : 0.

Said this, one may think of the axis system r/, together with its figure, moved so tha t its origin O ~ falls on O, and that each axis ~r falls on its homologous axis ~r (*). In such a hypothesis, the last equation expresses evidently that the two figures are found to be in polar or involutory correlation with respect to the quadric cone (in n - 1 dimensions)

~ + ~ + ... + ~ = 0

that has its vertex on O. Hence, one can always convert a correlation of first degree of the above type, through a motion of one of the figures, into a polar or involutory correlation with respect to a quadric cone (in n - 1 dimensions) having its vertex on the common center of the two figures overlaid.

In the case of n = 2, this general proposition yields the very well known theorem that two homographic bundles of rays can always be overlaid in such a way that they consti tute a quadratic involution of rays.

In the case that n = 3, one has the theorem, also known, that two correlative stars (i.e., such tha t to every ray in one corresponds a plane in the other, and vice-versa) can always be overlaid in such a way that they become reciprocal polar with respect to a quadric cone having its vertex on the common center.

One can interpret the analytic theorem in another way, and recover other geometric properties in the cases n = 2, and n = 3. If with

YlYx + Y2Y2 -[- ... + YnYn "b 1 = 0

one represents a locus of first order (S~_l) lying in the space S~, to every orthogonal substi tution of the form

Ys -" ~qbsq~q (*) Something that is possible by having AB = 1.

On Bilinear Functions 15

applied on the local coordinates y, corresponds an identical orthogonal substi tut ion

Y8 = 2CqbsqEq

applied to the tangential coordinates Y. Given this, suppose that between the x's and y's one institutes the n - 1 relations that result from setting equal the n ratios

Y" ( , = I , 2, ... '0. ClrXl + C 2 r X 2 + . . . + CnrXn

This is equivalent to considering two homographic stars with centers on the points 0 and 0 ~. With such hypotheses, the equation

ylY1 + y2Y2 + ... + ynY,~ = 0,

which corresponds to this other equation in the ~?-axis system

~?IE1 + rl2E: + ... + ~?~E,~ = 0,

is equivalent to the following relation between the x's and the y's

~rsCrsXrYs = O,

and this is in turn reducible, with two simultaneous orthogonal substitutions, to the canonical form

71fiE1 + 72~2E2 + ... + %f,~En = 0.

Now the relation that this establishes between the new tangential coordinates E cannot differ from that contained in the third to last equation; hence it must be

~ . - - . . . - - .

' T I ~ I " T 2 ~ 2 "Tn~n

From these equations, which are nothing else than the relations of homography, expressed in the new coordinates ~ and ~?, it emerges evidently that the axis ~1 and r/l, ~2 and ~72, ... ~n and r/n are pairs of corresponding straight lines in the two stars. Since it is thus possible to move one of the stars in such a way that the ~ axes and the ~7 axes coincide one for one, one concludes that , given two homographic stars in an n dimensional space, one can always overlay one upon the other so that the n double rays acted upon by the overlaying, constitute a system of n orthogonal cartesian rays.

From this, setting n = 2 and n = 3, one deduces that two homographic groups of rays can always be overlaid so that the two double rays are orthogonal; and that the two homographic stars can always be overlaid so that the three double rays form an orthogonal cartesian triple.

Returning now to the hypothesis of two arbitrary linear substitutions, but always acting to give to the transformed function ~ the canonical form r = EmTm~m~?m, one solve the equations (3) with respect to the quantities h and k, respectively, with which one finds the two typical equations, mutually equivalent,

(6) Bhrs = Brs%, Akr, = Ars%,

16 E. Beltrami

in which At,, B,s are the algebraic complements of the elements ara, bra in the respective determinants A, B. The equations (4) are nothing else than particularizations of these.

Let ff be a second bilinear function, given by the expression

f = Z~clrszrys ,

and suppose we want to transform simultaneously, with the very same linear substitutions (1), the function f into the canonical form ~o and the function f into the canonical form

= Z,~7,~,~y,~.

Indicating with h ~ and k ~ quantities analogous to the h and k for the second function, together with the equations (6), these further equations must hold

(6)' Bh~,~ = B ~ 7 : , Akin, = A~,7: .

Dividing the first two equations of each of the pairs (6) and (6)' one by the other, one obtains

hra - Aah~rs = 0 where As = "Y---ts 7; '

o r

(7) ( C l r -- A s C t l r ) a l s Jr" (C2r - - AsCt2r )g2s Jr" .. . Jr" (Cnr -- A s C t n r ) a n . = O.

Setting r = 1, 2, ...n in this last equation and eliminating the quantities als, a2s, ... arts from the n equations obtained in this way, one arrives at the equation

C l l - - AC~l c12 - ~c~2 .. . s ~ c ~ n

O - - c21 - A c ~ l c22 - AC~2 "" c2n - Act2n -- 0 ..o . o . . ~ ..o ..~ . . . . . . . ~ ,~ ,~176

Cnl - ACtnl cn2 - Actn2 .. . C n n - ~ C : r ,

which must be satisfied by the n values AI, A2, ... A n of A. One can convince oneself of this by another way, by observing that | is the determinant of the bilinear function f- A f, and that hence one has by virtue of the general theorem on the transformation of this determinant (*)

O = AB('),I- A ' y ~ ) ( " [ 2 - A " [ 2 ) . . . ( ' ~ ' r ~ - A'] /~) ,

whence emerges just as expected, for A and B to be different from zero, that the equation | = 0 has for its roots the n ratios

71 72 Vn ~' ~' ...~,,.

Of the rest, since the two series of quantities V and V ~ are not determined except by these ratios, it is useful to assume for more simplicity that 7~ = V~ . . . . . V~ = I, and hence % = A~.

(*) f r o m th i s one sees t h a t t h e d e t e r m i n a n t of a b i l inea r f u n c t i o n is zero on ly w h e n t h e f u n c t i o n i t se l f can be r e d u c e d to c o n t a i n two fewer va r i ab les , a p r o p e r t y t h a t one can eas i ly show d i rec t ly .

On Bil inear Functions 17

Here then is the procedure that leads to the solution of the problem: Find the n roots (real or imaginary) A1, A2, ..., A,~ of the equation | = 0; then substitute them successively into the equations of the form (7). One obtains in this way, for each value of s, n linear and homogeneous equations, one of which is a consequence of the other n - 1, so that one can recover only the values of the ratios az, : a2s : . . . : arts. Having chosen arbitrarily the values of the quantities al , , a2s, ..., an, so that they have these mutual ratios, the equations that are of the form of the second part of (6) ~ yield

I A I I C2rA2s Cnr (8) b,, - - Czr z, + + ... + An,. CIA

and in this way all the unknown quantities can be determined.

One can observe that if for the coefficients at, one substitutes the products paara, something that is legitimate by a previous observation, the coefficients bra, determined by the last equation, are converted into ~ . This is the same as saying that if for the variables ~, one substitutes p,~,, the variables ~?s are converted to ~ ; but this change leaves unaltered the transformed functions 7~ and 7~ ~.

If one denotes by | the algebraic complement of the element cr, - Adr, in the determinant $ , and by Sr,(A,) what results by setting A = A, in this complement, it is easy to see that the equations (7) are satisfied by setting

a,. , = c ~ , O , . , ( , L ) .

In this way, if one forms the equations analogous to (7) and containing the coefficients b in place of the coefficients a, they can be satisfied by setting

b,, =/~,o,,(~,,).

The quantities aa and ~a are not determined completely: but the equations analogous to (2) determine the product aa~a. In any case, these factors are not essential, since by writing ~a in place of aa~a and ~?, in place of ~,~7,, they can be removed. From the expressions

a.. = O . . (A . ) . b . = O . . (A . ) .

that result from this supposition, it is evident that this important property emerges, that ~ i.e. when the two bilinear functions are associated harmonically when Cr, = C,r, %, = C,r,

to tWO quadratic functions, the substitutions that reduce them both simultaneously to canonical form are subs tant ia l ly mutua l ly identical.

If we suppose that the second bilinear function ft has the form

f l : XlYl + x2Y2 + ... + XnYn,

the general problem just treated assumes the following form: Reduce the bilinear function f to the canonical form

18 E. Beltrami

with two simultaneous linear substitutions, so that the function

X l Y l + x 2 Y 2 q" . . . q" Z n Y n

is transformed into itself. In this case, the general formula (8) becomes

whence

and hence

Abe, = A~,,

Ay~ = A r l r / 1 q- A r 2 r / 2 + . . . q- Arnrh~,

~Ta = a l s Y l + a28Y2 + ... + anaYn.

This last formula shows that the two substitutions (a) and (b) are inverses of each other, something that follows necessarily from the nature of the function that is transformed into itself. The relations among the coefficients a and b that follow from this

~rarsbr, = 1, ~,arsbrs = 1 Y~rarsbr,, = 0, ~,arsbr,s = 0,

A B = I

make a perfect contrast to those that hold among the coefficients of one orthogonal substitution, and they reduce to them when ara = br,, such a case arising (by what we have recently seen) when the form f is associated to a quadratic form. In the special problem which we have mentioned (and which has already been treated by Mr. CHRISTOrFEL at the end of his Memoirs), the equation 0 = 0 takes on the form

C l l - - )~ s . . . C l n

C21 C22 - - '~ " " C'2n - - O~

. . . . . . . . , , . . . . . . . . . . . . . . . . . . . .

Cnl Cn2 . . . Cnn - -

and under the hypothesis cr, = csr (that we have just alluded to), it is identified, as is natural, with what the analogous problem leads to via a true quadratic form.

We will not add any word on the geometric interpretation of the preceding results, since their intimate connection with the whole theory of homogeneous coordinates is evident.

Likewise, we will not discuss for now the transformations of bilinear functions into themselves, an important argument but less easy to handle than the preceding ones, for which M r . C H I t I S T O F F E L has already presented the treatment in the case of a single function and a single substitution.

P A R T 1

K E Y N O T E P A P E R S



21

I M P L I C I T L Y R E S T A R T E D A R N O L D I / L A N C Z O S M E T H O D S A N D L A R G E S C A L E S V D A P P L I C A T I O N S

D.C. SORENSEN Department of Computational and Applied Mathematics Rice University P. O. Box 1892 Houston, TX 77251 sorensen @rice. edu

ABSTRACT. Implicit restarting is a technique for combining the implicitly shifted QR mechanism with a k-step Arnoldi or Lanczos factorization to obtain a truncated form of the implicitly shifted QR-iteration for eigenvalue problems. The software package ARPACK that is based upon this technique has been successfully used to solve large scale symmetric and nonsymmetric (generalized) eigenvalue problems arising from a variety of applications. The method only requires a pre-determined limited storage proportional to n times the desired number of eigenvalues. Numerical difficulties and storage problems normally asso- dated with Arnoldi and Lanczos processes are avoided. This technique has also proven to be very useful for computing a few of the largest singular values and corresponding singular vectors of a very large matrix. Biological 3-D Image Reconstruction and Molecular Dynamics are two interesting applications. The SVD plays a key roll in a classification procedure that is the computationally intensive portion of the 3-D image reconstruction of biological macromolecules. A relatively new application is to analyze the motions of the proteins using the SVD instead of normal mode analysis. The primary research goal is to pick out the non-harmonic phenomena that usually contains most of the interesting behavior of the protein's motion. This paper reviews the Implicitly Restarted Arnoldi/Lanczos Method and briefly discusses the biological applications of the SVD.

KEYWORDS. Lanczos/Arnoldi methods, implicit restarting, image reconstruction, molecular dynamics, computational biology.

1 I N T R O D U C T I O N

Large scale eigenvalue problems arise in a variety of applications. In many of these, relatively few eigenvalues and corresponding eigenvectors are required. The Lanczos method

22 t D. C. Sorensen

for symmetric problems and the Arnoldi method, which is a generalization to nonsymmetric problems, are two important techniques used to compute a few eigenvalues and corresponding vectors of a large matrix. These methods are computationally attractive since they only require matrix-vector products. They avoid the cost and storage of the dense factorizations required by the more traditional Q R-methods that are used on small dense problems.

A number of well documented numerical difficulties are associated with the Lanczos process [11]. These include maintenance of numerically orthogonal basis vectors or related techniques to avoid the introduction of spurious approximations to eigenvalues; required storage of basis vectors on peripheral devices if eigenvectors are desired; no a-priori bound on the number of Lanczos steps (hence basis vectors) required before sufficiently accurate approximations to desired eigenvalues appear. Since the Arnoldi technique generalizes the Lanczos process, these difficulties as well as the additional difficulties associated with computing eigenvalues of nonsymmetric matrices are present [13]. The Arnoldi method does not lead to a three term recurrence in the basis vectors as does Lanczos. Hence considerable additional arithmetic and storage is associated with the more general technique.

Implicit restarting is a technique for combining the implicitly shifted QtL mechanism with a k-step Arnoldi or Lanczos factorization to obtain a truncated form of the implicitly shifted Qtt-iteration for eigenvalue problems. The numerical difficulties and storage problems normally associated with Arnoldi and Lanczos processes are avoided. The technique has been developed into a high quality public domain software package ARPACK that has been useful in a variety of applications.

This software has proven to be very effective for computing a few of the largest singular values and corresponding singular vectors of a very large matrix. Two applications of the SVD occur in Biological 3-D Image Reconstruction and in Molecular Dynamics. The SVD plays a key roll in a classification procedure that is the computationally intensive portion of the 3-D image reconstruction of biological macromolecules. A relatively new application is to analyze the motions of the proteins using the SVD instead of normal mode analysis. The primary research goal is to pick out the non-harmonic phenomena that usually contains most of the interesting behavior of the protein's motion.

This paper will review the implicit restarting technique for eigenvalue calculations and will discuss several applications in computing the SVD and related problems.

2 S O F T W A R E F O R LARGE SCALE E I G E N A N A L Y S I S

The Arnoldi process is a well known technique for approximating a few eigenvalues and corresponding eigenvectors of a general square matrix. A new variant of this process has been developed in [14] that employs an implicit restarting scheme. Implicit restarting may be viewed as a truncation of the standard implicitly shifted QR-iteration for dense problems. Numerical difficulties and storage problems normally associated with Arnoldi and Lanczos processes are avoided. The algorithm is capable of computing a few (k) eigenvalues with user specified features such as largest real part or largest magnitude using 2nk + O(k2)storage. No auxiliary storage is required. Schur basis vectors for the k-dimensional eigen-space are computed which are numerically orthogonal to working precision.

Implicitly Restarted Arnoldi/Lanczos Methods 23

This method is well suited to the development of mathematical software. A software package ARPACK [10] based upon this algorithm has been developed. It has been designed to be efficient on a variety of high performance computers. Parallelism within the scheme is obtained primarily through the matrix-vector operations that comprise the majority of the work in the algorithm. The software is capable of solving large scale symmetric, nonsymmetric, and generalized eigenproblems from significant application areas.

This software is designed to compute approximations to selected eigenvalues and eigenvectors of the generalized eigenproblem

Ax = AMx, (1)

where both A and M are real n x n matrices. It is assumed that M is symmetric and positive semi-definite but A may be either symmetric or nonsymmetric.

Arnoldi's method is a Krylov subspace projection method which obtains approximations to eigenvalues and corresponding eigenvectors of a large matrix A by constructing the orthogonal projection of this matrix onto the Krylov subspace Span{v , Av, ..., A k - l v } . The Arnoldi process begins with the specification of a starting vector v and in k steps produces the decomposition of an n x n matrix A into the form

A V = V H + f e T, (2)

where v is the first column of the matrix V E l:t nxk, v T v = Ik; H E R, kxk is upper Hessenberg, f E R n with 0 = v T f and ek E R k the kth standard basis vector. The vector f is called the residual.

This factorization may be advanced one step at the cost of a (sparse) matrix-vector product involving A and two dense matrix vector products involving V T and V. The explicit steps are:

1 . / ~ = Ilfll; v + - / 1 ~ ; v +- ( v , v); H +- Zekr .

2. z ~ Av;

3. h ~ VTz; f ~-- z - Vh;

4. H .-- (H, h);

The dense matrix-vector products may be accomplished using level 2 BLAS. In exact arithmetic, the columns of V form an orthonormal basis for the Krylov subspace and H is the orthogonal projection of A onto this space. In finite precision arithmetic, care must be taken to assure that the computed vectors are orthogonal to working precision. The method developed in [4] may be used to accomplish this.

Eigenvalues and corresponding eigenvectors of H provide approximate eigenvalues and eigenvectors for A. If

H y = y O , and we put x = V y .

24 D. C. Sorensen

Then x, 0 is an approximate eigenpair for A with

liAr, - xS[I = []f[lleTy[,

and this provides a means for estimating the quality of the approximation.

The information obtained through this process is completely determined by the choice of the starting vector. Eigen-information of interest may not appear until k gets very large. In this case it becomes intractable to maintain numerical orthogonality of the basis vectors V and it also will require extensive storage. Failure to maintain orthogonality leads to a number of numerical difficulties. Implicit restarting provides a means to extract interesting information from very large Krylov subspaces while avoiding the storage and numerical difficulties associated with the standard approach. It does this by continually compressing the interesting information into a fixed size k dimensional subspace. This is accomplished through the implicitly shifted QR mechanism. An Arnoldi factorization of length k + p is compressed to a factorization of length k by applying p implicit shifts resulting in

AVk++p + + T = Yk+pHk+ p + fk+pek+pQ , (3)

where Vk~ p = Vk+pQ, Hk+ p+ = QTHk+pQ, and Q = Q1Q2"" "Qp, with Qj the orthogonal matrix associated with the shift #j. It may be shown that the first k - 1 entries of the

T vector ek+pQ are zero. Equating the first k columns on both sides yields an updated k - s t ep Arnoldi factorization

Av: = V:H+ + (4)

with an updated residual of the form f+ = Vk+_4.pek+l~k + fk+po'. Using this as a starting point it is possible to use p additional steps of the Arnoldi process to return to the original form. Each of these applications implicitly applies a polynomial in A of degree p to the starting vector. The roots of this polynomial are the shifts used in the Q R process and these may be selected to filter unwanted information from the starting vector and hence from the Arnoldi factorization. Full details may be found in [14].

The software ARPACK that is based upon this mechanism provides several features which are not present in other (single vector) codes to our knowledge

�9 Reverse communication interface

�9 Ability to return k eigenvalues which satisfy a user specified criterion such as largest real part, largest absolute value, largest algebraic value (symmetric case), etc.

�9 A fixed pre-determined storage requirement suffices throughout the computation. Usually this is n , O ( 2 k ) + O ( k 2) where k is the number of eigenvalues to be computed and n is the order of the matrix. No auxiliary storage or interaction with such devices is required during the course of the computation.

�9 Eigenvectors may be computed on request. The Arnoldi basis of dimension k is always computed. The Arnoldi basis consists of vectors which are numerically orthogonal to working accuracy.

�9 Accuracy: The numerical accuracy of the computed eigenvalues and vectors is user specified and may be set to the level of working precision. At working precision,

Implicitly Restarted Arnoldi /Lanczos Methods 25

the accuracy of the computed eigenvalues and vectors is consistent with the accuracy expected of a dense method such as the implicitly shifted QR iteration.

�9 Multiple eigenvalues offer no theoretical or computational difficulty other than additional matrix vector products required to expose the multiple instances. This cost is commensurate with the cost of a block version of appropriate blocksize.

3 C O M P U T I N G T H E SVD W I T H A R P A C K

In later sections we shall discuss large scale applications of the Singular Value Decomposition (SVD). To avoid confusion with the basis vectors V computed by the Arnoldi factorization, we shall denote the singular value decomposition of a real m x n matrix M by

M = U S W T

and we consider only the short form with U E R mx'~ , W E l:t "~x'~ and S = d i a g ( a l , a2, ..., an)

with o'j ~_ aj+l . We assume m > n and note that u T u = w T w = In.

The applications we have in mind focus upon computing the largest few singular values and corresponding singular vectors of M. In this case (with m > n ) it is advantageous to compute the desired right singular vectors W directly using ARPACK

z ~ A v where A = M T M

If Wk is the matrix comprised of the first k columns of W and Sk is the corresponding leading principal submatrix of S then

A W k -- W k ( S k 2)

The corresponding left singular vectors are obtained through the relation

vk = M W k S [ ~.

Although this mechanism would ordinarily be numerically suspect, it appears to be safe in our applications where the singular values of interest are large and well separated. Finally, it should be noted that the primary feature that would distinguish the implicit restarting method from other Lanczos style iterative approaches [1] to computing the SVD is the limited storage and the ability to focus upon the desired singular values.

An important feature of this arrangement is that space on the order of 2nk is needed to compute the desired singular values and corresponding right singular vectors and then only m k storage locations are needed to compute the corresponding left singular vectors. The detail of the matrix vector product z ~ A v calculation is also important. The matrix M can be partitioned into blocks M T = (MT, M T, ..., M T) and then matrix-vector product z ~ A v may be computed in the form

26 D. C. Sorensen

�9 for j = 1, 2, ..., b,

z ~ z + M T ( M j v ) ; 1.

end.

Moreover, on a distributed memory parallel machine or on a network or cluster of work- stations these blocks may be physically distributed to the local memories and the memory traffic associated with reading the matrix M every time a matrix vector product is required could be completely avoided.

4 L A R G E S C A L E A P P L I C A T I O N S OF T H E SVD

In this section we discuss two important large scale applications of the SVD. The first of these is the 3-D image reconstruction of biological macromolecules from 2-D projections obtained through electron micrographs. The second is an application to molecular dynamical simulation of the motions of proteins. The SVD may be used to compress the data required to represent the simulation and more importantly to provide an analytical tool to help in understanding the function of the protean.

4.1 ELECTRON MICROGRAPHS

We are currently working with Dr. Wah Chiu and his colleagues at the Baylor College of Medicine to increase the efficiency of their 3-dimensional image reconstruction software. Their project involves the identification and classification of 2-dimensional electron micrograph images of biological macromolecules and the subsequent generation of the corresponding high resolution 3-D images. The underlying algorithm [16] is based upon the statistical technique of principal component analysis [17]. In this algorithm, a singular value decomposition (SVD) of the data set is performed to extract the largest singular vectors which are then used in a classification procedure. Our initial effort has been to replace the existing algorithm for computing the SVD with ARPACK which has increased the speed of the analysis by a factor of 7 on an Iris workstation. The accuracy of the results were also increased dramatically. Details are reported in [6]. The next phase of this project will introduce the parallel aspects discussed previously.

There are typically several thousand particle images in a working set, and in theory, each one characterizes a different viewing angle of the molecule. Once each image is aligned and processed to eliminate as much noise as possible the viewing angle must be determined. The set is then Fourier transformed, and each transformed image is added to a Fourier composite in a position appropriate to the this angle. The composite is then transformed back to yield a high resolution 3-D image.

If we consider each 2-D image to be a vector in N dimensional space, where N is the number of pixels, we know that we can describe each image as a weighted sum of the singular vectors of the space defined by the set. The distribution of the images in this N dimensional coordinate system is then used to break up the set into classes of viewing angles, and the images in a class are averaged. Determining the left singular vectors of


the space of images is presently the computational bottleneck of the process. As is true in many applications, the classification can be accomplished with relatively few of the singular vectors corresponding to the largest singular values.

In a test example (using real data) the largest 24 singular triplets uiaiw T were extracted from a data set of size 4096x1200, corresponding to 1200 64x64 images. The decomposition was done with an existing software package as well as with a newer algorithm with the hopes that the computational bottleneck would be resolved. The existing package ran with 24 iterations of its method in roughly 29 minutes of CPU time. The new routines performed 7 iterations in just under 4 minutes, an improvement in speed by a factor of 7. In this test, the data set fits into memory. The ultimate goal is to process data sets consisting of 100,000 images each 128x128 pixels and this will require 6.4Gbytes of storage. I/O will then be a severe limitation unless a parallel I/O mechanism as described above can be implemented.

4.2 MOLECULAR DYNAMICS

Knowledge of the motions of proteins is important to understanding their function. Pro- teins are not rigid static structures, but exist as a complex dynamical ensemble of closely related substates[7]. It is the dynamics of the system that permit transitions between these conformational substates and there is evidence that this dynamic behavior is also responsi- ble for the kinetics of ligand entry and ligand binding in systems such as myoglobin. These transitions are believed to involve anharmonic and collective motions within the protein.

X-ray crystallography is able to provide high-resolution maps of the average structure of a protein, but this structure only represents a static conformation. Molecular dynamics, which solves the Newtonian equations of motion for each atom, is not subject to such approximations and can be used to model the range of motions available to the protein on short (nanosecond) time-scales. Time-averaged refinement extends the molecular dynamics concept, restraining the simulation by requiring the time-averaged structure to fit the observed X-ray data[2]. With this methodology, an ensemble of structures that together are consistent with the dynamics force-fields and the observed data is generated. This ensemble represents a range of conformations that are accessible to proteins in the crystalline state.

One difficulty with time-averaged refinements is the plethora of generated conformations. No longer is one structure given as the model, rather, thousands of structures collectively describe the model. Although it is possible to analyze this data directly by manually examining movies of each residue in the simulation until some pattern emerges, this is not a feasible approach. Nor is it feasible to watch an animation of an entire protein and discern the more subtle global conformation states that may exist. The SVD can provide am important tool for both compressing the data required to represent the simulation and for locating interesting functional behavior.

In order to employ the SVD we construct a matrix C from the simulation data. A column of C represents the displacement of the atoms from their mean positions at a given instant of time in the simulation. Each group of three components if this column vector make up the spatial displacement of a particular atom from its mean position. As discussed in Section 2 above, we construct an approximation to C using the leading k singular values

28 D. C. Sorensen

and vectors.

C ~ Ck = Uk&V T.

For a visually realistic reconstruction of the simulation (ie. run the graphics with Ck in place of C one can typically set k around 15 to 20).

This representation not only compresses the data but classifies the fundamental spatial modes of motion of the atoms in the protein. The SVD also includes temporal information in a convenient form. It is the temporal information that sets it apart from the more traditional eigendecomposition analysis. This transform can also be thought of as a quasi- normal mode calculation based on the dynamics data rather than the standard normal mode analysis obtained by an approximation to the system's potential energy function. This standard approach only admits harmonic motions. Using the SVD, it is possible to extract information about the distribution of motion in the protein, and to classify the conformational states of the protein and determine which state the system is in at any point in the simulation. Projections of the entire trajectory onto the SVD basis can also be used for understanding the extent of configuration space sampling of dynamics simulations and the topography of the system's energy hypersurface[3].

In [12], we have computed a time-averaged refinement of a mutant myoglobin. Using the SVD, we have characterized the motions within the protein and begun to automate this analysis. We have also used the SVD to distinguish between different conformational states within the dynamics ensemble. This study exposes anharmonic motions associated with the ligand binding sites through analysis of the distribution of the components of the right singular vectors. It also documents the efficiency of computing the SVD through ARPACK in this application.

5 G E N E R A L A P P L I C A T I O N S OF A R P A C K

AR.PACK has been used in a variety of challenging applications, and has proven to be useful both in symmetric and nonsymmetric problems. It is of particular interest when there is no opportunity to factor the matrix and employ a "shift and invert" form of spectral transformation,

fi ~ ( A - a I ) -1 . (5)

Existing codes often rely upon this transformation to enhance convergence. Extreme eigenvalues {#} of the matrix A are found very rapidly with the Arnoldi/Lanczos process and the corresponding eigenvalues {A} of the original matrix A are recovered from the relation A = 1/# + a. Implementation of this transformation generally requires a matrix factorization. In many important applications this is not possible due to storage requirements and computational costs. The implicit restarting technique used in ARPACK is often successful without this spectra/transformation.

One of the most important classes of application arise in computational fluid dynamics. Here the matrices are obtained through discretization of the Navier-Stokes equations. A typical application involves linear stability analysis of steady state solutions. Here one lin- earizes the nonlinear equation about a steady state and studies the stability of this state


through the examination of the spectrum. Usually this amounts to determining if the eigenvalues of the discrete operator lie in the left halfplane. Typically these are parametrically dependent problems and the analysis consists of determining phenomena such as simple bifurcation, Hopf bifurcation (an imaginary complex pair of eigenvalues cross the imaginary axis), turbulence, and vortex shedding as this parameter is varied. ARPACK is well suited to this setting as it is able to track a specified set of eigenvalues while they vary as functions of the parameter. Our software has been used to find the leading eigenvalues in a Couette-Taylor wavy vortex instability problem involving matrices of order 4000. One interesting facet of this application is that the matrices are not available explicitly and are logically dense. The particular discretization provides efficient matrix-vector products through Fourier transform. Details may be found in [5].

Very large symmetric generalized eigenproblems arise in structural analysis. One example that we have worked with at Cray Research through the courtesy of Ford motor company involves an automobile engine model constructed from 3D solid elements. Here the interest is in a set of modes to allow solution of a forced frequency response problem (K - AM)x = f ( t ) , where f ( t ) is a cycnc forcing function which is used to simulate expanding gas loads in the engine cylinder as well as bearing loads from the piston connecting rods. This model has over 250,000 degrees of freedom. The smallest eigenvalues are of interest and the ARPACK code appears to be very competitive with the best commercially available codes on problems of this size. For details see [15].

Another source of problems arise in magnetohydrodynamics (MHD) involving the study of the interaction of a plasma and a magnetic field. The MHD equations describe the macroscopic behavior of the plasma in the magnetic field. These equations form a system of coupled nonlinear PDE. Linear stability analysis of the linearized MHD equations leads to a complex eigenvalue problem. Researchers at the Institute for Plasma Physics and Utrecht University in the Netherlands have modified the codes in ARPACK to work in complex arithmetic and are using the resulting code to obtain very accurate approximations to the eigenvalues lying on the Alfven curve. The code is not only computes extremely accurate solutions, it does so very efficiently in comparison to other methods that have been tried. See [9] for details.

There are many other applications. It is hoped that the examples that have been briefly discussed here will provide an indication of the versatility of the ARPACK software as well a the wide variety of eigenvalue problems that arise.

Acknowledgemen t s

The author is grateful to Ms. Laurie Feinswog and to Mr. Tod Romo for working with ARPACK in the 3-D image reconstruction and molecular dynamics applications mentioned above. They provided most of the information conveyed in Section 3.

References

[1] M. W. Berry, "Large scale singular value computations", IJSA ,6,13-49,(1992).

30 D. C. Sorensen

[2] J.B. Clarage, G.N. Phillips, Jr. "Cross-validation Tests of Time-averaged Molecular Dynamics Refinements for Determination of Protein Structures by X-ray Crystallog- raphy," Acta Cryst., D50,24-36,(1994).

[3] J.B. Clarage, T.D. Romo, B.M. Pettit, G.N. Phillips, Jr., "A Sampling Problem in Molecular Dynamics Simulations of Macromolecules, Keck Center Rept., Rice Univer- sity, (1994).

[4] J. Daniel, W.B. Gragg, L. Kaufman, G.W. Stewart, "Reorthogonalization and stable algorithms for updating the Gram-Schmidt QR factorization," Math. Comp.,30, 772- 795,(1976).

[5] W.S. Edwards, L.S. Tuckerman, R.A. Friesner and D.C. Sorensen, "Krylov Methods for the Incompressible Navier-Stokes Equations," Journal of Computational Physics, ~0,82-102 (1994).

[6] L. Feinswog, M. Sherman, W. Chiu, D.C. Sorensen, "Improved Computational Meth- ods for 3-Dimensional Image Reconstruction" , CRPC Tech. Rept., PAce University (in preparation).

[7] H. Frauenfelder, S.G. Sligar, P.G. Wolynes, " The Energy Landscapes and Motions of Proteins, Science, 254,1598-1603, (1991).

[8] G.H. Golub, C.F. Van Loan. Matrix computations. North Oxford Academic Publishing Co., Johns Hopkins Press,(1988).

[9] M.N. Kooper, H.A. van der Vorst, S. Poedts, and J.P. Goedbloed, "Application of the Implicitly Updated Arnoldi Method with a Complex Shift and Invert Strategy in MHD," Tech. Rept., Institute for Plasmaphysics, FOM Rijnhuizen, Nieuwegin, The Netherlands,(Sep. 1993) ( submitted to Journal of Computational Physics).

[10] R. Lehoucq, D.C. Sorensen, P.A. Vu, ARPACK: Fortran subroutines for solving large scale eigenvalue problems, Release 2.1, available from [email protected], (1994).

[11] B.N. Parlett, The Symmetric Eigenvalue Problem , Prentice-Hall, Englewood Cliffs, NJ. (1980).

[12] T.D. Romo, J.B. Clarage, D.C. Sorensen, and G.N. Phillips, Jr., "Automatic Iden- tification of Discrete Substates in Proteins: Singular Value Decomposition Analysis of Time Averaged Crystallographic Refinements," CRPC-TR 94481, Rice University, (Oct. 1994).

[13] Y. Saad, Numerical Methods for Large Eigenvalue Problems, Halsted Press-John Wiley & Sons Inc., New York (1992).

[14] D. C. Sorensen, "Implicit application of polynomial filters in a k-Step Arnoldi Method," SIAM J. Matt. Anal. Apps.,13 , 357-385, (1992).


[15] D.C. Sorensen, P.A. Vu, Z. Tomasic, "Algorithms and Software for Large Scale Eigen- problems on High Performance Computers," High Performance Computing 1993- Grand Challenges in Computer Simulation,Adrian Tentner ed., Proceedings 1993 Sim- ulation Multiconference, Society for Computer Simulation, 149-154, (1993).

[16] M. Van Heel, J. Frank, "Use of Multivariate Statistics in Analysing the Images of Biological Macromolecules," Ultramicroscopy, 6 187-194, (1981).

[17] S. Van Huffel and J. Vandewalle, The Total Least Squares Provblem: Computational Aspects and Analysis, Frontiers in Applied Mathematics 9, SIAM Press, Philadel- phia,(1991).



33

I S O S P E C T R A L M A T R I X F L O W S F O R N U M E R I C A L A N A L Y S I S

U. HELMKE Department of Mathematics University of Regensburg 93040 Regensburg Germany Uwe.Helmke@mathematik. uni-regensburg.de

ABSTRACT. Current interest in analog computation and neural networks has motivated the investigation of matrix eigenvalue problems via eigenvalue preserving differential equations. An example is Brockett's recent work on a double Lie bracket quadratic matrix differential equation with applications to matrix diagonalization, sorting and linear programming. Another example is the Toda flow as well as many other classical completely integrable Hamiltonian systems. Such isospectral flows appear to be a useful tool for solving matrix eigenvahe problems. Moreover, generalizations of such flows are able to compute the singular value decomposition of arbitrary rectangular matrices. In neural network theory similar flows have appeared in investigations on learning dynamics for networks achieving the principal component analysis. In this lecture we attempt to survey these recent developments, with special emphasis on flows solving matrix eigenvalue problems. Discretizations of the flows based on geodesic approximations lead to new, although slowly convergent, algorithms. A modification of such schemes leads to a new Jacobi type algorithm, which is shown to be quadratically convergent.

KEYWORDS. Dynamical systems, Riemannian geometry, SVD, Jacobi method.


Isospectral matrix flows are differential equations evolving on spaces of matrices, which have the defining property that their solutions have constant eigenvalues. Such equations have been studied over the past 25 years, appearing in diverse areas such as classical and statistical mechanics, nonlinear partial differential equations, numerical analysis, optimization, neural networks, signal processing and control theory. The early development of isospectral flows goes back to the pioneering work of Rutishauser on the infinitesimal analogue of

34 U. Helmke

the quotient - difference algorithm. However Rutishauser's work in the 1950s has stayed isolated and it took more than 20 years that interest into such flows did renew through the discovery of Flaschka, Symes and Kostant that the Toda flow from statistical mechanics is a continuous - time analogue of the QlZ - algorithm on tridiagonal matrices. More precisely the QI~ - algorithm was seen as a discretization of an associated isospectral differential equation, the discretization being at integer time. This work on the Toda flow has spurred further investigations of Chu, Driessel, Watkins, Elsner and others on continuous time flows solving matrix eigenvalue problems. More recently, work by Brockett and others has considerably broadened the scope of applications of isospectral flows. In this paper I will focus on the applications within numerical linear algebra. More specifically the focus will be on matrix eigenvalue methods. After a brief outline on the historical development of isospectral flows the double bracket equation of Roger Brockett is studied. This is a quadratic matrix differential equation which is derived as a gradient flow for a least squares cost function. Approximations of the flow based on short length geodesics are introduced. These lead to linearly convergent algorithms involving calculations of matrix exponentials. A variant of the classical Jacobi algorithm is introduced which is based on optimization of the least squares cost function along several geodesics. This algorithm, termed Sort-Jacobi algorithm, has attractive properties such as quadratic convergence and incorporates sorting of the eigenvalues. The extension to the SVD case is straightforward. This is a survey paper. Accordingly most proofs are omitted. I have tried to preserve the structure of my talk at the workshop in as much as possible.

2 I S O S P E C T l Z A L F L O W S

A differential equation on matrix space ~nxn

A(*) = F(A(t)), A e I~ '~• (1)

is called isospectral, if the eigenvalues of the solutions A(t) are constant in time, i.e. if

spectrum(A(t)) = spectrum(A(0)) (2)

holds for all t and all initial conditions A(0) E I~ nxn. A more restrictive class of isospectral matrix flows are the self-similar flows on I~ nxn. These are defined by the property that

A(t) = S(t)A(O)S(t) -1 (3)

holds for all initial conditions A(0) and times t, and suitable invertible transformations S(t) E GL(n, n~). Thus the Jordan canonical form of the solutions of a self-similar flow does not change in time. It is easily seen that every self-similar flow (1) has the Lie-bracket form

= [A, f(A)]

= A f ( A ) - f (A )A (4)

Isospectral Matrix Flows f o r Numerical Analysis 35

for a suitable matrix-valued function of the coefficients of A. Conversely, every differential equation on I~ nxn of the form (4)is isospectral; see e.g. Chu [7], Helmke and Moore [19].

Of course, instead of considering.time-invariant differential equations (1) we might as well consider time-varying systems A = F ( t , A ( t ) ) . More generally we might even consider isospectral flows on spaces of linear, possibly infinite-dimensional, operators, but we will not do so in the sequel.

E x a m p l e 1 (Linear Flows)

The simplest example of an isospectral flow is the linear self-similar flow

A = [ A , B ] (5)

for a constant matrix B E Ii~ nxn. The solutions are

A ( t ) = e - t s A(O)e t s ,

which immediately exhibits the self-equivalent nature of the flow.

E x a m p l e 2 (Brocke t t ' s Flow)

The second example is Brockett's double Lie bracket equation

= [A, [A, B]] = A 2 B + B A 2 - 2 A B A . (6)

For this equation there is no general solution formula known; we will study this flow in section 3.

Example 3 (QR-Flow)

This is a rather nontrivial looking example. Here

= [A, (log A)_] (7)

where B_ denotes the unique skew-symmetric part of B with respect to the orthogonal decomposition

0 -b21 . . . . bnl * . . . . . . *

B = b21 "'. "'. + 0 "'. :

b,n . . . bn,,~-i 0 0 . . . 0 ,

into skew-symmetric and upper triangular matrices. This differential equation implements exactly the QR-algorithm in the sense that for all n E N:

A ( n ) = n - th step of Qtt-algorithm applied to A(0).

Let us now briefly describe some of the earlier developments on isospectral flows.

36 U. Helmke

2.1 KORTEWEG-DE VRIES EQUATION

An important part of the theory of isospectral matrix flows is linked to comparatively recent developments in the theory of nonlinear partial differential equations and, in particular, with the theory of solitons. This is explained as follows.

Let us consider the Korteweg-de Vries (KdeV)

~ = 6 ~ - ~ . , ~ = ~ ( ~ , t ) . ( 8 )

This nonlinear partial differential equation models the dynamics of water flowing in a shallow one-dimensional channel. By a beautiful observation of P. Lax the KdeV equation has the appealing form

Lt = [P, L] (9)

where

9 2 /-, = -~-'~'z2 -F u (x , t ) ( I0)

denotes the SchrSdinger operator and

o3 P = -4~--dx 3 + 6u + 3ux. (11)

Therefore the KdeV equation is equivalent to the isospectral flow (9) on a space of differential operators. In particular, the spectrum of the SchrSdinger operator (10) yields invariants for the solutions u(z,t) of (t). It is thus obvious that the isospectral nature of (9) may lead, and indeed has led, to further insights into the solution structure of the KdeV equation.

Following this fundamental observation of Lax several other important nonlinear partial differential equations (such as, e.g., the nonlinear SchrSdinger equation or the Landau- Lifshitz equation) have also been recognized as isospectral flows. These developments then have culminated into what is nowadays called the theory of solitons, with important contributions by Gelfand-Dikii, Krichever, Novikov, Lax, Adler and van Moerbeke and many others. For further information we refer to [1], [2].

2.2 EULElZ EQUATION

One of the many mechanical systems which do admit an interpretation as an isospectral flow is the classical Euler equation

Isospectral Matrix Flows for Numerical Analysis 37

I=~= = (Ia - h)~'l~oa

Ia~aa = ( h -/2)~01~,=.

(12)

Using the notation

B'= 0 I~ "1 0 , A: = r/3 0 -r/1 (13) 0 0 I~ -1 -72 771 0

where

(~h, ~12, */3) = (/1 r I2w2, I3w3 )

a simple calculation shows that

/[ = [A2, B] = [A, AB -i- BA]. (14)

Thus the Euler equation (12) is equivalent to the quadratic isospectral matrix flow (14) on the Lie algebra so(3) of 3 x 3 skew-symmetric matrices. In particular we conclude that

IIAII2 2 2 2 2 i3w32 = Ii wl + I~02 +

is an invariant of the Euler equation. Moreover, from the identity tr(A[B, C]) = tr([A, B]C) it follows that

2tr(BAfi) = tr(BA + AB),;i = tr((AB + BA)[A, AB + BA]) = O.

Thus

tr(BA 2) = / 1 ( / 2 + I3)w 2 + I2(I1 + I3)w~ + I3(I~ + /2 )w 2

is another invariant of the Euler flow.

Thus for c _> 0, d >_ 0 the ellipsoids

Mc = {A e so(3) [ tr(A2B) = -c}

as well as the spheres {A e so(3) lllAII 2 = d} are invariant submanifolds of (14). From this the familiar phase-portrait description of the Euler equation on II~ 3 is easily obtained; see e.g. Arnol'd [2]. Thus the isospectral form (14) of the Euler equation leads to a complete integration of the flow. For a further analysis of generalized Euler equations defined on arbitrary Lie algebras we refer to Arnol'd [2], Fomenko [16].

38 U. He lmke

2.3 TODA FLOW

The Toda flow is one of the earliest examples of an isospectral flow which yields a continuous analogue of a numerical matrix eigenvalue problem. The original motivation for the flow stems from statistical mechanics; cf. Flaschka (1974, 1975), Moser (1975), Kostant (1979).

Consider the one-dimensional lattice of n idealized mass points Zl < . . . < xn, located on the real axis I~. We assume that the potential energy of the system is described by

Y(xl , . . . , z~) = ~ e~ -~+ ~ k=�92

?%

X o ' = - o o , x n + l " = +oo; while the kinetic energy has the usual form ] ~] x~. k = l

Thus, if the particles are sufficiently far apart, then the total potential energy is very small and the system behaves roughly like a gas. The Hamilton function of the system is (~ = ( ~ , . . . , ~ ) , v = (v~,..., y~))

lfi fi H ( ~ , v ) = ~ v~+ ~ - ~ + ~ k = l k = l

(15)

where Yk = :~k denotes the velocity of the k-th mass point. The associated Hamiltonian system is

OH 'xk = = Yk

Oyk OH

~]k = OXk = eXk- l -xk -- eXk-xk+l (16)

for k = 1 , . . . , n. These are the Toda lattice equations on 11~ 2n. Equivalently, the Toda lattice is described by the second order differential equations

X k - - s _ e x k - - x k + l , k = 1,. . . , n . (17)

To relate the Toda system to an isospectral flow we make the following (not invertible) change of variables (this observation is due to Flaschka).

( x l , . . . , x n , Y l , . . . , Y n ) ' ) ( a l , . . . , a n - l , b l , . . . , b n )

with

1 -1 (18) ak -- e (xk-xk+l)/2 bk -- Yk k - 1, n. ' T ~ " ' ' '

Then, in the new variables, the Toda system becomes (ao = 0, bn+l = 0).


hk = ak(bk+l -- bk)

bk = 2 ( a ~ - a ~ _ l ) , k = 1 , . . . , n . (19)

This is called the Toda flow.

Thus we have associated to every solution ( x ( t ) , y ( t ) ) o f (16) a corresponding solution (a(t),b(t)) of (19). Conversely, if (a( t ) ,b( t )) is a solution of (19) with initial conditions satisfying al(0) > 0 , . . . , an-l(0) > 0~ then one can uniquely reconstruct from (a(t),b(t)) via (18) the velocity vector (y l ( t ) , . . . , yn( t ) ) as well as the pairwise differences x l ( t ) - x2( t ) , . . . , Xn_l ( t ) - xn(t) of the positions of the mass points. Moreover, if (x(0), y(0))is an initial condition of (16), then by conservation of momentum we have

�9 ~(t) = t. ~ w(0)+ ~ ~ ( 0 ) k = l k = l k = l

Thus, given any initial condition (x(0), y(0))of (16) and a corresponding solution (a(t), b(t)) of (19), one can explicitly determine the solution (x(t) ,y( t)) of (16). In this sense the systems (16) and (19) are equivalent.

In order to exhibit the isospectral nature of the flow (19) we store the state variables as components of a tridiagonal matrix�9 Let

bl al 0

A:= al b2 "'. (20)

� 9 � 9 an_ 1 0 at,-1 bn

A �9

0 al 0

--al 0 "'.

�9 �9 " �9 an-1 0 an-1 0

(21)

With these notations the flow (19) is easily verified to be equivalent to the isospectral Toda flow on tridiagonal matrices

,~i = [A, A_] (22)

The convergence properties of the Toda flow (22) are summarized in the following theorem, whose proof is due to Moser [23], Flaschka [15], Delft, Nanda and Tomei [11].

Theorem 1.1

(a) The solutions of the Toda flow (22)

40 U. Helmke

A = [A,A_]

exist and are tridiagonal for all time.

(b) Every tridiagonal solution of (22) converges to a diagonal matrix

lim A(t) = d i a g ( A 1 , . . . , An),

where ) u , . . . , A n are the eigenvalues of A(O). For almost every initial condition A(O) in the set of tridiagonal Jacobi matrices the solutions converges to diag(Ax,. . . , An) with )u < . . . < A n .

(c) The Toda flow interpolates the QR-algorithm in the sense that for all k E N"

e A(k) - k - th QR step, applied to e A(O).

(d) Let N = d iag(1 ,2 , . . . ,n ) and A1 < . . . < An. The Toda flow is the gradient flow of n

the function t r ( N A ) = ~ iaii the isospectral manifold of tridiagonal matrices with fixed i=1

eigenvalues A1, . . . , An.

It follows immediately from this theorem that the solutions of the Toda lattice equation behave as follows: The distances I x i - xi+ll between subsequent mass points xx(t) < . . . < xn(t) go to infinity while the velocities ~] ( t ) , . . . , kn(t) converge to their limiting values given by the eigenvalues of A(0). Moreover, for generic initial conditions the limiting velocities are sorted according to their magnitude. This sorting property of isospectral flows such as the Toda flow is an important aspect of the theory and has been further analyzed by Brockett [5], Bloch, Brockett and Ratiu [4].

It is instructive to consider an example. The simplest nontrivial case is n = 3. Here the set of 3 x 3 real tridiagonal matrices with fixed distinct eigenvalues Ax < A2 < A3 can be shown to be a Riemann surface of genus 2; compare e.g. Tomei [29]. There are 3! = 6 equilibrium points corresponding to the isospectral diagonal matrices and every solution converges to one of them. The unique local attractor of the flow is diag(A1, A2, A3) with A1 < A2 < A3. For a complete phase portrait analysis of the Toda flow we refer to Faybusovich (1992).

2.4 RUTISHAUSEI~'S FLOW

The perhaps earliest example of a continuous-time isospectral matrix flow is that of Rutis- hauser (1954). Rutishauser constructs an infinitesimal analogue of the well-known QD- algorithm. Rutishauser's infinitesimal QD-algorithm is described as follows.

Let F(t) denote a function of a real variable t. For any h > 0 let

F( h) , o , 1 , 2 , .,

denote the sequence of sampled values of F. Consider the following modified quotient difference scheme:

Isospectral Matrix Flows f o r Numerical Analysis 41

Quo t i en t Dif ference Scheme

r _ S~ -) O~,,) : = ~k ~,(~')

h i r ~"J k

D~') .= '~k h

E(~) -(,,+1) D~) k : = E k - 1 nt.

k + l : ~

Rutishauser then considers the limit process where h ~ 0, hu --. t. He thus proposes the following

Rutishauser's inf ini tes imal QD-scheme

Sl( t ) := F(t)

Qk(t) "= S'k(t) = ( log Sk(t))' Sk(t)

Dk( t ) = Qtk(t )

Ek( t ) = Dk( t ) + Ek+l (t)

Sk+l(t) - - E k ( t ) S k ( t ) , k E N (23)

The computational capabilities of the algorithm are demonstrated by the easily estab- oo

lished fact, that for a convergent Dirichlet series F( t ) = ~ cke ~kt then lim Qk(t ) = ,kk k = l t - - , o o

holds for all k E N; see [24] for a proof.

A simple reformulation of the Rutishauser system (23), using

ak(t): = Ek( t ) , bk(t):-- Qk(t) , (24)

yields the following system of quadratic differential equations.

R u t i s h a u s e r ' s Flow

hk = ak(bk+x -- bk)

bk = ak -- ak-1 �9 (25)

Note the striking similarity with the Toda flow!

The isospectral nature of Rutishauser's flow is seen as follows: Consider the non-symmetric tridiagonal matrices

42 U. Helmke

A"

A * - - -

bl 1 0 al b2 1 0

0 a2 b3 "'. �9 �9 o �9

�9 � 9 1 4 9 � 9 1 4 9 ~ 1

0 "" 0 an-t br~

0 0 al 0 0

0 a 2 " ' . �9 �9 , ,

�9 , , �9 �9 , ,

0 ' ' ' 0 an-1 0

Then Rutishauser's flow (25) is equivalent to the isospectral flow

= [ A , A _ ] , (26)

evolving on the set of tridiagonal matrices of the above form. Again it can be shown that (26) interpolates the LR-algorithm in the sense that for any solution A ( k ) of (26)

e A(k) "- k - th LR iteration for e A(O).

For an elementary proof of this fact, as well as for further results on the LR-flow (26) we refer to [30], Watkins and Elsner [31].

2.5 Q1R,-FLOW

There has been extensive work on studying a continuous-time analogue of the QR-algorithm. As an incomplete list of references let us mention here Symes (1982), Deift, Nanda and Tomei (1983), Chu (1984), Watkins (1984), Watkins and Elsner (1989).

For any A E ~nx,~ let

A m ~"

0 - - a 2 1 . . . . a n l

a21 �9 �9 �9

�9 " " �9 - - a n , n - 1

a n l " " �9 a n , n - 1 0

denote the skew-symmetric part. Similarly we can compute

(log A)_

for any choice of a logarithm of A.

The result then is that the QR-algorithm is obtained via constant time sampling of the isospectral flow


.,4 = [A, (log A)_].

That is, for any k E N0,

A(k) = k - th step in the qR-algorithm applied to A(0).

Similar results also exist for versions of the QR-algorithm incorporating shifts; see Watkins and Elsner [32].

S T H E D O U B L E B R A C K E T F L O W

Brockett [5] has introduced a new class of isospectral flows on the set of symmetric matrices. He considers the double Lie bracket equation

)( = [X, [X,A]]

= A X 2 + X 2 A - 2 X A X (27)

where [A, B] = A B - B A denote the Lie bracket and A, X are real symmetric n x n matrices. Here the matrix A plays the role of a parameter to guide the system to a desired equilibrium state. The matrix differential equation (27) has remarkable properties" it can be used to diagonalize symmetric matrices, solve linear programming problems and for sorting lists of real numbers. It generalizes both the Toda flow as well as matrix Riccati equations. Moreover related differential equations serve as learning equations for neural networks and can model arbitrary finite automata.

In this paper we focus on applications to numerical linear algebra, viz. symmetric matrix diagonalization and singular value decomposition. As Brockett's equation (27) is a gradient flow which minimizes a matrix least squares distance problem with spectral constraints we first discuss such minimization tasks.

3.1 A MATRIX LEAST SQUARES MINIMIZATION PROBLEM

Let

A '=d iag(A~, . . . ,An) , A~_>. . .>A~. (28)

The set of real symmetric n x n matrices with eigenvalues A1,..., An is the isospectral set

M(A) = {OAOT I ooT = I ) .

M(A) is an orbit of the Lie group action

(29)

,7: SO(n) x ~"• ~ ~nxn

( e , X ) ~ e x e T (30)

44 U. Helmke

and is therefore a smooth, compact and connected submanifold of IR nx'~. Here SO(n) denotes the Lie group of real orthogonal n x n matrices with determinant one. In fact, M(A) is a homogeneous space (a generalized flag manifold) and is thus a well known space from topology.

The fact that M(A) is a homogeneous space of a Lie group action simplifies the subsequent geometric analysis considerably and enables us to compute in a rather straightforward way tangent spaces and Riemannian metrics on M(A).

Given any symmetric n x n matrix

A = A T E n~ n•

we consider the smooth least squares distance function

fA" M(A) , I~

fA(X)--IIA- X l l 2

--IIAll 2 + IIXll 2 - 2tr(AX) (31)

for the Frobenius norm []Xll 2 - t r ( X X T ) . From (31) and orthogonal invariance of the

Frobenius norm IIXll 2 = IIA[I 2 = ~ A~ we deduce that minimization of f A ( X ) i s equivalent i=1

to maximization of the trace function t r ( A X ) on M(A). Our first rcsult is slight extension of a result of Brockett [5]; see also Helmke and Moore (1994).

T h e o r e m 2.1

(a) A matrix X e M(A) is a critical point of fA: M(A) --* IR if and only if A X = X A .

(b) The local minima coincide with the global minima.

(c) The set of global minima is connected.

(d) f A : M ( A ) ~ I~ is a Morse-Bott function.

P r o o f

For a proof of (a), (b) we refer to Brockett [5], Helmke and Moore [19]. Part (d) is shown in Duistermaat, Kolk and Varadarajan (1983), while (c) follows from the explicit characterization of the global minima, given in Brockett [5]. D

We specialize the above result to two important cases

Case 1: Diagonal iza t ion

For matrix diagonalization we choose A as

A = d iag(# l , . . . , #~) , #1 > . . . > #~.

Then Theorem 1 specializes to

Coro l l a ry 2.2 Let A be as above. Then


(a) fA" M(A) ~ tt~ has ezactly n! critical points. These are diagonal matrices

X = diag(A~(1),..., A~(n))

where r : {1 , . . . , n} ---, {1 , . . . , n} is any permutation.

(b) There is a unique local = global minimum

X = diag(A1,...,A,~) , A~ >_ . . . >_ A~.

All other critical points diag(A,r(1),..., A~(n)) for r # id are saddle points.

(c) The Hessian of fA" M(A) ~ I~ at each critical point is nondegenerate, for A1 > . . . > An. El

Case 2: Block Diagonalization

This case is of obvious interest for principal component analysis. Here we choose

A = d iag(1 , . . . , 1 , 0 , . . . , 0 ) ,

the eigenvalue 1 appearing k times.

Theorem 1 then specializes to

C o r o l l a r y 2.3 Let A be as above. Then

(a) The critical points of fA" M(A) ---, [~ are all block diagonal matrices

X = diag(Xll,X22) E M(A)

with Xlx E IR kxk.

(b) The set of all local = global minima consists of all block diagonal matrices

X = diag(Xll,X22)

where

Xxl has eigenvalues ~1 >_ . . . >__ Ak

and

X22 has eigenvalues Ak+l >_ . . . _> A,~.

3.2 THE DOUBLE BRACKET FLOW

The above results show that we can characterize the tasks of symmetric matrix diagonalization or block diagonalization as a constrained optimization problem for the cost function fA :M(A) ~ IR. A standard approach to tackle optimization problems on manifolds is by a steepest descent approach, following the solution of a suitable gradient flow. What is then the gradient flow of fA: M(A) ---, IR? To compute the gradient of fA one have to specify a Riemannian metric on M(A). As the answer depends crucially on the proper choice of a Riemannian metric it is important to make a good choice.

46 U. Helmke

It turns out that there is indeed (in some sense) such a best possible metric, the so-called normal Riemannian metric. With respect to this normal metric the double bracket flow (27) is then seen as the gradient flow of fA: M(A) ~ II~; see Bloch, Brockett, and Ratiu (1990). All other choices of Riemannian metrics lead to much more complicated differential equations.

T h e o r e m 2.4 (Brocke t t (1988), Bloch, B rocke t t and l:tatiu (1990)). Let A = A T E I~ nXn.

(a) The gradient flow of fA: M(A)---, R is the double bracket flow

)( = IX, [X, A]]

on symmetric matrices X = X T E I~ nx'~.

(b) Th~ diff~,'~ntial ~quatio,~ (27) d~n~ a~ i~osp~ctrat flow on th~ ~t o/,'~al ~ymm~t,'i~ matrices X E ]~nxn.

(c) Every solution X( t ) of (27) exists/or all t e It~ and converges to an equilibrium point X ~ . The equilibrium points of (27) are characterized by [A, X~] = 0, i.e. by A X ~ = X ~ A .

(d) Zet A = d i a g ( m , . . . , ~ ) with #1 > . . . > #,~. Every solution X( t ) of (27) converges to

lim X ( t ) = diag()~TrO),... )~r(n)) t----~oo '

for a suitable permutation 7r: {1 , . . . , n} ~ {1 , . . . , n}.

There exists an open and dense subset U C M(A) such that for every initial condition X(O) E U then

lim Z ( t ) = diag(, \ l , . . .An) , ~1 >_ ..._> ~ . t - - * o o

Moreover, convergence of X( t ) to this unique attractor of the flow is exponentially fast.

Thus solving the double bracket flow leads us to a diagonalization method for symmetric matrices. Of course, this is a rather naive method which is not recommended in practice!

Doub le Bracke t Flow Diagonal iza t ion M e t h o d

�9 The to be diagonalized matrix X0 is the initial condition of (27).

�9 Choose, e.g., A = d i a g ( n , n - 1 , . . . , 1 ) .

�9 Solve the differential equation (27)! Then

X ( ~ ) = diag(A1,... , An)

with ) q , . . . , An the eigenvalues of X0.


3.3 DISCRETIZATIONS

A major drawback of the double bracket flow diagonalization method is that the solutions of a nonlinear ODE have to be computed. For this it would be most helpful if explicit solutions of the double bracket flow were known. Unfortunately this appears to be possible only in one simple case.

3.3.1 Exact Solutions Suppose that X is a projection operator, that is X 2 = X holds. Then the double bracket equation (27) is equivalent to the matrix Riccati equation

= A X + X A - 2 X A X

For this Riccati equation explicit solutions are well-known. We obtain the formula

(32)

x( t ) = r177 Xo + d'AXo)-~r 'A (33)

for the solution of the double bracket flow (27) with X02 = X0, X0 T = X0. Note that (33) defines indeed a projection operator for each t E I~.

We use this formula to show that discretizing (27) at integer time t = k E N is equivalent to the power method for the matrix e A. For simplicity let us restrict to the case where rkXo = 1.

C o n n e c t i o n wi th t he P o w e r M e t h o d

Suppose X(0) is a rank 1 projection operator. Then the solution formula (33) simplifies to

etAX(O)e tA X ( t ) - tr(e2tAX(O)).

Thus

eAX(k)e A x(k + 1)= t~(~2AX(k))" (34)

Since X is a rank 1 projection operator we have X = ~ T for a unit vector ~ E I~ n. The above recursion (34) is then obtained by squaring up the power iterations for the matrix eA:

eA~k

O

3.3.2 Geodesic Approximation As we have mentioned above, it is in general not possible to find explicit solution formulas for the double bracket equation. Thus one is forced to look for approximate solutions. A straightforward although naive approach would be to discretize numerically the ODE on the space of symmetric matrices, using Euler type

48 U. Helmke

or Runge-Kutta methods. Such a standard discretization method is however not recommended, as it may lead to large errors in the eigenvalues. The difficulty with such an approach is that the fixed eigenvalue constraints are not satisfied during the iterations.

Thus our goal is to find suitable approximations which preserve the isospectral nature of the flow. Such approximate solutions are given by the geodesics of M(A). The geodesics of the homogeneous space M(A) can be explicitly computed. They are of the form

e t~Xe -t~ , X E M(A)

where f~ is any skew-symmetric matrix in the tangent space of M(A) at X. We are thus led to consider the following

Geodes ic A p p r o x i m a t i o n Scheme

Xk+l = e~k[Xk'A]Xke-~k[Xk'A] (35)

for a still to be determined choice of a step-size factor ak. We assume, that there exists a function aA:M(A) ~ It~ such that ~k = aA(Xk) for all k. Moreover we require that the step-size factor satisfies the following condition.

B o u n d e d n e s s Cond i t ion

There exist 0 < a < b < oc, with C~A:M(A)---, [a,b] continuous everywhere, except possibly at those points X E M(A) where A X = X A.

We then have the following general convergence result of (35).

P r o p o s i t i o n ([22]) Let A = diag(#l, . . ., I.tn) with #1 > . . . > #n and let aA" M(A) ~ [a, b] satisfy the Boundedness Condition. For Xk E M(A) let ~k = aA(Xk) and define

A f A ( X k , ak) = t r ( A ( X k - Xk+l ))

where Xk+l is given by (35). Suppose

A f A ( X k , ak) < 0 whenever [A, Xk] # O.

Then

(a) The fixed points X e M(A) of the discrete-time iteration (35) are characterized by A X = X A .

(b) Every solution Xk, for k e N, of (35) converges as k ---, oc to a diagonal matrix Xoo = diag()~r(1), . " , )~r(n)), :r a permutation.

[]

Note that the above result does not claim that Xk converges to the optimal fixed point A, which minimizes fA" M(A) ~ ~. In fact, the algorithm will converge to this local attractor A = diag(A1,..., A~), with A1 _> . . . >_ A,~, only for generic initial conditions. For


a nongeneric initial condition the algorithm may well get stuck in a saddle point. This is in sharp contrast to the globally convergent Sort-Jacobi algorithm which will be introduced later.

The following examples yield step-size selections ak which satisfy the assumption of the Proposition.

Example 1: Cons t an t Step-Size ([22])

1 o~ : 4IIAll " i lXol l (36)

Example 2" Variable Step-Size I ([22])

1 I[A, Xdll = ) ~k--II[A, Xk]ll log(1 + IlXo : [i[-~:~.:x~]]ll (a7)

Example 3: Variable Step-Size II ([6])

211[A, Xk]ll 2 ~k -- II[A, [A, Xk]]ll" II[[A, Xk], Xk]ll

(as)

In all three cases the isospectral algorithm (35) converges to a diagonalization. While the last variable step-size (38) is always larger than the (36) or (37), the on-line evaluation of the norms of Lie brackets, in comparison with the constant step size, causes additional computational problems, which may thus slow down overall convergence speed.

What are the advantages and disadvantages of the proposed geodesic approximation method?

1. The algorithm (35) is isospectral and thus preserves the eigenvalues.

2. Linear convergence rates hold in a neighbourhood of the global attractor.

3. By combining the method with known quadratically convergent optimization schemes, such as the Newton method or a conjugate gradient method, local quadratic convergence can be guaranteed. However, the domains in M(A) where quadratic convergence holds can be rather small, in addition to the hugh computational complexity one faces by implementing e.g. Newton's method. Thus these quadratically convergent modifications are not suitable in practice. See the recent Ph. D. thesis of R. Mahony [21] and S. Smith [26] for coordinate free description of such methods.

4. Despite of all such possible advantages of the above geodesic approximation algorithm, the bottleneck of the method really is the need to compute on-line matrix exponentials. This makes such gradient-flow based schemes hardly practical. In section 5 we thus describe a way how to circumvent such difficulties.

50 U. Helmke

4 G E N E R A L I Z A T I O N S

The previous theory of double bracket flows can be extended in a straightforward way to compute eigenvectors or the singular value decomposition. Moreover, the whole theory is easily generalized to flows on Lie algebras of compact semisimple Lie groups.

4.1 FLOWS ON ORTHOGONAL MATRICES

Let SO(n) denote the Lie group of n x n real orthogonal matrices of determinant one and let A, X0 be n x n real symmetric matrices. We consider the task of optimizing the smooth function

CA,Xo: SO(n) ----* I~ , CA,Xo = [[A- OXoOTII2; (39)

this time the optimization taking place on the Lie group SO(n) rather than on the homogeneous space M(Xo) . We have the following analogous result of Theorem 2.1 by Brockett. Let us assume for simplicity that A = d iag(# l , . . . , #n) with #1 > . . . > #~.

Theorem 3.1 (Brockett (1988))

(a) A matrix O e SO(n) is a critical point of CA,x0 : SO(n) ---, I~ if and only if[A, OXo| T] = O. Equivalently, for A = d iag(# l , . . . ,#n) , with #1 > . . . > #n, Q E SO(n) is a critical point if and only if the row vectors of | form an orthonormal basis of eigenvectors of Xo.

(b) The local minima of CA,Xo : S O ( n ) ~ ~ coincide with the global minima.

(c) CA,X0: S O ( n ) ~ I~ is a Morse-Bott function. D

The gradient flow of CA,X0: SO(n) ---, It~ with respect to the (normalized) Killing form is easily computed to be the cubic matrix differential equation

(9 = [|174 , | E SO(n). (40)

Moreover, the solutions | E SO(n) exist for all t E It( and converge, as t ~ =t=~, to an orthogonal basis of eigenvectors of X0; see Brockett [5].

The associated recursions on orthogonal matrices which discretize the gradient flow (40) are

Ok+l = eak[Okx~174 (41)

where ak is any step-size selection as in section 3.3. In particular, if ak is chosen as in (36) - (37) (with Xk = QkXoQT), then the convergence properties of the gradient flow (2) remain in force for the recursive system (40); see [22] for details. Thus

Ok+~ ' e~[O~X~ , o~ = 1 / (411Xol l IIAII), ( 4 2 )


converges to a basis of eigenvectors of X0. The rate of convergence in a neighbourhood of the stable equilibrium point is linear.

4.2 SINGULAR VALUE DECOMPOSITION

One can parallel the analysis of double bracket flows for symmetric matr ix diagonalization with a corresponding theory for flows computing the singular value decomposition of rectangular matrices�9 For a full analysis we refer to [18], [25].

Let A, E E IR '~xm, m >_ n, be rectangular matrices of the form

#1 "'" 0 .

A : = : ".. :

0 . . . # , ~

�9 O~,x (~_,-,) (43)

o'1 .." 0

E ' = : ".. : : 0~x(.~_~) (44)

0 " ' ' (7" n .

with al _> �9149149 > a,~ >_ 0 and #1 > . . . > #n > 0. The Lie group O(n) x O(m) acts on the matr ix space I~ n x m by

~: o ( ~ ) • o ( m ) • ~ • --~ ~ •

((u, v ) , x ) ~-~ u x y T (45)

Thus the set M ( E ) of n x m real matrices X with singular values a l , . . . , an

M ( E ) = { u E v T I u e O(n), V E O(m)} (46)

is an orbit of a and therefore M ( E ) is a smooth, compact manifold. Consider the task of optimizing the smooth least squares distance function

FA: M(E) ---* II~

F A ( X ) = IIA- XII 2

= E (o? + ~ ) - 2t~(AX~). i-1

(47)

For proofs of the following results we refer to [18], [25]; see also [9].

52 U. Helmke

T h e o r e m 3.2

(a) A matrix X E M ( 2 ) is a critical point of FA: M ( 2 ) - - , IR if and only if A X T = X A T and A T x = X T A holds. Equivalently, for A as in (~3) and al > . . . > an > O, X is a critical point if an only if

X = (diag(exa~(1),...,ena~(,~)), O,~•

with ei E {q-l} and rr: {1 , . . .n} --~ {1 , . . . , n} is a permutation.

(b) The local minima coincide with the global minima.

(48)

T h e o r e m 3.3

(a) The gradient flow of the least squares distance function FA: M(E) ---, IR, FA(X) = IIA- Xll 2, is

2 = ( A X T - x A T ) X - x ( A T X - x T A ) . (49)

This differential equation defines a self-equivalent, i.e. singular value preserving, flow on I~ r~ • m.

(b) The solutions X ( t ) of (~9) exist for all t e I~ and converge to a critical point of FA: M ( E ) ~ I~. In particular, every solution X ( t ) converges to a matrix

Xr = (diag(Ax,...,)~n), Onx(m-,~)) (50)

where ])ql, i = 1 , . . . , n, are the singular values of X(O) (up to a possible re-ordering). []

Similarly also recursive versions of (49) and corresponding flows for the singular vectors exist. We refer to [22] for details.

4.3 FLOWS ON LIE ALGEBRAS

Brockett's double bracket equation generalizes in a straightforward way to flows on Lie algebras. The importance of such generalizations lies in the ability to cope in a systematic way with various structural constraints, such as for skew-symmetric or Hamiltonian matrices.

The standard example of a Lie algebra is the Lie algebra of skew-symmetric matrices. 1 More generally let us consider an arbitrary compact semi-simple Lie group G with Lie algebra ~. Then G acts on ~ via the adjoint representation

Ad: G x ~ ----* r

(g, ~) ~ Ad(g) .

1The set of symmetric matrices is not a Lie algebra, but a Jordan algebra!

(51)


where Ad(g). ~ is defined as

d . t~ Ad(g) . ~ = --~(ge g-1)lt=o.

By differentiating Ad(g).~ with respect to the variable g we obtain the Lie bracket operation

ad: ~ x ~ ~

(~,,7) --- , [~, hi. (52)

We also use the notation ad~(~)= [~, 7/]. Then the orbit

M(~) = {Ad(g). ~ l g e G) (53)

is a smooth compact submanifold of the Lie algebra qS.

If G is semi-simple, then the Killing form

(~, r/): = -tr(ad~ o adn) (54)

defines a positive definite inner product on @.

Consider for a fixed element 7/E q5 the trace function

Cn:M(~) ~ ~ , Cn(~) = ((,77). (55)

We now have the following natural generalization of Theorems 2.1, 2.4 to arbitrary compact semi-simple Lie algebras.

Theorem 3.4 (Bloch, Brockett, Ratiu (1990))

(a) There exists a Riemannian metric on M(~) such that the double Lie bracket flow

= [~, [~, ~]] (56)

is the gradient flow of Cn: M(~) ~ ~ .

(b) Every solution ((t) of (17) converges to an equilibrium point ~oo, characterized by [~oo, ~] = o.

Therefore the double bracket flow (17) converges to the intersection of tow sets, one being defined by the a spectral constraints ~oo E M(~) while the other set is defined by the linear constraint [~oo, r/] = 0. Thus such flows are capable of solving certain structured inverse eigenvalue problems. This can in fact be done in more generality; cf. [10] for further examples and ideas.

54 U. Helmke

R e m a r k s

(a) It is not quite proved (nor even stated!) in [4] that every solution r of the gradient flow (56) really converges to a single equilibrium point, rather than a set of equilibria. While this would be a trivial consequence of standard properties of gradient flows if (56) were known to have only finitely many equilibrium points, it is not so obvious in case (16) has a continuum of equilibria. However, as r M(~) ~ I~ can be shown to be a Morse- Bott function, convergence to a single equilibrium point then it follows from appropriate generalizations of standard convergence results of gradient flows; see e.g. Helmke and Moore (1994), Prop. 1.3.6.

(b) There is a step-size selection ak similar to (38), such that the recursion

~k+l -- Ad(e-~ �9 ~k

has the same convergence properties as (56); see Brockett (1993).

(57)

5 SORT-JACOBI ALGORITHM

In the previous sections we have seen that double Lie bracket isospectral flows and their respective discretizations lead to convergent algorithms for matrix diagonalization. Although such algorithms are attractive from a theoretical viewpoint, they do lead only to highly unpractical, slowly convergent algorithms. Our goal in this section is to show how modifications of such algorithms yield efficient, quadratically convergent algorithms for matrix diagonalization. The algorithm which we derive is actually a modification of the classical Jacobi algorithm, which also incorporates sorting of the eigenvalues. This part is forthcoming joint work with K. Hiiper, where full details will appear.

5.1 MATRIX DIAGONALIZATION

We use the notation of section 3.

Let ei E I~ n denote the i-th standard basis vector of I~ n. For 1 _< i < j _ n and t E I~ let

Gij ( t ) = cos(t). (eie T + eje T) - sin(t). (eie T - eje T) (58)

denote the Jacobi-rotation in the (i,j)-plane. Using any fixed ordering of {( i , j ) E 1~21 1 _< i < j _< n} we denote by

l n ( n - 1) (59) G ~ ( t ) , . . . , G N ( t ) , lV =

the Jacobi rotations of I~ n.

The Sort-Jacobi algorithm is build up similarly to standard numerical eigenvalue methods via the iteration of sweeps.


k- th S o r t - J a c o b i Sweep

Define

X 0) .= Gl ( tO) )XkGl ( - t (1))

X~ 2) .= G2(t!2))X~I)G2(-t!2) )

X~ N) "---- GN(t!N)~ y(N-1)GN( )

where

(60)

t! 0 = arg re. in .{fA(Gi(t ~ ( i - 1 �9 J"k ) G i ( - t ) } . (61) tE[0,2~rJ

f o r / = 1 , . . . , N .

Thus X~ i) is recursively defined as the minimum of the least squares distance function

fA: M(Zo) ~ ~ , when restricted to the i-th Jacobi geodesic {Gi(t)X~i-1)Gi(- t) l t e I~} r(~-~)

containing ~k �9

The Sort-Jacobi algorithm then consists of the iteration of sweeps.

S o r t - J a c o b i A l g o r i t h m

�9 Let X o , . . . , X k E M(Xo) be given for k E H0.

�9 Define the recursive sequence

(1) X~2) x~N) Xk , , . . . ,

as above (sweep).

�9 Set Xk+l"= X~ N). Proceed with the next sweep.

The convergence properties of the algorithm are established by the following result.

T h e o r e m 4.1 Let Xo = XTo E Xn• be given and A = diag(#l,. . . ,#n) with #1 > . . . > #n. Let Xk, k = O, 1, 2, . . . , denote the sequence generated by the Sort-Jacobi algorithm. Then

(a) Global convergence holds, that is

with

lim Xk = diag(11, . . . 1,~) (62) k---, oo

A1 >_ . . . _> A~. (63)

56 U. Helmke

Here hx,...,)~n are the eigenvalues of Xo.

(b) The algorithm is quadratically convergent.

R e m a r k s

(a) The above algorithm automatically sorts the eigenvalues of X0 in a given order. If the diagonal entries #i in A were, e.g., chosen in an increasing way, than the eigenvalues A1,. . . ,An of X0 would also appear in an increasing order. Similarly for any other permutation. This property does not hold for standard versions of the aacobi algorithm.

(b) The theorem remains in force for any a priori chosen order, in which the sweep steps are applied.

(c) The basic difference to standard textbook Jacobi algorithms is that we are maximizing the linear trace function tr(AX) in each step, rather than minimizing the quadratic Jacobi cost function Iloffdiag(X)ll ~. This is important for the proof, as the critical point structure of the trace function on M(A) is much simpler that that of I]offdiag(X)ll 2. This difference in the choice of cost functions is the main reason why our theory appears to be simpler than the classical theory of aacobi algorithms.

P r o o f of T h e o r e m 4.1 (Sketch)

For details we refer to our forthcoming work Hiiper and Helmke (1994).

First we establish global convergence. By construction

/A(Xk+,) < /A(Zk).

It is easily seen that equality holds if and only if Xk+l = Xk is a critical point of fA: M(Xo) Ii~. By a Lyapunov type argument we deduce that the sequence (Xk) converges to a critical point of fA: M(Xo) ~ Ii~; i.e. to a diagonal matrix X~ . Suppose Xr162 were a saddle point. Then a single sweep involving permutations, applied to Xk for k sufficiently large, would bring us arbitrarily close to the unique global minimum. Contradiction. Thus (Xk) converges to the global minimum of the cost function.

To prove quadratic convergence we consider the algorithm in a neighbourhood of the global minimum A. Using the implicit function theorem one shows that there exists a smooth map ~p: U + U, defined on an open neighbourhood U C M(A) of A, such that for all sufficiently large k

Xk+~ = ~(Xk)

holds.

Thus the algorithm consists in iterating a locally defined smooth map of M(A). A calculation shows that the first derivative of ~: U ~ U at the fixed point A is zero! Thus the result follows from a standard Taylor expansion argument. Q.E.D. []


5.2 SVD

In an analogous way a Sort-Jacobi algorithm which computes the singular values of a matrix is obtained. Let

A" = (d i ag (# : , . . . , #n), 0nx(m-~)) (64)

E : = (diag(a: , . . . ,a ,~) , 0n•

with #: > . . . > #n > 0 and a: _> .. >_ an _> 0. Using the well-known transformation [0A f~'-- A T 0 ' 0

(65)

we can reduce the problem to the symmetric case. Thus for any matrix Xo E I~ n• with singular values a l , . . . , an we can apply the Sort-Jacobi algorithm to the symmetric (m + n) • (m + n) matrix )(0. The only difference now is that we only perform those Jacobi iterations which do preserve the structure of )(0. That is we are applying, in each sweep, only Jacobi-iterations of the form G:XG2 E M ( X ) . We then have an analogous convergence theory for the SVD Sort-Jacobi algorithm.

T h e o r e m 4.2 Let Xo E I~ n• and let A be defined by (64). Wi th#: >_ ...>_ #n >_ 0 let Xk, k = O, 1, 2 , . . . , denote the sequence generated by the Sort-Jacobi SVD algorithm. Then

(a) Global convergence holds, i.e.

lira Xk = (diag(a:, . . . an) On• (66) k-,,,,+ ~ '

with

a: >_. . .>an>_O.

Here a : , . . . , an are the singular values of Xo.

(b) The algorithm converges quadratically fast.

(67)

A c k n o w l e d g e m e n t

This work was partially supported by the German-Israeli-Foundation for scientific research and development, under grant 1-0184-078.06/91.

58 U. Helmke

References

[1] Ablowitz, M.J. and Clarkson, P.A., Solitons, Nonlinear Evolution Equations and In- verse Scattering, London Math. Soc. Lecture Notes Series 149, Cambridge Univ. Press, 1991

[2] Arnold, V.I., Mathematical Methods of Classical Mechanics, Second Edition, Graduate Texts in Mathematics 60, Springer, New York, 1989

[3] Bloch, A.M., Steepest descent, linear programming and Hamiltonian flows, Contempo- rary Math. 114, 77-88, 1990

[4] Bloch, A.M., Brockett, R.W. and Ratiu, T., A new formulation of the generalized Toda lattice equations and their fixed point analysis via the moment map, Bull. Amer. Math. Soc., 23,447-456, 1990

[5] Brockett, I~.W., Dynamical systems that sort lists and solve linear programming problems, Proc. 27th Conference on Decision and Control, Austin, TS, 779-803. See also Linear Algebra Appl. 146 (1991), 79-91, 1988

[6] Brockett, I~.W., Differential geometry and the design of gradient algorithms, Proceed- ings of Symposia in Pure Mathematics, 54, 69-91, 1993

[7] Chu, M.T., The generalized Toda flow, the QR algorithm and the center manifold theory, SIAM J. Disc. Meth., 5,187-201, 1984b

[8] Chu, M.T., A differential equation approach to the singular value decomposition of bidiagonal matrices, Linear Algebra and its Appl., 80, 71-80, 1986

[9] Chu, M.T. and Driessel, K.R., The projected gradient method for least squares matrix approximations with spectral constraints, SIAM J. Numer. Anal. 27, 1050-1060, 1990

[10] Chu, M.T., Matrix differential equations: A continuous realization process for linear algebra problems, Nonlinear Analysis, TMA 18, 1125-1146, 1992a

[11] Deift, P., Nanda, T. and Tomei, C., Ordinary differential equations for the symmetric eigenvalue problem, SIAM J. Numer. Anal. 20, 1-22, 1983

[12] Duistermaat, J.J., Kolk, J.A.C. and Varadarajan, V.S., Functions, flow and oscillatory integrals on flag manifolds and conjugacy classes in real semisimple Lie groups, Compositio Math. 49,309-398, 1983

[13] Faybusovich, L., Toda flows and isospectral manifolds, Proc. American Mathematical Society 115,837-847, 1992

[14] Flaschka, H., The Toda lattice, I, Phys. Rev. B 9, 1924-1925, 1974

[15] Flaschka, H., Discrete and periodic illustrations of some aspects of the inverse methods, in dynamical system, theory and applications, J. Moser, ed., Lecture Notes in Physics, 38, Springer-Verlag, Berlin, 1975


[16] Fomenko, A.T., Symplectic Geometry, Advanced Studies in Contemporary Mathemat- ics, Vol. 5, Gordon and Breach Publ., New York, 1988

[17] Golub, G.H. and Van Loan, C.F., Matrix Computations, Second Edition, The John Hopkins University Press, Baltimore, 1989

[18] Helmke, U. and Moore, J.B., Singular value decomposition via gradient and self equivalent flows, Linear Algebra and its Appl. 69,223-248, 1992

[19] Helmke, U. and Moore, J.B., Optimization and Dynamical Systems, CCES, Springer- Verlag, London, 1994

[20] Kostant, B., The Solution to a generalized Toda lattice and representation theory, Advances in Mathematics 34, 195-338, 1979

[21] Mahony, R., Optimization Algorithms on Homogeneous @aces, Ph.D. Thesis, Can- berra, ANU, 1994

[22] Moore, J.B., Mahony, R.E. and Helmke, U., Numerical gradient algorithms for eigenvalue and singular value decomposition, SIAM J. of Matrix Analysis, Appl., Vol. 15, 881-902, 1994

[23] Moser, J., Finitely many mass points on the line under the influence of an exponential potential- An integrable system, in J. Moser, Ed., Dynamical Systems Theory and Applications, Springer-Verlag, Berlin - New York, 467-497, 1975

[24] Rutishauser, H., Ein InfinitesimaIes Analogon zum Algorithmus, Arch. Math., (Basel), 5, 132-137, 1954

Quo tienten- Di ff erenzen-

[25] Smith, S.T., Dynamical systems that perform the singular value decomposition, Systems and Control Letters 16,319-328, 1991

[26] Smith, S.T., Geometric optimization methods for adaptive filtering, PhD Thesis, Har- vard University, 1993

[27] Symes, W.W., Systems of the Toda type, inverse spectral problems and representation theory, Inventiones Mathematicae 59, 13-51, 1980

[28] Symes, W.W., The QR algorithm and scattering for the finite nonperiodic Toda lattice, Physica 4D, 275-280, 1982

[29] Tomei, C., The topology of isospectral manifolds of tri-diagonal matrices, Duke Math. Journal 51,981-996, 1984

[30] Watkins, D.S., Isospectral Flows, SIAM Rev., 26,379-391, 1984

[31] Watkins, D.S. and Elsner, L., Self-similar flows, Linear Algebra and its Appl. 110, 213-242, 1988

[32] Watkins, D.S. and Eisner, L., Self-equivalent flows associated with the singular value decomposition, SIAM J. Matrix Anal. Appl. 10, 244-258, 1989



61

T H E R I E M A N N I A N S I N G U L A R V A L U E D E C O M P O S I T I O N

B.L.R. DE MOOR E S A T - S I S T A , Kathol ieke Univers i te i t Leuven

Kard inaa l Merc ier laan 9~

3001 Leuven , Be lg ium bart. demoor@esat , kuleu ven. ac. be

ABSTRACT. We define a nonlinear generalization of the SVD, which can be interpreted as a restricted SVD with Riemannian metrics in the column and row space. This so-called Rie- mannian SVD occurs in structured and weighted total least squares problems, for instance in the least squares approximation of a given matrix A by a rank deficient Hankel matrix B. Several algorithms to find the 'minimizing' singular triplet are suggested. This paper reveals interesting and sometimes unexplored connections between linear algebra (structured matrix problems), numerical analysis (algorithms), optimization theory, (differential) geometry and system theory (differential equations, stability, Lyapunov functions). We also point out some open problems.

KEYWORDS. (Restricted) singular value decomposition, gradient flows, differential geometry, continuous algorithms, total least squares problems, power method.

1 T H E R I E M A N N I A N SVD: M O T I V A T I O N

Since the work by Eckart-Young [14], we know how to obtain the best rank deficient least squares approximation of a given matrix A E Itr215 of full column rank q. This approximation follows from the SVD of A by subtracting from A the rank one matrix u.a .v T, where (u, a, v) is the singular triplet corresponding to the smallest singular value a, which satisfies

A v = uo" ~ uTu = 1 ,

A T u : va , vT v : 1. (1)

Here u E li~ p and v E ~q are the corresponding left resp. right singular vector. When formulated as an optimization problem, we obtain

min I]A-BI] ~ subject to B y = 0 , B E It~ T M y T y = 1 , (2)

y EIt~ q

62 B.L.R. De Moor

the solution of which follows from (1) as B = A - uav T, y - v. Here, a is the smallest singular value of A and it is easy to prove that minB I IA- BII ~ = a 2 if S is required to be rank deficient. This problem is also known as the Total (Linear) Least Squares (TLS) problem and has a long history [1] [2] [14] [16] [17] [20] [25] [27] [28]. The vector y describes a linear relation between the columns of the approximating matrix B, which therefore is rank deficient as required.

In this paper we consider two generalizations of the TLS problem. First of all, in the remainder of this Section, we describe how the introduction of user-given weights and/or the requirement that the approximant B is to be structured (e,g. Hankel or Toeplitz), leads to a (nonlinear) generalization of the SVD, which we call the R i e m a n n i a n SVD. In Section 2, we show that the analysis and design of continuous time algorithms provides useful insight in many problems and how they also provide a unifying framework in which several mathematical engineering disciplines meet. In Section 3, we discuss the power method in more or less detail to illustrate this claim. We also derive some new continuous-time algorithms, that are not yet completely understood. In Section 4, we discuss some ideas to generate algorithms for the Riemannian SVD. Conclusions are presented in Section 5.

There are at least two important variations on the Total Least Squares problem (2) 1 .

The first one is called the Weighted TLS problem: p q

min ~_~ ~ ( a i j - bij)2wij subject to By.o . .-_ 0,1 B 6 n~pxq i=lj=l tJ~u '

y 6 N q

(3)

where the scalars wij E 1~+ are user-defined weights. An example is given by chosing wii - 1/a~j in which case one minimizes (a first order approximation to) the sum of relative errors squared (instead of the sum of absolute errors squared as in (2)). Another example corresponds to the choice wij E {0, 1) in which case some of the elements of B will be the same as corresponding elements of A (namely the ones that correspond to a weight wij - 1, see also the Example in Section 4.2). Yet other examples of weighted TLS problems are given in [10] [11].

The second extension consists of adding a so-called structural constraint to the optimization problem (2), in which case we have a Structured TLS problem. For instance, we could require the matrix B to be a Hankel matrix. The problem we are solving then is to approximate a given (possible structured) full column rank matrix A by a rank deficient Hankel matrix B. Rank deficient Hankel matrices in particular have important applications in systems and control theory. But in general there are many other applications where it is required to find rank deficient structured approximants [10] [11] [12] [13]. The results of this paper apply to affine matrix structures, i.e. matrices that can be written as a linear combination of a given set of fixed basis matrices. Examples are (block) Hankel, Toeplitz,

1The list of constraints discussed in this paper is not exhaustive. For instance, one could impose constraints on the vectors y in the null space, an example that is not treated here (see e.g. [9]).

The Riemannian SVD 63

Brownian, circulant matrices, matrices with a given zero pattern or with fixed entries, etc. . .

The main result, which we have derived in [10], applies to both Weighted and/or Structured TLS problems and states that these problems can be solved by obtaining the singular triplet that corresponds to the smallest singular value that satisfies:

A v - D~ua , uTDvu = 1,

ATu---- Duva , v T D u v - 1. (4)

Notice the similarity with the SVD expressions in (1). Here A is the structured data matrix that one wants to approximate by a rank deficient one. Du and Dv are nonnegative or positive definite matrix functions of the components of the left and right singular vectors u and v. Their precise structure depends on the weights in (3) and/or the required affine structure of the rank deficient approximant B.

To give one example (which happens to be a combination of a Weighted and Structured TLS problem), let us consider the approximation of a full column rank Hankel matrix A �9 ~pxq, p _> q, rank(A) = q, by a rank deficient Hankel matrix B such that [[A- S[[~ is minimized. In this case, the matrix D~ has the form Dv - T v W - 1 T T where

W = d i a g [ 1 2 3 . . . qq . . . q . . . 3 2 1 ] , y -

( p - q + l ) t i m e s

and Tv is a banded Toeplitz matrix (illustrated here for the case p - 4, q - 3) of the form" iv v v oool 0 V 1 V 2 V 3 0 0

T~ = 0 0 Vl v2 va 0 "

0 0 0 vx v2 v3

The matrix Du is constructed similarly as D~, - T u W - 1 T T. Obviously, in this example, both Du and D~ are positive definite matrices. Observe that B has disappeared from the picture, but it can be reconstructed as

B = A - multilinear function of (u, a, v),

(see the constructive proof in [11] for details). Observe that the modification to A is no longer a rank one matrix as is the case with the 'unstructured' TLS problem (2). Instead, the modification is a multilinear function of the 'smallest' singular triplet, the detailed formulas of which can be found in [10] [11]. We are interested in finding the smallest singular value in (4) because it can be shown that its square is precisely equal to the object function. For more details and properties and other examples and expressions for Du and Dv for weighted and structured total least squares problems we refer to [10] [11] [12] [13].

In the special case that D~ = Ip and Dv = Iq we obtain the SVD expressions (1). In the case that D~ and Dv are fixed positive definite matrices that are independent of u and v, one obtains the so-called Restricted SVD, which is extensively studied in [7] together with some structured/weighted TLS problems for which it provides a solution. In the Restricted

54 B.L.R. De Moor

SVD, D~ and D~ are positive (or nonnegative) definite matrices which can be associated to a certain inner product in the column and row space of A. Here in (4), D~ and D~ are also positive (nonnegative) definite, but instead of being constant, their elements are a function of the components of u and v. It turns out that we can interprete these matrices as Riemannian metrics, an interpretation which might be useful when developping new (continuous-time) algorithms. For this reason, we propose to call the equations in (4), the Riemannian SVD 2.

2 C O N T I N U O U S - T I M E A L G O R I T H M S

As discussed in the previous section, in order to solve a (weighted and/or structured) TLS problem, we need the 'smallest' singular triplet of the (Riemannian) SVD. The calculation of the complete SVD is obviously unnecessary (and in the case of the Riemannian SVD there is even no 'complete' decomposition). For the calculation of the smallest singular value and corresponding singular vectors of a given matrix A, one could apply the power method or inverse iteration to the matrix AT A (see e.g. [16] [24] [29]), which will be discussed in some more detail in the next Section.

One of the goals of this paper is to point out several interesting connections between linear algebra, optimization theory, numerical analysis, differential geometry and system theory. We also would like to summarize some recent developments by which continuous time algorithms for solving and analyzing numerical problems have gained considerable interest the last decade or so. Roughly speaking, a continuous time method involves a system of differential equations. The idea that a computation can be thought as a flow that starts at a certain initial state and evolves until it reaches an equilibrium point (which then is the desired result of the computation) is a natural one when one thinks about iterative algorithms and even more, about recent developments in natural information processing systems related to artificial neural networks 3. There are several reasons why the study of continuous time algorithms is important. Continuous time methods can provide new additional insights with respect to and shed light upon existing discrete time iterative or recursive algorithms (such as e.g. in [22], where convergence properties of recursive algorithms are analysed via an associated differential equation). In contrast to the local properties of some discrete time methods, the continuous time approach often offers a global picture. Even if they are not particularly competitive with discrete time algorithms, the combination of parallel- lism and analog implementation does seem promising for some continuous-time algorithms. In many cases, continuous-time algorithms provide an alternative or sometimes even better understanding of discrete-time versions (e.g. in optimization, see the continuous-time version of interior point techniques in [15, p.126], or in numerical analysis, the self-similar

2This name is slightly misleading in the sense that we do NOT want to suggest that there is a complete decomposition with rain(p, q) different singular triplets, which are mutual 'independent' (they are 'orthogonal') and which can for instance be added together to give an additive decomposition of the matrix A (the dyadic decomposition). There might be several solutions to (4) (for some examples, there is only one), but since each of these solutions goes with a different matrix D~ and D~, it is not exaclty clear how these solutions relate to each other, let alone that they would add together in one way or another to obtain the matrix A.

SSpecifically for neural nets and SVD, we refer to e.g. [3] [4] [21] [23] (and the references in these papers).


iso-spectral (calculating the eigenvalue decomposition) or self-equivalent (singular value decomposition) matrix differential flows, see the article by Helmke [19] in this Volume and its rather complete list of references). The reader should realize that it is not our intention to claim that these continuous-time algorithms are in any sense competitive with classical algorithms from e.g. numerical analysis. Yet, there are examples in which we have only continuous-time solutions and for which the discrete-time iterative counterpart has not yet been derived.

To give one example of a continuous-time unconstrained minimization algorithm, let us derive the continuous-time steepest descent method and prove its convergence to a local minimum. Consider the minimization of a scalar object function f ( z ) in p variables z E I~ p. Let its gradient with respect to the elements of z be Vzf(z). It is straightforward to prove that the system of differential equations

~(t) = -V~f (z ) , z ( 0 ) = z0, (5)

where z0 is a given initial state, will converge to a local minimum. This follows directly from the chain rule:

d f ( z ) = ( V z f ( z ) ) T L " = _llVzf(z)ll2 (6) dt

which implies that the time derivative of f ( z ) is always negative, hence, as a function of time, f ( z ) is non-increasing. This really means that the norm of the gradient is a Lyapunov function for the differential equation, proving convergence to a local minimum.

In Section 3.8 we will show how to derive continuous-time algorithms for cons tra ined minimization problems, using ideas from differential geometry.

3 C O N T I N U O U S - T I M E A L G O R I T H M S F O R T H E E I G E N V A L U E P R O B L E M

The symmetric eigenvalue problem is the following: Given a matrix C - C T E IR pxp, find at least one pair (x, A) with x E I~ p and A E I~ such that

C x - " x~ , x Tx -- 1 . (7)

Of course, this problem is of central importance in most engineering fields and we refer to [16] [24] [29] for references and bibliography. Here we will concentrate on power methods, both in continuous and discrete time. The continuous-time power methods are basically systems of vector differential equations. Some of these have been treated in the literature (see e.g. [6] [18] [26]). Others presented here are new. The difference with the matrix differential equations referred to in the previous section is that typically, these vector differential equations compute only one eigenvector or singular vector while the matrix differential equations upon convergence deliver all of them (i.e. the complete decomposition). Obviously, when solving (structured/weighted) TLS problems, we only need the 'minimal' triplet in (1) or (4). For the TLS problem (2) one might also (in principle) calculate the smallest eigenvalue of A T A and its corresponding eigenvector.

56 B.L.R. De Moor

3.1 GEOMETRIC INTERPRETATION

Consider the two quadratic surfaces xTcx = 1 and xTx = 1. While the second surface is the unit sphere in p dimensions, the first surface can come in many disguises. For instance, when C is positive definite, it is an ellipsoid. In three dimensions, depending on its inertia, it can be a one-sheeted or two-sheeted hyperboloid or a (hyperbolic) cilinder. In higher dimensions, there are many possibilities, the enumeration of which is not relevant right now. In each of these cases, the vectors Cx and x are the normal vectors at x to the two surfaces. Hence, when trying to solve the symmetric eigenvalue problem, we are looking for a vector x such that the normal at x to the surface x T c x = 1 is proportional to the normal at x to the unit sphere xTx -- 1. The constant of proportionality is precisely the eigenvalue A.

3.2 THE EXTREME EIGENVALUES AS AN UNCONSTRAINED OPTIMIZATION PROBLEM

Consider the unconstrained optimization problem:

f ( z ) with f ( z ) - ~(zTCz) / ( zTz) . (8) min z E ~ P

It is straightforward to see that

Vzf(z) = ( C z ( z T z ) - z (zTCz)) / ( zTz) 2 , (9)

from which it follows that the stationary points are given by

C ( z / zVq~z) = ( z / ~ ) ( z T C z ) / ( z T z ) .

This can only be satisfied if x = z/[[z[[ is an eigenvector of C with corresponding eigenvalue x T c x . Obviously, the minimum will correspond to the minimal eigenvalue of C (which can be negative, zero or positive). The maximum of f ( z ) will correspond to the maximum eigenvalue of C.

3.3 EXTREME EIGENVALUES AS A CONSTRAINED OPTIMIZATION PROBLEM

We can also formulate a constrained optimization problem as:

min f ( x ) subject to xTx--1 where f(x)----xTcx. (10) x ElmP

The Lagrangian for this constrained problem is

L(x, A)= x T c x -1- A ( 1 - xTz) ,

where A E R is a scalar Lagrange multiplier. The necessary conditions for a stationary point now follow from VxL(x , A) = 0 and V~L(x, A) = 0, and will correspond exactly to the two equations in (7). Observe that these equations have p solutions (x, A) while we are only interested in the minimizing one.


3.4 THE CONTINUOUS-TIME P O W E R METHOD

Let us now apply the continuous-time minimization idea of (6) to the unconstrained minimization problem (8). We then find the 'steepest descent' nonlinear differential equation

~. -- - ( C z ( z T z) -- z ( z T C z ) ) / ( z T z ) 2 . (11)

An important property of this flow is that it is i sonormal , i.e. the norm of the state vector z is constant over time: []z(t)[[ = [Iz(O)[[, Yt > O. This is easy to see since d[[z[[2/dt = 2 zT2; = O, hence [Iz(t)[[ is constant over time. This means tha t the state of the nonlinear system (11) evolves on a sphere with radius given by [[z(0)[ I. We can assume without loss of generality that [Iz(0)[]- 1. Hence (11)evolves on the unit sphere. Replacing z by x with Ilxll 2 - x T x -- 1 we can rewrite (11)

x T C x = - c z + z . ( 1 2 ) x T x

We know about this flow that it is isonormal and that it will converge to a local minimum. Hence, (12) is a system of nonlinear differential equations tha t will converge to the eigenvector corresponding to the minimal eigenvalue of the matrix C. This flow can also be interpreted as a special case of Brockett 's [5] double bracket flow (see [19] for this interpretation).

3.5 THE CONTINUOUS-TIME P O W E R METHOD FOR THE LARGEST SINGULAR VALUE

The preceeding observations can also be used to derive a system of nonlinear vector differential equations that converges to the largest singular value of a matrix A E I~ TM, p _> q. It suffices to choose in (12)

( ) 0 A a n d z = . ( 1 3 ) C = A r 0 v

This matrix C will have q eigenvalues given by a~(A), q eigenvalues equal to -a~(A) and p - q eigenvalues equal to 0. Hence the smallest eigenvalue is -amax(A). The first q components of z will go to the corresponding left singular vector u while its last q components converge to the right singular vector v.

3.6 THE DISCRETE-TIME P O W E R METHOD

The surprising fact about the nonlinear differential equation (12) is that it can be solved analytically. Its solution is

�9 ( t ) - - e-c~(o)/ll~-C~x(O)ll, ( 1 4 )

which can be verified by direct substitution. If we consider the analytic solution at integer times t = k = 0, 1 ,2 , . . . , we see that

x(k + 1) = e-Cx(k) / l Ie-Cx(k)[ I ,

68 B.L.R. De Moor

i ............... i ................ ~ .......... i

o.8 ..... : .... :. i ............. i .............. i ................ : .................................

0 . 6 "~): - . . . . . . . . . . . i-:~ . . . . . . . . . . . i . . . . . . . . . . . . . . : . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . �9 . . . . . . . . . . . . . . . .

0 . 4 I \ , ~ . . . . . . . . . . ! .x.~.:-~ . . . . . . . . . ; . . . . . . . . . . . . . . . . : . . . . . . . . . . . . . . . . : . . . . . . . . . . . . . . . . : . . . . . . . . . . . . . . . .

0 2 ~ - \ . . x . . . . . . i . . : : - , . . : . k . . . . . i . . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . �9 ~ : . . . . . % , : . : : :

\ : ' . . . ' . . ' . ~ : ~ ~ ~ : : :

0 . . . . . . . . . . ~...i . . . . . . . . . . . ".'.'::~;; ; f , T : . ~ , . , , , ' , ~ , = . 7 . : . : . -_ .~ - ~ - . ~ . . ~ ~ , , ~ _ , . . ~ . . ~ . . . . . . ~. - _ .

-o.2 .............. ::.." ....... :..i ................ ! ............... :: ................ i ...............

- 0 . 4 . . . . . . . . . . . . . . :: . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . . :: . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . . :: . . . . . . . . . . . . . . .

- o . ~ . . . . . . . . . . . . . . . " . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . . i. . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . .

- o . 8 . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . . :: . . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . . i ................

-1

0 0.5 1 1.5 2 2.5 3

Figure 1: As a function of time, we show the convergence of the elements of the vectors u(t) and v(t) of the differential equation (12) with C given by (13), where A is a 4 x 3 'diagonal' matrix with on its diagonal the numbers (5, 3, 1). The initial vector z0 = (u(O) T v(O)T) T is random. This differential system converges to the eigenvector corresponding to the smallest eigenvalue of C in (13), which is -amax(A) = -5 . Hence, this flow upon convergence delivers the 'largest ' singular triplet of the matrix A. The picture was generated using the Matlab numerical integration function 'ode45'.

which shows that the continuous time equation (12) interpolates the discrete time power method 4 for the matrix e -c'. This implies that we now have the clue to understand the global convergence behavior of the flow (12). Obviously, the stat ionary points (points where

= 0) are the eigenvectors of C, but there is only one stat ionary point tha t is stable (as could be shown by linearizing (12) around all the stationary points and calculating the eigenvalues of the linearized system). It will always convergence to the eigenvector corresponding to the smallest eigenvalue of C, except when the initial vector x(0) is orthogonal to the smallest eigenvector. These observations are not too surprising since the solution for the linear part in (12) (the first term of the right hand side) is x( t ) = e x p ( - V t ) x ( O ) . The second term is a normalization term which at each time instant projects the solution of the linear part back to the unit sphere. We can rewrite equation (12) as

xx T . 5~ = ( I - ~-~x ) C x .

This clearly shows how ~ is obtained from the orthogonal projection of the vector C x (which is the gradient of the unconstrained object function 0 . h z T C x ) onto the hyperplane which is tangent to the unit sphere at z.

4...which is exactly the reason why (12) is called the continuous-time power method.


3.7 A D I F F E R E N T I A L EQUATION DRIVEN BY THE RESIDUAL

Another interesting interpretation is the following: Define the residual vector r(t) - Cx - x (xTCx/xTx) . Then the differential equation (12) reads

= - ~ ( t ) .

Hence, the differential equation is driven by a residual error and when r(t) = 0, we also have ~ = 0 so tha t a zero residual results in a s tat ionary point. Moreover, using a little known fact in numerical anaylsis [24, p.69] we can give a backward error interpretation as follows. Define the rank one matr ix M(t) - r(t).x(t) T. Then we easily see tha t

( C - M(t))x(t) = x(t))~(t) with A(t) = ( x ( t ) ) T c x ( t ) / ( x ( t ) T x ( t ) ) .

The interpretat ion is that , at any time t, the real number )~(t) is the exact eigenvalue of a modified matrix, namely C - M(t). The norm of the modification is given by IIM(t)l] = IIr(t)ll, which is the norm of the residual vector and from the convergence, we know tha t

3.8 DERIVATION AS A GRADIENT F L O W

Gradient flows on manifolds can be used to solve constrained optimization problems. The idea is to consider the feasible set (i.e. the set of vectors tha t satisfies the constraints) as a manifold, and then by picking out an appropriate Riemannian metric, determining a gradient flow.

Let us illustrate this by deriving a gradient flow to solve the constrained minimization problem (10). The set of vectors tha t satisfies the constraints is the unit sphere, which is known to be a manifold:

M = { z e~Pl x r z = l } .

The tangent space at x is given by the vectors z tha t belong to

T x M = { z e I~Pl z T x = O } .

The directional derivative Dxg(z) is the amount by which the object function changes when moving in the direction z of a vector in the tangent space:

Dxg(z) -~ xTCz �9

Next, we can choose a Riemannian metric represented by a smooth matr ix function W(x) which is positive definite for all x E M. It is well known (see e.g. [18] that , given the metric W(x), the gradient Vg can be uniquely determined from two conditions:

1 Compatibility: Dzg(z)= z T C x - zTW(x)Vg.

2 Tangency: Vg E T~M r x T V g = O.

70 B.L.R. De Moor

The unique solution to these two equations applied to the constrained optimization problem (10) results in the gradient given by

xrW(z)-lCz). Vg(~) = W ( ~ ) - ~ ( C ~ - �9 ~rW(~)-~

~,From this we obtain the gradient flow:

= - w ( ~ ) - ~ ( c ~ _ ~ ~rw(~) -1C~ ~rw(~)-~ )" (15) It is easily seen that the stationary points of this system must be eigenvector-eigenvalues of the matrix C. Convergence is guaranteed because one can easily find a Lyapunov function (essentially the norm of the gradient) using the chain rule (see e.g. [18] for details). Observe that the norm IIx(t)ll is constant for all t. This can be seen from

1 dllzll 2 = xT ~ = _ z T w ( z ) _ X C x + z T w ( z ) _ X C x = 0 2 dt

Hence, if IIx(0)ll = 1, we have IIx(t)l I = 1,Vt > 0. If we choose the Euclidean metric, W ( x ) = I~, we obtain the continuous-time power method (12). An interesting open problem is how to chose the metric W ( x ) such that for instance the convergence speed could be increased and whether a choice for W(x) other than Ip leads to new iterative discrete-time algorithms.

3.9 NON-SYMMETRIC CONTINUOUS TIME POWER METHOD

So far we have only considered symmetric matrices. However most of the results still hold true mutatis mutandis if C is a non-symmetric matrix. For instance, the analytic solution to (12) is still given by (14) even if C is nonsymmetric. For the convergence proof, we can find a Lyapunov function using the left and right eigenvectors of the non-symmetric matrix C. Let X E ]tip• be the matrix of right eigenvectors of C while Y is the matrix of left eigenvectors, normalized such that

A X - X A , y T x - - I p ,

A T y = Y A , X Y T - Ip . For simplicity we assume that all the eigenvalues of C are real (although this is not really a restriction). The vector y~n E ~P is the left eigenvector corresponding to the smallest eigenvalue. It can be shown that the scalar function

L(z) = (xTymin) 2 / ( x T y y T x )

is a Lyapunov function for the differential equations (12) because L > 0, Yr. Note that x - x ( y T x ) , which says that the vector y T x contains the components of x with respect to the basis generated by the column vectors in X (which are the right eigenvectors). The denominator is just the norm squared of this vector of components. The numerator is the component of x along the last eigenvector (last column of X), which corresponds to the smallest eigenvalue. Since g > 0, this component grows larger and larger relative to all other ones, which proves convergence to the 'smallest' eigenvector.


3.10 NON-SYMMETRIC CONTINUOUS-TIME P O W E R METHOD FOR THE SMALL- EST SINGULAR VALUE

We can exploit these insights to calculate the smallest singular triplet of a matr ix A 6 i~p• as follows. Consider the 'asymmetric ' continuous-time power method:

= - ~ ( 1 6 ) i) A T -~Iq v v '

with

= ~ ( u r ~ - v rv) I (~r~ + ~rv) "

Here c~ is a user-defined real scalar. When it is chosen such that a > amin(A), u and v will converge to the left resp. right singular vector corresponding to the smallest singular value of A. This can be understood by realizing that the 2q (we assume that p _> q) eigenvalues of the matrix

A T -a Iq

are Ai = 4-~/a2 _ a2(A) and that there are p - q eigenvalues equal to c~. Hence, if c~ > hi, the corresponding eigenvalues Ai are real, else they are pure imaginary. So if a > amin, the

smallest eigenvalue is A ~/c~ 2 2 to which (16) will converge. This is illustrated in _ _ _ O'mi n Figure 2.

3.11 CHRISTIAAN'S FLOW FOR THE SMALLEST SINGULAR VALUE

The problem with the asymmetric continuous time power method (16) is tha t we have to know a scalar ~ that is an upper bound to the smallest singular value. Here we propose a new continuous-time algorithm for which we have strong indications tha t it always converges to the 'smallest ' singular value, but for which there is no formal proof of convergence yet. Let A 6 n~ T M (p >_ q) and r 6 I~ be a given (user-defined) scalar satisfying 0 < r < 1. Consider the following system of differential equations, which we call Christiaan's flow 5:

(it)(aIPi) = - CA T - a r (17)

It is our experience that (u(t), a(t), v(t)) converges to the smallest singular triplet of A. The convergence behavior can be influenced by r (for instance, when r --* 1, there are many oscillations (bad for numerical integration) but convergence is quite fast in time; when r --* 0, there are no oscillations but convergence is slow in time). There are many similarities with the continuous-time algorithms discussed so far. When we rewrite these equations as

it = A v - u a , iJ - - r va) , (18)

5... after one of our PhD students Christiaan Moons who one day just tried it out and to his surpise found out that it always converges.

72 B.L.R. De Moor

! ! i 1 1 ! ! !

0.8 0.8 " i "

0 . 6 0 . 6

0.4 0.4

o~ o ~ i

-0.2 -0.2 - L

-0.4 -0.4 . . . . . 1

-0.6 -0 .6 -

: : : : : - 0 . 8

- 0 . 8 . . .

-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ": . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i !

, .11 o , I 0 0.5 1 1.5 2 2.5 3 0 0.5 1 1.5

!

i 2

!

2.5 3

! !

! i

3 . 5

Figure 2: Convergence as a function of time of the components of u(t) and v(t) of the asymmetric continuous-time power method (16) for the same matrix A and the same random initial vectors u(0) and v(0) as in the previous Figure. The left picture shows the behavior for a = 2, which is larger than the smallest singular value and therefore converges. The right picture shows the dynamic behavior for a - 0.5. Now, all the eigenvalues of the system matrix are pure imaginary. Therefore, there is no convergence but instead, there is an oscillatory regime.

we easily see from (1) that both equat ions are 'driven' by the (scaled) residual error (see Section 3.7). Another intriguing connection is seen by rewriting (17) as

- I 0 0 A u u (0) (0 01), which compares very well to (15), except that the metric here is indefinite! As for a formal convergence proof, there are the following facts: It is readily verified that the singular triplets of A are the stationary points of the flow. When we linearize the system around these stationary points, it is readily verified that they are all unstable, except for the stationary point corresponding to the 'smallest' singular triplet of A. However, at this moment of writing, we do not have a formal proof of convergence (for instance a Lyapunov function that proves stability) ~.

4 A L G O R I T H M S F O R T H E R I E M A N N I A N S V D

In this section, we'll try to use the ideas presented in the previous section to come up with continuous time algorithms for the Riemannian SVD (4) and hence for structured/weighted total least squares problems. The ideas presented here may be premature but on the other

6...and as we have done before, we offer a chique dinner in the most exquis restaurant of Leuven if somebody solves our problem.


! I !

0.8

o . 6 ~. . . . . . ' : . . . . . . . . . i . . . . . . . . . i . . . . . . . . . . ! . . . . . . . . . i . . . . . . . . . i . . . . . . . . .

~ : : : : : :

~ : �9 i i i i i 0.4 . . . . . . ~ i . . . . . . . . . . . ~ . . . . . . . . . i . . . . . . . . . . ! . . . . . . . . . ~ . . . . . . . . . i . . . . . . . . .

0 2 : ~ : i i i i

�9 .

i i i -0, 2 3 4 5 6 7 8 9 10

Figure 3: Example of the convergence behavior of Christiaan's flow (17) for the same matrix A and initial vectors u(0) and v(0) as in the first Figure. The full lines are the components of u, the dashed ones those of v. Both vectors converge asymptotically to the left and right singular vectors of A corresponding to the smallest singular value.

hand, they offer intriguing and challenging perspectives, which is the reason why we mention them. It should be noted that we have derived a heuristic iterative discrete-time algorithm for the Riemannian SVD in [12], the basic inspiration of which is the classical power method. While this algorithm works well in most cases, there is no formal proof of convergence nor is there any guarantee that it will convergence to (an at least local) minimum. Therefore, we try to go via these continuous-time algorithms to find an approach which would be guaranteed to converge to a local minimum. So far we have not succeeded in doing so, but we hope that the elements presented in this section provide enough new material to convince the reader of the usefulness of the presented approach.

4.1 AN OPTIMIZATION PROBLEM

Let us try to solve (see the equations (4):

uTAv vTATu a = min -- min .

u,v uTDvu u,v vTDuv

A nice property which can often be used in manipulating formulas in this framework, is that for every vector u and v, we always have uTDvu = vTDuv (see [12] for a proof). The fact that D= is independent of v and Dv is independent of u allows us to apply the continous-time algorithm (5) and derive the system of differential equations

da 1 uTAv it -- du = uTDv"-'--"~ ( - A v + Dvu urDvu) ,

da 1 vTATu -- ~ ( - A T u -~" Duv ) �9

iJ = dv vT Du v vT Du v

74 B.L.R. De Moor

While this system will converge to a local minimum, it is not this local minimum that solves our structured/weighted TLS problem. Indeed, to see why, it suffices to consider the special case where Dv = Ip and Du - Iq in which case we recover the system (12)-(13) which converges to -amax(A) and the corresponding singular vectors (and NOT amin(A)!).

4.2 CHRISTIAAN'S FLOW FOR THE RIEMANNIAN SVD

Although we are completely driving on heuristics here, it turns out that the following generalization of Christiaan's flow (17) works very well to find the minimal singular triplet of the Riemannian SVD (4).

with r E II~ a user-defined number satisfying 0 < r < 1. The only difference between (17) and (19) is the introduction of the positive definite metric matrices D,~ and D,~. We have no idea whatsoever about possible convergence properties (that are formally provable), except for the fact that in a stationary point, the necessary conditions (4) are satisfied.

Yet, our numerical experience is that this system of nonlinear differential equations converges to a local minimum of the object function. As an example, let A 6 n~ 6x5 be a given matrix (we took a random matrix), that will be approximated in Frobeniusnorm by a rank deficient matrix B, by not modifying all of its elements but only those elements that are 'flagged' by a '1' in the following matrix

0 1 0 0 0 1 0 1 0 1

V = O 0 1 1 0 1 1 1 0 1 ; Let W =

0 1 1 1 1 0 0 1 1 1

oo 1 oo oo oo

1 c~ i oo i

oo oo i i oo

I I i oo i

oo I I 1 I

oo oo 1 i i

be the elementwise inverse of V. The matrix W contains the weights as in (3) and the element oo means that we impose an infinite weight on the modification of the corresponding element in A (which implies that it will not be modified and that the corresponding element in B will be equal to that in A). It can be shown [11] that the metric matrices D~ and Dv for this flow are diagonal matrices given by

D ~ = d i a g ( V v~ ) , OH =d iag (Y T u~ )"

Here vi and ui denote the i-th component of v, resp. u and u 2 and v 2 are their squares. As an initial vector for Christiaan's flow (19) we took [u(0) T v(0) T] = leT/v/6 eT/v/5] where ek E irk is a vector with all ones as its components. The resulting behavior as a function of


4).

0

x~xxxx ~

2 4 6 8 10 1 14 16 1 20

Figure 4: Vectors u(t) and v(t) as a function of time for Christiaan's flow (19), which solves the problem of least squares approximation of a given matrix A by a rank deficient one, while not all of its elements can be modified as specified by the elements in the matrix V which belong to {0, 1}. The vector differential equation converges to the same solution as the one provided by the discrete-time algorithm of [11], which on its turn is inspired by the discrete-time power method.

time is shown in Figure 4.

5 C O N C L U S I O N S

In this paper, we have discussed how weighted and/or structured total least squares problems lead to a nonlinear generalization of the SVD, which we have called the Riemannian SVD. Next, we have derived several interesting interpretations of the continuous-time power method (geometric, unconstrained and constrained optimization, gradient flow, residual driven differential equation). We have also discussed Christiaan's flow, which is a set of nonlinear differential equations that seems to converge to the 'smallest' singular triplet, both of the SVD and the Riemannian SVD. As of now, there is however no formal proof of convergence.

There are interesting analogies between the SVD and the Riemannian SVD, which are enumerated in the following table: There are still several open problems, which all together form a complete research program: Proofs of convergence of some of the algorithms we have introduced here, discretization (e.g. based on geodesic approximations (see e.g. [26])), decreasing the rank with more than 1 while preserving the structure, etc . . . . Also in the applications there are several interesting sideways that could be followed. For instance, in [8] we have shown how our approach leads to an H2-model reduction algorithm, while in [12] [13] we show how the structured total least squares problem with a 'double' Hankel matrix of noisy inputs and outputs is the

76 B.L.R. De Moor

Problem Applications

Decomposition

Approximation B Algorithms

'Euclidean' 'Riemannian'

TLS (Eckart-Young) Weighted/Structured TLS Linear relations, noisy data

SVD of A 'Identity' metric

Hankel matrices Model reduction, Dynamic TLS Relative error TLS Maximum likelihood Riemannian SVD of A Positive definite metric Du, Dv

Rank 1 modification Multilinear matrix function Power method Power method Christiaan's flow Christiaan's flow

same as the L2-optimal so-called errors-in-variables problem from system identification.

Acknowledgements

The author is a Senior Research Associate of the Belgian National Fund for Scientific Research. This work was supported by grants from the Federal Ministry of Scientific Pol- icy (DWTC, with grants IUAP/PAI-17 (Modelling and Control of Dynamical Systems), IUAP/PAI-50 (Automation in Design and Production)), the Flemish NFWO-project no. G.0292.95 (Matrix algorithms for adaptive signal processing systems, identification and control) and the SIMONET (System Identification and Modelling Network) supported by the Human Capital and Mobility Program of the European Commision.

I would sincerely like to thank Christiaan Moons, Jeroen Dehaene and Johan Suykens for many lively discussions as well as Thomas De Moor for letting me use his balloon in my attempts to illustrate the equivalence principle of general relativity in Gene Golub's car.

References

[1] Adcock R.J. Note on the method of least squares. The Analyst, Vol IV, no.6, Nov. 1877, pp.183-184.

[2] Adcock R.J. A problem in least squares. The Analyst, March 1878, Vol V, no.2, pp.53- 54.

[3] Baldi P., Hornik K. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, Vol.2, pp.53-58, 1989.

[4] Bourlard H., Kamp Y. Auto-association by multilayer perceptrons and singular value decomposition. Biol. Cybern., 59, pp.291-294, 1988.

[5] Brockett R.W. Dynamical systems that sort lists and solve linear programming problems. Linear Algebra and its Applications, 146, pp.79-91, 1991.


[6] M.T. Chu, On the continuous realization of iterative processes, SIAM Review, 30, 3, pp.375-387, September 1988.

[7] De Moor B., Golub G.H. The restricted singular value decomposition: properties and applications. Siam Journal on Matrix Analysis and Applications, Vol.12, no.3, July 1991, pp.401-425.

[8] De Moor B., Van Overschee P., Schelfhout G. H2-model reduction for SISO systems. Proc. of the 12th World Congress International Federation of Automatic Control, Syd- ney, Australia, July 18-23 1993, Vol. II pp.227-230.7

[9] De Moor B., David J. Total linear least squares and the algebraic Riccati equation. System ~z Control Letters, Volume 18, 5, pp. 329-337, May 1992 s.

[10] De Moor B. Structured total least squares and L2 approximation problems. Special issue of Linear Algebra and its Applications, on Numerical Linear Algebra Methods in Control, Signals and Systems (eds: Van Dooren, Ammar, Nichols, Mehrmann), Volume 188-189, July 1993, pp.163-207.

[11] De Moor B. Total least squares for aJi~nely structured matrices and the noisy realization problem . IEEE Transactions on Signal Processing, Vol.42, no.11, November 1994.

[12] De Moor B. Dynamic Total Linear Least Squares. SYSID '94, Proc. of the 10th IFAC Symposium of System Identification, 4-6 July 1994, Copenhagen, Denmark.

[13] De Moor B., Roorda B. L2-optimal linear system identification: Dynamic total least squares for SISO systems. ESAT-SISTA TR 1994-53, Department of Electrical En- gineering, Katholieke Universiteit Leuven, Belgium. Accepted for publication in the Proc. of 33rd IEEE CDC, Florida, December 1994.

[14] Eckart C., Young G. The approximation of one matrix by another of lower rank. Psy- chometrika, 1, pp.211-218, 1936.

[15] Fiacco A.V., McCormick G.P. Nonlinear programming; Sequential unconstrained minimization techniques. SIAM Classics in Applied Mathematics, 1990.

[16] Golub G.H., Van Loan C. Matrix Computations. Johns Hopkins University Press, Bal- timore 1989 (2nd edition).

[17] Golub G.H., Van Loan C.F. An analysis of the total least squares problem. Siam J. of Numer. Anal., Vol. 17, no.6, December 1980.

[18] Helmke U., Moore J.B. Optimization and dynamical systems. CCES, Springer Verlag, London, 1994.

[19] Helmke U. Isospectral matrix flows for numerical analysis. This volume.

7Reprinted in the Automatic Control World Congress 1993 5-Volume Set, Volume 1: Theory. 8Forms also the basic material of a Chapter in "Peter Lancaster, Leiba Rodman. Algebraic Riccati

Equations. Oxford University Press, 1994.

78 B.L.R. De Moor

[20] Householder A.S., Young G. Matrix approximation and latent roots. Americ. Math. Monthly, 45, pp.165-171, 1938.

[21] Kung S.Y., Diamantaras K.I., Taur J.S. Neural networks for extracting pure/constrained/oriented principal components. Proc. of the 2nd International Work- shop on SVD and Signal Processing, June 1990.

[22] Ljung L. Analysis of recursive stochastic algorithms. IEEE Transactions on Automatic Control, Vol.AC-22, no.4., August 1977, pp.551-575.

[23] Oja E. A simplified neuron model as a principal component analyzer. J. Math. Biology, 15, pp.267-273, 1982.

[24] Parlett B. The symmetric eigenvalue problem. Prentice Hall, Englewood Cliffs, NJ, 1980.

[25] Pearson K. On lines and planes of closest fit to systems of points in space. Phil. Mag., 2, 6-th series, pp.559-572.

[26] S.T. Smith, Geometric Optimization Methods for Adaptive Filtering, PhD Thesis, Har- vard University, May 1993 Cambridge, Massachusetts, USA.

[27] Van Huffel S., Vandewalle J. The Total Least Squares Problem: Computational Aspects and Analysis. Frontiers in Applied Mathematics 9, SIAM, Philadelphia, 300 pp., 1991.

[28] Young G. Matrix approximation and subspace fitting. Psychometrika, vol.2, no.l, March 1937, pp.21-25.

[29] Wilkinson J. The algebraic eigenvalue problem. New York, Oxford University Press, 1965.


79

C O N S I S T E N T S I G N A L R E C O N S T R U C T I O N A N D C O N V E X C O D I N G

N.T. THAO Department of Electrical and Electronic Engineering Hong Kong University of Science and Technology Clear Water Bay, Kowloon Hong Kong

M. VETTERLI Department of Electrical Engineering and Computer Science University of California, Berkeley Berkeley, CA 9~720 U.S.A.

ABSTRACT. The field of signal processing has known tremendous progress with the development of digital signal processing. The first foundation of digital signal processing is due to Shannon's sampling theorem which shows that any bandlimited analog signal can be reduced to a discrete-time signal. However, digital signals assume a second digitization operation in amplitude. While this operation, called quantization, is as deterministic as time sampling, it appears from the literature that no strong theory supports its analysis. By tradition, quantization is only approximately modeled as an additive source of uniformly distributed and independent white noise.

We propose a theoretical framework which genuinely treats quantization as a deterministic process, is based on Hilbert space analysis and overcomes some of the limitations of Fourier analysis. While, by tradition, a digital signal is considered as the representation of an approximate signal (the quantized signal), we show that it is in fact the representation of a deterministic convex set of analog signals in a Hilbert space. We call the elements of the set the analog estimates consistent with the digital signal. This view leads to a new framework of signal processing which is non-linear and based on convex projections in Hilbert spaces.

This approach has already proved effective in the field of high resolution A/D conversion (oversampling, Sigma-Delta modulation), by showing that the traditional approach only extracts partial information from the digital signal (3dB of SNR are "missed" for every octave of oversampling).

80 N. Z Thao and M. Vertterli

The more general motivation of this paper is to show that any discretization operation, including A/D conversion but also signal compression, amounts to encoding sets of signals, that is, associating digital signals with sets of analog signals. With this view and the framework presented in this paper, directions of research for the design of new types of high resolution A/D converters and new signal compression schemes can be proposed.

KEYWORDS. A/D conversion, digital representation, oversampling, quantization, Sigma- Delta modulation, consistent estimates, convex projections, set theoretic estimation, coding.


Although numbers are usually thought of real continuous numbers in theory, signal processing is nowadays mostly performed digitally. Traditionally, digital signals are considered as the encoded version of an approximated analog signal. In many cases, the approximation error is considered negligible and digital numbers are thought of quasi-continuous. How- ever, this assumption starts to be critical in more and more emerging fields such as high resolution data conversion (oversampled A/D conversion) and signal compression.

In this paper, we ask the basic question of the exact correspondence which exists between analog signals and there encoded digital signals. This starts by reviewing the existing foundations of analog-to-digital (A/D) conversion. It is known that A/D conversion consists of two discretization operations, that is, one in time and one in amplitude. While a strong theory (Shannon's sampling theorem) describes the operation of time discretization, we will see in Section 2 that the analysis of the amplitude discretization, or quantization, is only approximate and statistical. This approach turns out to be insufficient in fields such as oversampled A/D conversion. To find out what the exact analog information contained in a digital signal is, it is necessary to have a more precise description of the whole A/D conversion chain.

In Section 3 we define a theoretical framework which permits a more precise description of A/D conversion. To do this, we go back to the basic description of an analog signal as an element of a Hilbert space (or Euclidean space in finite dimension), and wedescribe any signal transformation geometrically in this space, instead of using the traditional Fourier analysis which is limited to time-invariant and linear transformations. In this framework, we show that the precise meaning of a digital signal is the representation of a deterministic convex set of analog signals. The elements of the set are called the analog estimates consistent with the digital signal. Because of the convexity property, we show that, given a digital signal, a consistent estimate must be picked as a necessary condition for optimal reconstruction.

With this new interpretation, digital signal processing implies a new framework of (nonlinear) signal processing based on convex projections in Hilbert spaces and presented in Section 4.

In fact, the case of A/D conversion which is thoroughly considered in this paper is only a particular case of digitization system. The more general motivation of this paper is to

Consistent Signal Reconstruction and Convex Coding 81

show that the basic function of any digitization system, including high resolution data acquisition systems (Section 5) but also signal compression systems, is to associate digital representations with sets of analog signals, or, to encode sets of analog signals. Not only does this view give a genuine description of their functions, but it indicates new directions of research for the design of A/D converters and signal compression systems.

2 C L A S S I C A L P R E S E N T A T I O N OF A / D C O N V E R S I O N

The term of "digital signal processing" often designates what should be actually be called "discrete-time signal processing" [1]. Thanks to Shannon's sampling theorem, it is known that any bandlimited analog signal can be reduced to a discrete-time signal, provided that the sampling rate is larger than or equal to the Nyquist rate, that is, twice the maximum frequency of the input signal. Mathematically speaking, there exists a invertible mapping between bandlimited continuous-time signals x(t) and sequences (Zk)keZ such that

1 xk = x (kTs) , provided that T, = ]'8 >_ 2fro, where fm is the maximum frequency of x(t). Therefore, any processing of the continuous-time signal x(t) can be performed in the discrete-time domain. This constitutes the foundation of discrete-time processing.

However, digital signal processing assumes that a second discretization in amplitude, or quantization, is performed on the samples, as indicated by Figure 1. The digital output

bandlimited real real integer

x(t) xk _! quantizer l c~ d k -Isamplerl -I I

time discretization

ampfitude discretization

_1 coder I -I I

linear scaling

Figure 1: Analog-to-digital (A/D) conversion

sample dk of an A/D converter is an integer representation of ck which is a quantized version of the continuous-amplitude sample xk. The transformation from Xk to Ck is known to introduce an error ek " - C k - Xk, called the quantization error. While the time discretization process is supported by a solid theory, it appears from the literature that there only exists an approximate analysis of the quantization process. Either the quantization error is neglected and the quantization operation is considered as "transparent", or, when some close analysis is needed, it is commonly modeled as a uniformly distributed and independent white noise [2, 1] This leads to the classical mean squared quantization error of ~- where q is the

�9 1 2

quantization step size. However, this model, which is in fact only accurate under certain conditions [3, 4], does not take into account the deterministic nature of the quantization operation.

This is particularly critical when dealing with oversampled A/D conversion. Oversam- piing is commonly used in modern data conversion systems to increase the resolution of conversion while using coarse quantization. While the independent white noise model validity conditions become less and less valid with oversampling [4], it is still used as basic model to recover a high resolution estimate of the source signal from the oversampled and

82 N.T. Thao and M. Vertterli

coarsely quantized signal. With this model, a frequency analysis of the quantized signal shows in the frequency domain that only a portion of the quantization error energy lies in the input baseband region. Thus, the total energy of quantization noise can be reduced by the oversampling factor R = 2f-~, by using a linear lowpass filter at cut off frequency f,n (see Figure 2). In practice, the lowpass filtering is performed digitally on the encoded

lowpass filter cut off = fm

=Isampler quantizer I I (~)

c(o)) X ( c o ) ~

in-band E ((o),~ W j error

-rr, 2~/R rr,

(b)

v

Figure 2: Oversampled A/D conversion. (a) Principle: the sampling is performed at the frequency f s > 2fro. (b) Power spectrum of the quantized signal (Ck)keZ with the white quantization noise model.

version (dk)keZ of (Ck)keZ. On Figure 2(a), only the equivalent discrete-time operation is represented.

Although this noise reduction can be observed in practice under certain conditions, this does not tell us how much exactly we know about the analog source signal from the oversampled and quantized signal. We can already give a certain number of hints which tell us that a linear and statistical approach of the quantization process is not sufficient to give a full analysis of the signal content process.

First, it is not clear whether the in-band noise which cannot be canceled by the lowpass filter is definitely irreversible. Because quantization is a deterministic process, there does exist some correlation between the input signal and the quantization error signal, even after filtering. Second, with the linear filtering approach, it appears that the quantization mean squared error (MSE) has a non-homogeneous dependence with the time resolution and the amplitude resolution. Indeed, the MSE is divided by 4 when the amplitude resolution is multiplied by 2 (that is, q is divided by 2), whereas it is divided by 2 only when the time resolution is multiplied by 2 (that is, R is multiplied by 2). This is a little disappointing when thinking of A/D conversion as the two dimensional discretization of a continuous graph.

In fact, an example can already be given which shows by some straightforward mechanisms that the in-band noise is indeed not irreversible. Figure 3 shows a numerical example


2q ! ! ' , . . . : : ,. ~ ~ , ~

3q/2 . . . . . . . . . . . . . . . . . :-- * - * ' - - : - :- * - ". z~* -~ i ' : " ' . i X ', i ' . i '

i ,

. . . . i . . . . i . . . . ) .... i 1 :2 3 4 5 6 '7 '8 9 fO 1.1 1~2 13 1'4 1.5 1'6 17

quantization threshold

analog input signal : x ( t )

�9 quantized signal c k

,, estimate ck obtained from linearfiltering of c k

~, projection of the estimate ckonC

I remaining error

Figure 3: Example of oversampling and quantization of a bandlimited signal with reconstruction by linear filtering.

of a bandlimited signal x(t), shown by a solid line, which is oversampled by 4 and quantized, giving a sequence of values (ck)keZ represented by black dots. The classical discrete-time reconstruction (~k)keZ obtained by lowpass filtering (ck)keZ is shown by the sequence of crosses. Some error represented by grey shades can be observed between the signal reconstruction ((3k)keZ and the samples of the input signal. We know that this error forms a signal located in the frequency domain in the baseband region. However, some anomalies can be observed in the time domain. At instants 11 and 12, it can be seen that the values of (dk)kez are larger than q, while the given values of the quantized signal (ck)keZ tell us that the input signal's samples necessarily belong to the interval [0, q]. Not only is the sequence (ck)keZ not consistent with the knowledge we actually have about the source input signal, but this knowledge also gives us a deterministic way to improve the reconstruction estimate (ck)keZ. Indeed, although we don't know where exactly the samples of the input signal are located within the interval [0, q] at instants 11 and 12, we know that projecting the two respective samples of (ck)keZ on the level q leads to a necessary reduction of the error (see Figure 3). This shows that the in-band error is not irreversible.

These hints show that a new framework of analysis is necessary.

3 N E W A N A L Y S I S OF A / D C O N V E R S I O N

3.1 SIGNAL ANALYSIS FRAMEWORK

The goal is to define a framework where quantization can be analyzed in a deterministic way with the given definition of an error measure.

Bandlimited analog signals are usually formalized as elements of the space /~2(R) of square summable functions, where Fourier decomposition is applicable. Thanks to Shan- non's sampling theorem, the analog signals x(t) bandlimited by a maximum frequency fm can be studied as elements of the space s of square summable sequences, thanks to the

84 N. T Thao and M. Vertterli

invertible mapping

z ( t ) , (xk)k~Z E L2(Z) where xk = x (kT , ) , 1 under the condition that fa = ~ _ 2fro. Errors between the bandlimited analog signals are

measured using the canonical norm of s and can also be evaluated in the discrete-time space/:2(Z) using its own canonical norm, thanks to the relation:

• f~ I~(t)l ~t = ~ I~kl ~ T8 R kEZ

Unfortunately, this framework cannot be used to study quantization because the quantized version (ck)keZ of an element (Zk)keZ of s is not necessarily an element of s (or, is not necessarily square summable). For example, using the quantization configuration of Figure 3, although a sequence (Xk)kE Z may be decaying towards 0 when k goes to infinity, its quantized version (ck)keZ never goes below ~ in absolute value. On the other hand, while the MSE type of error measure can be applied for the analysis of quantized signals, it cannot be applied to the elements of s since it would systematically lead to the value 0.

Therefore, we propose to confine ourselves to another space of bandlimited signals which can be entirely defined on a finite time window [0, T0]. Precisely, we assume that the sinusoidal components of the Fourier series expansion of x(t) on [0, T0] are zero as soon as the corresponding frequencies are larger than fro. This is equivalent to saying that the T0-periodized version of z(t) defined on [0, To] is bandlimited by the maximum frequency f,n. Under this assumption, we have a finite time version of Shannon's sampling theorem.

N-1 It can be easily shown that, under the condition ~ _ 2fro equivalent to the Nyquist condition, there is an invertible mapping between z ( t ) and its discrete-time version X = (Xk)l<k<N E I~ N where zk = x ( k ~ ) f o r k = 1, .... ,N [5, 6]. In this context, we can evaluate the error between two bandlimited input signals using the mean squared sum"

M S E ( x ( t ) , x ' ( t ) ) = ~ =o

This error can be in fact evaluated in the discrete-time domain using the mean squared sum

M S E ( X , X ' ) = IIX' Xll 2 1 N - = -~ ~ I~'~- ~1 ~, k-1

thanks to the relation, easy to show [5, 6]"

__1 TO lz,(t) _ z ( t ) [2d t= _ ~ [Ztk _ zk[2. To -o

Now, the quantized version of the discrete-time signal X = (Zk)l<k<N is an element C = (Ck)lSkSN of the same space p N, where Ck =Q[zk] for k = 1, . . . ,N, and Q is the scalar quantizer function. Note that the MSE function can be applied to any element of p N

whether it is a continuous-amplitude signal or a quantized signal.


3.2 QUANTIZATION ANALYSIS

The quantization operation can be deterministically defined as a mapping Q from R, N to

it. N. But unlike the sampling operation, this is a many-to-one mapping. While, under the Nyquist condition, a discrete-time signal is characteristic of a unique analog bandlimited signal, a quantized signal C is characteristic of a whole set of possible continuous-amplitude and discrete-time signals. Mathematically, this set is simply the inverse image of C through the mapping Q, usually denoted by Q-I[C] C R g. If a sequence X is only known by its quantized version C, the exact knowledge about X available from C is that X belongs to the set of signals Q-I [C] . We call the elements of the set C =Q-I [C] the estimates of l~ N

consistent with the quantized signal C.

Figures 4(a) and (b) show the form of the set C of consistent estimates in the cases N = 1

X t C

x 3

i l j j 2

X C J coder =

(x 1 . . . . . . N c 1 . . . . . . N i I d I . . . . . dN)

(a) (b)

Figure 4: Quantization as a many-to-one mapping of R N. (a) Case N = 1. (b) Case N > 1.

and N > 1 respectively. In the case N = 1, C is equal to the whole quantization interval which contains the given quantized value c. Using the classical configuration of uniform quantization, the value c appears to be the particular consistent estimate located at the mid-point of the interval C. In the traditional view point, the digital output d of a quantizer is a binary encoded version of the quantized value c. In our approach, we consider that d is a digital representation of the complete set C. In the case N > 1, C is obviously the N dimensional cross-product of real intervals and therefore forms geometrically a hypercube of R N parallel to the canonical axes. As a generalization of the case N = 1, the quantized signal C appears to be the particular consistent estimate located at the geometric center of C. As in the case N = 1, we consider that the digital output D is the encoded version of the whole set C, not of the signal C.

In the case of oversampled A/D conversion, the quantization operation is performed, not on any element of R N, but on the sampled version of bandlimited signals only. Indeed, it is easy to see from the assumption of Section 3.1 that the bandlimited signals have a finite Fourier series expansion containing not more than 2f,.,To + 1 components. As a

86 N. I". Thao and M. Vertterli

consequence, they belong to a space of finite dimension equal to W = [2freT0 + 1], where [y] designates the smallest integer greater than or equal to y. As a second consequence, their sampled version also belongs to a W dimensional space, since the sampling operation applied on bandlimited signals is a linear and invertible mapping. Because of the Nyquist rate condition ~ > 2fro, note that we necessarily have W < N. Therefore, the sampled

versions of the bandlimited signals belong to a W dimensional subspace S of l:t N. By abuse of language, we call 5 ̀ the space of bandlimited discrete-time signals. It can be shown that the dimensional ratio N coincides approximately with the oversampling ratio R.

To recapitulate, in the oversampling context, the inputs to the quantizer are elements of the subspace S C R g. Once X E 5' is quantized into C, the complete knowledge which is available about X is that X belongs to the set S N 17 where 17 =bf Q-I[C]. We will say that S g117 is the set of estimates consistent with C. This set is geometrically represented in Figure 5.

.......... 1

Figure 5: Geometric representation of oversampled A/D conversion.

3.3 NECESSITY FOR CONSISTENT RECONSTRUCTION

In the previous section, it was shown that when an input signal is quantized, the exact information which remains available to us is that it belongs to the set of consistent estimates. However, nothing tells us until now that we must pick a consistent estimate if we want to estimate the input signal from its quantized version. We show in this section that this is in fact the case in a certain sense.

It can be easily shown from the previous section that sets of consistent estimates are convex. We recall that .,4 is a convex set if and only if for any couple of elements X, Y E `4, the segment [X, Y] is entirely included in ,4. Because the considered norm I1" II in R g is a euclidean norm, the convexity property will appear to play an important role thanks to the following lemmas:

L e m m a 3.1 [7] Let X be an element of I~ N and A C I~ N be a convex set. There exists a unique element X ' of the closure -~ of A such that for all Y E `4, I l Z ' - Y[[ < [I z - Yli. The transformation from X to X ~ is then a mapping of R N called the convez projection on


A~

L e m m a 3.2 [8] I f X I is the convex projection of X on a convex set A C R N and X q~ "A, then for all Xo e ,4, [ I X ' - Xo[[ < [ I X - Zo[[.

These lemmas are illustrated by Figure 6. They lead to the following proposition.

Figure 6: Geometric representation of convex projection.

P r o p o s i t i o n 3.3 Let Xo E I~ N be the input of a quantizer, Co the output of the quantizer and X an element of R y which does not belong to the set A of estimates consistent with Co. Then, although we don't know where the input Xo is located within the set A, the distance between X and Xo can be deterministically reduced 1 by a convex projection of X on .A.

We recall that .A is equal to Q-I[C0] without oversampling, and SNQ-I[C0] with oversam- piing. In any case, the operation of convex projection on .A is uniquely determined by the knowledge of Co.

As a conclusion, when reconstructing a discrete-time signal from its quantized version Co, any non-consistent estimate is by necessity non-optimal and can be deterministically improved using the knowledge of Co.

3.4 DETERMINISTIC ANALYSIS OF OVERSAMPLED A/D CONVERSION

We have already seen from Figure 3 an example where the reconstruction estimate C = (ck)l<k<Y proposed by the classical and linear approach of oversampled A/D conversion is not necessarily consistent with the quantized signal information and can be improved. The previous section gave the formal reason why in general a non-consistent estimate can always be improved. Now the reason why the estimate C is not necessarily consistent can be seen geometrically in the euclidean space R N as shown in Figure 5. The sequence (~ = (ck)l<k<g defined as the lowpass filtered version of C = (ck)l~k<N is more precisely the bandlimited discrete-time signal which coincides in the frequency dom~n with C = (Ck)l<k<N in the baseband region. As a consequence C is the element of the space A of bandlimitecl" discrete- time signals which is closest to C in the MSE sense, or equivalently, in the sense of the euclidean norm of R g. According to Lemma 3.1, (~ is in fact the convex projection of C on S. As shown in Figure 5, while C is the geometric center of the hypercube C, there is no reason for its convex projection C on S to remain necessarily in S, and therefore, to be consistent.

1The reduction of distance is strict if X does not belong to the closure of .4, according to Lemma 3.2.

88 IV. Z Thao and M. Vertterli

The second important question is now to know what the performance yielded by consistent estimates is in terms of MSE. The following result was recently shown in [5, 9]:

T h e o r e m 3.4 Let x(t) be a bandlimited and To-periodic signal which has a time density of quantization threshold crossings larger than or equal to the Nyquist rate. At the oversampling ratio R = N let X E R N be the sampled version of x(t), C the quantized version of X and X ~ E ~t 1~ any estimate consistent with C. Then, there exists a constant a > 0 which only depends on x(t) and the definition of the quantizer, such that

M S E ( X , X ' ) <_ - ~ .

Qualitatively speaking, this theorem implies that under a certain condition on the input's quantization threshold crossings, signals chosen in the set of consistent estimates yield an MSE which asymptotically decreases with R in (.9(R-2), instead of (.9(R -1) as it is the case with the classical linear reconstruction. This represents a faster decrease of MSE over the classical method by 3dB per octave of R. With this new result, the symmetry of the MSE dependence with the amplitude and the time resolutions is recovered.

4 C O N V E X P R O J E C T I O N B A S E D S I G N A L P R O C E S S I N G

We have seen that a digital signal is not the representation of a single estimate, but of a complete set of estimates called the consistent estimates. Although the convex set corresponding to a given digital signal is deterministically known, the problem of using this knowledge to retrieve a consistent estimate or at least partially improve a non-consistent estimate, is not trivial. Although the space of analysis R g is of finite dimension, N may be "infinitely" large compared to the finite time window of operation of the working processor. For this reason, the existing algorithms derived from the field of non-linear and linear programming [10] to retrieve an estimate satisfying convex constraints, may be not feasible.

More feasible algorithms may be derived from the field of set theoretic estimation [11] in euclidean spaces or, for the case of infinite dimension, in Hilbert spaces. The basic idea is that a convex set .A can be often decomposed as intersection of a certain number p of convex sets Ci, i = 1, ...,p with simple structure and on which the convex projections are implement able. For example, in oversampled A/D conversion, the convex set of consistent estimates is the intersection S I"1 C, where 5" and g are two convex sets of relatively simple structure. While it is difficult to find directly an estimate in 8 M g, the convex projections on S and g respectively are easily defined. The hypercube g of R y can be itself seen as the intersection of 2N convex sets which are half-spaces of R N. The projection on each of these convex sets is trivial and only implies local operations in time. Figure 7 shows in general how polygonal sets can be decomposed as intersection of half-spaces.

Assuming that the set of consistent estimates has the following decomposition ,4 P - - CIi= 1 Ci

and that the convex projection on each set r is implement able , we already have a way to partially improve any non-consistent estimate. Indeed, if X ~ .A, there exists by necessity i E {1, ..., p} such that X ~ gi. The projection of X on gi will reduce the distance of X with any element of Ci, and therefore any element of A, since ,4 C gi. This is exactly what was performed in the example of Figure 3. After noticing that (~ does not belong to Q-I[C], it

89

Xn

Figure 7: Decomposition of a polygon into half-spaces and representation of the parallel projection algorithm.

can be easily shown that the time domain operation indicated in Figure 3 is the projection of C on the convex set C =Q-I[C] and leads to a necessary improvement since C includes S N C. This process can be in general reiterated as long as the current estimate X does not belong to .A (see Figure 8).

Consistent Signal Reconstruction and Convex Coding

Figure 8: Geometric representation of the alternating projection algorithm.

It was in fact proved in [8] that by applying convex projections onto C1,C2, ...,Cp alternately and periodically, one converges to an element of the intersection 2. We formalize this property as follows:

T h e o r e m 4.1 Let CI,...,Cp be p convex sets in a Hilbert space 7-l, P1,.-.,Pp be the convex projections on Ca, ...,Cp respectively, and (Xn)neN be a sequence in 7-l such that

X,,+I = Pn mod n+l [Xn], for n e N.

p Then the sequence (Xn)nEN converges to an element of A = Ni=IC, in the sense of the Hilbert norm of 7-[.

This is often called the algorithm of alternating projections or the POCS algorithm (Pro- jection Onto Convex Sets). This algorithm became popular in signal processing with the work by Youla [12].

2Rigorously, one converges to an element of the intersection of Cl, C2, ...,C"pp,in the case where Cl, e2, ..., Cp have not been specified as closed sets.


There exists a more general version of this algorithm including relaxation coefficients (a,~),~N and based on the following operation:

X,~+I = a,~. P,~ mod p+l[ X,~] + (1 - a,~). X,~.

Note that the choice a,~ = 1 brings us back to the simple case of alternating projections. For the general case an ~ 1, it is shown that this single operation reduces the distance 3 of the current estimate X,~ with any element of ,4. It is also shown that the infinite iteration converges to an element of A if there exists e > 0 such that Vn E N, a,~ E [e, 2 - e]. In practice, the speed of convergence can be often accelerated by empirical adjustments of the coefficients an in ]1, 2[.

One drawback of the alternating projection algorithm is that it does not permit parallel processing. A new algorithm involving parallel projections was recently introduced by Combettes [13] and based on the following operation

Xn+l -- ~ 'Wi,n" Pi[Zn], where Wi,n >_ 0 and ~ wi,,~ = 1, ieln ieI,,,

and where In is a subset of indices of {1,...,p}. Qualitatively speaking, at each step n, a certain number of sets among C1, ..., Cv is selected (the set of the indices of the selected sets is called In) and the convex projections of Xn on these selected sets are applied. This forms a set of points {Pi[X,~] / i E In) and Xn+l is chosen in the convex envelop of this set. These operations are illustrated in Figure 7. The distance of the estimate X,~ with any element of ,4 is shown to be reduced by this transformation [14] and the infinite iteration is proved to converge to an element of ,4 under certain conditions on the sequence (In)heN [13]. The admissible choices of (I~)~eN include two particular cases:

(i) I,~ = {1, ...,p}: This is the case where all convex projections are performed in parallel at each step.

(ii) I,~ = {n mod p + 1}: This fails back to the case of alternating projections.

A version with relaxation coefficients is also introduced in [13] as:

X,,,+I = an" ~ wi,,~Vi[X,-,] + (1 - an)" X~,. iE I,.,

The convergence to an element of ,4 is shown to be guaranteed if 3e > 0, gn E N, an E [e, 2Lr, - e] where

_ E ~ . ~,~ I I P ~ [ X ~ ] - X~l l ~

- It- , 7o itl

5 A P P L I C A T I O N T O H I G H R E S O L U T I O N D A T A C O N V E R S I O N

Although the deterministic approach was introduced on the simple version of oversampled A/D conversion in Section 3, it is also applicable to modern techniques of high resolution data conversion such as oversampled ~A [15, 16]. The conversion scheme is similar to that of Figure 2, but the quantizer is replaced by a more sophisticated circuit called a ~A

3The reduction of distance is strict when Xn ~t S rood v+l and ~n E [0, 2].


modulator, including an integrator, a quantizer and a feedback loop (see Figure 9). This

I , s a m p l e r ZA I

I TM

ck

-I I (a)

V% (b)

Figure 9: 2A modulation. (a) Overall principle. (b) Detail of the ~A modulator.

type of data conversion allows the use of very coarse quantization (down to one bit), and thus simple circuitry, while reproducing a high resolution estimate after lowpass filtering.

Although the conditions of validity of the white quantization noise model are not really applicable here [4], ~A modulation is still classically analyzed using this model [16]. In this context, it is shown that a ~A behaves like an additive source of independent noise whose spectrum is "shaped" as shown in Figure 10. Then, it is easy to show that the

~ , . in-band / - 1 i ~ error C(o)) f I

2rdR

Figure 10: Power spectrum of the output of a ~A modulator with the assumption of white quantization noise.

portion of noise energy contained in the baseband of the quantized signal decreases with the oversampling ratio R in R -3, which represents a decrease of 9dB per octave of R. In spite of the limited validity of the assumed model, this result is observed in practice. More sophisticated architectures of ~A modulation exist, which include a higher number of integrators [17]. In general, for an n th order ~A modulator, the noise energy remaining in the baseband of the quantized signal depends on R in R -(2n+1).

Now, the same kind of question as in Section 3 can be raised here. What do we know exactly about a bandlimited signal after it is oversampled and processed through a ~A modulator ?

Like a single quantizer, a F~A modulator can also be studied as a many-to-one mapping of R g . The set C of estimates consistent with the output of a ~A modulator can be obtained


by inversion of this mapping. It is shown in [5, 6] that the set is no longer a hypercube, but a parallelepiped (the edges are no longer perpendicular). However, this is still a convex set, and it is shown that the quantized signal C = (r is still located at its geometric center. As in Section 3, the set of consistent estimates is S N C. This is geometrically represented in Figure 11. .Mthough the distance between X and C, due to the in-band

C

X

-of-band

l s ~ ............. ~ .... I- ...................

Figure 11: Geometric represetation of oversampled lEA modulation.

error remaining in the quantized signal C, decreases with R faster than in the case of simple quantization, it appears that C is still not necessarily a consistent estimate. In fact, numerical experiments performed on bandlimited and T0-periodic signals [5, 6] show that the MSE yielded by consistent estimates decreases in average w i t h / / i n O(/ / -4) instead of (.0(//-3). In general, for an n th order EA modulator~ it was shown that the average MSE of consistent estimates behaves in O(//-(2~+2)) instead of 0(R-(2~+1)), implying, as in the case of simple quantization, a faster decrease of MSE by 3dB per octave of R, regardless of the order n.

With a deterministic approach, these experiments show that the output of a lEA modulator contains more information about the input signal than that recovered with the classical approach of A/D conversion.

6 C O N C L U S I O N A N D R E L A T E D R E S E A R C H

The full meaning of a digital signal is obtained by a deterministic analysis of the digitization process as a many-to-one mapping. Thus, a digital signal is the representation of, not a single estimate, but a whole set of analog signals, called the set of consistent estimates. This set plays two roles:

(i) it gives the exact knowledge of the possible locations of the original analog signal,

(ii) it is the set where a signal should be picked when estimating the original signal from its digital version.

The second item is due to the convexity of the set, as observed on classical quantization schemes. With this approach, not only is a more precise analysis of the A/D conversion process given, but, in the context of oversampling and ~A modulation, it also leads to the conclusion that a digital output signal contains more analog information about the input


signal than that traditionally recovered by the classical analysis of A/D conversion. Namely, the MSE of consistent estimates decreases with the oversampling ratio R fastest than that of the classical linear reconstruction estimate by 3dB per octave. This new approach of digital signals implies a new framework of signal processing based on convex projections in Hilbert spaces, derived from the field of set theoretic estimation.

This past research leads to the new idea that the intrinsic function of an A/D converter is to split the space of analog input signals into convex sets, and assign a digital representation to each of them. For this reason, we say that an A/D converter is a convex coder. The intrinsic performance of a convex coder can be evaluated by its ability to split the input space into small sets with respect to the considered error measure. Recent research has been done to measure the intrinsic performance of an oversampled A/D converter or a EA modulator [18, 19, 20]. Figures 12 and 13 show that the evolution of the intrinsic performance of a ]CA modulator with the oversampling ratio R or with the order n of the modulator can be graphically observed by the set partition it defines in the input space. The intrinsic performance of the encoder can be measured by the average MSE of optimal

SINGLE-LOOP, R=4

/ ~ 3 ~ \ \ \ ' \ '

0 .4

0.2 , i- X / ~ o o~ -0.2 o o

-0 .4

-0 .6 ..

'-,, , / -1 -0.5 0 0.5 1

sin component

(a)

SINGLE-LOOP, R=6

0.6

0.4

'~ 0 .2

8 ~ -0.2

-0.4

-0 .6

-1 -0.5 0 0.5

sin component

(b) Figure 12: Partition defined by a first order EA modulator in the 2 dimensional space of To- periodic sinusoids with arbitrary amplitude and phase: (a) Case of oversampling ratio R = 4 . (b) C a s e R = 6 .

reconstruction which consists of picking for each cell of the input space partition its centroid. It was shown in [19, 20] that optimal reconstruction yields the same MSE behavior in R as consistent reconstruction. This input space view can be a new direction for the design of high resolution data converters, traditionally designed using the noise shaping approach.

The convex coding approach can also be applied to signal compression [21]. Although this field implies a digital to digital transformation, the input signal is usually considered as quasi-continuous in amplitude. In this context, it is shown in [21] that classical signal compression schemes such as block DCT coding can be analyzed as convex coding schemes. An example of new signal compression scheme is proposed by a direct and active control of the encoded sets.

/

94 N. T Thao and M. Vertterli

SINGLE-LOOP, R=4

(~ -0.5

-,/ -2 -1.5 -~ -0'.5 0 0.5

sin component 1 1.5 2

oo

~o-0.

~

DOUBLE-LOOP, R=4 . . . . . .

i "

I

-2 -1.5 -1 -0.5 0 0.5 1 .5 2 sin component

(~) (b)

Figure 13: Partition defined by a ~A modulator in the 2 dimensional space of T0-periodic sinusoids with arbitrary amplitude and phase at oversampling ratio R = 4. (a) Single-loop case. (b) Double-loop case.

References

[1] A.V.Oppenheim and R.W.Shafer, Discrete-Time Signal Processing. Prentice Hall, 1989.

[2] N.S.Jayant and P.Noll, Digital Coding of Waveforms. Prentice-Hall, 1984.

[3] W.R.Bennett, "Spectra of quantized signals," Bell System Technical Journal, vol. 27, pp. 446-472, July 1948.

[4] R.M.Gray, "Quantization noise spectra," IEEE Trans. Information Theory, vol. IT-36, pp. 1220-1244, Nov. 1990.

[5] T.T.Nguyen, "Deterministic analysis of oversampled A/D conversion and ~A modulation, and decoding improvements using consistent estimates," PhD. dissertation, Dept. of Elect. Eng., Columbia Univ., Feb. 1993.

[6] N.T.Thax) and M.Vetterli, "Deterministic analysis of oversampled A/D conversion and decoding improvement based on consistent estimates," IEEE Trans. on Signal Proc., vol. 42, pp. 519-531, Mar. 1994.

[7] D.G.Luenberger, Optimization by vector space methods. Wiley, 1969.

[8] L.M.Bregman, "The method of successive projection for finding a common point of convex sets," Soviet Mathematics - Doklady, vol. 6, no.3, pp. 688-692, May 1965.

[9] N.T.Thao and M.Vetterli, "Reduction of the MSE in R-times oversampled A/D conversion from O(1/R) to O(1/R2)," IEEE Trans. on Signal Proc., vol. 42, pp. 200-203, Jan. 1994.


[10] D.G.Luenberger, Linear and nonlinear programming. Wiley, 1984.

[11] P.L.Combettes, "The foundations of set theoretic estimation," Proc. IEEE, vol. 81, no. 2, pp. 1175-1186, Feb. 1993.

[12] D.C.Youla and H.Webb, "Image restoration by the method of convex projections: part 1 - theory," IEEE Trans. Medical Imaging, 1(2), pp. 81-94, Oct. 1982.

[13] P.L.Combettes and H.Puh, "A fast parallel projection algorithm for set theoretic image recovery," Proc. IEEE Int. Conf. ASSP, vol. V, pp. 473-476, Apr. 1994.

[14] P.L.Combettes and H.Puh , Personal communication, May 1994.

[15] J.C.Candy, "A use of limit cycle oscillations to obtain robust analog-to-digital converters," IEEE Trans. Commun., vol. COM-22, pp. 298-305, Mar. 1974.

[16] J.C.Candy and G.C.Temes, eds., Oversampling delta-sigma data converters. Theory, design and simulation. IEEE Press, 1992.

[17] S.K.Tewksbury and R.W.Hallock, "Oversampled, linear predictive and noise shaping coders of order N > 1," IEEE Trans. Circuits and Systems, vol. CAS-25, pp. 436-447, July 1978.

[18] S.Hein, K.Ibraham, and A.Zakhor, "New properties of sigma-delta modulators with dc inputs," IEEE Trans. Commun., vol. COM-40, pp. 1375-1387, Aug. 1992.

[19] N.T.Thao and M.Vetterli, "Lower bound on the mean squared error in multi-loop 2EZX modulation with periodic bandlimited signals," Proc. 27th Asilomar Conf. on Signals, Systems and Computers, Pacific Grove, CA, Nov. 1993.

[20] N.T.Thao and M.Vetterli, "Lower bound on the mean squared error in oversampled quantization of periodic signals," IEEE Trans. Information Theory. Submitted in June 1993, revised in Sept. 1994.

[21] K.Asai, N.T.Thao, and M.Vetterli, "A study of convex coders with an application to image coding," Proc. IEEE Int. Conf. ASSP, vol. V, pp. 581-584, Apr. 1994.


P A R T 2

A L G O R I T H M S A N D T H E O R E T I C A L C O N C E P T S


SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) �9 1995 Elsevier Science B.V. All rights reserved. :t.

99

T H E O R T H O G O N A L Q D - A L G O R I T H M

U. VON MATT Institute for Advanced Computer Studies University of Maryland College Park, MD 207~2 U.S.A. na. von matt @na- ne t. ornl. gov

ABSTRACT. We present the orthogonal qd-algorithm to compute the singular value decomposition of a bidiagonal matrix. This algorithm represents a modification of Rutis- hauser's qd-algorithm, and it is capable of determining all the singular values to high relative precision. We also introduce a generalization of the Givens transformation, which has applications besides the orthogonal qd-algorithm.

KEYWORDS. Generalized Givens transformation, Newton's method, Laguerre's method, orthogonal qd-algorithm, singular value decomposition.


In 1990 J. Demmel and W. Kahan showed that all the singular values of a bidiagonal matrix are determined to high relative accuracy by the entries of the matrix [2]. They modified the SVD-algorithm by Golub and Reinsch [6] as implemented in the LINPACK library [3] by introducing special zero shifts to compute the small singular values to high relative accuracy. Their code is now a part of the LAPACK library [1].

A different approach is proposed by K. V. Fernando and B. N. Parlett [4]. They use a modification of Rutishauser's qd-algorithm (cf. [5, 9, 10, 11, 12, 13]) to compute the singular values of a bidiagonal matrix. However, their algorithm can no longer be expressed as a sequence of orthogonal transformations applied to the bidiagonal matrix. Consequently, it is not possible to compute the singular vectors simultaneously with the singular values.

Our approach is based on Rutishauser's qd-algorithm, too. But the orthogonal qd- algorithm can be expressed as a sequence of Givens rotations applied from the left and the right to the bidiagonal matrix. This enables us to compute the singular vectors along with the singular values.

1 O0 U. von Matt

2 G E N E R A L I Z E D G I V E N S T R A N S F O R M A T I O N

Usually the Givens transformation

- -8 C

with c 2 + s 2 = 1 is determined such that

z2 0 "

The matr ix G is therefore used to selectively annihilate elements in a vector or a matrix. But it is also possible to introduce another value a different from zero:

~ �9 X2 (7

The v~ue of~ is given b y e ' - +~/~ + ~ - ~2.

Obviously, this transformation is only possible if lal ___ ~/x~ + z~. It is easily verified that

1

8 Xl 2 -{- X 2 X2 - - X l 0.

We will also need a variant of the generalized Givens transformation, its so-called differential form�9 In this case we determine the matrix G such that

G [ x l ] = rl X2 r2 '

where ~1 := ~ i g n ( ~ l ) ~ / ~ 2 - ~,

~ = sign(~2)J~2 + 0"2.

Of course, this is only possible if ]0"[ _< [Xl[. It is easily verified that the values of c and s are given by

~ - ~ + ~ ~2 - ~ 1 r2 "

3 O R T H O G O N A L Q U O T I E N T - D I F F E R E N C E S T E P S

In what follows, we will refer to the following n-by-n lower bidiagonal and upper bidiagonal matrices L and U:

C~l

i : - ]~1 '" U : - ~

/~n--I O/n

71

" � 9 ~ n - 1

7n

The Orthogonal QD-algorithm 101

| 81

|

O/2

8 2 0/3

" . .

~r

81

~

P O"

~

O/2

8~

P

ff

0/3

0

81

| 82

|

~3

~

*)'1 ~1

0 3'2 82

0 % �9 ~ �9 .�9 �9149 o�9

P

P

O"

�9 �9149 J ~149149

Figure 1: Orthogonal Left lu-Step with Shift s.

Def in i t ion 3.1 Let Q be an orthogonal (2n)-by-(2n) matrix. We call the transformation

(2 aI = ~/a 2 + s2I

an orthogonal left lu-step with shift s.

This transformation exists if and only if the condition Isl _< a~n(L) holds. The singular values of L are diminished by the amount of the shift s, i.e.

=

The matrix Q can be constructed by the sequence of Givens rotations depicted in Figure 1. The quantity p is an abbreviation for V'a2+ s 2. Note that every other step consists of a generalized Givens transformation. In order to avoid numerical problems for small shifts s it is necessary to use the differential form of the generalized Givens transformation.

Def in i t ion 3.2 Let Q denote an orthogonal (2n)-by-(2n) matrix. We call the transformation

Q : v/a2 -~- S2I

an orthogonal left ul-step with shift s.

This transformation represents the dual version of the orthogonal left lu-step. It can be carried out if and only if Isl < a~n(U) . The singular vMues of U are reduced by the amount of the shift s, i.e.

:

102 U. von Matt

| 1 7 4 ' ~

�9176 ~

a l 0 ~1

" ~

,~

0

0~2 0

�9 ~ ~

#2 ~3 ".

Figure 2: 0rthogonal Right ul-Step.

~

An orthogonal left ul-step, too, can be executed by a sequence of Givens rotations. The mechanism is the same as in Figure 1, with the only exception that the transformations are applied from bottom to top.

Def in i t ion 3.3 Let Q denote an orthogonal n-by-n matrix. We call the transformation

I QT Q = aI

an orthogonal right ul-step.

This transformation can always be executed, and it leaves the singular values of U unchanged. If 7n = 0 we have an = fin-1 = 0 after the transformation. This property is useful to deflate a matrix U with Vn = 0.

The sequence of Givens rotations necessary for an orthogonal right ul-step is depicted as Figure 2.

Def in i t ion 3.4 Let Q denote an orthogonal n-by-n matrix. We call the transformation

an orthogonal right lu-step.

This transformation represents the dual version of the orthogonal right ul-step. It can also be executed unconditionally, and it preserves the singular values of the matrix L. If the first row of L is zero, i.e. if a l = 0, we get a matrix U with 3'1 = 61 = 0. We can therefore use this transformation to deflate a matrix L with a l = 0.

An orthogonal right lu-step can also be carried out by a sequence of Givens rotations. The same technique is used as shown in Figure 2, except that the transformations are applied from bottom to top.

We will refer to the four transformations that have been introduced in this section by the generic term of orthogonal qd-steps.

4 O R T H O G O N A L Q U O T I E N T - D I F F E R E N C E A L G O R I T H M

Let B denote a given n-by-n lower bidiagonal matrix whose singular value decomposition is desired. The orthogonal qd-algorithm transforms the (2n)-by-n matrix

B ] A:= 0


by a sequence of orthogonal qd-steps into the (2n)-by-n matrix

0 ] E '

where IE denotes an n-by-n diagonal matrix with the singular values of B. In matrix terms this process can be described by the equation

where P denotes an orthogonal (2n)-by-(2n) matrix, and Q denotes an orthogonal n-by- n matrix.

5 D E F L A T I O N

In this section we analyse the conditions that allow us to set to zero an entry of the lower bidiagonal matrix L. This reduced matrix will be called L. We compare the singular values of the original matrix

A := a I

with those of the modified matrix

A.:= a I "

We require that all the singular values of A and A agree within the precision of the computer arithmetic. The analysis in [14, pp. 143-149] shows that we can set the diagonal entry ak to zero if the condition

applies numerically. It should be noted that/30 =/3,~ = 0. The numerical criterion for an off-diagonal element ~k reads

~2 + I~kl(l~kl + min(l~kl, I~k+~l))= ~2.

Similar criterions can be derived for an upper bidiagonal matrix U.

6 S I N G U L A R V E C T O R S

If the orthogonal transformations are accumulated the orthogonal qd-algorithm delivers the decomposition (1). In most cases, however, we would like to get the singular value decomposition

B = Vr~V T (2)

of the n-by-n lower bidiagonal matrix B, where U and V denote orthogonal n-by-n matrices. It is straightforward to identify the matrix Q from (1) with the matrix V in (2). The

104 U. von Ma t t

matrix U can then be obtained from the QR-decomposition (without pivoting) of B V :

B V = U R .

Care must be taken that the diagonal elements of R remain nonnegative.

We get an alternative way of computing the left singular vectors by partitioning the orthogonal matrix P in (1) into n - b y - n submatrices:

[ Pll P12 ] P = /~ /~ "

If B is nonsingular then we must have Pll = P22 = 0, and P12 is an orthogonal matrix consisting of the left singular vectors of B. Even in the presence of rounding errors we can force Pll and P22 to go to zero by an appropriate convergence criterion.

7 S H I F T S

The performance of the orthogonal qd-algorithm mainly depends on the choice of the shift s in each step. In this section we describe two different shift strategies based on Newton's and Laguerre's method to compute the zeros of a polynomial.

The zeros of the characteristic polynomial

p(A) := det(U w U - AI)

are the eigenvalues of u T u , which are equal to the squares of the singular values of U. Thus if we use Newton's or Laguerre's method to approximate the smallest zero of p(A) we can also get an approximation for the smallest singular value of U. Newton's method can be described by the iteration

?(~k) )~k+l = )~k - p,()~k) ,

and in the case of Laguerre's method we have

p(~k) ~ . Ak+l F(~k)

1 + i ( n - 1)(nP'C~)~-P( ~)p''(~)p,(~)~ - 1)

We choose A0 = 0 as our initial value. It is well-known (cf. [7, 8] and [15, pp. 443-445])that both methods will then converge monotonically to the smallest zero of p(A). In particular we have

0 - ~0 _~ )~1 _~ Amin(UTU) �9

Laguerre's method enjoys cubic convergence (cf. [8, pp. 353-362] and [15, pp. 443-445]). On the other hand Newton's method will converge only quadratically [15, p. 441].

We will use v / ~ as the shift in an orthogonal qd-step. Thus we define Newton's shift and Laguerre's shift as follows:

./ p(O) 3Newton V p,(o)'


"-'- i p(o) l[ n 8Laguerre -p,(o) p'(0)~ - 1)

1

Both Newton's and Laguerre's shift can be computed by recurrence formulas. Special attention is necessary to avoid overflow and underflow problems. More details may be found in [14, pp. 149-151].

8 N U M E R I C A L RESULTS

In this section we compare the performance of the orthogonal qd-algorithm with the subroutine sbdsqr from the LAPACK library [1] (see also [14, pp. 152-155]). This subroutine represents an implementation of the work of Demmel and Kahan [2]. Both algorithms compute all the singular values to high relative accuracy. We also observe that Laguerre's shift is quite expensive to evaluate. If n is the size of the matrix, the calculation of Laguerre's shift needs on the order of O(n) operations. On the other hand, Demmel and Kahan use Wilkinson's shift which only needs 0(1) operations. Wilkinson's shift also requires fewer iterations per singular value to converge. Unfortunately it can only be applied in conjunction with the QR-algorithm since it does not compute a lower bound for the smallest singular value. Laguerre's shift, on the other hand, always gives us a lower bound on the smallest singular value of the bidiagonal matrix, which is essential for the orthogonal qd-algorithm.

If only the k smallest singular values are desired the orthogonal qd-algorithm can compute them in O(kn) operations. If the singular vectors are also needed the operation count increases to O(kn2). This represents a significant savings compared to O(n 3) which is the computational complexity of a complete singular value decomposition.


We have presented the orthogonal qd-algorithm to compute the singular values of a bidiagonal matrix to high relative accuracy. Our approach differs from the qd-algorithm by Fernando and Parlett as we do not transpose the bidiagonal matrix in each step. This enables us to accumulate the orthogonal transformations and thus obtain the singular vectors.

We use Newton's and Laguerre's method to compute the shifts for the orthogonal qd- steps. Although Laguerre's shift does not quite attain the efficiency of Wilkinson's shift it has the advantage that it always computes a lower bound on the smallest singular value of the bidiagonal matrix.

We have also presented two generalizations of the Givens transformation. They come in very handy in the context of the orthogonal qd-algorithm, but they should also be useful in other applications.

Acknowledgmen t s

Our thanks go to W. Gander, G. H. Golub, and J. Waldvogel for their support. The author also thanks G. W. Stewart for his helpful comments.

106 U. von Matt

References

[1] E. Anderson, Z. Bal, C. Bischof, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Ostrouchov and D. Sorensen. LAPA CK Users' Guide. SIAM Publications, Philadelphia, 1992.

[2] J. Demmel and W. Kahan. Accurate Singular Values of Bidiagonal Matrices. SIAM J. Sci. Stat. Comput. 11, pp. 873-912, 1990.

[3] J. J. Dongarra, C. B. Moler, J. R. Bunch and G. W. Stewart. LINPA CK Users' Guide. SIAM Publications, Philadelphia, 1979.

[4] K. V. Fernando and B. N. Parlett. Accurate singular values and differential qd algorithms. Numer. Math. 67, pp. 191-229, 1994.

[5] W. Gander, L. Molinari and H. Svecovs Numerische Prozeduren aus Nachlass und Lehre yon Prof. Heinz Rutishauser. Internat. Ser. Numer. Math., Vol. 33, Birkhguser, Basel, 1977.

[6] G. H. Golub and C. Reinsch. Singular value decomposition and least squares solutions. Numer. Math. 14, pp. 403-420, 1970.

[7] E. Hansen and M. Patrick. A Family of Root Finding Methods. Numer. Math. 27, pp. 257-269, 1977.

[8] A. M. Ostrowski. Solution of Equations in Euclidean and Banach Spaces. Third Edition of Solution of Equations and Systems of Equations, Academic Press, New York, 1973.

[9] H. Rutishauser. Der Quotienten-Differenzen-Algorithmus. ZAMP 5, pp. 233-251, 1954.

[10] H. Rutishauser. Der Quotienten-Differenzen-Algorithmus. Mitteilungen aus dem Insti- tut fiir angewandte Mathematik Nr. 7, Birkh~iuser, Basel, 1957.

[11] H. Rutishauser. Uber eine kubisch konvergente Variante der LR-Transformation. ZAMM 40, pp. 49-54, 1960.

[12] H. Rutishauser. Les propridtds numdriques de l'algorithme quotient-diffdrence. Rapport EUR 4083f, Communaut~ Europ~enne de l'Energie Atomique - EURATOM, Luxem- bourg, 1968.

[13] H. Rutishauser. Lectures on Numerical Mathematics. Birkh~iuser, Boston, 1990.

[14] U. von Matt. Large Constrained Quadratic Problems. Verlag der Fachvereine, Ziirich, 1993.

[15] J. H. Wilkinson. The Algebraic Eigenvalue Problem. Clarendon Press, Oxford, 1965.


107

A C C U R A T E S I N G U L A R V A L U E C O M P U T A T I O N W I T H T H E J A C O B I M E T H O D

Z. DRMA(~ Fachbereich Mathematik, Fern UniversitSt Hagen, Postfach 9~0, D-5808~ Hagen, Germany. zlatko, drmac @fern uni-hagen, de

ABSTRACT. The main interest in this work is implementation of the Jacobi method as reliable software for computing the singular values of a general real matrix. The modifications of formulas for computing the rotation angle and modified Jacobi transformation are given in order to ensure reliable computation of singular values in the full range of floating point numbers. If the gradual underflow is used, then the effect of underflow on the accuracy properties of the modified Jacobi SVD method is not worse than the uncertainty caused by roundoff errors.

KEYWORDS. Jacobi method, relative accuracy, singular values.


The following is a very short summary of recent results in the field of singular value computation:

The singular values of general matrices are not perfectly conditioned ( in the sense of Weyl's theorem ), if we consider relative perturbation estimates. In other words, if a > 0 is singular value of A E R mxn, then small perturbation

5A = A o E, 5Aij = AijEij, IEijl <_ �9 << 1

can cause ~a with ]3al/a > e. The estimate based on Weyl's theorem implies for a r 0

]~a[ _~ [[~A[]2 _ V/-~g2(A)e, ~2(A)= [IA[12[[A~[]2. o" o"

This can be improved for full column rank A by

I~1 A + ~A = (Q + ~AR-1)R ==~ <_ [15AR-1112 <_ v / ~ 2 ( A s ) e , (1) (7

108 Z. Drrna~'

where A = QR is the QR factorization of A and As = Adiag (llAe~llz) -~. The estimate (1) is valid/or IlaAe~llz _< ellAe~ll2, 1 _< i _< n, �9 < 1/(v/'ff~2(As)). This is the essence of the perturbation theory of Barlow and Demmel [1] and Demmel and Veselid [~] which introduced ~2(As) as a condition number for relative perturbations. Since ~2(As) < v'ff~z(A) and since it is possible that ~;2(As) << ~;2(A), the bounds yrom [I], [4] are better than those derived from the Weyl's theorem.

The use of orthogonal transformations in the singular and eigenvalue computation is not per se the guarantee for the relative accuracy of the computed output. For example, the singular values of A can be insensitive to small pointwise perturbations Aij ~ Aij h- r and small pointwise perturbations of A ~ = AV, V some orthogonal matrix, can cause loss of information on the smallest singular values of A.

The nonorthogonal transformations, if properly used, are useful tool in the matrix computation, even in solving the singular value and the symmetric eigenvalue problems.

The new development introduced the following approach" Describe a set of matrices for which the singular values can be computed with relative accuracy and find a method capable of achieving that accuracy.

For full column rank A with moderate ~2(As) the algorithms of choice are the implicit Jacobi SVD of Demmel and Veselid [4] and the implicit Cholesky SVD of Fernando and Parlett [9].

In this work we discuss reliable floating point implementation of the Jacobi SVD method. The presented material is short summary of [7], [5], [8], [6].

2 J A C O B I S V D

Let A E t t m• have full column rank. The Jacobi SVD algorithm generates the sequence

A (~ = A, A (k+l) - A(k)u (k), k - 0 , 1 , 2 , . . .

where U (k) is Jacobi rotation in the (p(k),q(k)) plane. Demmel and Veselid have shown

that the relative errors in the computed singular values depend on maxk>o ~2(A (k)) (A (k) - A(k)diag (llA(k)eill2) -1) and not on ~2(A). If the elements of A already have uncertainties, then even exact computation will not improve on Jacobi. This claim of Demmel and Veselid is based on well behaviour of maxk>0 ~2(A(k))/~2(As). However, the usual floating point implementation of the algorithm is not reliable software in the sense of [3]. We want to recommend modification of the Jacobi SVD method which satisfies the criterion of reliability [3].

2.1 TO ROTATE OR NOT TO ROTATE

Let ti = A[ep, eq] = [ap, aq] e R rex2 and

r = [ ,l~,,~ (a~,~)] j = (a~,~) ,,~,,~ ' cos r sin r

- sin r cos r , j ~ r J = diag (2)

Singular Value Computation with the Jacobi Method 109

where

cot 2r -- Ilaqll2 - Ilapll2 tan r = sign cot 2r

2(av, aq) I cot 2r + X/1 + cot2 2r (3)

The standard approach in the implicit Jacobi-like methods reads 1. Compute the Gram matrix of pivot cohmns; 2. Decide whether to rotate or not; 3. If to rotate then

3.1. Compute J using (3) ( or some other symmetric 2 x 2 eigensolver ); 3.2. Perform ft. := AJ using some library routine.

First observation is that, because of squaring ( 11%112 2, Ilaqll~ ), the computation is possible for (roughly) a(A) C (v/'5, x / ~ , where a(A) denotes the set of singular values of A and u, co axe the underflow and the overflow treshold, respectively. Next, the dot product (av, %) can underflow and that additionally reduces the range of safe computation to the interval (roughly) ( x / ' ~ , V/w), where e denotes the roundoff unit. This restriction can be removed by rewritting the formula for cot 2r to

Ila.ll~ Ila.l12 cot 2r = [lapl]2 - Ilaqll2

2 (ap, aq) Ilapll2[laqll2

where for Ilapll211aqll2 ~ ( v / 6 , a , ) , (ap, a~)lll%ll2111aqll2 is computed as

[]ap[]21]aql]2 []ap[]2 ' l]aq[]2 ) -- []ap]]2 Ilaq]]2 " . _

Let t o l be the tresh for performing the Jacobi rotation and let

l(ap, aq)l llapll211aqll2

> toi. (4)

Usually t o l E {me, v/'me}. It is reasonably to assume 1;ol << 1. The importancy of the criterion (4) for the relative accuracy is explained in [1], [4]. The bound (4)implies that cot 2r overflows ( [cot 2r > co ) only if

max 11%112'11ap112 > 2coto l .

Assume for simplicity [lapll2 > I]aqll2. We shah analyse the situation in the case f l (cot 2r e {Inf ,--Inf}, i.e. 11%112 >> Ilaq[12 �9 Note, in that case t an r is computed as zero and the transformation is identity, thus useless and the whole process may not converge. This means that for ~2(A) > co (roughly) the floating point implementation depends on tc2(A). We have to simulate the acting of the Jacobi rotation with I tanr < v. The insight into the geometric structure of the transformation in such situation is possible for

Ila~l12 > Ilaqll~

110 Z. Drmat'

In that case

Ilaqli2 11~ll2 Ila~ll2 cot 2 r I lapll2- Ilaqll2 ~ -Ilaqll~ , Icot2r ~ 1

and

tanr ~ ~cot2r ~ Ilapll~ hold with relative error e. Thus, I tan r -< v~ and the cosine will be computed as one. Note that

! aq = aq "F ap tan r ~, aq

ap, aq)

performs the Gram-Schmidt orthogonalization of aq against ap and that I ap = ap - aq tanr

changes ap by

I tanr ~ I(ap, aq)l Ilaqll~ < Ilaq I~ Ila~l1211a~l12 Ila~l12 - I1% ~llapll2 _< ellapll2. In the case of overflow due to II~pll2/llaqll2 > ~ t o l , the relative change of ap is negligible. On the other side, the change of aq is bounded by I cos | aq)lllaqll2 and it cannot be neglected. In this situation we perform the Jacobi rotation in the following way:

! ap = ap, (7 )

Thus, although there is no machine representation for cot 2r and tan r because of overflow and underflow, respectively, performing Jacobi rotation is possible thanks to the connection between the Jacobi rotation and the Gram-Schmidt orthogonalization. It is important to note that large cot 2r reflects relative relations between geometric terms and cannot be removed by scaling cA, c 6 R. Note, the transformation (7, 8) in matrix notation reads

E 1 0 ][ apaq 1 1 [a~ a~q] = lap aq] Ilapll2 1 I l ap l l2 o , , 1 Ilapll211aqll2 o Ilaqll2 "

o II~qll2 o 1 Three nonorthogonal transformations are used to simulate the orthogonal one.

2.2 HOW TO ROTATE

Our next modification is motivated by the -qmaf option I of the X L - F O K T K A N 2.0 compiler on IBM RISC System/6000. The -qmaf option means using the mar ( multiply-and-add )

1This option is default.


function which computes a . x + y with one rounding, i.e. for floating point reals a, x, y

Let us shortly discuss application of the Jacobi rotation in the floating point arithmetic providing the mar function. The Jacobi SVD algorithm does not explicitly introduce zeros into the matrix, thus inaccurate tangents affect the convergence ( and, therefore, the overall perturbation analysis ) but not the one-step floating point error analysis. Let t be the computed value of the tangent of the Jacobi angle. The computed value of the cosine is estimated as follows:

~ 1,1+ x/ll ~ l + e 2 + t 2 l+ei = f z ( t = ) = (1 + ~=), I~,1 _ 2~ + ~= I~=1, I~=1 < ~.

The transformation is implemented in the factorized form . q

1 t C 0 ] = [(~ - ty)~, (y + t~)~] ~, y e R ~ (9) [x,y] - t 1 0 c ' �9

This can be treated as exactly orthogonal transformation

1 1 [~, y] ~ [(~ - ty) v' l + t2' (y + t~) v'l + t2],

followed by the multiplicative perturbation I + Ec, IIEcll2 ___ ec, which causes ec relative error in the singular values. Note that the condition number of the current matrix does not impact relative errors in singular values caused by I + E~. This means that r is a small part of the perturbation, as compared with relative perturbations caused by performing the transformation. Thus, in the case of forward stable rotation technique, the condition number takes place after maximal possible number of nonoverlapping rotations. This suggests a sort of optimal preconditioning in computing the singular values of full column rank A E R m• - performing the maximal number of nonoverlapping rotations between pivot pairs with largest cosines of the angles between corresponding columns. An additional advantage of (9) is that it can exploit small tangent t for faster computation. Indeed, for Itl <_ v/~ only the "fast part" of the transformation is performed.

Consider now de Rijk's [2] linked triads rotation technique. Our main interest is forward stable implementation of the linked triads technique in the floating point environment providing the mar function. We give short forward error analysis which supports the claim: The linked triads implementation of the Jacobi rotation should determine the smaller vector in the first triad. Consider the two possible linked triads factorization of the rotation,

1 0 i ~ cosr 0 (10) [ap, a'q] = [ap, aq] _ tan r 1 0 1 0 cosr '

i li 1 0 [ os0 0 ] [ ;] 1 tanr 1 1 (11) alp,a = [ap,aq] 0 1 - ~ s i n 2 r 1 0 "

COS ~)

112 Z. Drmag

Let t be the computed value of the tangent and let ~ = f l ( t / ( 1 + t2)). Then the first triad in (10) can in the floating point computation be represented by 2

a~ := (r + Z ) ( a ~ - ta~), IIEIl~ < r

The next one reads aq := ( I + F)(aq + a ( I + Z)(ap - taq))

= (x + F ) ( ~ + , , ( ~ - ta~))+ ~(I + F ) E ( ~ - t ~ ) , IIFII~ < ~, and we easily see the forward manifestation of the wrong choice. 3 In the case of grading ( [lapl]2 > I]aqll2 ) the first triad in (11)is replaced by the modified transformation, and the next one is skipped.

2.3 OVERALL ERROR BOUND. RELIABILITY

In some applications of the SVD procedure is the input matrix quadratic. For example A E R mx'~ is factorized ATr = QR by the QR factorization with optional column pivoting and the SVD of R is computed. This is reasonable e.g. for m >> n. (In the eigenvalue computation positive definite H is factorized, ~r*HTr = LI, ~, by Cholesky factorization with optional pivoting and the SVD of/_, is computed.) Due to the unitary invariance of the spectral norm, ~2(As) = ~2(Rs), where As = Adiag ([]Aei[[2) -1, Rs = Rdiag ([[Rei[]2) -1. The accelerated Jacobi method [4] runs on R r (on L in the eigenvalue computation of r~Hlr = LLr) . The following two theorems show that the perturbations of singular values caused by the Jacobi procedure are of the same order as those caused by the QR (Cholesky) factorization.

T h e o r e m 2.1 Let A E l:t mxn and let al >_ ".>_ an > 0 be its singular values. Let R be the upper triangular matrix computed by Givens QR factorization in the I E E E arithmetic with relative precision r and let 6A be the backward perturbation (A + 6A = QR). I f the overall proces can be performed in p parallel steps, then the singular values al ~ "'" ~ an of R satisfy

I~ O'il _< v~llAts[12((1 + 6r p - 1), 1 ___ i _< n, (12)

where q < n denotes the maximal number of nonzero entries in any row of 6A and provided that the right hand side in (12) is less than one. For m >> n and suitably chosen pivot strategy p ~ log 2 m + (n - 1) log 2 log 2 m.

T h e o r e m 2.2 I f one sweep of the Jacobi SVD method with some pivot strategy can be performed in p parallel steps on A E R nxn, then after l sweeps

l a i - ail _ v/-ff{id~sR{]2(( 1 + 10r _ 1), 1 <_ i <_ n, AsR = diag ({le~d[12)-lA,(13) ryi

where al >_ . . . >_ an > 0 and ~r 1 >_ . . . >_ ~,~ are the singular values of A and the matrix obtained at the end of the i th sweep, respectively, and provided that the bound in (13) is less than one.

~For the sake of technical simplicity we do not consider underflow exceptions. STake e.g. Ilapl[2 ~ HaqH2 and ap and aq nearly parallel.


The main difference between our and the analysis of Demmel and Veselid is that we estimate relative errors in matrix rows while transforming its columns. The point is that the condition number of row-scaled matrices remains unchanged during the process because of the unitary invariance. 4

The conclusion of the error analysis is: If the gradual underflow is used and the singular values of the stored matrix belong to (v, r then the effect of underflow on the accuracy of the modified Jacobi SVD method is not worse than the uncertainties caused by roundoff e r r o r s .

2.4 FAST CONVERGENCE ON GRADED MATRICES. WHY?

As we know, if the pivot columns are differently scaled, then the Jacobi rotation behaves like the Gram-Schmidt orthogonalization. On the other side, the modified Gram-Schmidt procedure follows the pat tern (1, 2), ( 1 ,3 ) , . . . , ( n - 1, n), just as the row-cyclic 3acobi process does. With proper pivoting, introduced by de Rijk [2], the Jacobi SVD algorithm is forced to behave on graded matrices like the modified Gram-Schmidt orthogonalization, s (Initial sorting of matr ix columns in nonincreasing sequence additionally supports this similarity.) This explains fast convergence of Jacobi SVD method on graded matrices.

2.5 CONDITION BEHAVIOUR. AN EXAMPLE

Explaining the excellent behaviour of maxk tc2(A(sk))/tc2(As) is stated in [4] as an important open problem. For the sake of compatibility with [4], [10] we consider hereafter the

symmetric two-sided Jacobi method. Set H = ArA, Hs = ArsAs, H (k) = (A(k))*A (k),

k = 1 , 2 , . . . Mascarenhas [10] found examples where maxk>ox2(H(k))/~2(Hs) can be as large as n/4. Although this is not bad because of factors of n already in the floating point errors, the condition behaviour in the Jacobi method remains interesting problem. Mas- carenhas used S(n ,a) = ( 1 - a)I + aee ~, e = ( 1 , . . . , 1 ) ~, 0 < a < 1, with n = 2 t and pivot strategy that produced recursively matrices of the same type. In the first step he obtained S (1) = (1 - a)In/2 • (1 + a)S(n/2, (1 - a ) / (1 + a)); in the next step the same

strategy was applied to the lower right 2 z-1 x 2 l-1 submatrix of S (1) etc. To the growth of the condition of scaled matrices shown in [10], we added in [8] the fonowing observation: The matrix S(n ,a) is already optimally scaled, i.e. ~2(S(n,a)) <_ ~2(DS(n,a)D) for any diagonal nonsingular D, and the same property have all matrices S (i) generated by Jacobi. Similar analysis holds for the row-cyclic strategy.

Thus, the matrix S(n, a) is actually an example of perfect behaviour of Jacobi rotations.

A c k n o w l e d g e m e n t s

The material presented in this note is contained in the authors PhD Thesis, made at the Department of Mathematics, University of Hagen, and under the supervision of Prof. K.

4A similar result is obtained independently by Mathias [11]. 5The same analysis holds for the J-symmetric Jacobi method [12].

114 Z. Drma~'

Veseli6. The author wishes to express his appreciation and thanks for many useful comments to Prof. K. Veseli6.

References

[1] J. Barlow and J. Demmel. Computing accurate eigensystems of scaled diagonally dominant matrices. SIAM J. Num. Anal., 27(3):762-791, 1990.

[2] P. P. M. de Rijk. A one-sided Jacobi algorithm for computing the singular value decomposition on a vector computer. SIAM J. Sci. Stat. Comp., 10(2):359-371, 1989.

[3] J. Demmel. Underflow and the reliability of numerical software. SIAM J. Sci. Stat. Comp., 5:887-919, 1984.

[4] J. Demmel and K. Veseli~. Jacobi's method is more accurate than QR. SIAM J. Matrix Anal. Appl., 13(4):1204-1245, 1992.

[5] Z. Drma~.. On relative errors caused by Jacobi and Givens rotations. Submitted to Numer. Math., October 1993.

[6] Z. Drma~. Accurate singular value computation by Jacobi method. Submitted to SIAM J. Sci. Comp., March 1994.

[7] Z. Drma~. Computing the Singular and the Generalized Singular Values. PhD thesis, Lehrgebiet Mathematische Physik, Fernuniversit~t Hagen, 1994.

[8] Z. Drma~. On the condition behaviour in Jacobi method. Submitted to SIAM J. Matrix Anal. Appl., January 1994.

[9] K. V. Fernando and B. N. Parlett. Implicit Cholesky algorithms for singular values and vectors. Technical report, Center for Pure and Applied Mathematics, University of California, Berkeley, August 1993.

[10] W. F. Mascarenhas. A note on Jacobi being more accurate than QR. SIAM J. Matrix Anal. Appl., 15:215-218, 1993.

[11] R. Mathias. Fast accurate eigensystem computations by Jacobi methods. Technical report, University of Minnesota, Minneapolis, May 1993.

[12] K. Veseli6. A Jacobi eigenreduction algorithm for definite matrix pairs. Numer. Math., 64:241-269, 1990.


115

N O T E O N T H E A C C U R A C Y O F T H E E I G E N S O L U T I O N O F M A T R I C E S G E N E R A T E D B Y F I N I T E E L E M E N T S

Z. DRMAr K. VESELIC Fachbereich Mathematik, Fern Universit~t Hagen, Postfach 9~0, D-58084 Hagen, Germany. [email protected], kresimir, [email protected]

ABSTRACT. Matrices generated by finite elements usually have high condition numbers which may spoil the accuracy of their lowest eigenvalues. This note considers some common classes of such matrices. These matrices are determined by some parameters which are not merely their non-zero elements. These parameters determine well the eigenvalues and the eigenvectors and the Cholesky factor. The latter is shown to be a perfect condition estimator and rank revealer. Accurate computation of this factor is still an open problem, we have succeeded to perform it only for some very simple cases: the simple chain a ad the simple loop with an arbitrary number of earth connections.

KEYWORDS. Finite elements, relative accuracy, singular values.

1 INTRODUCTION AND SUMMARY

To find the singular values ai of a matrix with a high relative accuracy means to get the relative errors 6ai/ai small even then when ai differ greatly in their magnitude:

I~1 ~ c ~ (1) for the corresponding singular values ai, ai + ~ai, respectively. Here C is a mild constant ([1]) and ~ is the machine precision. Algorithms of QR type with this kind of accuracy exist for bidiagonal matrices ([3, 6]). The same error bound holds for totally acyclic matrices (and for no other sparsity pattern) treated by a bisection algorithm ([2]). On the other side there exist broad classes of dense matrices for which (1) holds. These matrices are characterised not by a sparsity pattern but by a "scaled" condition number. In fact, if

G = DB , B has full column rank , (2) where D is diagonal and nonsingular then

C .~ condB, (3)

116 Z. Drma~ and K. Veseli~

([1, 4, 8]). A nearly best D has the diagonals equal to the Euclidean norms of the rows of G. The matrix G for which g2(DG) with some diagonal nonsingular D is a moderately growing function of n is called well-scaled from left. Since the transposed matrix G T has the same singular values on a square matrix G the right scaling G = B1D1 has the same consequence. On these matrices Jacobi methods are accurate ([4, 5, 7]). We are interested here in matrices arising e.g. in finite element formulation of vibrational systems. A vibrational system is governed by the differential equation

Ks + M x = 0,

where K (stiffness matrix) and M (mass matrix) are n x n real symmetric and positive definite. The eigenfrequencies ai of this system are the square roots of the eigenvalues of the problem Kx = AMz , or, equivalently, the generalized singular values of any pair (G1, G2) with K = GTG1, M = GTG2.

We consider here the matrices of the form

K = ~ kiziz T , M = diag ( m t , . . . , m ~ ) , (4) i

where ki, mi are positive parameters and zi are some incidence vectors. This again leads to the singular value problem for the matrix G = diag (gi)A0diag (#i) with A~ = [zlz2...],

gi = v/~s, #i = mTsl/2. Quite generally it is advisable to compute the eigenvalues of a positive definite matrix K by the singular values of its factor, especially if that factor can be assembled directly. The reason is that in this way we avoid squaring which may spoil small eigenvalues. Our situation is even more special: here the singular values depend on the parameters ai, #i "with the condition one". Indeed, the perturbation ki + ~ki, mi -F ~mi with I~kil <_ ~ki, [~mil_ Emi implies for all x E R n

( 1 - ~)xT g x <_ x T ( g + ~ g ) x < (1 + ~)xT g x , (5)

(1 - -e )xTMx <_ xT(M + ,SM)x <_ ( I + e ) x T M x . (6) As an immediate consequence of the minimax principle this implies (1) with C replaced by approximately 1 t. It would be ideal to have an algorithm, based on the input data ai, #i, which computes the singular values with that accuracy. In want of genuine direct algorithms we may try to transform "safely " the initial matrix e.g. into a bidiagonal one. No way of doing this seems to be known yet. Another possibility would be to arrive at a well-scaled matrix by a "safe" QR decomposition GM-1/2P = Q[R.TO] T, where Q is orthogonal and R is upper triangular. In other words, R is the Cholesky factor of H = p T M - 1 / 2 K M - 1 / 2 P , K = GrG. Here P is a permutation. Our first result (w 2) is" Let K be an irreducible diagonally dominant symmetric M-matrix; take the permutation P such that the (diagonal) matrix P M P T is increasingly ordered. Then the scaled condition of R from the left is at most nx/(n + 1)/2 independently of K, M. Here two points are remarkable: (i) the pivoting used is not the one of the standard "optimal" column pivoting, putting at every step the largest column to the first position, (ii) the factor R is a perfect condition estimator and rank-revealer in the sense that the singular values ai satisfy the

a Note that the estimate for a general bidiagonal matrix is obtained just in transforming it to a scaled bidiagonal form, the larger constant obtained there (it is linear in n) is due to the loss upon this transformation ([1, 3]).

The Accuracy o f the Eigensolution o f Matrices 117

following estimate

J n(n + 1)IR.I < ~i < J'~IR.I , (7)

independently of K, M.

The other problem: to transform K, or better G, safely to this R seems to be much tougher. We have succeeded to do this only for two types of oscillation nets: the chain and the simple loop with an arbitrary number of earth connections; see w 3. In both cases the QR factorization of G is even forward stable.

2 W E A K L Y D I A G O N A L L Y D O M I N A N T M - M A T R I C E S

We consider here symmetric matrices with the properties:

1. the diagonal is positive, 2. the off-diagonal is non-positive, 3. the sum of each column is non-negative.

LFrom this it follows that K is positive semidefinite. Without loss of generality we can assume that K is irreducible. An example of such K is

g = ~ k i jwi jw T , (8) i<j

with kij >_ 0 and

ei, for i = j, (ei canonical basis vector). wij = ei - ej, for i ~t j

Conversely, it is immediately seen that all symmetric matrices with the properties 1, 2, 3 above are of the form (8) 2. Obviously, for an irreducible K the positive definiteness means that one of kil is positive. Note that the Cholesky factor of K coincides with the QR factor of the matrix G for which K = G ' G .

For instance, taking in (8) n = 3 and the coefficients k11, k12, k23, k13 > 0 we obtain

[ J kll + k12 q- k13 -k12 -k13 K;12 -~12 0

K = -kx2 k12 + k23 -k23 G = ~13 0 -~13 . (9) ' 0 /~23 --/~23

-k13 -k23 k23 + k13 /~11 0 0

In spite of the mentioned fact that the parameters kij determine perfectly the singular values of G and their squares i.e. the eigenvalues of K, by taking above k12 = k13 = k23 - 1, kll = 10 -16 the MATLAB function svd(G) gave here the singular values, rounded to 8

2One should distignuish between the matrices (kij) and K, the latter having elements Kij. Of course, ki.i = -Ki j for i ~r j.

118 Z. Drma~ and K. Veseli~

places: [ +000 ] 1.7320508e + 000 . O.O000000e + 000

Thus, no correct digit is obtained in the smallest singular value 10-16/~/3 + (.9(10 -32) ,,~ 5.773502691896258.10 -17 although the machine precision of MATLAB is 10 -16. Also, the triangular factor computed by MATLAB's qr(G) had large relative error in the (3, 3) entry.

We now come to the central result of this section. Any given upper triangular R with the properties 1, 2, 3 above can be scaled as

R = D ( I - S~) , D = diag ( R l l , . . . , R n n ) �9 (10)

Here S,~ is a strictly upper triangular non-negative matrix of order n with the row sums not exceeding one.

T h e o r e m 2.1 Let Sn be as above�9 Then

1 1 . . . 1 0 1 . . . 1

O <_ A,~ = ( I - S,~) - I <_ T,~ = . .. .. . . �9 �9 . ,

0 . . . 0 1

If, in addition, all nonvanishing row sums of S,~ are one then the last column of A,~ has all o n e s .

Since the spectral norm is a pointwise monotone norm on positive matrices we have

[[A,~[12 _< [IT,~[12 _< ~ n ( n 2 + 1) . (11)

Here the first inequality is attainable, while the second is very sharp since

1 2n

2 sin ~ r 4 n + 2

Thus, we have the following estimates for the extremal singular values of I - S,~

(i s~) < 2

~ . ( I - s~) > ~(~ + 1)' ~ ~ - v ~

The matrix R from (10) is well-scaled from the left; in particular, the inequalities (7) follow from Ostrovsky's theorem. The same estimates follow for matrices of type n x m:

= D [ ( I - S , ~ ) , ] , D = d i a g ( R 1 1 , . . . , R , ~ ) . R (12) C o r o l l a r y 2.2 An y K of the form (8) has a Cholesky factor or, equivalently, any corresponding [ wll ]

a = diag ( k l l , k 1 2 , . . . ) w12

The Accuracy of the Eigensolution of Matrices 119

has the QR-factor which is well scaled.

Note also that all assertions above are absolutely independent of the permutation of rows and columns in K or, equivalently, of columns in G. If the matrix K is substituted by M - 1 / 2 K M -1/2, M diagonal then instead of R above we obtain R M -1/2 which will inherit the well-scaledness of R, if the diagonals of M are increasing. Otherwise the left scaled condition of R M -1/2 may be spoiled. This necessitates the coresponding prepermutation of K, M or equivalently, G, M which, however, is different from the standard column pivoting OI1 e .

3 F O R W A R D S T A B L E Q R F A C T O R I Z A T I O N

In this section we describe the floating point implementation of the QR factorization of matrices from Corollary 2.2. Let ~, e, w be the underflow treshold, the roundoff unit and the overflow treshold, respectively. We use the standard model of the floating point arithmetic, i.e. for floating point reals a, b, c in the absence of underflow and overflow

f l ( a o b) = (a o b)(l + eab), leabl _< e, o e { + , - - , ' , / } ,

f t ( f i ) = f i ( ~ + ~), I~ _< ~. We first consider quadratic matrices with the following two properties: P1. Each nonzero column contains exactly two nonzero elements. P2. Each nonzero row contains exactly two nonzero elements and their sum is zero.

Let 7~n C R n• denote the set of matrices that have the properties P1 and P2. For A E Pn the QR factorization can obviously be divided into n - 1 ( at most ) steps where each step consists of one Givens rotation and eventually one row interchange.

T h e o r e m 3.1 Let A E 7),~ and let A = Q R be the QR factorization of A constructed by a sequence of Givens rotations in some prescribed order. Let R = i:l + 5R be the triangular factor computed by the floating point implementation of the same process. Then in the absence of underflow and overflow

15Rij] <_ Lx _ kx _/ - 1 IRijl, 1 < i < j < n, (13)

hold, where X+ = 1 + e and Pij <_ i - 1 depend on the number of changes of relevant elements in the past of the position (i, j ) .

This analysis immediately implies the following perturbation estimate.

T h e o r e m 3.2 Let A E 79n be given by incidence matrix Ao and n parameters h i , . . . , h n . Let A' E 79n have the same incidence matrix Ao and perturbed parameters h~ = hi + ~hi, 15hi] <_ elhil, I <_ i <_ n. Let R, R' be the triangular factors of A, A', determined by the same order of Givens rotations. Then

'R~ j - RiJ' < [ (1+ 1 - e 1] .Rij., l < i < j < _ _

The forward stability property in the QR factorization of A E P~ remain if the structure is fixed against rigid-body motion. We give another interesting physical example as introduction to the next step in the algorithm and perturbation theory development.

120 Z. Drmal and K. Veseli~

In the nodal analysis of e.g. electrical resistance network the node conductance matrix of the Julie bridge is

TAB ~" TAD n c TEA --TAB 0 --TAD

K = --TAB TAB q- 7 B C -[- 7BE - - T B C 0

0 --TBC 7BC @- 7CD "4- 7CE --TCD "

--TAD 0 --TCD TAD 2r 7CD 2r ")'DE

Then K = A~A with

A

~V~7~ - ~V'q~ o o o o - ~

o ~ - ~Vq-~ o o o ~ - ~

o o o o ~ o o o o ~ o o o o

The upper 4 x 4 part of A is an element from 7)4 and can be transformed into the triangular form in a forward stable way. To get the triangular factor of K the update by the lower 4 x 4 part of A is needed�9 This leads us to the following problem: Given M-mat r ix g = ~ k i j w i j = R~VR with upper triangular M-matr ix R and [( = K § ~_,~=1 wieie~, wi >__ O, .find the Cholesky factor of [( in a forward stable way, using R. and w l , . . . , w n . Without loss of generality we take wl > 0, w2 . . . . . wn = O. The sign paterns are as follows:

~ D � 9 m

. . . . .

. . . .

A - ~

+

+

R ~ I , A r A = I ( = 021 el

. . . . . .

m + . . . . .

: : : " . �9 �9 . �9

. . . .

Rotating at (1, n § 1) in A produces

A : =

o �9 �9

. . . .

+ + + . . . +

and this transformation is obviously forward stable�9 Now, rotate at (2, n § 1). The sign patern of the transformation of pivot rows is

- + + + + . . . + 0 + + . . . + "

All 'pluses' can be computed in a forward stable way. The minus signs ( @ ) are secure because /~ is an M-matrix , but in the application of the rotation they are computed by

The Accuracy of the Eigensolution of Matrices 121

proper substraction which does not obey the forward stability property. We can easily handle this problem by using formulas from the Cholesky factorization of K. Indeed,

A2j := ~ 1 ( / ( 2 j - A12Alj), 3 <_ j <_ n, /'122 ~

<0 >o

are forward stable formulas for the rest of the second row. Generally, Aii is stable function during the QR factorization and

Aij "= ( K i j - AkiAkj), i + 1 <_ j <_ n, k = l

are also forward stable formulas. In simple words, the diagonals of R are computed .from the QR factorization and the off-diagonals from the Cholesky factorization, if needed.

The single precision QR factorization followed by one sided single precision Jacobi algorithm gave 53 = 0.57735030.10 -18 as smallest singular value of the matrix (9), with relative error less than 6 .10 -s.

References

[1] J. Barlow and J. Demmel, Computing accurate eigensystems of scaled diagonally dominant matrices, SIAM J. Num. Anal., 27 (1990), pp. 762-791.

[2] J. Demmel and W. Gragg, On computing accurate singular values and eigenvalues of acyclic matrices, Linear Algebra Appl., 185 (1993), pp. 456-467.

[3] J. Demmel and W. Kahan, Accurate singular values of bidiagonal matrices, SIAM J. Sci. Stat. Comp., 11 (1990), pp. 873-912.

[4] J. Demmel and K. Veselid, Jacobi's method is more accurate than QR, SIAM J. Matrix Anal. Appl., 13 (1992), pp. 1204-1245.

[5] Z. Drma~, Computing the Singular and the Generalized Singular Values, PhD thesis, Lehrgebiet Mathematische Physik, Fernuniversitgt Hagen, 1994.

[6] K. V. Fernando and B. N. Parlett, Accurate singular values and differential qd algorithms, Numer. Math., 67 (1994), pp. 191-229.

[7] R. Mathias, Fast accurate eigensystem computations by Jacobi methods, technical report, University of Minnesota, Minneapolis, May 1993.

[8] K. Veselid and I. Slapni~ar, Floating-point perturbations of Hermitian matrices, Linear Algebra Appl., 195 (1993), pp. 81-116.



123

T R A N S P O S E - F R E E A R N O L D I I T E R A T I O N S F O R A P P R O X I M A T I N G E X T R E M A L S I N G U L A R V A L U E S A N D V E C T O R S 1

M.W. BERRY Department of Computer Science University of Tennessee i07 A yres Hall Knoxville, TN 37##6-I301 USA berry @cs. utk. edu

S. VARADHAN Department of Computer Science University of Tennessee 107 A yres Hall Knoxville, TN 37996-1301 USA varadhan @cs. utk. edu

ABSTRACT. Block Lanczos algorithms are commonly used to compute extremal eigen- pairs or singular triplets of large sparse matrices. For well-conditioned matrices A, the block Lanczos algorithm can be used to compute extremal singular values and singular vectors by solving the equivalent symmetric eigensystem of A TA. Each iteration requires multiplication by both A and A T. An alternative algorithm based on the block Arnoldi method for the nonsymmetric eigensystem [A]0 ] is presented. This method does not require multiplication by A T up to restarting. The performance of both methods in the number and type of sparse matrix multiplications is presented for a few selected test matrices. For diagonally-dominant matrices, Inverse Iteration (INVIT) is one acceleration alternative that can be used to compute higher-accurate restart vectors for a single-vector Arnoldi-based SVD method. Both single-vector and block Arnoldi SVD methods may be more suitable for architectures in which multiplication by A T does not equal that by A subject to data structure constraints.

KEYWORDS. Arnoldi, extremal, singular, sparse, transpose-free, values, vectors.

1This research was supported by the National Science Foundation under grant Nos. NSF-CDA-9115428 and NSF-ASC-92-03004.

124 M.W. Berry and S. Varadhan


In applications such as information retrieval and seismic reflection tomography [3], the singular value decomposition (SVD) of large sparse matrices is needed in the shortest possible time. Given the growing availability of high-performance computer systems, there has been great interest in the development of efficient implementations of the singular value decomposition. SVDPACKC [4] is an ANSI C software library specifically designed to compute the SVD of large unstructured sparse matrices.

Without loss of generality, suppose A is a sparse m by n (m _> n) matrix with rank(A) = r. The singular value decomposition (SVD) of A can be defined as

A = UEV T, (1)

where uTU = v T v = In and E = diag(al, ..., an), ai > 0 for 1 __ i < r, ai = 0 for i > r + 1. The first r columns of the orthogonal matrices U and V define the orthonormalized eigenvectors associated with the r nonzero eigenvalues of AA T and A TA, respectively. The singular values of A are defined as the diagonal elements of E which are the nonnegative square roots of the n eigenvalues of AA T. The set {ui, ai, vi} is called the i-th singular triplet. The singular vectors (triplets) corresponding to large (small) singular values are called large (small) singular vectors (triplets).

SVDPACKC uses Lanczos, block-Lanczos, subspace iteration, and trace minimization methods to approximate extremal singular values and corresponding singular vectors. Two canonical sparse symmetric eigenvalue problems are solved by SVDPACKC routines to (indirectly) compute the sparse SVD. For well-conditioned matrices A, the n x n eigensystem of AT A is the desired eigenvalue problem. For SVDPACKC methods such as B[.52, which implements a block Lanczos iteration similar to [5], each step of the Lanczos recursion requires multiplication by A and A T. If memory is not sufficient to store both A and A T, the matrix A is normally stored in an appropriate compressed sparse matrix format [2]. Subsequently, the multiplication by A T may perform poorly as compared to multiplication by A simply due to excessive indirect addressing enforced by the data structure in A [3]. Can the multiplication by A T be avoided altogether? Probably not, but as demonstrated in the next section, the block Arnoldi method applied to the m • m matrix [AI0 ] is an alternative scheme which can suppress the multiplication by A T up to restarts.

2 AI : tNOLDI SVD M E T H O D

Sand and Schultz [8] [9] originally proposed the GMI:tES method for solving large sparse nonsymmetric linear systems based on the Arnoldi process [1] for computing the eigenvalues of a matrix. Can Arnoldi's method, which is essentially a Gram-Schmidt method for computing an orthonormal basis for a particular Krylov subspace, be used to compute the SVD in (1) without having to reference A T explicitly? Consider the block Arnoldi iteration below.

Transpose-free Arnoldi Iterations 125

A l g o r i t h m 2.1 Transpose-Free ARnoldi (T[=AR) SVD iteration.

1. Choose V1 (m • b) such that vTv1 = /b, and define B = [AlO], so that B is an m • m nonsymmetric matrix, and A is a given m • n (m _> n) sparse matrix.

2. For j = 1, 2, ...k

W = B •

Hid = viTw, i = l , 2 , . . . , j

J W = W - y ~ V i •

i=1 W = QR, QTQ = Ibm and R is upper triangular,

Hj+I,j - R.

Let V = IV1, V2, . . . , Vk], and assemble the block upper Hessenberg matr ix H from the b • b submatrices, Hi, j , i = 1 , . . . , k + 1;j = 1 , . . . , k , generated in Step 2 of the Arnoldi iteration above. Define the n • bk matrix V = [In l0] • V so that

is an orthogonal factorization of 1)" where Q k is n x bk and has orthonormal columns. It can be shown that

AQ, k = VCk + Vk+l HTk+l,k • GT]~-I, (2)

where Ck = HkR -1 , Hk is the bk• block upper Hessenberg matr ix constructed by deleting the last b • b submatrix of / tk , and G T is the b • bk matrix defined by G T = [0, . . . , 0 I/b].

Via (2), the extremal singular triplets of the m • n matrix A are approximated by the k singular triplets of the k • k Hessenberg matrix Ck. If the SVD of Ck = (]kEkV T , where uT~rk = ~'TVk = Ik and ~k = diag[~l, a2, . . . , ak], then

A(Ok~/k) = (V~fk)Ek + (Vk+~HT+~,k)(aTk-~ fZk), (3)

and the i-th diagonal element of Ek is an approximation to the exact singular value, hi, of A in (1).

Now, define Yk = QkVk = [Yl, Y2, . . . , Yk] and Zk = YUk = [zl, z2, . . . , Zk], where Yk and Zk are n x k and m x k, respectively. Then, (3) becomes

AYk -- Zk~k + (Vk+l HT )(GT/~-IVk) (4) k+l,k

where the columns of Zk and Yk are approximations to the left and right singular vectors of A, respectively. If the columns of Yk are not suitably accurate approximations of the right singular vectors of A, Algorithm 2.1 can be restarted by setting

vxr [(A Tzk~;~ r r = ) 10~_~,b], (5)

126 M. W. Berry and S. Varadhan

as an improved approximation to b-largest right singular vectors of A. Hence, b multiplications of A T are required upon each restart of Algorithm 2.1.

If one can easily solve systems of the form

(AT A - b2I)y k+l = 0, (6)

then Inverse Iteration (INVIT) can be used to greatly improve the accuracy (see [7]) of the restart vectors stored as columns of V1. The use of INVIT for restarting Algorithm 2.1 is referred to as Arnoldi with INVerse iteration or AINV.

3 PRELIMINARY PERFORMANCE

The performance of Algorithm 2.1 or TFAR and BL52 from SVDPACKC [4] is compared with respect to the number of required multiplications by A and A T. All experiments were conducted using MATLAB (Version 4.1) on a Sun SPARCstation 10. For the four test matrices mentioned below, the p-largest singular triplets, where p = 2q, q = 1, 2, 3, 4, are approximated to residual accuracies [Iri[[2 = []A~i- aifii[]~ of order (..9(10 -3) or less. Deflation was used in both methods to avoid duplicate singular value approximations.

In Figure 1, the well-separated spectrum of the 374 x 82 term-document matrix from [3] is approximated. Initial block sizes b = p and Krylov subspace dimension (bk) bounds of 2p were used in both methods. The TFAR scheme required, on average, 60% fewer multiplications by A T (represented by y=A'x) with a 40% increase in multiplications by A only for the cases p = 6, 8. In Figure 2, a well-separated portion of the spectrum of the 457 x 331 constraint matrix from the collection of linear programming problems assembled in [6] was approximated. These experiments used initial block sizes of 2, 4, 6, 4 and corresponding Krylov subspace dimension bounds of 12, 24, 24, 20 for p = 2, 4, 6, 8 triplets, respectively. On average, TFAR required 65% fewer multiplications by A T but unfortunately at the cost of over 200% more multiplications by A.

Figures 3 and 4 compare the performance of AINV, BLS2, and TFAR on two 50 x 50 synthetic matrices with clustered spectra. The spectra of these test matrices (see table below) are specified by the function r fl, 6, k,n,~x, ima=) = (ak + t3) + 6i, where k = 0 ,1 , . . . , kmax is the index over the clusters, i = 1 , 2 , . . . , irnax is the index within a duster, 6 is the uniform separation between singular values within a duster, and a, fl are scalars.

Number of Matrix Clusters CLUS1 13 CLUS2 5

Cluster Separation (6)

10 -3 10-6

Maximum Cluster Size

4 10

~.(a,13,6, kmax, ima=) r 12,4) r

In approximating the p-largest triplets of CLUS1, both TFAR and BL52 used initial block sizes of 2, 4, 6, 4 and corresponding Krylov subspace dimension bounds of 12, 24, 24, 20 for p = 2, 4, 6, 8, respectively. For CLUS2, initial block sizes of 2, 4, 6, 8 and corresponding Krylov subspace dimension bounds of 4, 8, 12, 16 were used. An implementation of AINV using a Krylov subspace dimension bound of 16 and deflation for a static block size b = 1


470

300

cg~ 100

= BLS2 (y=Ax) --m- BLS2 (y=A'x) / ,,4~4"

" TFAR (y=Ax) / ,,,4"," - ,a- TFAR (y=A'x) / /("

Number of Singular Triplets

Figure 1: Performance of BLS2 and TFAR on the 374 • 82 matrix ADI for 10 -3 residual

accuracy.

470

3oo

~2oo

100

= BLS2 (y=Ax) - - i - BLS2 (y=A'x) -*-4 TFAR (y=Ax) . ~

�9 1 " ~ " 8


Figure 2: Performance of BLS2 and TFAR on the 457 x 331 matrix SCFXM1 for 10 -3 residual accuracy.


: AINV (y=Ax) - 4- AINV (y=A'x) , ;

: BLS2 (y=Ax) , " s - g- BLS2 (y=A'x)

�9 TFAR (y=Ax) ..o --~- TFAR (y=A'x)

~ 100-

I I

1 - - - - - - - O - 2 " 1 ~ 8


Figure 3: Performance of AINV, BLS2, and TFAR on the 50 • 50 matrix CLUS1 for 10 -3 residual accuracy.

s I : mNV (y=Ax)

| - 4 - AINV (y=A'x) / . " | = BLS2 (y=Ax) ~ "

/ - - - BLS2 (y=A'x) / . " .2~2oo.~ �9 TFAR (y:Ax) / ~ ' "

sssssSSS S

2 4 6 8 Number of Singular Triplets

Figure 4: Performance of AINV, BLS2, and TFAR on the 50 x 50 matrix CLUS2 for 10 -3 residual accuracy.


(i.e., a single-vector iteration) was also applied to these synthetic matrices. In Figure 3, about 85% fewer multiplications (on average) by A T and 35% fewer multiplications (on average) by A are required by TFAR.

In Figure 4, the reduction factors in multiplications by A and A T for TFAR are about 50% and 80%, respectively. At the cost of solving the systems in (6), both Figures 3 and 4 demonstrate that AINV requires approximately 42% and 96% fewer multiplications by A and A T, respectively, than that of BIS2.

• Mults by A

l Mults by A'

20 40 60 80 Subspace Dimension

Figure 5: Distribution of sparse matrix-vector multiplications for TFAR as the Krylov subspace dimension increases. The largest singular triplet of the 374 x 82 term-document matrix ADI is approximated to a residual accuracy no larger than 10 -3.

Figure 5 illustrates how the distribution of sparse matrix-vector multiplications by A or A T varies as the Krylov subspace dimension (k from Algorithm 2.1) increases. Here, k ranges from 10 to 75 as we approximate the largest singular triplet { u l , a l , Vl} of the 374 x 82 ADI matrix. Clearly, the need for restarting (see (5)) diminishes as the subspace dimension increases and thereby requires fewer and fewer multiplications by A T at the cost of requiring more storage for the Arnoldi vectors (columns of V in Algorithm 2.1).

4 F U T U R E W O R K

Although extensive testing of TFAR and AINV is certainly warranted, the results obtained thus far for selected matrices is promising. Future research will include comparisons with recent Arnoldi methods based on implicitly shifted QR-iterations [10]. Although there does not appear to be a way to completely remove the multiplication by A T in computing the SVD of A, there can be substantial gains for methods such as TFAR (or AINV) when multiplication by A T is more expensive than multiplication by A. Alternative restarting


procedures may improve the global convergence rate of the TI=AR method and further reduce the need for multiplications by A T .

References

[1] W. E. Arnoldi, The Principle of Minimized Iteration in the Solution of the Matrix Eigenvalue Problem, Quart. Appl. Math., 9 (1951), pp. 17-29.

[2] R. Barrett et al., Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, SIAM, Philadelphia, 1994.

[3] M. W. Berry, Large Scale Singular Value Computations, International Journal of Su- percomputer Applications, 6(1992), pp. 13-49.

[4] M. W. Berry et al., SVDPACKC: Version 1.0 User's Guide, Tech. Rep. CS-93-194, University of Tennessee, Knoxville, TN, October 1993.

[5] G. Golub, F. Luk, and M. Overton, A Block Lanczos Method for Computing the Sin- gular Values and Corresponding Singular Vectors of a Matrix, ACM Transactions on Mathematical Software, 7 (1981), pp. 149-169.

[6] I. J. Justig, An Analysis of an Available Set of Linear Programming Test Problems, Tech. Rep. SOL 87-11, Systems Optimization Laboratory, Stanford University, Stan- ford, CA, August 1987.

[7] B. N. Parlett, The Symmetric Eigenvalue Problem, Prentice Hall, Englewood Cliffs, NJ, 1980.

[8] Y. Saad, Krylov Subspace Methods for Solving Large Unsymmetric Linear Systems, Mathematics of Computation, 37 (1981), pp. 105-126.

[9] Y. Saad and M. H. Schultz, GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems, SIAM Journal of Statistical and Scientific Com- puting, 7 (1986), pp. 856-869.

[10] D. C. Sorensen, Implicit Application of Polynomial Filters in a K-Step Arnoldi Method, SIAM Journal of Matrix Analysis and Applications, 13 (1992), pp. 357-385.


131

A L A N C Z O S A L G O R I T H M F O R C O M P U T I N G T H E L A R G E S T Q U O - T I E N T S I N G U L A R V A L U E S I N R E G U L A R I Z A T I O N P R O B L E M S

P.C. HANSEN UNI. C, Building 304 Technical University of Denmark DK-2800 Lyngby, Denmark Per. [email protected]

M. HANKE Institut fiir Praktische Mathematik Universitgt Karlsruhe, Englerstrasse 2 D-76128 Karlsruhe, Federal Republic of Germany hanke @ipmsun l. mathematik, uni-karlsruhe, de

ABSTRACT. In linear regularization problems of the form min{llA x - bll 2 + )~21IB xll2}, the quotient singular value decomposition (QSVD) of the matrix pair (A, B) plays an essential role as analysis tool. In particular, the largest quotient singular values and the associated singular vectors are important. We describe an algorithm based on Lanczos bidiagonalization that approximates these quantities.

KEYWORDS. Lanczos bidiagonalization, QSVD, regularization.


If the coefficient matrix of a (possibly overdetermined) system of linear equations is very ill conditioned, then the solution is usually very sensitive to errors (measurement errors, approximation errors, rounding errors, etc.), and regularization is therefore needed in order to compute a stabilized solution which is less sensitive to the errors.

Perhaps the most popular regularization method is the method due to Tikhonov and Phillips, see [7]. In this method, the regularized solution is computed as the solution to the following problem

rnJn {llAx - bl] 2 + )~211B zl]~). (1)

132 P.C. Hansen and M. Hanke

Here, A is the coefficient matrix and b is the right-hand side of the linear system to be solved. In many regularization problems in signal processing, such as image restoration and other deconvolution problems, the coefficient matrix A is either (block) Toeplitz or (block) Hankel. This structure of A should be utilized whenever possible.

The matrix B in (1) defines a seminorm, and typically B is a discrete approximation to a derivative operator. Without loss of generality we can assume that B has full row rank. We will also assume that B is reasonably well conditioned--otherwise it will not have the desired regularizing effect.

The parameter A is the regularization parameter, and it controls how much emphasis is put on minimizing the smoothness of the solution x, measured by lib zl12 , relative to minimization of the residual norm IIA x - b[12. How to choose )~ is an important topic which we shall not consider here; see, e.g., [9] for details.

In addition to the Tikhonov/Phillips method, many other regularization methods can be defined. Surveys of some of these methods can be found in, e.g., [4], [7] and [9]. Common for all these methods is the appearance of the matrix pair (A, B), where A is the coefficient matrix, and B is the regularization matrix.

For all these regularization methods involving the pair (A,B), it turns out that the Quotient Singular Value Decomposition 1 (QSVD) is a very valuable tool for analysis of the problem as well as for computing regularized sohtions. Let A and B have dimensions m x n and p x n, respectively, satisfying m __ n _> p. Then the QSVD takes the form

A = U ( ~0 In-,O ) X-I' B = V ( M , O ) X -1, (2)

where U and V have orthonormal columns, X is nonsingular, and E and M are p x p diagonal matrices,

= diag(ai), M = diag(#i). (3)

Then the quotient singular values are defined as the ratios

"r~ = ,~ lu~ , i = 1 , . . . , p . (4)

We note that if B is nonsingular, then the quotient singular values are the ordinary singular values of the matrix quotient A B -1.

The reason why the QSVD is so important in the analysis of regularization problems is that the regularized solution can often be expressed in terms of the QSVD components. For example, for Tikhonov regularization (1) the regularized solution is given by

~'r "/? + '~2 ffi i=p+l Here, ui and xi are the columns of U and X, respectively. Other regularized solutions can be obtained from (5) by replacing 72/(72 + ,~2) with the appropriate filter factors for the particular regularization method. Thus, we see that information about the QSVD components gives important insight into the regularization problem.

1The QSVD was previously known as the Generalized SVD.

A Lanczos Algorithm 13 3

In particular, the QSVD components corresponding to the largest quotient singular values 7i are important because they constitute the essentially "unregularized" component of the regularized solution, i.e., the component for which the associated filter factors are close to one. In fact, one can define a regularization method (called "truncated GSVD" in [10]) in which the corresponding filter factors are simply ones and zeros.

Another important use of these QSVD components is to judge the choice of the regularization matrix B. Only limited theoretical results are available regarding this choice, and often the user wants to experiment numerically with different B-matrices and monitor the influence of B on the basis vectors--the columns of X ~ f o r the regularized solution.

The largest quotient singular value is also needed as a scaling factor in connection with the v-method for iterative regularization, cf. [9, w

Hence, there is a need for efficient algorithms that compute the largest QSVD components when A is large and structured or sparse. A Lanczos-based algorithm for computing the largest QSVD components of a general matrix pair was proposed recently by Zha [14]. In this paper we present an alternative algorithm specifically designed for the case where B is a banded matrix with full row rank~which is the typical situation in regularization problems. By making use of this property of B, we obtain an algorithm which is faster than the more general algorithm. A related direct algorithm which involves a full SVD computation can be found in [11].

2 T H E QSVD L A N C Z O S B I D I A G O N A L I Z A T I O N A L G O R I T H M

The key idea in our algorithm is to apply the Lanczos bidiagonalization algorithmmfor computing the largest singular values, see, e.g., [1, w [2, w and [6, w the matrix A, but "preconditioned" from the right in a specialized way such that the quotient singular values and associated vectors are obtained.

The "preconditioner" is BtA, the A-weighted generalized inverse of B, see [3], which in terms of the QSVD can simply be written as

( M-1) vT (6) BtA = X 0 '

and we see that A BtA is then given by

0 (7)

Hence, the ordinary SVD of A BtA is identical to part of the QSVD of (A, B).

The use of this "preconditioning" for iterative methods was initially advocated in [8]. We emphasize that the purpose of the "preconditioner" is not to improve the spectrum of the matrix, but rather to transform the problem such that the desired quotient singular values are computed. More details can be found in [9].

In the Lanczos bidiagonalization algorithm we need to multiply a vector with A BtA and (A BtA) T. Hence, two operations with BtA are required, namely, matrix-vector multiplication


with BtA and (BtA) T. With the aid of some limited preprocessing, both operations can be implemented in ( . 9 ( ( n - p + l )n) operations, where I is the band width of B. The preprocessing stage takes the following form.

PREPROCESSING STAGE W .-- basis for null space of B A W = Q R ( QR factorization) S ~ R - 1 Q T A

/3 ~ augmented B (see text below) /} = ],/)" (L V factorization of/}) .

Usually, the n x (n - p) matr ix W can be constructed directly from information about B; see, e.g., [11] and (19) below. The QR decomposition of the "skinny" m x ( n - p) matrix A W is needed to compute its pseudoinverse (A W) t = R -1 QT. The computation of S requires O(m(n - p)2) operations plus n - p multiplications with A and with A T.

Next, the p x n matrix B is augmented with n - p rows such that the resulting mat r ix /3 is square and nonsingular, followed by L U factorization of J3. If B is a discrete approximation to the (n - p)th derivative operator of the form (1_1) (1_21)

B = ".. ".. or B = ".. ".. ".. ,

1 - 1 1 - 2 1

then one can set 0),

(s)

( ~ )

and no L U factorization is actually required because /} is lower triangular. If B is a discrete approximation to the Laplacian on a square N x N grid (in which case n = N 2 and p = n - 4), then one particular choice s of B has the block form

B(~ 0 B(1) B(2) B(1)

�9 . . . . . . ( ~ 0 )

B= B(i) B(~) BO) 0 B (~

where B (~ is ( N - 2) • N, B (1) and B (~) are N x N, and these matrices have the form

B (o) = ".. ".. ".. (11) 2 - 4 2

B (1) = d iag (2 ,1 ,1 , . . . , 1,2) (12)

2This particular choice of B corresponds to the MATLAB function del2.

A Lanczos Algorithm 135

- 4 0 1 - 4 1

B (2) = ".. ".. ".. . 1 - 4 1

0 - 4 Then one can obtain/} by augmenting B (~ as follows

( la)

(40...0) ~ ( o ) , _ (o) ,

0 . . . 0 4 (14)

and since B has bandwidth 2N + 1, an LU factorization without pivoting requires O ( n N 2) = O(N 4) operations, see [6, w For this choice of/}, pivoting is not required because/~ is column diagonally dominant (see [6, w

Given the matrices W and 5', and possibly an LUfactorization of/}, the two algorithms for multiplication with B?A and (B?A) T take the following form.

COMPUTE y = BtA z ,-- x (pad x with zeros according to B ~ / ~ )

y ~ /}-12

y ~ y - W b ' y .

COMP,TE V = ( B ~ ) r �9 X ~-- X-- s T w T x

fl , - ( i~-~ ) r ~ y *- ~ (extract elements of ~) according to B ~ B).

Both algorithms require ( .9((n- p + t)n) operations, where l is the band width of B. In particular, if B is given by (8) and (10), then 3 ( n - p)n and 2(N + 4)N 2 multiplications axe required, respectively.

3 H O W TO E X T R A C T T H E N E C E S S A R Y Q U A N T I T I E S

The Lanczos algorithm immediately yields approximations of the largest quotient singular values 7i and the associated singular vectors ui and v~ (the columns of U and V in (2)). From 7i, the quantities ai and #i in (3) can easily be computed from

7~ 1

" ' = J1 +-~' ~' : J1 +-y? (15) which follow from (4).

Finally, we need to be able to compute the columns xi of X corresponding to 7i. Using the QSVD (2) and the definition (6) of B?A, we get

B ~ v = x 0 ' (16)


from which we obtain the relation

zi = ~i BOA vi. (17)

Hence, xi can be computed from the corresponding vi by multiplication with B?A, which can be performed by means of the algorithm from the previous section.

Notice that the vectors ui and xi for i = p+ 1 , . . . , n cannot be determined by our Lanczos algorithm. However, the spaces span{xp+i , . . . , xn} and span{up+i , . . . , un}--which may be more important than the vectors--are spanned by the columns of the two matrices W and A W, respectively. Moreover, the vector

�9 o = ~ ~Tb ~i, i..---p+l

which is the component of the regularized solution x~ (5) in the null space of B, can be computed directly by means of the relation

xo = W (A W) t b = W R -1QT b, (iS)

where Q and R form the QR factorization of A W, cf. the preprocessing stage.

We emphasize that our single-vector Lanczos algorithm cannot compute the multiplicity of the quotient singular values. If this is important, one should instead switch to a block Lanczos algorithm, cf. [5].

4 N U M E R I C A L E X A M P L E S

We illustrate the use of our Lanczos algorithm with a de-blurring example from image reconstruction. All our experiments 3 are carried out in MATLAB using the REGULARIZATION TOOLS package [12]. The images have size N x N. The matrix A is N ~ x N 2, it represents Gaussian blurring of the image, and it is doubly Toeplitz, i.e., it consists of N x N Toeplitz blocks which are the same along each block diagonal; see [9, w167 and 8.2] for details. The matrix B is the discrete approximation to the Laplacian given by (10)-(13), and its size is ( g 2 - 4) x g 2. The starting vector is Ae/llAell2 with e = (1 , . . . , 1) T, and no reorthogonalization is applied.

The matrix W representing the null space of B is computed by applying the modified Gram-Schmidt process to a matrix W0 of the following form, shown for N = 4:

Ii 1 1 1 1 1 1 1 1 1 1 1 1 1 1 i\ w T = 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 " (19) 1 2 3 4 2 4 6 8 3 6 9 12 4 8 12 16

Our first experiment involves a small problem, N = 16, for which we computed all the 252 quotient singular values 7i explicitly by means of routine gsvd from [12]. The first five 7i axe very well separated in the range 175-46, the next 15 7i are less well separated in the range 38-6.3, and the next 30 7i are even closer separated in the range 5.5-0.67.

3The MATLAB routines used in these tests are available from the authors.

A Lanczos Algorithm 137

number of iterations

k 10 20 30 40 50

index i of exact quotient singular value 3'~ 3 6 10 14

1 2 4 5 7 8 9 11 12 13 15 16 17 1 1 1 1 3 2 2 1 1 1 1 1 1 4 3 3 2 2 1 1 1 1 5 4 4 3 2 2 2 2 1 7 5 5 3 3 2 2 2 2

1 1 1 1 1 1 1 1 1 2 1 1 1 1

19 18 20

Table 1: The convergence history of the quotient singular values in the small example with N = 16. The table shows the number of approximations to the ith exact quotient singular value 3'i, after k Lanczos iterations, whose absolute accuracy is better than 10 - l~ Multiple Vi, such as V3 and 74, are grouped.

O ~

k u(k) smallest converged 10 5 1.19.103 20 11 2.56 �9 102 30 14 2.56 �9 102 40 21 8.98.101 50 28 3.21.101

Table 2" The convergence of the quotient singular values for the large example with N = 64. The table shows the number u(k) of converged distinct quotient singular values after k Lanczos iterations, together with the smallest converged value.

The convergence of the approximate quotient singular values reflects this distribution, see Table 1 next page where we list the computed quotient singular values whose absolute accuracy is better than 10 - l~

We see that the first, well separated 7i are captured fast by the algorithm, while more iterations are required to capture the 7{ which are less well separated. We also observe the appearance of multiple (or "spurious") quotient singular values which is a typical phe- nomenon of the Lanczos process in finite precision and which is closely related to the convergence of the approximate 7i, cf. [13, w

Our next example involves a larger problem, N = 64, which is too large to allow computation of the exact quotient singular singular values by means of routine gsvd. Table 2 shows the number of converged distinct quotient singular values after k = 10, 20, 30, 40, and 50 Lanczos iterations, using a loose convergence criterion (10 -3) and identifying spurious values using the test by Cullum & Willoughby [2, w The largest quotient singular value is 71 = 7.66.103, and the table also lists the size of the smallest converged quotient singular value. The numbers u(k) are for N = 64 are not inconsistent with the results for N = 1 6 .


References

[1] /~. BjSrck. Least Squares Methods. In �9 P.G. Ciarlet and J.L. Lions (Eds.), Handbook of Numerical Analysis, Vol. I~ North-Holland, Amsterdam, 1990.

[2] J.K. Cullum and R.A. Willoughby. Lanczos Algorithms for Large Symmetric Eigen- value Computations. Vol.I Theory. Birkh~user, Boston, 1985.

[3] L. Eld~n. A weighted pseudoinverse, generalized singular values, and constrained least squares problems. BIT 22, pp 487-502, 1982.

[4] H.W. Engl. Regularization methods for the stable solution of inverse problems. Surv. Math. Ind. 3, pp 71-143, 1993.

[5] G.H. Golub, F.T. Luk and M.L. Overton. A block Lanczos algorithm for computing the singular values and corresponding singular vectors of a matrix. A CM Trans. Math. Soft. 7, pp 149-169, 1981.

[6] G.H. Golub and C.F. Van Loan. Matrix Computations. 2. Ed. Johns Hopkins University Press, Baltimore, 1989.

[7] C.W. Groetsch. Inverse Problems in the Mathematical Sciences. Vieweg, Wiesbaden, 1993.

[8] M. Hanke. Iterative solution of underdetermined linear systems by transformation to standard form. In : Proceedings Numerical Mathematics in Theory and Practice, Dept. of Mathematics, University of West Bohemia, Plzefi, pp 55-63 (1993)

[9] M. Hanke and P.C. Hansen. Regularization methods for large-scale problems. Surv. Math. Ind. 3, pp 253-315, 1993.

[10] P.C. Hansen. Regularization, GSVD and truncated GSVD. BIT 29, pp 491-504, 1989.

[11] P.C. Hansen. Relations between SVD and GSVD of discrete regularization problems in standard and general form. Lin. Alg. Appl. 141, pp 165-176, 1990.

[12] P.C. Hansen. Regularization Tools: A Matlab package for analysis and solution of discrete ill-posed problems. Numerical Algorithms 6, pp 1-35, 1994.

[13] B.N. Parlett. The Symmetric Eigenvalue Problem. Prentice-Hall, Englewood Cliffs, N.J., 1980.

[14] H. Zha. Computing the Generalized Singular Values/Vectors of Large Sparse or Struc- tured Matrix Pairs. Report CSE-94-022, Dept. of Computer Science and Engineering, Pennsylvania State University, January 1994; submitted to Numer. Math. A condensed version appears in these proceedings.


139

A Q R - L I K E S V D A L G O R I T H M F O R A P R O D U C T / Q U O T I E N T O F S E V E R A L M A T R I C E S

G.H. GOLUB Computer Science Department Stanford University Stanford, CA U.S.A. golub @sccm. stanford, edu

K. SOLNA Computer Science Department Stanford University Stanford, CA U.S.A. solna @sccm. stanford, edu

P. VAN DOOREN Cesame Universitd Catholique de Louvain Louvain-la-Neuve Belgium vandooren @an ma. ucl. ac. be

ABSTRACT. In this paper we derive a new algorithm for constructing unitary decomposition of a sequence of matrices in product or quotient form. The unitary decomposition requires only unitary left and right transformations on the individual matrices and amounts to computing the generalized singular value decomposition of the sequence. The proposed algorithm is related to the classical Golub-Kahan procedure for computing the singular value decomposition of a single matrix in that it constructs a bidiagonal form of the sequence as an intermediate result. When applied to two matrices this new method is an alternative way of computing the quotient and product SVD and is more economical than current methods.

KEYWORDS. Numerical methods, generalized singular values, products of matrices, quotients of matrices.

140 G.H. Golub et al.


The two basic unitary decompositions of a matrix A yielding some spectral information are the Schur form A = UTU* - where U is unitary and T is upper triangular - and the singular value decomposition A = UF~V* - where U and V are unitary and ]E is diagonal - (for the latter A does not need to be square). It is interesting to notice that both these forms are computed by a QR-l ike iteration [4]. The SVD algorithm of Golub-Kahan [3] is indeed an implicit QR-algorithm applied to the Hermitian matrix A*A. When looking at unitary decompositions involving two matrices, say A and B, a similar implicit algorithm was given in [6] and is known as the QZ-algorithm. It computes A = QTaZ* and B = QTbZ*

where Q and Z are unitary and T~ and Tb are upper triangular. This algorithm is in fact the QR-algorithm again performed implicitly on the quotient B - 1 A . The corresponding decomposition is therefore also known as the generalized Schur form.

This is not the case, though, when considering the generalized singular value decomposition of two matrices, appearing as a quotient B - 1 A or a product B A . In this case the currently used algorithm is not of QR type but of a Jacobi type. The reason for this choice is that Jacobi methods extend to products and quotient without too much problems. The bad news is that the Jacobi algorithm typically has a (moderately) higher complexity than the Q R algorithm. Yet, so far, nobody proposed an implicit QR-l ike method for the SVD of a product or quotient of two matrices.

In this paper we show that, in fact, such an implicit algorithm is easy to derive and that it even extends straightforwardly to sequences of products/quotients of several matrices. Moreover, the complexity will be shown to be lower than for the corresponding Jacobi like methods.

2 I M P L I C I T S I N G U L A R V A L U E D E C O M P O S I T I O N

Consider the problem of computing the singular value decomposition of a matrix A that is an expression of the following type :

A = A ~ . . . . . A~ 2. A~ 1, (1)

where si = + l , i.e. a sequence of products of quotients of matrices. For simplicity we assume that the Ai matrices are square n x n and invertible, but as was pointed out in [2], this does not affect the generality of what follows. While it is clear that one has to perform left and right transformations on A to get U * A V = E, these transformations will only affect A K and A1 Yet, one can insert an expression Q*Qi = In in between every pair asi+a A~ i

�9 " * i + 1

in (1). If we also define QK - U and Qo - V, we arrive at the following expression :

U * A V = �9 ,K (f),as2 (f) ,asl ( Q K A K Q K - I ) " �9 Q1). Q0) (2) �9 . . k ~ 2 ~ - ~ 2 \ ~41- ' -~1 �9

With the degrees of freedom present in these K + 1 unitary transformations Qi at hand, one can now choose each expression O*A si - ~ i - i Qi-1 to be upper triangular. Notice that the expression .~,O*.AS.iQi_l._, = T~ ~ with Ti upper triangular can be rewritten as �9

�9 A - - - * Q i _ l A i Q i Ti for si . (3) Qi i Q i - l = Ti forsi 1 , = - 1

A QR-like SVD Algorithm 141

/,From the construct ion of a normal Q R decompostion, it is clear tha t , while making the matr ix A upper t r iangular , this "freezes" only one matr ix Qi per mat r ix Ai. The remaining uni tary mat r ix leaves enough freedom to finally diagonalize the matr iz A as well. Since meanwhile we computed the singular values of (1), it is clear tha t such a result can only be obtained by an iterative procedure. On the other hand, one intermediate form tha t is used in the Golub-Kahan SVD algori thm [3] is the bidiagonalization of A and this can be obtained in a finite recurrence. We show in the next section tha t the matrices Qi in (2) can be constructed in a finite number of steps in order to obtain a bidiagonal Q*KAQo in (2). In carrying out this task one should t ry to do as much as possible implicitly. Moreover, one would like the to ta l complexity of the algori thm to be comparable to - or less than - the cost of K singular value decompositions. This means tha t the complexity should be O ( K n 3) for the whole process.

3 I M P L I C I T B I D I A G O N A L I Z A T I O N

We now derive such an implicit reduction to bidiagonal form. Below ~ / ( i , j ) denotes the group of Householder t ransformat ions having ( i , j ) as the range of rows/columns they operate on. Similarly ~(i , i + 1) denotes the group of Givens t ransformat ions operat ing on rows/columns i and i + 1. We first consider the case where all si = 1. We thus only have a product of matrices Ai and in order to i l lustrate the procedure we show its evolution operat ing on a product of 3 matrices only, i.e. A3A2At. Below is a sequence of "snapshots" of the evolution of the bidiagonal reduction. Each snapshot indicates the pa t t e rn of zeros ( '0 ') and nonzeros ( ' x ' ) i n the three matrices.

First perform a Householder t ransformat ion Q~X) 6 ~ ( 1 , n) on the rows of A1 and the

columns of A2. Choose Q~I)" to annihilate all but one element in the first column of A1 "

X X X X X 2; X 2; 2; X X 2; 2; 2; 2;

2; 2; X 2; 2; 2; X 2; 2; 2; 0 2; 2; 2; 2;

X X 2; 2; 2; 2; 2; 2; 2; 2; 0 2; 2; 2; 2; .

2; 2; 2; 2; 2; 2; 2; 2; X 2; 0 2; 2; 2; 2;

2; 2; 2; 2; 2; 2; 2; 2; X X 0 2; 2; X X

Then perform a Householder t ransformat ion Q~I) 6 T/(1, n) on the rows of A2 and the

columns of A3. Choose Q t21)' to annihilate all but one element in the first column of A2 "

X X 2; X 2; X 2; 2; 2; 2; X 2; 2; X 2;

2; x 2; 2; x 0 2; 2; 2; x 0 x x x x

x x x x x 0 x x x x 0 x x x x .

x x x x 2; 0 2; x x 2; 0 x 2; 2; 2; x x 2; x 2; 0 2; 2; x 2; 0 x 2; 2; 2;

Then perform a Householder t ransformat ion Q0) E ~/(1, n) on the rows of A3. Choose


Q i 1) t o a n n i h i l a t e a l l b u t o n e e l e m e n t in t h e f i r s t c o l u m n o f A3 �9

Z Z Z Z Z Z Z Z Z Z Z Z Z Z Z

0 x x x x 0 x x x z 0 x x x x

0 x x x x 0 x x x z 0 x x x x .

0 x x x x 0 x x x x 0 x x x x

0 x z x z 0 x z x z 0 z z z x

N o t i c e t h a t t h i s t h i r d t r a n s f o r m a t i o n y i e l d s t h e s a m e f o r m a l so f o r t h e p r o d u c t o f t h e t h r e e m a t r i c e s �9

Z Z Z Z Z Z Z X Z Z

0 z z x z 0 z z z x

0 z z z z 0 z z x z

0 x z z x 0 x z x z

0 z z z z 0 z z z z

z z z z z x X x x X

0 z z x z 0 z z z z

0 z z z z = 0 x z z z .

0 x z z z 0 z z x z

0 x x x x 0 z x x x

A t t h i s s t a g e w e a r e i n t e r e s t e d in t h e first row o f t h i s p r o d u c t ( i n d i c a t e d in b o l d f a c e

a b o v e ) . T h i s r o w c a n b e c o n s t r u c t e d as t h e p r o d u c t o f t h e f i r s t r o w o f A3 w i t h t h e m a t r i c e s

t o t h e r i g h t o f i t , a n d t h i s r e q u i r e s o n l y O ( K n 2) f lops . O n c e t h i s r o w is c o n s t r u c t e d w e c a n

f i nd a H o u s e h o l d e r t r a n s f o r m a t i o n Q(1) e 7-/(2, n ) o p e r a t i n g o n t h e l a s t ( n - 1) e l e m e n t s

w h i c h a n n i h i l a t e s a l l b u t t w o e l e m e n t s '

= o o o ] . (4)

T h i s t r a n s f o r m a t i o n is t h e a p p l i e d t o A1 o n l y a n d t h i s c o m p l e t e s t h e f i r s t s t a g e o f t h e b i d i a g o n a l i z a t i o n s ince

x x 0 0 0

0 Z X X X

Q ( ~ ) ' A Q ~ ) = 0 �9 �9 �9 �9

0 x x x x

0 X X X X

N o w p e r f o r m a H o u s e h o l d e r t r a n s f o r m a t i o n Q~2) E 7 / ( 2 , n ) o n t h e r o w s o f A1 a n d t h e

c o l u m n s o f A 2 . C h o o s e Q~2) t o a n n i h i l a t e a l l b u t t w o e l e m e n t s in t h e s e c o n d c o l u m n o f A1

X Z X X X X X X 27 X X X X X X

0 z z x z 0 z z z z 0 z x z x

0 z x x x 0 x z x z 0 0 x x z .

0 z x z z 0 z z z z 0 0 z z z

0 x x z x 0 z x x x 0 0 z x z

T h e n p e r f o r m a H o u s e h o l d e r t r a n s f o r m a t i o n Q~2) e 7~(2, n ) o n t h e r o w s o f A2 a n d t h e

c o l u m n s o f A3 . C h o o s e Q~2) t o a n n i h i l a t e a l l b u t t w o e l e m e n t s in t h e s e c o n d c o l u m n o f A2

X X X X X X X X X X X X X Z X

0 z x x z 0 z x z x 0 x z z x

0 x x x z 0 0 z z x 0 0 x z x .

0 x x x z 0 0 x z z 0 0 z z z

0 x x z z 0 0 x x x 0 0 z z x


Then perform a Householder transformation Q(2) 6 7-/(2, n) on the rows of A3 and choose it to annihilate all but two elements in the second column of A3 "

x 2; z x z 2; x 2; 2~

0 z z x x 0 x x x x

0 0 z z z 0 0 z z z

0 0 x x x 0 0 z x x

0 0 x z z 0 0 x z z

For the product we know that �9

2; X 2; 2; 2; 2; 2; X X 2;

0 2; Z 2; Z 0 Z 2; Z Z

0 0 2; 2; 2; 0 0 2; Z 2;

0 0 2; Z 2; 0 0 Z Z Z

0 0 Z 2; 2; 0 0 Z Z Z

x x x 2; x

0 z x z x

0 0 z z x .

0 0 x z x

0 0 x x z

z z z z z z z 0 0

0 z z z z 0 x x x

0 0 z z z - - 0 0 z x

0 0 z z z 0 0 z z

0 0 z z z 0 0 z z

0

x

x .

x

x

At this stage we are interested in the second row of this product (indicated in boldface above). This row can be constructed as the product of the second row of A3 with the matrices to the right of it, and this again requires only O(K(n-1) 2) flops. Once constructed

we can find a Householder transformation Q(2) 6 ~ ( 3 , n ) operating on the last ( n - 2) elements which annihilates all but two elements :

[o �9 �9 o o ] . (5) This transformation is then applied to A1 only, completing the second step of the bidiagonalization of A :

x x 0 0 0

0 x x 0 0

Q ~ ) ' Q ~ ) ' A Q ( 0 ~ ) Q ~ ) = 0 0 ~ ~ ~

0 0 z z x

0 0 z z x

after n - 1 stages �9

X 2; 2; X X

0 X X X X

0 0 z z z

0 0 0 z x

0 0 0 0 z

It is now clear from the context how to proceed further with this algorithm to obtain

x x x x x x z x x x x z 0 0 0

0 z z x z 0 z z z x 0 z z 0 0

0 0 z z z 0 0 z x x - - 0 0 x x 0 .

0 0 0 z z 0 0 0 x x 0 0 0 z z

0 0 0 0 z 0 0 0 0 z 0 0 0 0 z

Notice that we never construct the whole product A = A3A2A1, but rather compute one

its rows when needed for constructing the transformations Q(00. The only matrices that o f

are kept in memory and updated are the Ai matrices and possibly ~ K and Q0 if we require the singular vectors of A afterwards.

The complexity of this bidiagonalization step is easy to evaluate. Each matrix Ai gets pre and post multiplied with essentially n Householder transformations of decreasing range. For updating all Ai we therefore need 5Kn3/3 flops, and for updating QK and Q0 we need 2n 3 flops. For constructing the required row vectors of A we need ( K - 1)n3/3 flops. Overall


we thus need 2 K n 3 flops for the construction of the triangular Ti and 2n 3 for the outer transformations Q K and Qo. Essentially this is 2n 3 flops per updated matrix.

If we now have some of the si = - 1 we can not use Householder transformations anymore. Indeed in order to construct the rows of A when needed, the matrices A~ "1 have to be trangularized first, say with a QR factorization. The Q R factorization is performed in an initial step. From there on the same procedure is followed, but using Givens rotations instead of Householder transformations�9 The use of Givens rotations allow us to update the triangularized matrices A~ "1 while keeping them upper triangular�9 Each time a Givens rotation detroys this triangular form, another Givens rotation is applied to the other side of that matrix in order to restore its triangular form. The same technique is e.g. used in keeping the B matrix upper triangular in the Q Z algorithm applied to B -1A. The bookkeeping of this algorithm is a little more involved and so are the operation counts, which is why we do not develop this here. One shows that when there are inverses involved, the complexity of the bidiagonalization step amounts to less than 4n 3 flops per updated

matrix�9

4 C O M P U T I N G T H E S I N G U L A R V A L U E S

The use of Householder and Givens transformations for all operations in the bidiagonalization step guarantees that the obtained matrices 7"/in fact correspond to slightly perturbed data as follows :

* - * A Ti = Qi(Ai + ,SAi)Qi-1, si 1, Tj = Qj - I ( j + ~Aj)Qj, sj = -1 , (6)

where

II~Ai[I <_ ecnllAi[I , IIQ~Qi - I,~11 <- ed~, (7)

with e the machine precision and cn, dn moderate constants depending on the problem size n. This is obvious since each element transformed to zero can indeed be put equal to zero without affecting the e bound (see [7], [4]).

Things are different with the elements of A since they are not stored in the computer. How does one proceed further to compute the generalized singular values of A ? Once the triangular matrices Ti axe obtained, it is easy and cheap to reconstruct the bidiagonal :

T ~ ' . . . ' T ~ ~ ' T ; ~ = B =

ql s 0 1 , 3 �9 �9 �9

q2 e3 �9

0 1 , n

O n _ 2 ) n �9

e n

qn

(8)

The diagonal elements qi axe indeed just a product of the corresponding diagonal elements of the Tj matrices, possibly inverted �9

8 K �9 82 81 qi = t K i i . . . " t 2 i i �9 t l i i ,


and the off diagonal elements ei can be computed from the corresponding 2 x 2 diagonal blocks (with index i - 1 and i) of the Tj matrices. It is clear that the qi can be computed in a backward stable way : all errors can e.g. be superposed on the diagonal elements of the first matrix T1. For the errors performed when computing then ei one needs a little more analysis, but it can be shown that the backward errors can be superposed on the corresponding 2 x 2 blocks of Tj, without creating any conflicts or error build up. In other words, the computed bidiagonal corresponds exactly to the bidiagonal of the product of slightly perturbed triangular matrices Tj, who in tu rnsa t i s fy the bounds (6,7). Unfortunately, nothing of the kind can be quaranteed for the elements oid in (8), who are supposed to be zero in exact arithmetic. The best bound we can obtain from the construction of the rows (4,5) is that :

Ioi,jl <- ec,~l]Tg(i, :)11" ]ITK-11] " . . . " 117"111,

which is a much weaker bound than asking the off diagonal elements of A to be e smaller that the ones on the bidiagonal. This would be the case e.g. if instead we had :

Ioi,j] <_ ecnllTg(i, :)TK-~ . . .Till.

Yet, this is the kind of result one would hope for if some singular values of A are small and still have to be computed to a high relative accuracy. These two bounds can in fact be very different when significant cancellations occur between the individual matrices, e.g. if

IIAII << I ] A ~ I I ' . . . " IIA~ 211" IIA~ 111.

One way to test the performance of this algorithm in cases with very small singular values is to generate powers of a symmetric matrix A = S K. The singular values will be the powers of the absolute values of the eigenvalues of S :

ai (A) = I)q(S)l K,

and hence will have a large dynamic range. The same should be true for the bidiagonal of A and the size of the oi,j will then become critical for the accuracy of the singular values when computed from the bidiagonal elements qi, ei. We ran several tests with matrices 5' of which we know the exact eigenvalues and observed a very high relative accuracy even for the smallest singular values. The only explanation we can give for this is that as the bidiagonalization proceeds, it progressively finds the largest singular values first and creates submatrices that are of smaller norm. These then do not really have cancellation between them, but instead the decreasing size of the bidiagonal elements is the result of decreasing elements in each transformed matrix Ai. In other words, a grading is created in each of the transformed matrices. We believe this could be explained by the fact that the bidiagonalization is a Lanczos procedure and that such grading is often observed there when the matrix has a large dynamic range of eigenvalues.

The consequence of all this is that the singular values of such sequences can be computed (or better, "estimated") at high relative accuracy from the bidiagonal only ! Notice that the bidiagonalization requires 2 K n 3 flops but that the subsequent SVD of the bidiagonal is essentially free since it is 0(n2).


5 S I N G U L A R V E C T O R S A N D I T E R A T I V E R E F I N E M E N T

If one wants the singular vectors as well as the singular values at a guaranteed accuracy, one can start from the bidiagonal B as follows. First compute the bidiagonal :

B = Q*K A Q o = T~:K . . . . . T~2 . T~ 1,

and then the SVD of B :

B = U E V * ,

where we choose the diagonal elements of l] to be ordered in decreasing order. We then proceed by propagating the transformation U (or V) and updating each Ti so that they remain upper triangular. Since the neglected elements oi,j were small, the new form :

, , $ ^ =

will be upper triangular, and nearly diagonal. This is the ideal situation to apply one sweep of Kogbetliantz's algorithm. Since this algorithm is qudratically convergent when the diagonal is ordered [5], one sweep should be enough to obtain e-small off diagonal dements.

The complexity of this procedure is as follows. If we use only Givens transformations we can keep all matrices upper triangular by a subsequent Givens correction. Such a pair takes 4n flops per matrix and we need to propagate n 2 / 2 of those. That means 2n 3 per matrix. The cost of one Kogbetliantz sweep is exactly the same since we propagate the same amount of Givens rotations. We arrive thus at the following total count for our algorithm :

2 K n 3 for triangularizing Ai ~ Ti

2n 3 for constructing Q K and Q0

8n 3 for computing U and V

2 K n 3 for updating Ti ---* ~'i

2 K n 3 for one last Kogbetliantz sweep.

The total amount of flops after the bidiagonalization is thus comparable to 2 Kogbetliantz sweeps, whereas the Jacobi like methods typically require 5 to 10 of those sweeps ! More- over, this method allows to select a few singular values and only compute the corresponding singular vectors. The matrices Q K and Q0 can e.g. be stored in factored form and inverse iteration can be performed on B to find its selected singular vector pairs, and then transformed back to pairs of A using Q g and Q0.

6 C O N C L U D I N G R E M A R K S

The algorithm presented in this paper nicely complements the unitary decompositions for sequences of matrices defined for the generalized QR [2] and Schur decompositions [1]. These decompositions find applications in sequences of matrices defined from discretizations of ordinary differential equations occurring e.g. in two point boundary value problems [8] or control problems [1]. We expect they will lead to powerful tools for analyzing as well as


solving problems in these application areas.

Acknowledgements

G. Golub was partially supported by the National Science Foundation under Grants DMS- 9105192 and DMS-9403899. K. Solna was partially supported by the Norwegian Council for Research. P. Van Dooren was partially supported by the National Science Foundation under Grant CCR-9209349.

References

[1] A. Bojanczyk, G. Golub and P. Van Dooren, The periodic Schur form. Algorithms and Applications, Proceedings SPIE Conference, pp. 31-42, San Diego, July 1992.

[2] B. De Moor and P. Van Dooren, Generalizations of the singular value and QR decomposition, SIAM Matt. Anal. ~ Applic. 13, pp 993-1014, 1992.

[3] G. Golub and V. Kahan, Calculating the singular values and pseudo-inverse of a matrix. SIAM Numer. Anal. 2, pp 205-224, 1965.

[4] G. Golub and C. Van Loan, Matrix Computations 2nd edition, The Johns Hopkins University Press, Baltimore, Maryland, 1989.

[5] J.P. Charlier and P. Van Dooren, On Kogbetliantz's SVD algorithm in the presence of clusters. Linear Algebra ~ Applications 95, pp 135-160, 1987.

[6] C. Moler and G. Stewart, An algorithm for the generalized matrix eigenvalue problem, SIAM Numer. Anal. 10, pp 241-256, 1973.

[7] J. H. Wilkinson, The algebraic eigenvalue problem, Clarendon press, Oxford, 1965.

[8] R. Mattheij and S. Wright, Parallel stable compactification for ODE with parameters and multipoint conditions, Int. Rept. Argonne Nat. Lab., IL, 1994.



149

, A P P R O X I M A T I N G T H E P S V D A N D Q S V D

S. QIAO Department of Computer Science and Systems McMaster University Hamilton, Ontario Canada LSS ~K1 qiao@maccs, dcss. mcmaster, ca

ABSTRACT. This paper presents an adaptive method for approximating the SVD of the product AB or the quotient AB -1 of two matrices A and B. Specifically, it computes an approximation of the complete orthogonal decomposition of AB or AB -1. The algorithm is an extension of the ULV decomposition algorithm.

KEYWORDS. PSVD, QSVD, complete orthogonal decomposition, ULV decomposition.


In this paper we consider the problem of the SVD of the product AB (PSVD) and the quotient AB -1 (QSVD) of two matrices A and B. We present an adaptive method for approximating the PSVD or the QSVD. Specifically, this method computes an approximation of the complete orthogonal decomposition [1] of the product AB or the quotient AB -1 without explicit matrix multiplication or inversion. The algorithm is a generalization of the ULV decomposition algorithm [2]. For example, it computes the decomposition

A B = U ( G E ) v H 0 F (1)

where U and V are orthonormal, G and F are upper triangular, and I[E[[~ + [IF[[~ is approximately the sum of the squares of the "small" singular values of AB.

This algorithm can be generalized to any finite number of matrices.

This paper is organized as follows. In section 2, we present a generalization of the estimator in [3]. It gives an estimated right singular vector associated with the smallest singular value of AB or AB -1. In sections 3 and 4, we extend the deflation and refinement

150 S. Qiao

schemes in [2] to the cases of A B and A B -1 . Finally, in section 5, we describe the updating procedures.

In the following sections, we use 2-norm for all vector and matrix norms and adopt MATLAB notations, e.g., Ai.j,k:l denotes the submatrix of A consisting the entries in rows from i to j and columns from k to I.

2 T H E E S T I M A T O R

To begin with, we describe the estimator for the right singular vector corresponding to the smallest singular value of the product A B . Without the loss of generality, we assume both A and B are upper triangular and of order n. The estimator consists of two stages: first, it finds a vector d of unit norm such that the norm of the solution z to the system

B H A H x = d

is large; second, it solves for v in A B v = x. Then v is an approximation of the right singular vector associated with the smallest singular value of A B and iixJi/ilvJJ is an approximation of the smallest singular value of A B . We only describe the first stage for the second one is straightforward.

The procedure of finding d is incremental in that d is determined one element at a time. Let A and/} be the order k - 1 leading principal submatrices of A and B respectively, then the kth order leading principal submatrices are:

0 Ak,k 0 Bk,k

where a = Al .k - l , k and b = Bx:k-l,k. Suppose that we have found a ( k - 1)-vector d of unit norm so that the norm of the solution ~ to the system/~H.~iH~ = d is large. Our goal is to increment d to a k-vector of unit norm and, at the same time, make the norm of the solution x to B ( k ) H A ( k ) H x = d large. Because of the unit norm of d, we can assume that

d = ( s d ~ where J c j 2 + l s J 2 = l . (3) \ ] s

Let ~ be the solution to the triangular system/~H~ = d, then, from (2) and (3), the solution y to B ( k ) H y = d is given by

where c* denotes the complex conjugate of a complex number c and

def H ~ ' - - b y.

Similarly, if ~ is the solution to the triangular system ~ H ~ = ~, then the solution to A ( k ) H x = y is

Approximating the PSVD and QSVD 151

where def = a H f c .

In solving a tr iangular system, we often keep a partial sum vector, for example, in this case

/~ de__f Al:k_l,k:nHfc.

Consequently, the first component of t3 is ~ and we have the parti t ion

~ =

where ~ = Al:k_l,k+l:n H fc.

Having solved for x, we can update the partial sum vector using

def H X , . P - - A l : k , k + l : n - - s p q- ~ A k k + l : n H

Since the partial sum p will affect the solutions in the subsequent steps, we choose c and s so that the norm

I1(;)[I is maximized.

After some algebraic manipulation, we obtain

x - W 1 8 8

where ( " " ) Wl = (A;,kB;,k) -~ 0 Bk,kAk,kz 1 -7r-B~,k,5 '

and

W2 = (A~:,kB;,k)-l(Ak,k+l:,~ H A~c,kB~,k~-(Tr + B;,k6)Ak,k+l:nH).

Thus (c, s) T is the normalized right singular vector associated with the larger singular value of the n • 2 matr ix

W = ( wl

Now we derive the est imator for the right singular vector corresponding to the smallest singular value of the quotient A B -1 where, again, both A and B are upper triangular. Similarly, the crucial step is to incremently find a vector d of unit norm such that the norm of the solution z to the system

B - H A H x - d

is large�9

Using the partit ions in (2) and (3), let ~)= BHd, then

Y = sb H d + cB~, k

152 S. Qiao

The solution x and its corresponding partial sum p are

and

p -- s~ n t- (sbHd n t- cB~, k - s~)Ak,k+l:n H/A*k,k

recalling that

i5 = w h e r e ~ = a g~, .

Denote

we have

Ak'k) B* bHd k,k

and

W 2 - �9 -1 �9 H �9 - ( A k , k ) ( B k , k A k , k + l ' n A k , k P n t- ( b g d - ~ ) A k , k + l : n g ) .

Thus we choose c and s so that (c ,s ) T is the normalized right singular vector associated with the larger singular value of the matr ix

w2 "

3 T H E D E F L A T I O N

Suppose v is the approximation of the singular vector computed by the est imator described in the previous section. To deflate A B , we first find a sequence of row rotations transforming v into the n th unit vector en = (0, . . . ,0,1) T. We then apply the rotations to the columns of B and, at the same time, restore the triangular structure of B by applying a sequence of rotations to the rows of B. These row rotations are propagated to A in order to maintain the product A B . Finally, we apply another sequence of row rotations to restore the tr iangular structure of A. We can show that

A B = U R V H

where both U and V are or thonormal and R is nearly block diagonal:

where Ilrll 2 + lal 2 approximates the square of the smallest singular value of A B .

If we denote QH as the sequence of rotations applied to the rows of B, then we can write

A = URAQ H and B = Q R B V H (5)

where RA and RB axe resultant triangular matrices and R A R B = R.


The deflation of the quotient A B -1 can be obtained by noticing that B - 1 V = (V HB) -1. Specifically, assuming v is the est imated right singular vector associated with the smallest singular value of A B -1. we first apply the sequence of rotations which transforms v into the n th unit vector to the rows of B and simultaneously restore the tr iangular s tructure of B by column rotations. Then, as in the case of the product AB, we propagate these rotations to A and restore the triangular structure of A.

Similarly, if we denote Q as the sequence of rotations applied to the columns of B (or the columns of A), then we have

A = URAQ H and B = V R B Q H (6)

where RARB 1 = R.

The deflation procedure can be repeated until the decomposition (1) is achieved depending on a tolerance for the numerical rank.

If v is the exact singular vector associated with the smallest singular value amin of, say AB, then

[O'min[ 2-- [[ABv[[ 2 = []r[[ 2 + [a[ 2

which implies that

T Re,., it On the other hand, since R and A B have the same singular values and a = e n , follows from the minimax theorem for singular values that

This shows tha t lal = lamJnl and r = 0, when v is the exact singular vector associated with the smallest singular value. Thus an accurate estimate of the singular vector can bring R close to block diagonal. To improve the accuracy of the singular vector approximation v computed by the est imator described in the previous section, we can apply the inverse power method using the estimate v as an initial vector. Alternatively, we can bring R arbitrary close to block diagonal and a arbitrary close to a ~ n using a refinement scheme presented in the following section.

4 T H E R E F I N E M E N T

Let C = AB, we first find a rotation on the ( n - 1, n)-plane to eliminate Cn-l ,n using Cn-l ,n-1. This rotat ion is determined by the five elements in A and B: A,~-1,,~-1, A,~-I,,~, An,n, Bn-l,r,, and Bn,n, since Cn-l ,n = An- l ,n - lBn- l ,n + An-l,nBn,n and Cn-l,n-1 = An- l ,n - lBn- l ,n -1 . Then, after applying this rotation to the n th and the n - 1st columns of B, we annihilate the ( n - 1, n)-entry of B using a row rotat ion on the ( n - 1, n)-plane.

154 S. Qiao

The following figure illustrates the procedure:

X X X X X X X X X X X X

B : X X X ~ X X X ~ X X X

X X ~ X | X

X ~ X X X X

The above row rotation must be propagated to A in order to maintain the product A B as shown in the following figure:

x x x x x x x x

A: X X X ~ X X X

X X X X

X X X

Now that both the resultant product A B and the resultant B have the structure:

x x x x

x x x (7) x

x x

it can be verified that the resultant A must have the same structure. This procedure can be repeated until A and B and their product have the following structure:

x x x

x x

x

x x x x

To restore the triangular structure, we start with applying a sequence of row rotations to triangularize B. These rotations are propagated to A to maintain the product A B . Finally we apply another sequence of left rotations to A to restore its triangular structure.

We can prove that if we iterate the above procedure, the off diagonal vector r in (4) converges to zero and a to the smallest singular value of A B .

Similarly, in the case of the quotient A B -1 , let C = A B -1 , then the rotation eliminating C,~-l,n is determined by C,~-l,n = -A,~-I , ,~-IB,~-I , ,~/(B,~-I , ,~-IBn, ,~)-t- A,~-l,n/Bn,,~ and Cn-l ,r , -1 = A , ~ - l , n - 1 / B n - x , n - 1 . Then we apply this rotation to the rows of B and annihilate the ( n - 1, n)-entry of B using a column rotation. Note that if B has the structure (7), then so does B -1. The rest of the refinement procedure for the quotient A B -1 is similar to that for the product A B .

Note that all the rotations applied to the columns of A can be accumulated in the matrix Q in (5) or (6).


5 U P D A T I N G

In updating the decomposition, we assume the form (5) or (6). In other words, the matrix Q is the product of all the rotations applied to the columns of A.

5.1 ADDING A ROW TO A

When a new row a T is added to A, updating the decomposition can be achieved by writing

A 0 0

aTQ ) QH.

Then RA is updated by a sequence of rotations eliminating a TQ. The rotations are propagated to the left and absorbed by U.

5.2 ADDING A ROW OR COLUMN TO B

When a row is added to B in the case of the quotient AB -1, the updating is symmetric to the procedure described above. In the case of the product AB, when a column b is added to B, we write

(B b) = Q(RB QHb) ( V H O) 0 1 "

Then RB is updated by a sequence of column rotations which is absorbed by V.

After updating, we need to apply deflation and possible refinement since the numerical rank may change.

A ck nowled g e m e nt s

This work is supported in part by an NSERC grant OGP0046301. The author is grateful to C.T. Pan for valuable discussions.

R e f e r e n c e s

[1] G.H. Golub and C.F. Van Loan. Matrix computations. Second Edition, Johns Hopkins Press, 1989.

[2] G.W. Stewart. Updating a rank-revealing ULV decomposition. SIAM J. Matrix Anal. Appl. 46, pp 479-492, 1993.

[3] C. Van Loan. On estimating the condition of eigenvalues and eigenvectors. Linear Algebra Appl. 88/89 , pp 715-732, 1987.



157

I B O U N D S O N S I N G U L A R V A L U E S R E V E A L E D B Y Q R F A C T O R I Z A T I O N S

C.-T. PAN Department of Mathematical Sciences Northern Illinois University De Kalb, IL 60115-2888 U.S.A. [email protected]

P.T.P. TANG Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Ave. Argonne, IL 60439-4801 U.S.A. [email protected]

ABSTRACT. We introduce a pair of dual concepts: pivoted blocks and reverse pivoted blocks. These blocks are the outcome of a special column pivoting strategy in Q R factorization. Our main result is that under such a column pivoting strategy, the Q R factorization of a given matrix can give tight estimates on any two a priori-chosen consecutive singular values of that matrix. In particular, a rank-revealing Q R factorization is guaranteed when the two chosen consecutive singular values straddle a gap in the singular value spectrum that gives rise to the rank degeneracy of the given matrix. The pivoting strategy, called cyclic pivoting, can be viewed as a generalization of Golub's column pivoting and Stew- art's reverse column pivoting. Numerical experiments confirm the tight estimates that our theory asserts.

KEYWORDS. Singular value decomposition, rank-revealing QR factorization, cyclic column pivoting.

1 INTRODUCTION

Recently, Hong and Pan [5] showed that for any matrix A E ]R mxn, m _> n and given k,

158 C. - T. Pan and P. T. P. Tang

1 < k < n, there exists a column permutation 1I such that

and

k n - k

AII = QR = Q ( Rll R12 ) 0 R22 '

an~n(Rll) ~ ak(A), O'max(R22) ~ ak+l(A), where Q E p mxn, QTQ = I, R E pj~x,~ is upper triangular, and

(i)

(2)

al(A) >_ a2(A) _>... >_ an(A)

are singular values of A. In particular, if k is the numerical rank of A, i.e., at(A) ::~ ar+l(A) ~ 0. ( [4, 5]), the factorization is rank revealing ([1]). Unfortunately, since the choice of the permutation II in (1) relies upon partially on information of the SVD of A, the proof does not render a practical algorithm to substitute the SVD in determining rank.

In this paper, we give an algorithm which identifies the permutation II without using any information on the SVD.

We introduce a pair of dual concepts: pivoted blocks and reverse pivoted blocks. They are the natural generalization of what we called pivoted magnitudes and reverse pivoted magnitudes, which are the results of the well-known Golub's column pivoting [3] and a less well-known column pivoting strategy proposed by Stewart [7]. The main result is that a pivoted block Rll (or equivalently a reverse pivoted block R22) ensures (2).

This result allows us to devise a column pivoting strategy, which we call cyclic pivoting, that produces the pivoted and reverse pivoted blocks.

In particular, if k is the numerical rank of A, the cyclic pivoting strategy guarantees a rank-revealing QR factorization.

It should be pointed out that our pivoting strategy is similar to that of Hybrid-III in [2], without using the SVD, but the two methods also are fundamentally different.

The rest of the paper is organized as follows. Section 2 establishes bounds on singular values derived from the properties of pivoted and reverse pivoted magnitudes. Section 3 presents our main theorems on pivoted blocks and reverse pivoted blocks. Section 4 presents the cyclic column pivoting algorithm used to find the pivoted blocks. Section 5 presents some numerical experiments used to confirm the theoretical results.

2 P I V O T E D M A G N I T U D E S

Given a matrix A, there are two well-known pivoting strategies for Q R factorization that produce tight bounds on amax(A) and anon(A) from particular entries in R. Those particular entries of R are important to our subsequent discussion. We make two definitions. Given an m-by-n matrix A, m _ n, let IIi,j be the permutation such that AIIi,j interchanges columns i and j of A. The pivoted magnitude of A, 7/(A), is defined to be the maximum

Singular Values Revealed by QR Factorizations 159

magnitude of the (1, 1) entry of the R factors of AIIl,t, 1 = 1 ,2 , . . . , n, thus:

~(A) aef = m a x I r n l ' A I I l , t = Q

r l l r 1 2 �9 . �9 t i n

r 2 2 �9 �9 �9 r2n r~ . , l = l , 2 , . . . , n . 13

i

r n n

Algorithmically, one can think of applying QR factorization with column pivoting [3] to A. Then, the magnitude of r n of the resulting R factor is ~?(A). Clearly,

~?(A) = maxllAejll2, j = 1,2, . . . ,n ,

where ej is the standard basis vector in ~'~.

We now define the reverse pivoted magnitude of A, r(A), to be the minimum magnitude of the (n,n) entry of the R factors of AIIt,,~, 1 = 1 , 2 , . . . , n , thus:

r l l r 1 2 �9 . . r l n

r 2 2 � 9 r 2 n

r(A)d=dmin I r n n l ' A I I t ' n = Q O ".. "

rr~r~

If A is nonsingular, we also have

, I = 1 , 2 , . . . , n .

~(A) = ~ / m ~ ll~}"A-~ll~, 3

as shown in [7], where Stewart calls a related column pivoting strategy the reverse pivoting strategy.

The following lemma is not new. The result for r (A) i s proved in [7] and [1]. The result for r/(A) is rather straightforward. We therefore only state the results"

L e m m a 1 Let A be an m-by-n matrix, m >_ n. Then,

r/(A) < amax(A) _< V~r/(A), (3)

and

(1/~/-Qr(A) < amin(A)_ r(A). (4)

Now we consider the Q R factorization

k n - k

A = Q R = Q ( R n R 1 2 ) 0 R22 "

Two related submatrices are important in subsequent discussions: /~11, the (k + 1)-by-(k + 1) leading principal submatrix of R, and R22, the ( n - k + 1)-by-(n- k + 1) trailing principal submatrix of R. The next two lemmas facilitate the discussions to follow.

160 C. - T. Pan and P. T P. Tang

Lemma 2 I f T(Rll) ~ f ~(/~22) for 80me 0 < f ~ 1, then

amin(Rll) <_ ak(A) <_ ( q k ( n - k + 1) / f ) . amin(R11) (5)

and

(1/V~). r(R11) _ ak(A) <_ (qk(n - k + 1) / f ) . r(R11). (6)

Proof. From the interlacing properties of singular values, we have amin(R~) _< ak(A) and amax(/~22) ~ ak(A). Furthermore,

~n~n(Rl~) > ~(R~1)/4~ ~> f ?'}(-R22) / V/~

> f ~n~x(k22)/~/k(~- k + 1)

>_ f ak(A)/qk(n- k + 1). Thus (5)is proved. From (5) and Lemma 1, (6) follows easily. [[

Lemma 3 I f T(A~ll ) ~ f Tl(R22) for 80me 0 <~ f ~ 1, then

( f / q ( n - k)(k + 1)). amax(R22) _< ak+l(A) <_ amax(R22), (7)

and

( f / q ( k + 1)(n - k)). r/(R22) _ ak+l (A) <_ ~/n - k . r/(R22). (8)

Proof. From the interlacing properties of singular values, we have ak+l(A) <_ amax(R22) and amin(/~ll) _< ak+l(A). Furthermore,

~n~x(R2~) < 4~ - k ~(R~)

< (~/f)~/(~- k),(kl~)

_< ( l / f ) 7 ( n - k)(k + 1)a~n(/~ll)

_< ( l / f ) x / ( ~ - k)(k + 1)~k+~(A). Thus (7)is proved. From (7) and Lemma 1, (8) follows easily. |

Roughly speaking, Lemma 2 says that if T(.Rll) ~ fT](/~22) with f not too small, then

r(Rll) ~ amin(Rll) ~ ak(A); (9)

and Lemma 3 says that if r(/~11) >_ fr/(R22) with f not too small, then

~(R22) ~ amax(R22) ~ ak+l(A). (10)

Same results (without f) can be found in [2], where the algorithm "Hybrid-III" is based on Lemmas 2 and 3.


3 P I V O T E D B L O C K S

Let k n - k

( R l l R12 ) A = QR = Q 0 R22 "

Our main result is that if Rll is a pivoted block, or equivalently, if R22 is a reverse pivoted block (both to be defined momentarily), then (9) and (10) hold simultaneously.

By a pivoted block, we mean the following. Consider the column permutations IIl,k, l = 1 , 2 , . . . , k , and

k o(0

AIIt k = Q(t)R(t) = Q(l) ~11 ' 0

r(l) r(0 .. r(0 n - - k 11 12 " i n

~(t) ~ r(t) .. r(l) "~12 ) = O(t) 22 " 2~ p(t) " " 2 2 0 " " �9 "

kkl r/(/~ for l = 1, 2 , . . . , k,

then we call Rll a pivoted block. Note that, although Ri~)= R~ 21 = �9 .. = R~k2 ), the .̀ `̀ 22 fi(')

are different in general. Therefore, by itself, the condition r(R~{ )) = ~(k~J2 )) for some j, 1 _ j _ k, cannot guarantee a pivoted block Rll .

We note here that when k = 1 the pivoted block is r (z) and -r/~ (l) 1,1 'lk 22 = ~(A). Thus one can view pivoted block as a generalization of the concept of pivoted magnitude defined in Section 3.

We also consider pivoted blocks relaxed by a factor f (we call them f-pivoted blocks), 0 < f _ 1, where the equalities above are replaced by the inequalities

rkkl _> f for 1 = 1 , 2 , . . . , k .

Among other benefits, the f-factor provides us with flexibility in algorithm implementation without sacrificing theoretic rigor.

T h e o r e m 1 Let k

Rll A = Q R = Q 0

If Rll is a pivoted block, then

n - k R12 ) R22 "

drain(R11) _< ak(A) <_ ~ / k (n - k + 1) amin(Rll),

and ~ / ( n - k)(k + 1)amax(R22) _< ak+l(A) <_ amax(R22).

162 C. - T. Pan and P. T. P. Tang

Moreover, ak(A) and ak+l(A) can be estimated by the quantities r(R11) and y(R22):

<_ <_ +

and + k)) _< _< k

In other words, the conclusions (5), (6), (7) and (8) of Lemmas 2 and 3 are all satisfied with f = 1. More generally, if Rlx is an .f-pivoted block, 0 < .f < 1, then (5), (6), (7) and (8) are all satisfied with the relaxation factor f .

In particular, if A has numerical rank k, we have a rank-revealing QR factorization with

O'min(.~ll ) ~> ( f / ~ / k ( n - k + 1))ak(A) and O'max(R22 ) <: (~ / (n - k)(k + 1) / f )ak+l (A) .

(See [6] for a proof).

Now we define what we call a reverse pivoted block. Consider the column permutations IIl,k+l, l = k + 1, k + 2 , . . . , n , and

r(0 r(0 .. r(0 k n - k 11 12 �9 1~

AIIt,k+I = Q (')R(') Q(') ( R(t) •(0 ) r (0 .. r (') __ I I " "12 (I) 22 " 2n

0 R (0 = Q 22 O " ' . "

i k+l,k+l[ for l = k + 1, k + 2 , . . . , n ,

we call R22 a reverse pivoted block. Note that when k = n - 1 the reverse pivoted block is r (0n,n and v ( / ~ ) = r(A). Thus one can also view reverse pivoted blocks as a generalization of the concept of reverse pivoted magnitude defined in last section.

We also consider reverse pivoted blocks relaxed by a factor f (we call them reverse f- pivoted blocks), 0 < f _ 1, where the equalities above are replaced by the inequalities

r r~ (0 ) > f It(0 ~- '̀11 - k+l,k+l[ for l = k + 1, k + 2 , . . . , n.

It turns out that pivoted blocks and reverse pivoted blocks are dual concepts.

T h e o r e m 2 Let k n - k

( R l l R12 ) A = QR = Q 0 R22 "

Rll is a pivoted block if and only if R22 is a reverse pivoted block. More generally, Rl l is an f-pivoted block if only if R22 is an reverse f-pivoted block, where 0 < f < 1.

(See [6] for a proof).


4 A L G O R I T H M

In the previous section, the main properties of pivoted and reverse pivoted blocks are discussed. Do these pivoted blocks always exist? The following algorithm answers this question positively. We use the notation 7Z(M) to denote the R factor of a matrix M.

A l g o r i t h m 1 Cyclic P ivo t ing . Given k, 1 <_ k < n, and a threshold f , 0 < f <_ 1, this algorithm produces a column permutation II such that AII = QR and Rl l is a f-pivoted block. As a result,

~=n(R~l) < ~k(A) < (X/k(~- k + 1)/ /)~=n(R~),

and ( f /~/(n - k)(k + 1)) amax(R22) _< ak+l(A) <_ amax(R22).

Step 0. Initialization: R := ~(AII) with (Golub) column pivoting, where II is the column permutation. Set i := k - 1.

Step 1. Iteration; cyclic pivoting

Step 1.1. If i = 0, exit algorithm. Step 1.2. Set R := 7~(R. Hi,k); II := II. IIi,k. Step 1.3. If ]rkkl >_ f ~7(//22), then set i := i - 1. Otherwise, perform exchange as follows: Find an l, k + 1 _< l _< n, such that

?](-i~22) ---- I I[rk , l , r k + l , l , . . . , r n , l ] T l l 2

Set R "= TC(RIIkj), II := 1I. IIk,l, and i := k - 1. Step 1.4. Go back to Step 1.1.

The iteration will terminate because whenever an exchange takes place in Step 1.3, the value I det(Rll)l strictly increases by a factor of at least 1/ f . Therefore this exchange can happen only a finite number of times. Then, at most k - 1 iterations can take place after the final exchange. Clearly, at termination, Rll is an f-pivoted block.

Algorithm 1 produces an f-pivoted block Rll . Likewise we can also propose an algorithm to make R22 a reverse f-pivoted block. For this kind algorithms, see [6] Algorithm 2 and 3.

5 N U M E R I C A L E X A M P L E S

In this section we present our preliminary numerical results illustrating the theory and algorithms proposed. All computations were done on a SUN SPARC IPC, using MATLAB 3.5i.

We only present results of Algorithm 1 here as the results from the other algorithms are similar. Our first experiment tests the ability of the algorithms proposed in detecting the gaps between two consecutive singular values. We applied Algorithm 1 with f = .99 to the following example.

164 C. - T Pan and P. T P. Tang

Table 1: Gap Detection Using Algorithm 1

k a �9 c~(k) n n n k ~ l l )

1 80.0453 2 7.5904 3 4.3518 4 1.3744 5 0.4588 6 0.1016 7 0.0763 8 0.0450 9 0.0097

10 *

ak(A)

100.0000 10.0000 8.0000 4.0000 1.0000 0.2000 0.1000 0.0500 0.0100 0.0001

am [ D(k-1) aXk-'~22 )

10.3392 8.5638 5.0602 2.1754 0.2887 0.2517 0.0804 0.0137 0.0001

E x a m p l e 1 Let A E ]R 12x10 be

A = / / 1 2 0 t t l o ,

where H,~ = I - L--eeT ,

n

with e T = (1,1, ..., 1)., and ]E = diag(100, 10, 8, 4,1, .2, .1, .05, .01, .0001).

The numerical results are summarized in Table 1.

In Table 1, the superscript of R!~ ) indicates the corresponding Rll E pJxt. From the results we know that the gaps (100, 10), (1, .2), (0.05, 0.01), and (0.01, 0.0001) are well detected.

(11)


The authors wish to thank Pete Stewart for his encouragement in this work.

References

[1] T. F. Chan, Rank revealing QR factorizations, Linear Algebra and Its Applications, 88/89:67-82, 1987.

[2] S. Chandrasekaran and I. Ipsen, On rank-revealing QR factorizations, Research Report YALEU/DCS/RR-880, 1991.

[3] G. H. Golub, Numerical methods for solving least squares problems, Numerische Math- ematik 7:206-216, 1965.

[4] G. H. Golub, V. Klema, and G. W. Stewart, Rank degeneracy and least squares problems, Report TR-456, Dept. of Computer Science, University of Maryland, 1976.


[5] Y. P. Hong and C.-T. Pan, Rank-revealing QR factorizations and the singular value decomposition, Mathematics of Computation 58:213-232, 1992.

[6] P. T. P. Tang and C.-T. Pan, Bounds on singular values revealed by QR factorizations, Tech. Report, MCS Argonne National Lab., MCS-P332-1092.

[7] G. W. Stewart, Rank degeneracy, SIAM Journal on Scientific and Statistical Comput- i'ng, 5:403-413, 1984.

[8] G. W. Stewart, Determining rank in the presence of error, Tech. Report TR-92-I08, Dept. of Computer Science, University of Maryland, 1993.



167

A S T A B L E A L G O R I T H M F O R D O W N D A T I N G T H E U L V D E C O M P O S I T I O N

J.L. BARLOW, H. ZHA Department of Computer Science and Engineering The Pennsylvania State University University Park, PA 16802-6106 U.S.A. [email protected], edu, zha @cse.psu. edu

P.A. YOON Department of Computer Science and Engineering and Applied Research Laboratory The Pennsylvania State University University Park, PA 16802 U.S.A. [email protected], edu

ABSTRACT. A common alternative to performing the singular value decomposition is to

fact~176 u ( C ) VTwhereUandVare~176176

is a lower triangular triangular matrix which indicates a separation between two subspaces by the size of its columns. These subspaces are denoted by V = (V1,1/2) where the columns of C are partitioned conformally into C = (C1,C2) with II c T IIF~ • ' Here e is some tolerance. If the matrix A results from statistical observations, it is often desired to remove old observations, thus deleting a row from C. In matrix terms, this is called a downdate. A downdating algorithm is proposed that preserve the above block structure in the downdated matrix C.

KEYWORDS. Rank revealing decompositions, downdating, chasing algorithms.


We give a method for downdating the ULV decomposition. For m x n matrix A where m >__ n, the decomposition can be characterized by writing A in the form

A + S A = u ( C ) V T O (1)

168 J.L. Barlow et al.

where ~A is an m x n matrix of errors, U E ~mxm and V E ~nxn are orthogonal, and C E ~nxn is lower triangular and has the form

L 0 O ) C = F G O

0 0 0

L E Nkxk, G E ~(p-k)x(p-k) lower triangular

II (F C)l iE< e. (2)

Here k is the numerical rank of A and e < v / p - k . tol where II L-~ IIq> tol, q = 1, 2 or c~ according to some condition estimator. We use ]]. I]2 to denote Euclidean norm, I]" [IF to denote Frobenius norm, and ]]. ]I2,F to denote expressions that hold in either norm. It is assumed that I] C ]]~ 1 which can be assured by setting I] C liE = 1. The ULV decomposition was described by Stewart [8] who also gives a method for updating it. It is a particular case of what Lawson and Hanson [5] called HRK decompositions. A difference here is that we separate out blocks that are exactly zero.

The downdating problem is that of obtaining the ULV decomposition of A where (wT) A = ft. (3)

and the ULV decomposition of A is known. The opposite computation, updating, consists of finding the ULV decomposition of A from that of ft..

The problem can be transformed into the problem of finding a lower triangular matrix C', orthogonal matrices Cr E ~(n+l)x(n+l), ~" E ~nxn such that (0)z There is never a need to explicitly compute b = riTz. The algorithm is posed in terms of C and z. Like the updating routine of Stewart [8] the matrices C,V, and U can be produced using O(n) Givens rotations, thus updating the factorization in O(n2).

The following are the main results of this paper:

1. A blockwise procedure for downdating the ULV decomposition where

/, o o) g'= P O o

0 0 0

L, G lower triangular II (F C)II~,r<ll (F C)II~,F,

(5)

where the blocks are conformal with (2).

2. We show that our procedure works when L TL - xz T is positive definite. That is, we can perform a downdate when the full rank portion of C can be downdated.

An error analysis showing the favorable stability properties of this algorithm is presented in an expanded version of this paper [2]. The procedures we need to develop our algorithm is described in section two. We then describe our ULV downdating algorithm in detail in section three.

D o w n d a t i n g t h e U L V D e c o m p o s i t i o n

2 A C H A S I N G R O U T I N E F O R A L O W E R T R I A N G U L A R M A T R I X

169

In this section we give a simple routine that finds orthogonal matrices 0 , V such that

u T c v = C' lower triangular,

?Tz = p~, p =11 z I1=. We call this an lchase operation.

We need routine for the formation of a plane rotation ( f o r m r o t ) and the application of one ( a p p l y r o t ) . For instance f o r m r o t ( a , b, c n , s n , c a b s ) produces the plane rotation that maps the vector ( a l , a2 ) T into ( c a b s , O) T with

c a b s - cabs(a1, a 2 ) - C a ~ + a~; c n - a l / c a b s ; s n - a s / c a b s ;

applyrot (v ,w,cn,sn ,n) applies a plane defined by f o r m r o t thus modifying the 2 x n matrix

w T according to

w T *-- W T �9 --871, C7~

We now describe a simple chasing routine for a lower triangular matrix. Consider the

5 x 5 case below. The right arrows denote rotations from the left on two particular

rows, whereas the downarrows I I denote rotations from the right on two particular columns. Here z denotes elements in the vector z, c denotes elements in C, and ~ or denotes an element about to be zeroed out.

1 1 1 1 z z z z ~ z z z z 0 z z z ~ 0

c r c

c c ~ c c ~ c c

C C C C C C C C C

C C C C ~ C C C C C C C C C

C C C C C ~ C C C C C C C C C C

$ $ z z z 0 0 z z ~ 0 0 z z 0 0 0

C C C

c c ~ c c ~ ~ c c

C C C C C C C ~ C C C

C C C C 0 C C C C C C C C

C C C C C C C C C C C C C C C

=:~


z ,~ 0 0 0 z 0 0 0 0 c ~ c ~: C C ~ ~ C C

C C C C C C

C C C C C C C C

C C C C C C C C C C

z 0 0 0 0 a

c c

c c c

C C C C

C C C C C

We state the procedure lchase formally below.

procedure lchase(z,C,n); (* This procedure performs a chasing operation on C *) for i +-- n - 1 , . . . , 1

formrot (z ( i ) , z ( i + 1),cn, sn, z(i)); ~ - ~ , ~(i, i); ~(i, i) ~- ~ , ~(i, i ) ;

applyro t (c ( i + 1: n , i ) , c ( i+ 1: n, i + 1),cn, sn, n - i); formrot (c(i-F 1, i+ 1),e, cn, sn, c(iH- 1, i+ 1)); applyro t (c ( i + 1,1: i), c(i, 1: i), cn, 8n, i);

endfor endlchase

Stewart[8] points out that if the matrix C is from a rank revealing decomposition with k "large" rows and n - k "small" rows, this algorithm can yield k + 1 "large" rows, thus the rank revealing nature of C may be lost.

3 A P R O C E D U R E F O R ULV D O W N D A T I N G

The following algorithm is introduced for the ULV decomposition. It has the advantage that for C in (2) we obtain II (F G)112,F-----II (F G)112,F for the corresponding blocks of the downdated matrix C'. Our procedure produces a downdated matrix C' if L TL - xx T remains positive definite. A similar assumption is made implicitly by Park and Elden [7] for their URV downdating routine.

A l g o r i t h m 3.1 (Procedure for ULV downda t ing ) Given a lower triangular matrix C of the form (2) and a vector z = (x T, yT, yo)T x E ~k,y E ~p-k, YO E ~n-p, this algorithm finds a lower triangular matrix C of the form (5 ) and orthogonal matrices ~] and V satisfying (3). The components in yo are ignored. This is justified by Proposition 3.2 at the end of this section. Throughout the description of this algorithm L(k) and G (k) denote lower triangular matrices, f~k)= [F(k)]Tel, and g~k)= eT[G(k)]el.

1. ( R e d u c e G and y ) Use the procedure lchase in section 2.1 on G and y to produce ~]1, V1 such that

G (1) = ~]TGv" 1 lower triangular

and so that

Downdating the ULV Decomposition 171

Also, compute

F (1) = ~rTF.

2. ( M a k e glaeT+l a s i n g l e t o n r o w ) Find an orthogonal matrix U'2 such that

h) ( 0) 0 g~ = o [ {i~,l} ~ ~ . (6)

Define F (2) = ( I - exeT)F(1),that is, as F (1) with its first row zeroed out.

3. ( P e r f o r m D o w n d a t e on L) Use the downdating algorithm in [4] (or alternatives [6]) f ind a vector a E ~k, scalar c~, and orthogonal matrix ~]3 such that

_ _ _ - - e 1 a

Then do

I x T 0

4. ( P e r f o r m d o w n d a t e on gll ) Also do

- a -1 (p - h Ta)

if t5 _ g~2) t h e n

else gll </5

- 6 p (7)

, (2) aTh). Define ~]4 J ( 1 , k + 1,0) as the plane rotation with where 6p = p - ( gl l + =

cn = cos tg, sn = sin O such that cn = gll /g l l or cn = O if g = 0 . Then

~] = diag(Ik+l , ~]l , In_p)diag(1, ~r2, In-p-1)~f 4diag( ~f 3, In -k )


5. ( R e s t o r e lower t r i angu la r fo rm ) Find an orthogonal matrix ~d2 such that

I s ~ ) r o ther~

( 0o) (L3 0 0) g ' = P d~ o = F(3) G(4) 0 .

0 0 0 0 0 0

~r = diag(Ik, ~ , In-p)diag(V2, In-p)

Go to step 7 else go to step 6.

a. (Reduce G fu~the~ if ~ingul~) II g~)= 0 t h ~ S~)= 0 at~o ,~.c~ it ~a~ Io~m~

from g~) using V2. Thus

k p - k

(L30) k F(3) G(4) = 1 0 0

p - k - 1 p ~(4) k p - k k 1 p - k - 1

(~(4) is a lower Hessenberg matrix. We then find an orthogonal matrix fr3 such that

n - k - 1 1

If we place the zero row at the end, C has the form

C =

k ( 00) p - k - 1 fl' G 0 n - p + 1 0 0 0

k p - k - 1 n - p + 1

We note here that

~r = diag(Ik, ~ , In_p)diag(V2, In_p)diag(Ik, V3, In-p).

7. (ULV decompos i t i on of L) Perform a ULV decomposition of L to determine its numerical rank. If we have determined that rank of L correctly, the rank of L should be k or k - 1. Make appropriate adjustments to P and G.

Downdating the UL V Decomposition 173

R e m a r k 3.1 We note that (7) in Step 4 is equivalent to computing

( ~ ) = ~'T (g~21)+(SP)O~ ) + (~p)a 5P=P--(ctg~21)+aTh)

since ~]T maps the additive noise vector (Sp)(a, aT) T into (So)el �9 Only h is of interest in the computation.

satisfies (5) as is stated the following proposition proven in [2].

P r o p o s i t i o n 3.1 For the matrix C resulting from Algorithm 3.1 we have (5).

In the absence of rounding error, we can show that the "additive noise" in step 4 and the act of ignoring Y0 actually make the matrix C closer to being orthogonally equivalent to A than C is to A. Again the proof of this result is simple and we give it below.

P r o p o s i t i o n 3.2 Assume that Algorithm 3.1 is performed in exact arithmetic, that U and V are exactly orthogonal, that ~] = U[7 satisfies Uel = el , and that z = VTw and pax = ~Ty are computed exactly. Also let V = Vf" and b = vTw. If

then (bT) A + 8Ao = ft. + 8ft. 0

Thus II 5Ao [12,F=II ,Sfi 112,F<_II 5A 112,F.

Proof: Let 17 1 = Vdiag(Ik, Vl,I~_p). Then

( Vo ) OT Af/i = A1 A2 A3

~]T( 0 )diag(ik ~1 i,~_p)= L (2) /re T 0 C ' ' F(2) G(a) 0

0 0 0

Thus from (8)II 5w 112=11 8x 1122 +5p2+ II yo [1~. Similarly note that

~' ?T, i,~_p)diag(?T, F(21 a(al 0

0 0 0

(0) Using the fact that fTel = el , we have t/Ao = 5fi~ . The result follows. [::l

(9)

The downdating algorithm presented here coupled with updating algorithm of Stewart [8] show that the ULV decompositions can be updated and downdated in O(n 2) flops (including the time for updating invariant subspaces) in a manner that preserves their structure. In a


report that is an expanded version of this paper we show that the above algorithm satisfies a blockwise stability criterion [2], similar to that in [1].

Acknowledgements

The research of Jesse L. Barlow and Hongyuan Zha was supported by the National Science Foundation under grant no. CCR-9201612 (Barlow) and grant no. CCR-9308399(Zha). The research of Peter A. Yoon was supported by the Office of Naval Research under the Fundamental Research Initiatives Program.

References

[1] J.L. Barlow, H. Zha, and P.A. Yoon. Efficient and stable algorithms for modifying the singular value decomposition and partial singular value decomposition. Technical Report CSE-93-19, Department of Computer Science and Engineering, The Pennsylvania State University, University Park, PA, September 1993.

[2] J.L. Barlow, P.A. Yoon, and H. Zha. A stable algorithm for downdating the ULV decomposition. Technical Report, in preparation.

[3] L. l~lden and H. Park. Perturbation analysis of block downdating of a Cholesky decomposition. Numerische Mathematik, to appear, 1994.

[4] P.E. Gill, G.H. Golub, W. Murray, and M.A. Saunders. Methods for modifying matrix factorizations. Math. Comp., 28:505-535, 1974.

[5] C.L. Lawson and R.J. Hanson. Solving Least Squares Problems. Prentice-Hall, Engle- wood Cliff, NJ, 1974.

[6] C.-T. Pan. A modification to the LINPACK downdating algorithm. BIT, 30:707-722, 1990.

[7] H. Park and L. l~lden. Downdating the rank-revealing URV decomposition. Tech- nical Report LiTH-MAT-1992-47, Department of Mathematics, LinkSping University, LinkSping, Sweden, December 1992.

[8] G.W. Stewart. Updating a rank-revealing ULV decomposition. SIAM J. Matrix Anal. Appl., 14"494-499, 1993.


175

'THE I M P O R T A N C E OF A G O O D C O N D I T I O N E S T I M A T O R I N T H E URV A N D ULV A L G O R I T H M S

R.D. FIERRO Department of Mathematics California State University San Marcos, CA 92096, U.S.A. fierro@th under, csusm, edu

J.R. BUNCH Department of Mathematics University of California, San Diego La Jolla, CA 92093, U.S.A. [email protected]

ABSTRACT. The URV and ULV decompositions are promising alternatives to the singular value decomposition for determining the numerical rank k of an m x n matrix and approximating its fundamental numerical subspaces whenever k ~ rain(m, n). In this paper we prove general a posteriori bounds for assessing the quality of the subspaces obtained by two-sided orthogonal decompositions (such as the ULV and URV decompositions). We show that the quality of the subspaces obtained by the URV or ULV algorithm depends on the quality of the condition estimator and not on a gap condition.

KEYWORDS. Two-sided orthogonal decompositions, condition estimator.

1 INTRODUCTION

The singular value decomposition (SVD)is a widely used computational tool and is the most reliable tool for detecting near rank-deficiency in a matrix [4, p. 246]. It has important applications, for instance, in matrix approximation, subset selection, spectral estimation, direction of arrival estimation, optimization, rank-deficient least squares, and total least squares. The SVD of A (see [4, w denoted

A = UEV T (1)

176 R.D. Fierro and J.R. Bunch

where, for m _> n,

k n - k

U = [ U k U o U • V=[VkVo] and E = [ E k 0 ] k ' 0 E0 n - k .

0 0 m - n

The nonnegative diagonal elements of E, denoted ai, are the singular values of A and are arranged in decreasing order. The numerical rank of A is k. Also, rl = ak+l/O'k, and the "gap" in the singular values of A is large when 1 - '1 is close to 1.

For recursive problems where a "noise" subspace must be adaptively estimated, the SVD is viewed as computationally demanding or difficult to update. Therefore, alternative decompositions have been considered which yield the numerical rank, subspace information, or matrix approximations that are nearly as reliable as the powerful but computationally demanding SVD.

G.W. Stewart [9, 10] introduced rank-revealing "two-sided" orthogonal (or complete) decompositions, so-called URV and ULV decompositions, as alternatives. While complete orthogonal decompositions have been around for some time (e.g., see [5]), Stewart's technique is quite promising because it is guaranteed to reveal the numerical rank. In this algorithm the rectangular matrix A is preprocessed by a QR factorization, and then condition estimation and plane rotations on both sides are employed to produce a rank revealing decomposition.

The URV, and ULV algorithms are designed for the case k ~ min(m, n), where k is the numerical rank of the m • n matrix. For the low-rank case k << min(m, n), more efficient algorithms are available, see [3]. We also mention that a perturbation analysis for two-sided orthogonal (or complete) decompositions is given in [1].

For a URV decomposition of A, there exist orthogonal matrices UR 6 ~mxm and VR 6 ~nxn such that

A = URR V T = [Unk URo U~] R [Vm: Vno] T, (2) where

k n - k

0 G n - k 0 0 m - n

is upper triangular and k _< n. For a ULV decomposition of A, there exist orthogonal matrices UL 6 ~,n• and VL 6 ~n• such that

A = ULL V T = [ULk ULO U~] L [VLk VLo] T (3) where

k n - k

H E n - k 0 0 m - n

A Good Condition Estimator in the URV and ULV Algorithms 177

is lower triangular. In [9] it is shown how so-called "left" and "right" iterations may be used to iteratively refine the decompositions (in the sense that the norm of the off-diagonal block decreases). Based on this refinement strategy, error bounds for estimating the singular values of the matrix A are also provided in [6, 9].

Ideally, it would be useful for the user to have some kind of a diagnostic measure to assess the quality of subspaces obtained by a two-sided orthogonal decomposition as compared to the reliable SVD. This has important applications in areas where the SVD might be used. The objectives of this paper are to

1. provide error bounds for the subspaces determined by any two-sided orthogonal decomposition

2. show the importance of a good condition estimator in the high-rank revealing URV and ULV algorithms.

In [2] we show how these decompositions may be more accurate alternatives to the SVD than RRQR.

The paper is organized as follows. In w we derive a posteriori error bounds for decompositions of the form (2) and (3). These bounds, which are independent of the numerical rank or the algorithm used to compute the decomposition, suggest that if IIHII ~ IIFII, a ULV decomposition may yield a more accurate estimate of the numerical nullspace than a URV decomposition, while the URV decomposition may yield a better estimate of the numerical range. In w our theoretical results show that the quality of the subspaces obtained by Stewart's high-rank revealing URV and ULV algorithms depend on the condition estimator, not on the gap in the singular values. This is demonstrated in numerical simulations in [2].

Finally, I1" II = I1" 112 unless otherwise indicated, C ( I : i, 1: i) denotes the leading submatrix of C of order i, and superscripts t and T denote the pseudoinverse and transpose, respectively.

2 S U B S P A C E B O U N D S

It is well known that a singular vector corresponding to a singular value in a cluster is extremely sensitive to small perturbations, but that the span of the singular vectors corresponding to the cluster is well determined, i.e., relatively insensitive to small perturbations. Thus we provide bounds for the error in approximating the span. The following definition defines subspaces associated with the SVD.

Defini t ion [7] Let A E Nmxn and let X C Nn and Y C Nm be subspaces of dimension 1. Then X and Y form a pair of singular subspaces for A if

(i) A X C Y (ii) A T y C X.

T/(Vk) and 7~(Uk) are subspace pairs of dimension k, and Ta,,(Vo) and 7~(Uo) are subspace pairs of dimension n - k, where 7~(C) denote the range of matrix C. 7~(Vo) is termed the


numerical nullspace of A, sometimes referred to as the "noise" subspace. T~(Uk) is termed the numerical range of A.

For the (nonunique) URV decomposition in (2), there exist matrices Q and P (see [7]) such that Ti(URk + URoQ) and Tt(VRk + VRoP) form a pair of singular subspaces for A with

211FIIF II[Q P]IIF --- ~ ( R k ) - Ilall'

where I1" l i E denotes the Frobenius norm and a~n(C) denotes the smallest singular value of the matrix C. If F = 0 then n(URk) and n(VRk) form a pair of singular subspaces for A as well as 7e(URo) and Ti(VRo). If liFII is small then n(Ui) and ~(V~) (i = Rk, RO) nearly form a pair of singular subspaces for A (similar statements can be made for the ULV).

However, we wish to determine the "distance" between the subspaces n(V0) and 7~(Va0) (and 7~(Vo) and T2~(VLo)), as well as Ti(Uk) and Ti(URk) (and T~(Uk) and "]~(VLk)). We will need the following definition for the distance between two subspaces.

Def in i t ion [4, p.76] Let W = [W1 W2] and Z = [Z1 Z2] be orthogonal matrices where W~, Z~ e N px(p-q) and W2, Z2 E N TM. If See = 7~()/Yoo)and Se = 7~(Zoo)then mst(&o, Be) - - I IW/Z~l l .

2.1 SUBSPACE BOUNDS FOR THE URVAND SVD

If sin 0 = dist(Soo, Se) then O is the (largest) subspace angle between See and Se. Let s i n 0 v R . - dist(Te(Yo),Te(VRo)) and let sin CuRv - d i s t ( n ( V k ) , n ( V R k ) ) . Based on the definition, it easily follows that

• ~i~O~Rv = I I V I V R o i l = IIVRrkVoli ~ a si~ r = IIV~rURkii = IIV~k Vkii,

where U~- =- [Uo U • and U~k =_ [UR0 U• However, for our analysis in w a more useful expression for sin CvRv is needed and requires some preliminary work. We will use the following result throughout the paper.

L e m m a 1 Given the usual SVD and URV factorizations of A, then

sin CuRv = [[uT URo[] = [[UTkUo[[.

Proof : See [2].

A corresponding lemma can be proven for the ULV decomposition. Now we are ready for the main result of this section.

T h e o r e m 1 ( U R V E r r o r B o u n d s ) Let A E R "x'~ have the usual SVD and URV decomposition, and define ~ = ak+l/ak. Then the distance between the numerical nullspace ze(Vo) ~ d th~ UaV avpro~imat~ .~U~pac~ Ze(VRo). a.d th~ di~ta~ b ~ t ~ . th~ .um~ical range T~(Uk) and its URV estimate Ti(URk) are bounded by:

(a) ilnllll+f~llall < dist(7~(V0), T~(VRo)) <_ -~~ l l f l l , R ' a 2 ~ k)-- karl

(b) dist(7~(Uk)7~(URk)) < Ilfiillall , _ ~-ilall 2"


P roo f i See [2].

It can also be shown IIFII IIFII ~rk+l

dist(TC(V0), 7r _< and dist(n(Uk), n(Unk)) <_ ~ ( n , ) - ~+~ O'min(R/c)- ?llall Note that it is possible to find a posteriori upper bounds by using Theorem 1 together

with the facts a~n(Rk) <_ ak, ~? <_ 1, and ak+l _ [[G[I. Given the above, the following corollary is immediate.

C o r o l l a r y 1 (A Posteriori B o u n d s for UP.V) Under the assumptions of Theorem 1, the following a posteriori bounds hold:

(~) tlFtL~,,_ IIFII ~mi.(R*) IIRII+I < dist(7C(Vo), 7r < .L.ia~)_l;vll~

(b) dist(7~(Uk) 7~(Unk)) < ,L] IFII Ilall , _ . (Rk)-Ilall 2'

These bounds show explicitly that when IIFII is small then the subspaces nearly coincide. In w we discuss a way to achieve a small IIFII for the high-rank case k ~ min(m, n) so that high quality subspaces are obtained.

2.2 SUBSPACE BOUNDS FOR THE ULVAND SVD

In this section we are concerned with the ULVdecomposition. Let A have the usual SVD and ULV as in w Let sin OULV =-- dist(TC(V0), Ti(VLo)) and sin CULV = dist(TC(Uk), Ti(ULk)). After the ULV factorization is complete, we wish to determine upper bounds on the errors in the approximate subspaces.

T h e o r e m 2 ( ULV E r r o r B o u n d s ) Let A E R m• have the usual SVD and ULV decomposition, and define ~7 = O'k+l/ak. Then the distance between the numerical nullspace Ti(Vo) and the (fLY approximate nullspace ~(VLo), and the distance between the numerical range Ti(Uk) and its ULV estimate 7"r are bounded by:

(a) dist(7~(Vo) ~'~(Vno)) < crk+lllHII , -- a~i a (Lk)-a~+l

(b) ,l~ll~ltj~,l < dist(TC(U~) 7r < L~'t'~=~o(~) rain L ~ k ) - k + l

Proof : See [2].

As before, it is possible to generate a posteriori upper bounds for the subspace angles in terms of computed ULV factors. The necessary facts are amin(Lk) <_ ak, r/ < 1, and ak+l _< IIEII, as well as Theorem 2.

C o r o l l a r y 2 (A Posteriori B o u n d s for ULV ) Under the assumptions of Theorem 2, the following a posteriori bounds hold:

IIHIIIIEI] (a) dist(7~(V0), Ti(VLo)) <_ ~,~(Lk)_IIEll2

IIHIJ < dist(TC(G) Ti(ULk)) <_ r 2 �9 ( b ) t,Ltl+JtEtt -- ' ~-i~(L~)tlHlt


This shows a large [IH[I translates to a large sin CULY. In the next section we discuss a way to produce a small IIH[] for the high-rank case k ~ min(m, n). Comparing the a posteriori bounds for the URV and ULV, we may conclude that the ULV can be expected to yield a higher quality estimate of the numerical nullspace than the URV. Tables 1-4 summarize the results of some typical experiments that verify this conclusion.

3 T H E I M P O R T A N C E OF A G O O D C O N D I T I O N E S T I M A T O R

As mentioned earlier, any URV or ULV decomposition may be refined iteratively using orthogonal transformations, cf. [9]. The purpose of refinement procedures is to concentrate less "energy" of R (or L) in the 1,2 (or 2,1) position so as to decouple the matrix as much as possible. Recall from w that when the triangular matrix is decoupled (i.e., the off-diagonal block is a zero matrix) then we have obtained singular subspaces for the matrix, and when the off-diagonal block is small then we have obtained good singular subspace approximations. In [9] it is shown how so-called "left" and "right" ("shiftless" QR) iterations may be used to iteratively reduce the norm of the off-diagonal block, and hence refine the subspaces. Based on this particular refinement strategy, error bounds for estimating the singular values of the matrix A are provided [6, 9]. In this section we show that a small off-diagonal block is achieved in the high-rank revealing URV and ULV algorithms by using a good condition estimator.

Now we turn our attention to a brief but important discussion of the algorithms by Stewart. At the i th step of the URV algorithm, we work with the upper triangular matrix

i n - i

0 Gi n - i

i corresponding to and a unit estimate v~.ti of the exact i x 1 right singular vector v ~ amin(Ri). Using plane rotations, find an i x i orthogonal matrix Qi such that (Qi)Tviea t = (0 , . . . , 0,1) T and RiQ i is upper Hessenberg. Then determine an i • i orthogonal matrix pi such that (Pi)T(RiQi) is upper triangular. Partition the updated triangular matrix by

i - 1 1 n - i

0 0 Gi n - i.

Analogously, at the i th step of the ULV algorithm we work with the lower triangular matrix

i n - i Li 0 i Hi Ei n - i

i corresponding to an~n(Li). and a unit estimate u~s t i of the exact i • 1 left singular vector umi n i i = (0 , . . . 0,1) T Using plane rotations, find an i x i orthogonal matrix pi such that P uea t

and PiLi is lower Hessenberg. Then use plane rotations to determine an i x i orthogonal matrix Qi such that (PiLi)(Qi)T is lower triangular. Partition the updated triangular


matrix by

i - 1 1 n - i

Hi(Qi) T Ei = h i eli 0 1 H~ ei Ei n - i.

The following result shows how the accuracy of the estimate ves t i is related to the norm of the subcolumn fi , the approximation of 2 amin(Ri) by r~i , and [[F[[, as well as how the accuracy of the estimate ues t i is related to the size of the subrow hi T, the approximation of ff2n(Li) by li~ , and IIH[I.

T h e o r e m 3 Using the notation above, let v.ti with unit e-norm denote an estimate of i the right singular vector of Ri corresponding to a~n(Ri) . I f i Oun v denotes the angle VmJn ,

between i and i then Vest VmJn ,

- a~n (Ri) < sin 8 UnV. Ilfi[I < sin 0/uRv and ~/r2i 2 i IIR~II - IIa~ll -

Analogously, using the notation above, let ues with unit 2-norm denote an estimate of i Urr~n , the left singular vector of Li corresponding to am_in(Li) I f i �9 CULY denotes the angle between

i and i then U est U mJn ,

IIh~ll IIL~ll

Moreover,

IIFll IIRII

�9 - a ~ n ( L i ) _< sin r and ~/12i 2 i]/il] _< sin r

< ~ n - k sin Dmax _ run v and ~ < v / n - k sin . / , m a x -- ~'ULV

where vuR vj~ ~ m a x { O ~ r R v , . . . , vURVJak+l ~ and ~max ,r "t wULV -- max{r v'ULVJ"

Proof : See [2].

This means good estimates of the right (left) singular vectors of Ri (Li) for i = n , . . . , k+ 1 lead to a small IIFII (IIHII). By Corollaries 1 and 2 this means the quality of the subspaces depends on the quality of the estimated singular vectors. Theorem 3 generalizes [11, Theo- rem 1] where it is proven that if all the estimates i (or i , ves t Uest) are indeed singular vectors, then F = 0 (or H = 0) and the relevant subspaces coincide.

Re fe r ences

[1] R.D. Fierro, Perturbation analysis for two-sided (or complete) orthogonal decompositions, PAM Technical Report 94-02, Department of Mathematics, California State Uni- versity, San Marcos, CA 92096.

[2] R.D. Fierro and J.R. Bunch, Bounding the subspaces from rank revealing Two-Sided Orthogonal Decompositions, to appear in SIAM J. Matrix Anal. Appl.


[3] R.D. Fierro and P.C. Hansen, Low-rank revealing two-Sided orthogonaldecompositions, PAM Technical Report 94-06, Department of Mathematics, California State University, San Marcos, CA 92096.

[4] G.H. Golub and C.F. Van Loan, Matrix computations (Second Edition), John Hopkins University Press, 1989.

[5] C.L. Lawson and R.J. Hansen, Solving least squares problems, Prentice Hall, Engelwood, Cliffs, N.J., 1974.

[6] R. Mathias and G.W. Stewart, A Block QR algorithm and the singular value decomposition, Lin. Alg. Appl. 182 (1993), pp. 91-100.

[7] G.W. Stewart, Error and Perturbation Bounds for Subspaces Associated with Certain Eigenvalue Problems, SIAM Review, 15 (1973), pp. 727-64.

[8] G.W. Stewart, An Updating algorithm for subspace tracking, IEEE Transactions on Signal Processing, 40, (1992), pp. 1535-1541.

[9] G.W. Stewart, On an algorithm for refining a rank-revealing URV decomposition and a perturbation theorem for singular values, Technical Report CS-TR 2626, University of Maryland, Dept of Computer Science, 1991.

[10] G.W. Stewart, Updating a rank revealing ULV decomposition, SIAM J. Matrix Anal. Appl., 4 (1993), pp. 494-499. Lin. Alg. Appl., 28, (1979), pp. 257-278.

[11] S. Van Huffel and H. Zha, An e]j%ient total least squares algorithm based on a rank revealing two-sided orthogonal decomposition, Numerical Algorithms, 4, (1993), pp. 101- 133.


183

L-ULV(A), A L O W - R A N K R E V E A L I N G ULV A L G O R I T H M

R.D. FIERRO Department of Mathematics California State University San Marcos, CA 92096, U.S.A. fierro @th u nder. csusm, edu

P.C. HANSEN UNI . C, Building 304 Technical University of Denmark DK-2800 Lyngby, Denmark Per. [email protected]

ABSTRACT. We present and analyze an algorithm L-ULV(A)for computing a rank- revealing ULV decomposition. Our algorithm is efficient whenever the numerical rank of the matrix A is much less than its dimensions. We also show that good estimates of the singular vectors, needed in the algorithm, produce a rank revealing form that yields good approximations to the singular subspaces of A.

KEYWORDS. ULV decomposition, low-rank problems.


Most of the current algorithms for computing rank revealing decompositions are tailored to the high-rank case. However, the are also applications for which the matrix A E ~mxn has low rank, i.e., where the numerical rank is much smaller than min(m,n). Matrices with low rank arise when a relatively small number of parameters suffice to describe a system, and "heavy" oversampling is used to reduce the influence of noise or errors in the data. Examples of such problems are linear prediction problems in spectroscopy [1], [13] and noise reduction in speech processing [6].

The high-rank revealing algorithms are not suited for such problems because they "peel off" all the small singular values one by one. Instead, one needs algorithms that are specially designed for the low-rank case. The main purpose of this paper is to present and analyze

184 R.D. Fierro and P.C. Hansen

an algorithm for computing a low-rank revealing ULV decomposition that produces good estimates of the fundamental numerical subspaces associated with a matrix.

If k denotes the numerical rank of A, then we want to compute approximations to the k largest singular values as well as approximate basis vectors for the two k-dimensional subspaces spanned by the first k left and right singular vectors of A. The approximations we seek are obtained directly from a rank revealing ULV decomposition of A, denoted by

A = U L VT ' L = ( LkH E0)" (1)

Here, UTU = In, v T v = I,~, Lk is a k • k lower triangular matrix whose singular values approximate al through ak, while II(H, E)I I is of the order ak+l. Then the columns of ULk -- (Ul, . . . ,uk) and VLk = (Vl,. . . ,Vk) are the desired approximate basis vectors, and the smaller the norm IIHI], the better the approximations [3], [8]. Here, and throughout the paper, I]" I] denotes matrix and vector 2-norms.

2 T H E L O W - R A N K A L G O R I T H M : L-ULV(A)

A careful analysis of two different low-rank ULV algorithms, plus some variants of these, was carried out by the authors in [4]. The first algorithm, L-ULV(L), requires an initial ULV factorization, and it is very similar to Stewart's original high-rank algorithm [12] except that it is based on estimates of the largest singular values (instead of the smallest). The second algorithm, L-ULV(A), avoids the initial ULV decomposition and starts off directly with the matrix A.

The latter algorithm was found to be the most efficient algorithm in most cases (essentially when the numerical rank is very small compared to m and n), and it takes the following form.

ALGORITHM L-ULV(A). Input an m x n data matrix A and rank tolerance r.

1. Initialize i ~ 1, U ~ Ira, and V ~ In.

2. Compute estimates a (1) and ~ (1) of the largest singular value and est ~ e s t

corresponding left singular vector of A. (,,(i) )

3. While \yes t > r and i < n do 4. Determine a Householder matrix P(0 such that

= (1, o, ..., o) eo v o 5. Form Ai ~ ( I i -1 A and update U .--

\ o (o/ \ o P(o/" 6. Determine a Householder transformation Q~,i) that annihilates

elements i + 1 through n of the ith row of Ai.

7. Update A ~ , ( / , _ l 0 ) a n d V ~ _ V ( ' , _ l 0 ) 0 Qci) 0 Q(0 "

8. Deflate: set i ~ i + 1 and compute estimates Ves t'(0 and U~s ) of the largest singular value and left singular vector of A(i: m, i: n).

End of ALGORITHM L-ULV(A).

L-ULV(A), A Low-rank Revealing ULV Algorithm 185

Table 1" Flop counts for computing various factors by means of the L-ULV(A) algorithm and the two SVD algorithms applied to a general m x n matrix with m >_ n. The number Wd(A) is the average work to compute the singular value and vector estimate.

Required L-ULV(A) L

L , V L ,U L, U~

L, U, V L, U1, V

Required E

E, V E,O

Z, ~ , ?

8mnk + 3(m + n)k + (k + 1)Wd(A) 8mnk + 3(m + n)k + 4n2k + (k + 1)Wd(A) 4m2k + 8mnk + 3(m + n)k + (k + 1)Wd(A)

12mnk + 3(m + n)k + (k + 1)Wd(A) 4m2k + 4(2m + n)nk + 3(m + n)k + (k + 1)Wd(A)

12mnk + 3(m + n)k + 4n2k + (k + 1)Wd(A) R-SVD Golub-Reinsch SVD

(m _ 5n/3) (5n/3 _ m >__ n) 2mn 2 + 2n 3 4ran 2 - 4n3/3

2mn 2 + l l n 3 4mn 2 + 8n 3 4m2n + 13n 3 4m2n + 8mn 2 6mn 2 + l l n 3 14mn 2 - 2n 3 4m2n + 22n 3 4m2n + 8mn 2 + 9n 3 6mn 2 + 20n 3 14mn 2 + 8n 3

In Table 1 we summarize the flop counts for our L-ULV(A) algorithm. We compare the flop counts with those of two SVD algorithms, cf. [5, w the Golub-Reinsch SVD algorithm and the R-SVD algorithm. An important consideration is "how much" of the ULV decomposition is required. For example, in many total least squares problems, only k and V are required. Therefore, we give flop counts for various combinations of the factors U, U1, L and V, and compare with the two SVD algorithms.

In the table, Wd(A) denotes the average work involved in a deflation step to compute the required singular value and vector estimate. We refer to w for more details about Wd(A), and to w for an evaluation of the algorithm.

As mentioned earlier, in [4] we present and analyze ALGORITHM ULV(L), which requires an initial "skinny" QR factorization of A. We note that if the matrix A has Toeplitz structure, then the initial skinny QR factorization of A can be computed much faster by means of specialized algorithms: R and U1 can be computed in mn+ 6n 2 and 12mn flops, respectively; see, e.g., [9] (a more accurate R can be computed in 2mn + 6.5n 2 flops by new algorithm by Park & Eld~n [10]). The Toeplitz structure is lost in the R factor. Altogether, the flop count for ALGORITHM L-ULV(L)is mn(6k + 13)+ n2(12k +6) + (k + 1)Wd(L) when A has Toeplitz structure and U1 and V are required explicitly with L. See also [4] for a discussion of an implicit ULV algorithm in which the L matrix is never explicitly computed but kept in factored form.


3 T H E I M P O R T A N C E OF A G O O D S I N G U L A R V E C T O R E S T I M A T E

In this section we prove that a very important part of the algorithms given above is the estimation of the largest singular value and associated singular vector in the ith deflation step. The following theorem reveals the importance of a good singular vector estimate during the ith stage of ALGORITHM L-ULV(A), and readily applies to ALGOttlTHM L- ULV(L) (see [4] for the proof.)

T h e o r e m 1 Let Ves t''(i) and west" (i) denote the estimate of the largest singular value a~ i) and associated left singular vector u~ i) of A(i: m, i: n) Let Oi denote the angle between. (i) and

�9 ' * e a t

u~ i). Let the orthogonal matrices P(i) and Q(i) come from ALGORITHM L-ULV(A), and partition the updated P(i)A(i: m, i" n)Q(i ) according to

1 n - i P(i)A(i:m,i 'n)Q(i)= ( l i 0 ) 1

hi L (i) m - i.

Then

aq )~ < sin Oi and - ~ < - - 1 - sin 0i

Further,

Ilhill 2 sin Oi ~('j(ij -< 1 - sin Oi

(2)

It follows that if we based our algorithm on the exact singular vectors of the submatrices A(i: m, i: n) in each step, then we would get H = 0, and Lk would be diagonal and simply consist of the largest k singular values of A. This would give a low-rank "partial" SVD. Further, if we set k = n, Theorem 1 simplifies to an existence proof [5, Theorem 2.5.1] for the SVD.

In practice, however, we can only supply an estimate of the singular vector, and the norm of the off-diagonal subcolumn hi in the ith stage directly depends on the accuracy of this estimate--as measured by Oi. When Oi is small, then Ilhil] is correspondingly small and ]lil is a good estimate of a~ i). If we assume good singular vector estimates are obtainable, then Theorem 1 serves as a proof for the existence of a rank revealing two-sided orthogonal decomposition of a matrix.

The following important theorem for ALGORITHM L-ULV(A) shows that good singular vector estimates yield a small [IHII (the theorem readily applies to ALGORITHM L-ULV(L); see [4] for the proof).

T h e o r e m 2 Let Oi denote the angle between U(ie2t and u~ i) at the i-th stage of the algorithm, where 1 < i < k. Define 0m~x = max {01, . . . , Ok). When the algorithm terminates, then

[]L[[ - 1 - sin0m~x'


Theorem 2 proves that the better the singular vector estimates, the smaller the IIHII. Thus, the effort one spends in computing good estimates is paid off in a small off-diagonal block. On the other hand, the norm of this block can always be reduced by block QR iterations (also known as VLYrefinement)as pointed out in [8].

4 E S T I M A T I O N OF T H E L A R G E S T S I N G U L A R V E C T O R

We now discuss two algorithms for estimation of the largest singular value Cres t and the corresponding left singular vector Uest of an m X n matrix A: the simple power method and the more sophisticated Lanczos bidiagonalization algorithm. The same methods were also proposed in [2] in connection with the low-rank revealing QR algorithm, but here we give a more detailed analysis of these algorithms.

Both methods need a good starting vector which has a nonzero component along the largest singular vector, and we propose to use the vector

Ae u0 = i] A el I , where e = (1 , . . . , 1) T.

In the rare case that e is a null vector of A, try random vectors until IIAell ~ 0. Our stopping criterion is based on the current singular value estimate. Due to the importance of the singular vector estimate, as proved in Theorem 2, one could also monitor the angle between successive approximate singular vector estimates, but this is computationally more expensive.

For convenience, let p and ~ denote the required number of iterations in the POWER and LANCZOS algorithms, respectively. Notice that if p = g = 1 then the two algorithms produce the same estimates.

The POWER METHOD only needs multiplications with A and A T plus computations of vectors norms, and a systolic implementation of this is described in [7]. Thus, the algorithm requires pm(4n + 3) flops for a full A, which reduces to pn(2n + 3) if A is triangular.

The LANCZOS METHOD requires additional vector operations as well as the computation of the SVD of a bidiagonal matrix. Hence, the algorithm is more expensive than the POWER METHOD and it does not map onto a systolic array. Since we are only interested in computing the largest singular value of A, the LANCZOS METHOD does not need any reorthogonalization! If, on the average, 1.5 iterations are required per singular value, then computation of the singular values in the j th step requires 22j 2 flops, while accumulation of the left singular vectors (once the algorithm has terminated) requires 3/3 + 4.5/2 flops. The whole algorithm requires about 4gmn + 8gm + 6in + 10g 3 flops for a full A, which reduces to 2in 2 + 14in + 10s 3 flops if A is triangular.

The average amount of work Wd(') in the various combinations of algorithms is summarized in Table 2, where we have neglected all lower-order terms, including those coming from the fact that the dimensions of L and A are actually reduced in each step (these terms are negligible as long as k << n).

In both the POWER METHOD and the LANCZOS METHOD for the symmetric eigenvalue problem, the convergence of the eigenvectors is governed by the square root of the corre-


Table 2: The average amount of work Wd(X) in the algorithms for estimating the largest singular value and corresponding vector of the matrix X. Here, p and I denote the average number of power and Lanczos iterations.

POWER METHOD Wd(L) 2p,~ 2 Wd(A) 4pmn

LANCZOS METHOD 2In 2 + 14in + 10/3

4lmn + 81m + 6In + 1013

sponding factor for the convergence of the eigenvalues, cf. [5, w and [11, p. 62 and (12-4-1)]. Hence, the comparison of the POWER and LANCZOS methods from [5, w for computing eigenvalues essentially carries over to the computation of singular vectors. As long as al is well separated from a2, the LANCZOS METHOD converges much faster than the POWER METHOD. If al ~ . . . ~ aq, then one should compare the angle between the estimated singular vector and the space span{vl , . . . , %}, and again the LANCZOS METHOD outperforms the POWER METHOD when aq is well separated from aq+l. Hence, we advocate using the LANCZOS METHOD in our low-rank algorithms.

5 E V A L U A T I O N OF T H E A L G O R I T H M

When we combine the information in Tables 1, [4, Table 2], and Table 2 we can evaluate for which combinations of m, n and k ALGORITHM L-ULV(A)is more efficient than L-ULV(L) (see [4] for details about L-ULV(L)). The flop counts for L-ULV(L) differ for m _> 1.5 n and n _< m < 1.5 n, and we therefore analyze both cases.

We see from Table 2 that for m and n large, the dominating amount of work in both the POWER and LANCZOS METHODS is Wd(L) = 2pn 2 and Wd(A) = 2pmn, where p is the number of either power or Lanczos iterations. Then we can derive the approximate dominating terms for L-ULV(L) and L-ULV(A) shown in Table 3, from which we easily find the break-even ratios. We conclude that if the numerical rank k of A is very low, i.e., k << n, then L-ULV(A)is usually more efficient than L-ULV(L). For example, if p = 3 (which is realistic) and if we compute either L or the pair L, V, then L-ULV(A) using the first approach is more efficient whenever n > 10k.

In [4, Table 2] we see by comparing dominating terms that the L-ULV(L) algorithm is faster than the two SVD algorithms. In Table 3 we compare the L-ULV(A) algorithm to the SVD algorithms" to the Golub-Reinsch SVD algorithm for 5n/3 >_ m >_ n, and to the R-SVD algorithm for m >_ 5n/3. Table 3 shows that in low-rank problems ALGORITHM L-ULV(A) is the faster algorithm. This substantiates the claim that L-ULV decompositions are computationally attractive alternatives to the SVD.

Finally, we conclude that approximations to the largest k singular values and vectors can also be computed by means of applying Lanczos bidiagonalization directly to A. This is particularly attractive when A is a structured matrix, and good results are reported in [13]. Unlike our algorithms, however, Lanczos bidiagonalization may experience loss of orthogonality in the orthogonal factors and is not guaranteed to detect the numerical rank.


Table 3: Comparison of approximate dominating terms of the flop counts for the low-rank ULV algorithms, and ALGORITHM L-ULV(A) versus the SVD algorithms. The integer p is the number of either power or Lanczos iterations.

Required L

L,V L,U L, U~

L, U,V L, U1, V

Required L

L ,V L,U L, U~

L, U, V L, U1,V

Approx. dom. term (m >_ 1.5 n) L-ULV(L) L-ULV(A) 2ran 2 4mnkp + 8mnk 2mn 2 4mnkp + 8mnk 4m2n - 2ran 2 4m2k + 4mnkp + 8rank 4mn 2 4mnkp + 12mnk 4 m 2 n - 2mn 2 4m2k + 4mnkp + 8rank 4mn 2 4mnkp + 12rank

L- ULV(A) is faster than L-ULV(L) when n/k > 2p + 4 n/k > 2p+ 4 n/k > 3(p+ 3)/2 n / k > p + 3 ~/k > 3(p+ 3)/2 n/k > p + 3

Approx. dom. term (1.5 n > m > n) L-ULV(L) L-ULV(A) 4mn 2 v 3 - - ~ n

4ran 2 - n 3 4 m 2 n - n 3 8mn 2 - ll n3

8mn 2 --

L-ULV(A) is faster

4mnkp + 8mnk 4mnkp + 8mnk 4ra2k + 4mnkp + 8rank 4mnkp + 12mnk 4m2k + 4mnkp + 4(2m + n)nk 4mnkp + 12mnk

than L- ULV(L) when ,~/k > ~(p + 2)/2 n/k > 4(p+ 3)/3 n/k > 4(p + 3)/3 n/k > p + 3 n/k > p + 4 ,~/k > 3(p + 3)/4

Est. Requ.

~ , U :~, Crl

:~, (r, ?

Est. Requ.

y~, fz r,, i] Y~,G

F~,U,V r~,fh,#

Approx. dom. term (m >_ 5 n/3) R-SVD L-ULV(A) 2mn 2 4mnkp + 8mnk 2mn 2 4mnkp + 8mnk 4m2n 4m2k + 4mnkp + 8rank 6mn 2 4mnkp + 12mnk 4m2n 4m2k + 4mnkp + 8mnk 6mn 2 4mnkp + 12mnk

L-ULV(A) is faster than R-SVD when n/k > 2p+ 4 n/k > 2p + 4 n/k > p + 3 ~/k > 2(p+ 3)/2 n/k > p + 3 ~/k > 2(p + 3)/3

Approx. dom. term (5n/3 >_ m >_ n) Golub-Reinsch SVD L-ULV(A) 4mn 2 - 4n3/3 4ran 2 + 8n 3 4m2n + 8ran 2 14mn 2 - 2n 3 4m2n + +Smn 2 + 9n 3 4mn 2 + 87~ 3

4mnkp + 8rank 4mnkp + 8rank + 4n2k 4m2k + 4mnkp + 8rank 4mnkp + 12mnk 4m2k + 4mnkp + 8mnk 4mnkp + 12mnk + 4n 2k

L-ULV(A) is faster than Golub-Reinsch SVD when n/k > 3(p+ 2)/2 ~/k > (p + 3)/2 =/k > (p + 3)/2 n/k > 1 + p/3 ~/k > (p + 4)/3 ~/k > (p+ 4)/2


References

[1] H. Barkhuijsen, R. De Beer and D. Van Ormondt. Improved algorithm for noniterative time-domain model fitting to exponentially damped magnetic resonance signals. J. Magnetic Resonance 30, pp 553-557, 1987.

[2] T.F. Chan and P.C. Hansen. Low-rank revealing QR factorizations. Num. I, in. AI9. Appl. 1, 33-44, 1994.

[3] R.D. Fierro and J.R. Bunch. Bounding the subspaces from rank-revealing two-sided orthogonal decompositions. To appear in SIAM J. Matrix Anal. Appl.

[4] R.D. Fierro and P.C. Hansen. Low-rank revealing two-sided orthogonal decompositions. Report UNIC-94-07, UNIoC, September 1994.

[5] G.H. Golub and C.F. Van Loan. Matrix Computations (Second Edition), John Hopkins University Press, Baltimore, 1989.

[6] S.H. Jensen, P.C. Hansen, S.D. Hansen and J.Aa. SCrensen. Reduction of broad-band noise in speech by truncated QSVD. Report ESAT-SISTA/TR 1994-16J, Dept. of Elec- trical Engineering, Katholieke Universiteit Leuven, March 1994; submitted to IEEE Trans. Speech Proc. A short version of this paper appears in these proceedings.

[7] F. Lorenzelii, P.C. Hansen, T.F. Chan and K. Yao. A systolic implementation of the Chan/Foster RRQR algorithm. IEEE Trans. Signal Processing 42, 2205-2208, 1994.

[8] R. Mathias and G.W. Stewart. A block QR algorithm and the singular value decomposition. Lin. Alg. Appl. 182, 91-100, 1993.

[9] J.G. Nagy. Fast inverse QR factorization for Toeplitz matrices. SIAM J. Sci. Comput. 14, 1174-1193, 1993.

[10] H. Park and L. Eld~n. Fast and accurate triangularization of Toeplitz matrices. Tech- nical Report LiTH-MAT-R-1993-17, University of LinkSping, Dept. of Mathematics, April 1993.

[11] B.N. Parlett. The Symmetric Eigenvalue Problem. Prentice-Hall, Englewood Cliffs, N.J., 1980.

[12] G.W. Stewart. Updating a rank revealing ULV decomposition. SIAM J. Matrix Anal. Appl. 14, 494-499, 1993.

[13] S. Van Huffel, P. Lemmerling and L. Vanhamme. Fast algorithms for signal subspace fitting with Toeplitz matrices and applications to exponential data modeling. These proceedings.


191

F A S T A L G O R I T H M S F O R S I G N A L S U B S P A C E F I T T I N G W I T H T O E P L I T Z M A T R I C E S A N D A P P L I C A T I O N S T O E X P O N E N T I A L D A T A M O D E L I N G

S. VAN HUFFEL, P. LEMMERLING, L. VANHAMME Katholieke Universiteit Leuven Kardinaal Mercierlaan 9~ 3001 Leuven Belgium sabine, vanh u f f el @esat. kule u yen. ac. be

ABSTRACT. Fast methods are presented for solving the following key problem: "Given an L x M Toeplitz matrix T of effective rank K << min{L,M}, find an estimate of its K-dimensional signal subspace". This arises in signal processing and system identification problems, such as exponential data modeling, direction-of-arrival estimation and subspace tracking. The SVD, proven to be a valuable and reliable tool for solving this problem, is computationally too expensive. Replacing this SVD by a Low Rank-Revealing (LRR) two-sided orthogonal decomposition is shown to speed up the computations, up to 10 times. Properties of this factorization are presented, showing that the obtained subspace estimate is exactly the same as the space generated from the start vectors used. I.e. the LRR factorization does not further improve the subspace accuracy and, therefore, iterative methods are more efficient when the rank is fixed. An iterative method, based on Lanczos recursions and block power iterations, is presented and shown to be up to 30 times faster than the SVD computation in an exponential modeling problem encountered in NMR data quantification.

KEYWORDS. Rank-revealing two-sided orthogonal decomposition, orthogonal iteration, Lanczos method, exponentially damped sinusoids, nuclear magnetic resonance data.


It is well known that the Singular Value Decomposition (SVD) of a rank K matrix:

T = U E V * = [ U1 U2 U• ] 0 E2 [Vl V2 ]* K M - K L - M 0 0 K M - K

with E1 = d iag(a l , . . . , o'g) , E2 = d i a g ( o ' g + l , . . . , O'M), and al _> . . . >_ aM

(1)

192 S. Van Huffel et al.

yields a basis V1 of the signal subspace 7Z(V1). L _> M is assumed and * denotes the conju- gated transpose. However, since O ( L M 2 T M 3) operations are needed, faster factorizations axe considered here, such as the rank-revealing (Rtt) URV decomposition:

T = U T ( K ) v * = [ U1 U2 Ux ] 0 G [ ~'1 V2 ]* (2) K M - K L - M 0 0 K M - K

where U and V axe unitary, D 6 C KxK and G axe upper triangular. The UI%V decomposition is rank revealing in the sense that the numerical rank K of T is exhibited in (2) from

IIFIl + Ilall + . . . +

where ak denotes the kth singular value of T. In the noise-free case, all the singular values a K + l , . . . , a M axe equal to zero in exact arithmetic. However, in a noisy environment, the singular values a g + l , . . . , aM correspond to the noise part of the data, and the numerical rank K has to be estimated. The 1~1~ UP~V decomposition was first introduced by Stewart [11] for matrices of which the dimension of the signal subspace is large, i.e. when K is only slightly smaller than M. Recently, it was adapted to the computation of a basis for the signal subspace when K << M, which is referred to as the Low RI~ (LRR) URV decomposition [4]. Alternatively, the Kl~ ULV decomposition can be computed in which T (K) is lower triangular. Here, we restrict ourselves to the 1~1~ UI~V decomposition.

2 S I G N A L S U B S P A C E E S T I M A T I O N B A S E D ON A L O W R R URV DE- C O M P O S I T I O N

We improve the low RI~ URV algorithm, presented in [4], in the following way.

First of all, the initial QI~ factorization of the matrix, T = QR, requiring O(LM 2) operations, is replaced by a fast triangularization method that fully exploits the Toeplitz structure of T and requires only O ( L M + M 2) operations. In the fast algorithms introduced by Bojanczyk et al. [1] and Chun et al. [2], each row of the triangular factor R is computed recursively via updating and downdating steps. However, when the problem is ill-conditioned, these algorithms may produce an inaccurate R factor due to the difficult downdating steps. A method that monitors the conditioning of the downdating problems and chooses a more stable alternative in computing the downdating transformation, and also an algorithm where the computed 1~ factor is refined using the method of corrected semi-normal equations are presented in [9]. These methods can be used for the triangularization of the Toeplitz matrix with a little extra cost for better accuracy when the problem is ill-conditioned.

Next, a ttl~ UttV decomposition of the triangular M x M factor R is computed. Instead of estimating the left singular vector Umax of R corresponding to its largest singular value al, as initially proposed by Fierro and applied in [10], we start from estimates Wl of the corresponding right singular vector Vmax. Based on the relation

IIRwlll2 ~ al and Rwl = tlQ*Qwl,

Signal Subspace Fitting with Toeplitz Matrices 193

for any unitary Q, we choose Q as a product of Givens rotations which reduces wl to the first unit vector el, up to a factor e ~r . The application of Q* from the right to R gives an upper Hessenberg form, and by another sequence of rotations, applied to R from the left, the upper triangular form is restored, giving a matrix R0). All the right transformations are accumulated into V which is initialized as an identity matrix. If wl would estimate the largest singular vector Vmax of R exactly, then, as a result of these transformations in exact arithmetic, the first column of the new triangular matrix would become the scaled unit vector ale~e ~r . The same procedure is then applied to the lower principal ( M - 1) • ( M - 1) submatrix of R (1) and after K such steps we have R(K) and ~'. Given R, the computation of R (g) and ~" requires O ( K M 2) operations, and is outlined below.

A lgo r i t hm: TURVV

Given. TL• Toeplitz, its numerical rank K, L >_ M > > K, or a tolerance tol.

Compute T = Q R, using a fast Toeplitz triangularization method [9] Initialize R (1) ~ R ( I ' M , I ' M ) , / ~ 1 ~ R , V (1) ~ I , i ~ l Compute an estimate wl of Vmax(R), [[Wl[[ = 1 While []Riwi[[ > tol or i _ g do

compute plane rotations Q(i) such that Q(i). 0 = e~r 1 1 wi 0 M - i

compute plane rotations p(0 such that R (i+1) = P(O(R(OQ (0) is upper triangular update V: V (i+1) .-- V(i)Q (i) set" Ri+l ~ R(i+l)(i + 1 �9 M , i + 1 �9 M) compute an estimate wi+l of Vmax(Ri+l), [[wi+l[] = 1 i * - - i + l

end while Z ~ i - 1 and V1 = v(K+I)( ", l ' g ) is a basis estimate of the signal subspace T~(V1).

The total complexity of the TURVV algorithm is O ( L M + M 2 + KM2) , which is clearly more efficient than the SVD computation provided K << M. In this paper, the estimates wi are computed by a few iterations, say n, of the power method [5] with start vector s/[[s[[ where sj = ~k(Ri ) jk . Alternatively, a Lanczos method could be used as done in section 3.

This TURVV algorithm, based on estimates of Vn~x(Ri), ensures a more accurate signal subspace estimate ~ turvv than the so-called TURVU procedure, based on estimates of Um~x(Ri) and outlined in [10], as proven below (see [7, Theorem 1] for a proof):

T h e o r e m 1. Given a matrix T E C LxM with SVD (1) and LRR URV decomposition (2). Superscripts turvv (resp., turvu) are added to indicate that the LRR URV decomposition is obtained by means of the TURVV (resp., TURVU) algorithm. I /d is t (n(0~u~u) , u(U~)) = d i s t ( n ( v ~ r ~ ) , ~(V~)) then

dist(T~(V~ turvv) ~(V1)) < ~+1 dist(n(ff~urvu), T~(V~)) ~ ~K+I dist(n(~turvu), n(V1)) , - - a m i n ( D t u r v u ) a K

The larger the gap a g + l / a g , the closer the TURVV signal subspace estimate V1 turvv approaches the desired singular subspace T~(V1), compared to the TURVU estimate, as illustrated in Table l(a). Using the RR ULV decomposition reverse conclusions hold.

194 S. Van Huff el et al.

In practice, the condition dist(7~(uturvu),n(U1)) = dJst(n(~rturvv),T~(V1))is roughly satisfied when the largest left and right singular vectors Umax(Ri) and Vmax(Ri) of the Ri, i = 1 , . . . , K, are equally well estimated. This latter condition is used in Table l(b). This is also clear from the next theorem which proves that the accuracy of the TURVV signal subspace estimate solely depends on the accuracy of the start vectors wi, i = 1 , . . . , K:

T h e o r e m 2. Given a matrix T E C L x M with SVD (1) and LRR URV decomposition (2). Using the notations of the T U R V V algorithm, we have:

Moreover, if wi = Vmax(Ri) exactly, i = 1 , . . . , K , then (I . I denotes the absolute value)"

] D l= El, F = 0, G = RK+x; V1 = V1 and Ux = U1, up to a factor ~.

See [7, Theorems 2 and 3] for proofs. The appearance of (I) in Theorem 2 is due to the fact that the singular vectors of a complex matrix T are only defined up to a scaling factor of the form e ar A similar theorem can be proven for TURVU. Theorem 2 clearly shows that the LRR URV decomposition is not able to improve the initial signal subspace estimate based on the computed start vectors wi. This is even more clear when one estimates the K first right singular vectors of T corresponding to a l , . . . , aK at once and then uses the columns of this start matrix, say Wstart, one by one as estimates of Vmax(Ri), i = 1 , . . . , K. In this case, the transformations Q(i), applied to the consecutive R (i) during formation of the LRR URV decomposition, must also be applied simultaneously to Wstar t. Under these conditions, it is easy to prove [7, Theorem 4] that

~ ? ~ v = W~t~rta = W~t~rtdi~g(~-~, . . . , ~-~') i.e., the LRR URV subspace estimate is exactly the same as the space generated by the start vectors used for the LRR URV computation. Moreover, this decomposition is also costly and at most one order of magnitude faster than the SVD. Further refining the obtained LRR URV decomposition, as described in [8], improves the signal subspace accuracy but this refinement is also costly, especially when K is not very low. Therefore, the use of the TURVV method for signal subspace fitting is only recommended in those applications where the rank of T must be revealed (e.g. when K is not fixed) or whert this subspace needs to be up- or downdated continuously. Note that all theorems presented here apply for general (not necessarily Toeplitz) matrices.

If K is fixed, which is often the case in NMR data quantification, then an iterative method is mostly a more efficient alternative.

3 I T E t t A T I V E S I G N A L S U B S P A C E E S T I M A T I O N

A basis of the signal subspace 7~(V1) can be estimated efficiently by first performing K iterations of the Lanczos bi- (or tridiagonalization) method [5] on T and then further refining this estimate by applying n more iterations of the block power method [5], as outlined below.


A l g o r i t h m : ITER

Given . T E C L x M Toeplitz of numerical rank K, L > M > > K and the number n of desired block power iterations�9

1. Apply K Lanczos recursions: Bi = U(O*TV(O =

al Z2 0

, i = l , . . . , K t o

0 CXi

obtain the first K right Lanczos vectors V(K) (making use of the FFT when computing matrix-matrix products of the form T X or T ' X ) .

2. Refine V (g) by n iterations of the block power method. For i = K + 1 , . . . , K + n do

V (i) ~ T*(TV6-1)) making use of the FFT twice. orthonormalize V (i) using a modifed Gram-Schmidt method

end for V (g+n) is a basis estimate for the signal subspace T~(V1).

Orthogonality of the generated Lanczos vectors in step 1 is only guaranteed in exact arithmetic while it fails to hold in inexact arithmetic when spurious singular values of T start to appear in Bi. The cure is to use selective or complete reorthogonalization or to identify the spurious singular values [5].

This algorithm performs best if N = 2 p is a power of 2. If not then 2 p - N zeros should be appropriately inserted in the original data sequence. The benefits of the use of the FFT are already apparent from N = 32 on. The total complexity of the algorithm ITER (including complete reorthogonalization in step 1 as done in this paper) is O(nKp2 p + n M K 2) flops.

4 A P P L I C A T I O N S

The efficiency and accuracy of the presented methods TURVV and ITER are compared with the classical SVD in the following exponential data modeling problem encountered in NMR spectroscopy�9 Suppose N uniformly sampled data points Yn are given and are to be fitted to the following model function:

K K Yn = ~ CkZ~ n+6) = Z(ake~r (-dk+~2~rfk)(n+6)/xt, n = 0 , . . . , N - 1 (3)

k=l k=l

where K is the model order, At is the constant sampling interval, and ~At is the time lapse between the effective time origin and the first sample. The objective is to estimate the frequencies fk, damping factors dk, amplitudes ak, and phases Ck, k = 1 , . . . , K.

To take advantage of the presented fast Toeplitz algorithms, the parameters are estimated as follows. First arrange the N data samples y,~ in an L x M Toeplitz matrix T, N = L + M-.. 1, with first column [YL-1,..., Yo] T and first row [YL-1,..., YN-1], then estimate a basis V1 of the K-dimensional right singular subspace 7~(V1) of T. From V1, a set of linear


4 0 0

3 5 0 4 3

3 0 O l

2 5 O 5

5 0

1 5 0 0 0 0 0 ~ 0 0 0 - S O 0 - 1 0 0 0 - X S O 0 - 2 0 0 0

O ' i /

0.1 0.2 0.8

ratio a K.t.1

21.3119 24.55

k method 4 HSVD

TURVU TUttVV

k A d~ ak Ck 1 -1379 208 6.1 15 2 -685 256 9.9 15 3 -271 197 6.0 15 4 353 117 2.8 15 5 478 808 17.0 15

9.9642 2.2053

(b)

12.45 10.87

5 HSVD TURVU TURVV

(a)

fk(Hz) dk(rad/s) ak(a.u.) Ck (deg.) -2.534-9.3 38.94-69 0.744-1.21 7.414-20 -2.374-10 57.94-110 1.104-2.62 8.684-21 -2.654-9.8 48.14-90 0.894-1.55 8.154-21 7.864-19 8.154-21 7.824-20

31.94-119 -0.434-1.87 -2.394-6.7 62.84-240 -0.534-2.03 -3.68:J::10 40.54-144 -0.494-1.92 -2.864-7.8

(c)

Table 1: (a) FFT and exact parameter values of a simulated 31p NMR signal modeled as in (3). Ck=r (b) Numerical illustration of Theorem 1 using the first 128 data points of (a) with added white Gaussian noise of variance 2a~ and arranged in a 65 • 64 matrix. The start vectors wi used in the algorithms TURVV and TUI~VU are such that the stated condition is satisfied, ratio=dist(T~(~'lturvu), T~(V1))/dist(T~(v~urvv), T~(V1) ). (c) Bias-t-standard deviation values of the parameter estimates of peaks 4 & 5 when noise with au = 1.5 is added, a.u.=arbitrary units. Start vectors wi for TUttVV and TUttVU are equally well computed via 5 power iterations. HSVD, TUItVV and TUttVU failed to resolve peaks 4 and 5 in respectively 4%, 7% and 7% of the 1000 runs performed.

equations VITX: ~, V1 ~ is derived where ~-T (resp., ~1) consists of the top (resp., bottom) M - 1 rows of V1. From the least squares (LS) solution X*ls of this set, estimates of the

nonlinear parameters are derived from the eigenvalues "zk = e(--dk+32~r'fk)A~ of Xls. Then, the linear parameters Ck, which contain the amplitudes ak and phases Ck, are estimated as the LS solution of a set Ac ,,~ b obtained by fitting the N model equations (3) to the data samples y,~ (and replacing each zk by its estimate zk). It is easy to show that this method is equivalent to Kung et al.'s state-space based method [6], called HSVD in NMR spectroscopy [3], which arranges the data in a Hankel matrix H and uses the K left singular vectors U1 of H. An improved variant, based on total least squares, is presented in [12].

Numerical tests performed with Matlab show that the increased accuracy of the computed TURVV signal subspace estimates -as proven in Theorem 1- also leads to a similar increase in the accuracy of the derived signal parameter estimates (in terms of bias and variance), as compared to TURVU (see Table l(b)).

At comparable parameter accuracy, it is shown that TURVV (resp., ITEP~) computes the estimate T~(V1) of the signal subspace typically 8 to 9 (resp., 20 to 30) times faster than HSVD for reasonable signal-to-noise ratios. For smaller ratios (around 0 dB) the gain


5000 . . . . . . . . .

4500 original signal HTLS residual --

4000 HSVD residual : ]

I 3500

300O

1500 ~'r NAA

1000

500 400 300 200 100 0 -100 -200 -300 -400 -500

Frequency (Hz)

K Rturvv niter

9 1 2 10 1 2 11 1 2 15 4 4 16 6 5

9.3 30.3 8.3 26.0 7.6 22.8 4.2 10.7 3.4 8.7

Figure 1: Ratio of the number of flops (in Matlab) needed by HSVD to that required by the algorithms TURVV and ITER (including parameter estimation) for fitting, reconstructing and removing the water peak (> - 70 Hz) of the 1H NMR signal (full line) prior to estimating the peaks of interest in the residual signal (dashed line). K denotes the chosen model order and n denotes the number of power iterations used in TURVV and ITER to achieve comparable parameter accuracy. The water peak (with modulus up to 40.000) has been cut off in order to visualize the small peaks of interest. N = 256 and L = M + 1 = 129.

in efficiency is less pronounced. This subspace computation is the computationally most intensive part of the estimation procedure. As an example, consider again the simulated alp NMR signal, as described in Table 1. For 0.5 _ a~ _< 2, this subspace calculation represents 98% of the computational work performed by the HSVD algorithm, while the execution of TURVV (using 2 power iterations) represents 88% and that of ITER (with n = 2 to 4) only 70 to 80% of the work involved in estimating the parameters.

Efficiency is of primary importance in medical MR spectroscopic imaging when disturbing water/fat contributions are to be removed from the signals in order to produce high-quality metabolite images [3]. Typically, these peaks can be well fitted by very few exponentially damped sinusoids (i.e. K << M). A single metabolite image may require as many as 32 • 32 = 1024 times removal of unwanted signals. In these applications, the fast algorithms presented here yield significant time savings, as shown in Figure 1.

Acknowledgemen t s

We wish to thank Rick Fierro for introducing us the LRR URV/ULV factorizations and providing us Matlab code of the LRR URV algorithm based on estimates of Um~x(Ri). Beyond this, the work described here was performed independently of the results reported in [4], since that report was not available to us. We are also grateful to Haesun Park, Lars Eld~n and Jim Nagy for providing us part of the Matlab code. The first author is a Research Associate with the N.F.W.O. (Belgian National Fund for


Scientific Research). This paper presents research results of the Belgian Programme on Interuniversity Poles of Attraction (IUAP-nr.50 and 17), initiated by the Belgian State, Prime Minister's Office for Science, Technology and Culture. The scientific responsibility rests with its authors.

References

[1] A. Bojanczyk, R. Brent, and F. de Hoog. QR factorization of Toeplitz matrices. Numer. Math. 49, pp 81-94, 1986.

[2] J. Chun, T. Kailath, and H. Lev-Ari. Fast parallel algorithms for QR and triangular factorization. SIAM J. Sci. Stat. Comput. 8, pp 899-913, 1987.

[3] R. De Beer and D. Van Ormondt. Analysis of NMR data using time-domain fitting procedures. In : M. Rudin (guest Ed.), In-vivo Magnetic Resonance Spectroscopy I: Probeheads, Radiofrequency Pulses, Spectrum Analysis, NMR Basic Principles and Progress series 26, Springer-Verlag, Berlin Heidelberg, pp 201-248, 1992.

[4] R.D. Fierro and P.C. Hansen. Low rank revealing two-sided orthogonal decomposition algorithms. CSUSM PAM technical report 94-06, California State University, San Marcos, CA, U.S.A., in preparation, 1994.

[5] G.H. Golub and C.F. Van Loan, Matriz computations (2nd ed). The Johns Hopkins Univ. Press, Baltimore, MD, 1989.

[6] S.Y. Kung, K.S. Arun and D.V. Bhaskar Rao. State-space and singular value decomposition-based approximation methods for the harmonic retrieval problem. J. Opt. Soc. Am. 73, pp 1799-1811, 1983.

[71 P. Lemmerling and L. Vanhamme. Effici~nte algoritmen voor medische NMR beeld- vorming en spectroscopie. M. Eng. thesis, Katholieke Universiteit Leuven, 1994.

Is] R. Mathias and G.W. Stewart. A block QR algorithm and the singular value decomposition. Lin. Alg. Appl. 182, pp 91-100, 1993.

[9] H. Park and L. Eld~n. Fast and accurate triangularization of Toeplitz matrices. Preprint 93-038, AHPCRC, Univ. of Minnesota, 1993.

[10] H. Park, S. Van Huffel and L. Eld~n. Fast algorithms for exponential data modeling. Proc. 1994 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), April 19-22, Adelaide, Australia, Vol. 4, pp 25-28, 1994.

[11] G.W. Stewart. An updating algorithm for subspace tracking. IEEE Trans. Signal Proc. 40, pp 1535-1541, 1992.

[12] S. Van Huffel, H. Chen, C. Decanniere and P. Van Hecke. Algorithm for time-domain NMR data fitting based on total least squares. J. Magnet. Reson., series A, Vol. 110, Sept. 1994 (to appear).


199

A B L O C K T O E P L I T Z L O O K - A H E A D S C H U R A L G O R I T H M

K. GALLIVAN Coordinated Science Laboratory University of Illinois Urbana, IL USA gallivan @csrd. uiuc. edu

S. THIRUMALAI Coordinated Science Laboratory University of Illinois Urbana, IL USA srikanth @csrd. uiuc. edu

P. VAN DOOREN Cesame Universitd Cathotique de Louvain Louvain-la-Neuve, Belgium vandooren@anma, ucl. ac. be

ABSTRACT. This paper gives a look-ahead Schur algorithm for finding the symmetric factorization of a Hermitian block Toeplitz matrix�9 The method is based on matrix operations and does not require any relations with orthogonal polynomials�9 The simplicity of the matrix based approach ought to shed new light on other issues such as parallelism and numerical stability.

KEYWORDS. Block Toeplitz matrix, Schur algorithm, numerical methods, look-ahead.


The Schur algorithm yields a method to compute the symmetric decomposition

T = U*DU, U upper-triangular

of an (n + 1) • (n + 1) Hermitian Toeplitz matrix

to tl �9 �9 �9 tn m �9

T = tl to "�9 tn-1 �9 � 9 � 9 1 4 9 1 4 9 [

t. t~-1 . . . to

(1)

(2)

200 K. Gallivan et al.

in O(n 2) operations [S]. This algorithm actually derives this decomposition for all leading principal submatrices as well via a simple vector recurrence, which explains the low complexity of the method. Another well known algorithm for the same problem is the Levinson algorithm [10]. Yet, the Schur algorithm has gained a lot of popularity over the Levinson algorithm for various reasons : (i) it is known to be better suited for fine grain parallelism [9], (ii) it constructs the factor U in (1) directly, rather than its inverse as in the Levinson algorithm [8], (iii) it exploits better matrix properties such as bandedness and low rank [5] and (iv) it has been shown to have better numerical properties for positive definite T [1].

Both algorithms, though, are known to be potentially unstable when T is indefinite. This is the case when the leading principal minors of T are (nearly) singular, since both algorithms implicitly use these submatrices in their recurrence. Remedies for this were proposed for the Schur algorithm [11] and for the Levinson algorithm [2] and were essentially based on a look-ahead technique, whereby one "jumps" over the singular submatrices. Although this requires a slight increase in complexity, this is in general quite an effective technique. These techniques are linked to the theory of orthogonal polynomials and can become quite involved in the case of look-ahead [6], [3]. In this paper we present a matrix based derivation of such a look-ahead method and give the algorithm directly for block Toeplitz matrices. This extension is quite easy because of the use of matrix manipulations rather than orthogonal polynomials.

2 SCHUR COMPLEMENTS AND DISPLACEMENT RANK

Let T be a general Hermitian, block Toeplitz matrix of dimension N x N and block size m x m, i.e.

To T~ . . . T~

T = T~' To "'. Tr-x , To = T~, N = m x (r + 1). (3)

T; T;:I . . . To

The purpose here is to find a factorization as in (1) where D is diagonal or block diagonal and U is upper triangular or upper block triangular�9 Schur type algorithms are based on the concept of displacement, which is defined as follows. Choose Z to be the block right shift matrix of the same dimension �9

0m Im

Z = . (4)

�9 " , ITr~

0m

The displacement rank of the matrix T is then defined as �9

a = rank ( T - Z * T Z ) <_ 2 m , (5)

and the displacement of the matrix T can therefore be factored as �9

T - Z * T Z = G*EG (6)

A Block Toeplitz Look-ahead Schur Algorithm 201

where the a x a matrix E equals i"

= [ I ~ - • , p , q < m . E (7)

This factorization can be automatically written for block Toeplitz matrices. For arbitrary matrices satisfying (6) with a << N, the factorization can be obtained from the Bunch Kaufman decomposition or also from the eigen decomposition of T - Z*TZ. Matrices with such a low displacement rank are said to be quasi block Toeplitz. The complexity of this preliminary decomposition is normally O(aN2).

It is well known that factorizations of the type (1) are working on Schur complements of the original matrix at each stage of their recurrence. We now derive updating formulas for the Schur complement of a matrix T with low displacement rank, and show that it also has low displacement rank. This part is related to the work of [7] as was pointed out to us, but is not contained in it. Partition T and Z conformally as :

T = T21 T22 ' 0 Z22 ' (8)

where Tn and Zn are of dimension mk • mk (a multiple of the block size) and Tll is assumed to be invertible (this is always possible by choosing k large enough). Define

X = T~lT12 , X* = T~2T~ 1, U= [ I I X , ,, , (9)

then it follows that

v . r v : [ T~ ] , r .~:T:~- �9 -1 T{2T{I T12, (10) T,~

where Tsc is the Schur complement of T with respect to Tll. Applying U*( . )U to (6) yields �9

U*TU - (U*Z*U-*) U*TU (U -1ZU) : U*G*EGU. (11)

Notice that

Using (10) and (12) we can reduce (11) t o :

r n Tsc ] 2h Z22 ] [ Tll] ] Zl1212 Z22 . (13)

Equating the (1,2) and (2,2) positions in the above equation we have

[:] AT,~ T,~ * Z - X * G* : - z~m~ ~ : z~T,l~l~ + [ l• ] ~a . (14)


Substituting for 2x2 from (12) we can further simplify M and ATsc to �9

Substituting for X in the matrix in the middle of the above equations we get

w z I~ ] T~ 2 T~ll [ Tll l T12 ] -~ G* ~ G

This expression can now be further simplified to prove that the rank of ATsc is at most ~. In order to prove this we first need the following lemma.

L e m m a 1 Let

~=[~ ~][~o ~ o] ~ ~ Fr2 F~2 ~2 F21 F22 (IS)

where E1 and Wll = F~lElF11 -4- F~l~2F21 are invertible. Then there always exists a transformation H such that

o ~ r,~ (19)

H F21 F22 = 0 F22

Proof . Let H = RQ, where R is block upper triangular and Q is unitary. We choose Q such that

Q F2~ = 0

Fll is full where B is upper triangular. Moreover, since Wll is assumed invertible, F21

rank and hence B is invertible as well. Let R be partitioned conformally with W as

R = [ RnO R22R12] , (22)

then H automatically satisfies (20) and Rxl B = Fll. Also, H will satisfy (19)if and only if

[~10][~o 0 [~1~ [~ 0]~ ~, ~;2 ~ 2 ~2 0 R22 = Q ~2 '


where the right hand side is now known. A decomposition of the type R*~R is known to exist iff the (1,1) block of the right hand side is invertible. Because of (21) this equals B-* (F~I ~I Fll + F~I ~2F21)B -1 which is invertible. []

In order to simplify (17) we now want to apply this lemma to construct a transformation H such that

H * ~1~1 ~ ] H = Till I ~ , (24)

i-/ [r~ r . ]z ~ ~" (25) G = 0 G2 '

where Tll and ~bll are matrices of size mk x ink, G has dimensions a x N and G2 has dimensions a x (N - ink). In order to apply the above lemma we only need to show that Wll is invertible since Tll is invertible by assumption. From (13) it follows that

Tll = Z~1T11Z11 + G~ZG1 ,

From (17), Wll equals

and since

Z = [Zl l Z I , ] 0 Z22 '

we have

w~ere C~-C[~] . (~6~

[ G = [G1 G2 ] , (28)

Wll = Z~ITll TI~ITll Z l l + G ~ G 1 = Tll ,

which thus shows that Wll is invertible as well.

(29)

Applying (24) and (25) to (17) we obtain

o o I TG a~ 0 s o

Inserting this in (15) and (16)yields

a2

~ - i , , o l ~ [ - ~ ~ :~1~1~1i~1~][-~ ~ :o

] T~:

(ao)

(31)


S i n c e M - Oand'~n aadTll areinvertible, wehave [T111Tla ] [ i X = 0, which

yields,

AT.e = (~i~(~2. (32)

This establishes a new displacement identity where ~ and (~2 are obtained from (24-25).

3 B L O C K L O O K - A H E A D A L G O R I T H M

The above construction thus suggests the following algorithm for block Toeplitz matrices.

A l g o r i t h m 1 Block-Toepl i tz

Construct generator ~(o), G(o) Use Bunch Kaufman on To to obtain To = U~EoUo and define

0 - 2 0 ' G(ol-" 0 EoU;-*T~ . . . 2oUo*T~

Apply block Schur algorithm

i = l

while Gi-1 # void Construct leading rows [TIIIT~2 ] from E(i_x), G(i-x) until Tll is well conditioned Compute Ui = Tn 1 [TnlT12] = [I IX ] Append Tll to D and Ui to U Apply a transformation H satisfying

0 ~(i) 0 such that

H[ [Tll T12 ]Z -- ~%11 ~%12 G(i-1) 0 G(i)

Increment i end while

0 E(~_~)

This algorithm is of course only conceptual. It does not describe how to construct the transformation H nor how to track the condition number of Tll. For the latter we refer to techniques as those described in [2, 6, 3]. For the construction of the Z]-unitary transformations H we can use skew Householder transformations of block versions of them (see [4]). In [4] issues of efficient parallel implementation of such transformations are also adressed. We point out that when Tll is well conditioned then the transformation H and its construction should give no numerical problems. It should be pointed out that the first

columns of t[ Tn Tn ] Z are zero which can be exploited in the factorization m

H G(i-1) 0 G(0


This is especially the case when there is no look-ahead needed (i.e. when k = 1). The above matrix has then 3m rows of which only 2m have to be processed. One checks that this economical version is precisely the usual block Toeplitz algorithm without look-ahead. The complexity of this method is O(m2N 2) in the best case (i.e. without need for look-ahead). With look-ahead of moderate size this will increase slightly as a function of k.

Finally, notice also that the results presented in this paper can be extended to the non Hermitian case, provided two generator are kept and updated instead of one. For simplicity, we did not develop this here.

Acknowledgments

We are indebted to V. Olshevsky for pointing out reference [7] to us. This research was supported by the National Science Foundation under Grants CCR 9209349 and CCR9120008 and by ARPA under Grant 60NANB2D1272

References

[1] A. Bojanczyk, R. Brent, F. de Hoog and D. Sweet. On the Stability of the Bareiss and Related Toeplitz Factorization Algorithms SIAM J. Matrix Anal. Appl. 15, pp , 1994.

[2] T. Chan and P.C. Hansen. A look-ahead Levinson algorithm for indefinite Toeplitz systems. SIAM J. Matrix Anat. Appl. 13, pp 490-506, 1992.

[3] R. Freund and H. Zha. Formally biorthogonal polynomials and a look-ahead Levinson algorithm for general Toeplitz systems. Linear Algebra and its Applications 1 8 8 - 1 8 9 ,

pp 255-303, 1993.

[4] K. Gallivan, S. Thirumalai and P. Van Dooren. On solving block Toeplitz systems using a block Schur algorithm, in Intern. Conf. Parallel Processing, Proc. ICPP-94, St. Charles IL, pp III-274-III-281, 1994.

[5] K. Gallivan, S. Thirumalai and P. Van Dooren. Regularization Teoplitz least squares problems. Int. Rept. CSL, Univ. Illinois at Urbana-Champaign, 1994.

[6] M. Gutknecht. Stable row recurrences for the Pad~ table and generically superfast look- ahead solvers for non-Hermitian Toeplitz systems. Linear Algebra and its Applications 188-189, pp 351-421, 1993.

[7] T. Kailath and A. Sayed. Fast algorithms for generalized displacement structrures, in Recent Advances in Mathematical Theory of Systems (H. Kimura, S. Kodoma, Eds.), Proc. MTNS-91, pp 27-32, 1992.

[8] T. Kailath, A. Vieira and M. Morf. Inverses of Toeplitz operators, innovations and orthogonal polynomials. SIAM Review 20, pp 106-119, 1978.

[9] S.-Y. Kung and Y. Hu. A highly concurrent algorithm and pipelined architecture for solving Toplitz systems. IEEE Trans. ASSP 31, pp 66-76, 1983.

206

[10] N. Levinson. The Wiener RMS (root-mean-square) error criterion in filter design and prediction. J. Math. Phys. 25, pp 261-278, 1947.

[11] D. R. Sweet. The use of pivoting to improve the numerical performance of Toeplitz solvers, in Advanced Algorithms and Architectures for Signal Processing (J.M. Speiser, Ed.), Proc. SPIE 696, pp 8-18, 1986.


207

T H E S E T O F 2 - B Y - 3 M A T R I X P E N C I L S - K R O N E C K E R S T R U C T U R E S A N D T H E I R T R A N S I T I O N S U N D E R P E R T U R B A T I O N S - A N D V E R S A L D E F O R M A T I O N S O F M A T R I X P E N C I L S

B. K.~GSTR6M Department of Computing Science Umed University S-901 87 Umed, Sweden [email protected]

ABSTRACT. The set (or family) of 2-by-3 matrix pencils A - ,~B comprises 18 structurally different Kronecker structures (canonical forms). We review the algebraic and geometric characteristics of the generic and the 17 non-generic cases. Moreover, we derive a graph describing the closure hierarchy of the orbits of all 18 different Kronecker structures for the set of 2-by-3 pencils. By labeling the nodes in the closure graph with their geometric characteristics and the vertices with the change in geometric characteristics for transiting to an adjacent node, we get a labeled graph showing necessary conditions on perturbations for transiting from one Kronecker structure to another. We also mention some results from an experimental study of how the non-generic Kronecker structures behave under random perturbations using the GUPTRI software (which is based on the singular value decomposition (SVD) for rank decisions). Assuming a fixed relative accuracy of the input data structure invariances and transitions of each non-generic case is studied as a function of the size of the perturbations added. For large enough perturbations all non-generic pencils turn generic (as expected). Some non-generic cases transit between several non- generic structures before turning generic. These transitions always go from higher to lower codimensions, along the arcs in the closure graph.

In [2] we also present computable normwise bounds for the smallest perturbations (~A, ~B) of a generic 2-by-3 pencil A-)~B such that (A+$A)-)~(B -F~B) has a specific non-generic Kronecker structure. Two approaches to impose a non-generic structure are considered. First explicit expressions for the perturbations that transfer A - ,~B to a specified non- generic form are derived. When deriving these, the SVD is frequently used. In this context tractable and intractable perturbations are defined. Secondly, a modified GUPTRI that computes a specified Kronecker structure of a generic pencil is used. In the first approach we compute a perturbation (~A, 5B) such that (A + 5A)- ,~(B + 5B) is guaranteed to be on

208 B. KdgstrOm

the closure of the manifold of a certain KCF. If the KCF found is the intended KCF, then the perturbation is said to be tractable. If the KCF found is even more non-generic then the perturbation is intractable. An intractable perturbation finds some other structure within the closure of the manifold, i.e., a structure that can be found by travelling along the arcs from the intended KCF in the closure graph. A summary of these perturbations will be presented in a perturbation graph, where the path to each KCF's node shows the tractable perturbation required to find that KCF starting from the generic KCF (an L2 block). Somewhat counter-intuitively, it is the intractable perturbations, which impose the most non-generic structure (with highest codimension) for a given size of the perturbations (e.g. the relative accuracy of the data), that are requested in applications (e.g. computing the uncontrollable subspace).

In the second part of the talk we will present some (of several new) results on general m x n matrix pencils A - $B from [1]. First, we show that the codimension of the o r b i t ( A - $B) can be expressed as the number of zero singular values of a matrix Z, which is a matrix representation of the normal space of A - AB. Finally, if the time permits, we will present a normal form for matrix pencils to which not only one specific matrix pencil, but an arbitrary family of matrix pencils close to it can be reduced to by means of a mapping smoothly depending on the elements of the matrix pairs. This form is a complete generalization of the normal form derived by Arnold for unsymmetric matrices.

The first part is joint work with Erik Elmroth [2], and the second part is joint work with Alan Edelman and Erik Elmroth [1].

References

[1] A. Edelman, E. Elmroth, and B. Ks A Versal Deformation Approach to Per- turbation Theory of Matrices and Matrix Pencils. Report UMINF-94.13, Department of Computing Science, University of Umes S-901 87 Umes Sweden, 1994. To appear.

[2] E. Elmroth and B. Ks The Set of 2-by-3 Pencils - Kronecker Structures and Transitions under Perturbations. Report UMINF-93.22, Department of Computing Science, University of Umes S-901 87 Umes Sweden, 1993. To appear in SIMAX.


209

J - U N I T A R Y M A T R I C E S F O R A L G E B R A I C A P P R O X I M A T I O N A N D I N T E R P O L A T I O N - T H E S I N G U L A R C A S E

P. DEWILDE DIMES, Delft University of Technology POB 5031, 2600GA Delft The Netherlands P. Dewilde @cas. et. tudelft, nl

ABSTRACT. The teachability space of J-unitary or symplectic matrices plays a crucial role in the theory of constrained interpolation and approximation of matrices and operators. For the standard solution to exist, it is necessary that the Pick operator connected to the space is non-singular. However, solutions are also possible when the Pick operator is singular. They consist of a J-unitary operator cascaded with a J-isometric diagonal operator which partially reduces the overall reachability space. In this paper we give a complete theory how such singular solutions can be constructed. Its importance derives from the fact that the optimal approximant usually corresponds to a singular Pick matrix.

KEYWORDS. Interpolation, approximation, J-unitary matrices, Pick matrices.

1 INTRODUCTION

Matrices and operators can be approximated through interpolation. There are two main techniques for this: one uses a specification for the structure of the approximant and/or its inverse, and is based on generalized Schur recursions, the other does not use prior knowledge and finds an approximant of minimum complexity, given a desired accuracy in a special norm, dubbed the 'Hankel norm'. Both methods are based on a transposition of results in complex function calculus to 'calculus on diagonals' of a matrix or operator, pioneered in [3, 5, 6]. The gist of both methods is the determination of a J-unitary operator with a given 'teachability space' specified by the data. Associated to this proces which is sometimes called 'embedding', there is a Lyapunov-Stein equation which has a Pick operator as its solution. If that Pick operator is non-singular, then the embedding has a unique minimal solution up to a constant factor. In this paper we first introduce the logic that leads to the Pick operator, and next, we examine what happens when the Pick matrix (or operator) is

210 P. Dewilde

singular�9 The results of this analysis are' (1) there always exists a J-unitary embedding, (2) the embedding can not always be reduced to minimal size through a 'non-dynamic' passive load, (3) in case of positive semi-definiteness, the minimal reduction always exists.

2 P R E L I M I N A R I E S

Following our earlier papers [8, 10], we view a matrix as a 'computational model' for linear transformations. A succinct description is as follows (the 'empty' set ~} is a valid space of dimension zero):

Input Space: a sequence of spaces r = [ ' " , • - 1 , ] • 0 ] , • 1 , ' " ]

O u t p u t Space" a sequence of spaces A/'= [ " ' , A f - l , ~ , A f l , " ' ]

The transfer operator T �9 12 ~ --. 12 H �9 u ~ y = uT bounded, with block-matrix

(tableau) representation T = �9 .�9 T - l , - 1 T - l , O T-1,1 � 9

. . �9 0 ]To,0[ T0,1 . . . I ]

i i

. . . 0 0 T1,1 �9 �9 �9 �9 �9 �9

Note that we mostly use a row-vector matrix multiplication to indicate the transfer. The matrix or operator T will be causal if the tableau is upper triangular. Hence we distinguish

upper-triangular (or causal): H the following types of operators: diagonal (or instantaneous)" T~

lower-triangular (or anticausal)" ~3

Diagonal operators have special interest since they represent non-dynamic or instantaneous input-output transformations, when the row indexing of the operator is interpreted computationally as sequencing in time.

Associated to the transfer matrix or operator is what we shall call the Hankel Operator:

Def in i t ion 1 The Hankel Operator associated to the transfer operator T is a collection of operators {Hk} , for each point k in the sequence, defined by the map:

Hk -" U[support<k ~'+ uT[support>k

The Hk's form a collection of matrices gleaned from the transfer function - see fig. 1. (The collection can actually be interpreted as an operator from an extended input space to an extended output space with Hilbert-Schmidt metrics - see [8] for details.) For each Hk, we define the following important spaces:

(.Ok = row-span = range of Hk = observability space 7~k = {column-span}* = corange of ilk = teachability space

The study of the reachability or observability spaces is crucial to the solution of interpolation problems, so we pursue it a little further. Let Q k be a (eventually orthonormal)

basis for the k th reachability space, and let us stack the bases, keeping track of the position

J-unitary Matrices for Algebraic Approximation 211

. . . . . . . . . . . . . .' . . . . . . . . . . . . . . _ * - * _ 1 _ _ _

T:,: T:,: "'" Ha

T 2 , 2 . . .

Figure 1" The collection of Hankel matrices for a given transfer matrix

[ q - 1

go ql qa

Figure 2: The stacked reachability spaces

of the elements in the sequence, as shown in fig. 2.

Due to the ordering of the Hankel matrices (see fig. 1), a 'restricted shift invariance property' must hold between the subsequent reachability spaces, namely (wherein V indicates the span of a collection vectors and II a projection matrix on the specified subspace):

V c V (:) A realization for T is a local computational scheme which gives for each stage in the computation the instantaneous map"

Ak Ck ] state, input --+ next state, output �9 [xk,Uk] ~ [Xk+l,Yk] = [Xk, Uk] Bk Dk "

Since the computations in the state space realization represent instantaneous linear maps, they have a logical representation as diagonal matrices:

A - diag { . . . , A _ : , [ - ~ , A I , . . . } ,B "- diag {. . . ,B_: ,[-B-7],B1, . . .} ,etc...

To obtain the formal representation of the transfer operator in terms of the realizing matrices, we define the causal shift:

Z = [Zi,j] : Zi,j = I whenj = i + 1, otherwise Zi,j = O.

Z maps a sequence of spaces of the type

[. . . , .A4_:,I.Mol,.AdI,. . .] to [ . . . , .hd_2,l .hd_:l , .A4o,. . . ] .

Assuming ( I - AZ) invertible in some sense, the resulting representation is: T = D + B Z ( I - AZ)- : C, in which the operators {A, B, C, D} are all diagonal. If the operator ( I - AZ) is boundedly invertible, then the following representation for the tableau of reachability

212 P. Dewilde

1 i i _ b l _ b ~

- I C "

Figure 3: The direction of energy flow splits inputs and outputs

spaces exists:

, . ~

�9 .. B*_ 1 o Q = ( I - Z'A*) -1Z-B• = ... A~B* 1 B~

A * A * ~ * * * "'" --1"0---1 A 1B 0 B~ . , , , , 0 , . 0 . . , . o .

Hence, it is clear that a realization can be based on any basis chosen for Q. If it is chosen orthonormal, then the realization is in input normal form:

A * A + B * B = I (2)

STATE SPACE TRANSFOItMATIONS

If D is a diagonal operator, we denote by D(-1) a diagonally one notch shifted up version of D: D (-1) = d i a g { . . . , D - 1 , D 0 , ~ . . . } The state transformation uses a non-singular (i.e. one-to-one, onto or invertible) transformation R, giving:

B D ~ B(R(-1)) -1 D " (3)

It can be put in input normal form through a judicious choice of It. This process will lead to a Lyapunov/Stein equation - see [8].

J-UNITARY OPERATORS

A further refining associates to input, state and output sequences mixed inner products. For that, we decompose the input and output spaces in two components [al bl], [a2 b2] respect, with al G l ~ 1, bl G l ~ 2, a2 E l~ 1, b2 E 12 H~, and define signature matrices:

"11=[ IM1 -I.~2 f~ the input J 2 = [ I H I ' -IH2 ] for the output space.

We define (indefinite) inner products on these spaces by:

II [~ b~] I1~,= [~ b~]J~[~ b~]" = ~ a ~ - b~b;, i = 1, 2

An inner product carries the interpretation of 'energy' with it: positive when 'incoming', negative when 'outgoing' or reflected. The direction of the energy flow should not be confused with the direction of the signal flow, which is a matter of choice. Fig. 3 shows our pictorial convention for the energy flow.

However, not only the input/output signal flow can be interpreted as flow of energy, much the same is true of the flow of the state. Let us decompose the state x in two sequences:


[x+ x_] with x+ E/~+, x_ E l~-, and define a signature matrix for the states:

jB = IB+ ] --1/3_

The energy transported by the state is then accounted for as follows:

II [~+ ~-] JJ~= ~+~ - ~ - ~ :

Traditionally, signals that carry energy given by an/2-norm are called 'waves', the place where they impinge on a 'medium' is called a 'port', and the operator which maps signals at a left port to signals at a right port is called a 'chain scattering operator' O:

O: [al bl] ~ In2 b2].

If energy is conserved, then 0 will be isometric: O J20" = ,/1, and will be J-unitary, if in addition, {9* J10 = J2.

Let 0 be a realization operator consisting of a block-matrix of diagonal operators:

0 = B1 Dll D12 , (4) B2 D21 D22

assume that I - AZ is boundedly invertible (i.e. the spectral radius a(AZ) < 1), and that 0 is J-unitary in the sense,

0* J8 0 = 0 = J~ J2 ' J2 J1 '

then the corresponding transfer operator 0 is J-unitary. A qualified converse of this property is valid: suppose that a bounded causal operator 0 has a finite (bounded) state space realization, then the latter can be chosen J-unitary in the sense of the two preceeding equations.

Next, we give a brief survey of the major classical interpolation problems, converted to the new algebraic setting. For a more thorough account, we refer to the literature, see especially [7].

ALGEBRAIC NEVANLINNA-PICK AND HERMITE-FEJER PROBLEMS

Given: data pairs of diagonal operators {Y~, si) : i = 1 . . . l Asked: a causal, contractive transfer operator S which is such that ( Z - V~) - I (S - 8i) is causal as well.

There is quite a difference between the strict solution when S is required to be strictly contractive, and a solution where S is allowed to be partially unitary as well. The solution to the interpolation problem may be formulated in terms of the reachability space of a J-unitary chain-scattering matrix O. It is given by:

214 P. Dewilde

The obvious choice for the A and B operator of the corresponding O matrix is given by:

E ~

A __ "~176 ' B 2 :

I , � 9 I

81 � 9 8 n

For the algebraic Hermite-Fejer problem, which is an extension of the previous case to higher multiplicities, the choice is as follows:

A

V~ I

v;( l I

I V~(k-~)

B1

B2

V~ I

~ o . . . o ~ o . . .

THE MODEL REDUCTION PROBLEM

Here we start out with a high order model for a given T, in output normal form (for convenience, we assume T to be strictly causal), T = B Z ( I - A Z ) - I C with AA* -4- CC* = I. Let, furthermore, be given a hermitian, diagonal matrix r , which interprets as the 'precision matrix' (normally this would be a scalar constant e, but here we have more freedom). The problem is to find a bounded, but not necessarily causal operator T' such that: IIF-~(T- T')II < 1.

The problem is transformed into an interpolation problem as follows" let B: and D: be diagonal operators defined by the condition that the block matrix of diagonals

Bu D~

is unitary (this condition is actually a unitary condition on each individual state transition matrix build from the diagonals).

The teachability space which appears in the solution (a so called Schur-Takagi interpolation), is defined by (see [8])"

I A

B~ B

J-unitary Matrices fo r Algebraic Approximation 215

bl

Figure 4: The chain-scattering operator terminated in a load S L.

THE NON-SINGULAR CASE

In all the interpolation cases detailed above, a non J-isometric pair of A , B = [B1,B2] matrices are given which, if the interpolation problem is to have a solution, should define the teachability space of a J-unitary operator. This will be the case if a state transformation can be found which makes the pair J-coisometric. Consider the Lyapunov-stein equation in the diagonM operator A:

* - * B A ( - 1 ) A * A A + B 1B1 B2 2 = (5)

A is the Gram-Schmidt matrix in the input Jl-metric, of the reachability space characterized by the given A, B data. Let II0 indicate the projection on the diagonal, and Q = ( I - Z ' A * ) -1 Z ' B * , then

A = IIoQR1Q* (6)

If A is strictly non-singular, then it produces the desired state transformation diagonal | R A ( R ( - 1 ) ) -1 ] _ .

matrix R: A = R * J ~ R which makes ~ / B ( R ( - 1 ) ) -1 ] coisometric for the metric defined J

by the signature ]

I -

J~ J1 "

Once the proper state transformation R is found, then the complete realization for a J- unitary operator O with the appropriate reachability space is closeby. It is not hard to see (for the discussion, see [8]) that the missing operators C = [C1 C2], D = [Di,j : i , j = 1, 2] can now easily be found by completing the J-coisometric part to a J-unitary realization.

The 'classical', strict, interpolation problems (Nevanlinna-Pick or Hermite-Fejer) find a solution when A is strictly positive definite, in which case the corresponding interpolant is causal and strictly contractive. In this case J~ = I, and the chain-scattering matrix describes the transfer from port 1 to port 2 of a so-called passive and lossless system, in which the energy transfer goes strictly with the propagation of the state. In this case, 0~~ (which by the way always exists as a bounded operator) will also be causal, and |174 will be causal and strictly contractive solution.

LOADS

Once the appropriate O is found (provided it exists), then any other solution to the interpolation problem is given by 'loading' the chain-scattering operator in any causal, contractive load S L - see fig. 4.

216 P. Dewilde

The expression for the solution is then given in terms of a so-called homographic transformation T(| of the load SL induced by the chain scattering matrix:

S = T(O, SL) = [012 - | -- 022] -1. (7)

It is alSO not too hard to see that each such S will indeed satisfy the interpolation conditions, the main reason being that loading the chain-scattering matrix will not disturb the reachability space of the input angle operator. To be more precise, consider:

I - ) [Br B~] [ 0 ~ - O ~ S ~ ] -~ e U.(S)

If 0 has a strictly positive definite A as Pick matrix for its teachability operator, then, as remarked before, 022 will be causally invertible, and we see that

( Z - A*)-I[B~ B~] I S ] E lg

which is equivalent to the interpolation condition in each of the cases detailed above. The converse, namely that all solutions are reached in this manner, is harder to prove, see [9, 8] for the original proofs. The load can be put to greater effect, namely in the case where the Pick matrix A is singular. This will be the theme of the last section.

3 T H E S I N G U L A R C A S E

The interpolation problem becomes singular when the Pick matrix or operator A is singular. How to proceed? It was already remarked by Schur [1] that the boarderline case where the Pick matrix is positive semidefinite and singular could be solved by restricting the load SL to constant unitary (in the restricted case considered by him). This procedure can be interpreted as tampering with the reachability space of the chain scattering operator so as to produce an 'isotropic' part.

Let ~ be a space and J1 a signature matrix for it. Let Q be a suitable basis for 7~ and let A = IIoQJ1Q* the corresponding Gram-Schmidt (Pick) matrix. We shall say that a vector x E 7~ is isotropic if Vy E 7~, xJly* = O. The set of isotropic vectors form a subspace of R, a basis for which can be obtained by computing the inertia of A. Let

h = U - ~ U*, Or

where U is a nonsingular diagonal matrix, Ip and Iq are diagonals of identity matrices and Or is a diagonal of zero matrices (p etc.., are sequences of dimensions e.g. p = [ . . . , pk,'" "]). Let QI = QU, then the vectors in QI which correspond to the r-rows form a basis for the isotropic subspace of Q.

P r o p e r t y 1 Let E = {e+ e_ e0} be a basis for a space with an indefinite metric J such that:


e+ is positive �9 e+Je~. >> 0 e_ is negative : e_Je*_ << 0 eo is isotropic : Vy E Tr : eoJy* = 0

then there exist basis vectors {e++, e__ } such that:

- dim e++ = dim e__ = dim eo - eo C V { e + + , e - - } -{~++,~__} • {~+,~-}

We omit the (elementary) proof, but remark that the spaces V e+ and We- are not unique, and can be chosen in strategic ways, e.g. so that V e+ is orthogonal in the tradit ional sense on the positive part of V eo, and V e_ on the negative part of V e0. Simply splitting e0 in its positive and negative parts then pulls the trick. We call the process described in property 1 'augmentat ion ' . The augmented space Vie+, e_, e++, e__] does not have an isotropic subspace, and is then called a Krein space for that mat ter .

We need a slightly more refined, recursive version of the previous property.

P r o p e r t y 2 Assume that the J-signed reference input space A4 is actually the (standard) direct sum of two Ji-signed spaces .Mi 5 : 1, 2), J : J l+J2, and let $ be a subspace of .M. Let Hi denote the natural projection on .Mi, and let ${ = HiE. Suppose moreover that $1 C Sit with $1t a non-singular (Krein) space. Then $ can be augmented to a non-singular (Krein) space $t such that IIiEt C $1t.

In other words: the augmentat ion can be done locally, in the last space, leaving restric- tions to previous subspaces undisturbed.

P r o o f : Let eo be a basis for the isotropic subspace of $ such that e = {eo~, e02} and {eo~} is a basis for the isotropic vectors in $ ["].h41 (they are necessarily in $1). e01 finds an augmentat ion in $1t. e02 is complementary to e01 in V e. The part V e02 ["].M1 is already in NElt , so we can concentrate on e02 f'].M2. Take a basis vector e 6 V e02 and split it e = el + e2 with el E .h41 and e2 E A42. If e2 (and hence also el) is non-neutral, then add e2 to the basis - $1t already generates el. If, on the other hand, e2 (and hence also el) is neutral, then e2 will have to be split in two non-neutral vectors, following the same procedure as in property 1. This procedure can be repeated with the next basisvector in e02, in fact, a generalization of the above to blocks works just as well. []

The construction of the augmentat ion for a reachability space which contains isotropic vectors then proceeds recursively. Without impairing generality, we may restrict ourselves to a reachability space with three stages. If we position ourselves at point k in the recursion, we may observe the collection of basis vectors G which represent the strict past, the basis vectors .T" for the present, and the basis vectors $ which stand for the future - see fig. 5.

Our goal will be to obtain non-singular augmentat ions, which respect the restricted shift property and which are minimal.

S t ep 1: start with V e = $, the last space, let e0 be its isotropic space and write:

eo : {e01, e02, e03}

where e01 is an isotropic basis and lays in A41, e02 is a complementary basis and lays

218 P. Dewilde

A,~ 1 .M 2 A~ 3

s

Figure 5: The three stages in the augmentation procedure for the reachability space

Figure 6: The decomposition of eo into basis vectors

in ~ 1 + M 2 and eo3 is the remainder complementary basis. (Although V eo is unique, the bases are not, and the order in which they are taken turns out to be important.) A graphical schema for eo is shown in fig. 6. By property 2, eo3 can be augmented leaving lower levels undisturbed, producing vectors e03+, eo3-, and this can be done in a minimal way.

Step 2" we now move to .T. Since the restriction of C to .~1+M2 is contained in .T, our guiding principle will be first to choose bases which generate the restriction, and then augment further to cover .T. The basis needed for the restriction of • can then be 'exported' to ~'. Let .To be the isotropic subspace of .T, and let V e02o = V e02 ~.To. Choose bases in F as follows: (1) Start with .To: let fox = .ToN.M1, and define a basis foll which generates em ~.To, and the complementary basis fro2 so that fml, fro2 generates fro. Next, continue the basis construction with fo21 which generates eo2o and complete with f022 SO that also the complete .To gets to be generated. The basis foil, fro2 must be exported one level down (to ~), while fmx, fro2 must be augmented locally to fo2x+, f021-, f022+, f022-. (2) Next continue with the non-singular parts and let .Ts be the complementary basis to .To for .T. In order to generate the necessary basis vectors in the correct sequence, we proceed as follows: first we generate the remainder of em with f81, the remainder of e02 with fs3, and finally, the remainder of .T with fs2. (3) We now have the basis vectors needed for orderly export to stage s fo21+,fo21-,f81, f83. Not yet augmented are fro1, fro2 whose augmentation will have to be done in the context of ~. The there obtained augmentations will have to be exported back to .T (for both) and to ~' (only for loll). Needless to say, the procedure is only possible because of the restricted shift property.

Step 3" take care of ~. Notice that we could proceed with this recursively, the previous step handing over ordered isotropic spaces that have to be augmented and whose augmentations have to be handed back up the line after completion. Hence we import loll, fro2 from the previous stage as the first base vectors to be taken care off. We the proceed as follows: (1) Let go be the isotropic part of ~, and let f011 ~ 0 be generated by go3, V{foll, fo12} ~ 0 by {go3, go2}, and go by {go3, go2, go~}. An these spaces have to be augmented to yield bases:

{go3+, go3-, go2+, go2-, gin+, gin- }. (2) Continue with a non-singular remainder: complete V foil with g83, next V{fml, fro2} with {gs3, gs2}, and finally, Gs with {gs3, gs2, gsl}.


(3) The step is terminated by a final export procedure of {go3+, go3-, go2+, go2-,ga3, gs2}.

This terminates the construction. The resulting non-singular augmentation then looks as follows:

level 1: {go3+, go3-, go2+, go2-, go1+, go1-, g83, gs2, g,l). level 2: {go3+,goa-,go2+,g02-, f021+, f021-, f022+, fo22-,ga3,gs2, fsl, fs3, fs2}. level 3: {go3+, go3-, f02+, f02-, e03+, e03-, ga3, f83, es3}.

This in turn adds up to the theorem:

T h e o r e m 1 A strict (minimal) augmentation of a teachability space to a non-singular space allways exists.

E x a m p l e 1: Let's consider a chain scattering operator which maps { ~ ] , C 2} to { [ ~ C 2, C2},

w i t h t h e u s u a l i n p u t a n d ~ J = [ 1 - 1 ] ThecorrespondingO

has the form (diagonal entries with one empty dimension are indicated by a ']' or a ' - - ' ) :

0 = 011 0 = D1 BIC2 �9 _ _ u

u m

The teachability space looks as follows:

(~ n = V s ~ .

Assume that the objective is to generate the teachability space generated by:

rn [11] [1 ~] [0 0]

(notice that the given space consists solely of isotropic vectors, and that it satisfies the restricted shift condition.)

IZ]

1 0 The augmentation clearly gives" 0 1

1 0 0 1

The corresponding realization is given by:

1 0 Co=O, Do=O; A o = 0 , B o = 0 1 '

i A1 = 0 1 ~B1 =

oo] 0 0 "

00] c1=[00] o1=[10 0 0 ' 0 0 ' 0 1

220 P. Dewilde

1 0 A2 - 0, B2 = 0, C2 - 0 1 , D2 - 0.

The | for which the augmented basis defines the reachability space is:

[~ 0 0 1 0 0 0 0 1

O = 1 0 0 0 (0) 0 1 0 0

The corresponding Hankel matrices are: Ii~ o o 1 o o 1 H i = 0 0 0 1 ' H 2 = 0 0 " (10)

0 0

E x a m p l e 2: Now we require the specified reachability space to be:

D

1 0 0 1 (11)

1 ~ [0 0 ] .

The augmented reachability space is just as in the previous example, and hence | as well. There is, however, a major difference, which will be the topic of the next paragraphs.

REDUCTION

Our next goal is to reduce the augmented reachability space back to size and to use for that exclusively a non-dynamic, i.e. diagonal load. Again, the three stage case covers generality, because it will again lead to a recursive construction. Fig. 7 gives a picture of the situation. The cascaded transfer function now becomes (with the Ci to be determined):

[OooCol Oo, cl Oo2C 011C1 012C2 {~1363 (12)

| | 033C3

The corresponding Hankel matrices are now:

002C2 003C3 , Ha = 013C3 . (13) H1 = [! O01C1 O02C2 O03C3 ],,H2 = k J

O23C3

The purpose of the Ci is to cut down on the ranges of the Hankel matrices. The column ranges starred correspond to the teachability operators. Clearly, Co plays no role and hence must be chosen to be a unit matrix. C3 has the easiest job: it can be allways be chosen such that the original//3 results, and, assuming that the base vectors are ordered


11

I T , , ,

I T

Figure 7: The chain scattering matrix terminated by a non-dynamic, diagonal load.

as {e+ ,e_ ,e++,e__} , will have the form:

I~ C3 = Iq3

I,./v~ With H2 the situation is more delicate. In normal circumstances, its first block column

will be rich enough to supply the necessary basis vectors, but it may happen that that is not the case because some basis vectors needed have already been collapsed by C3. It is possible to give precise conditions when that defectuous case will occur: (1) part of an earlier space is isometrically generated by a later space (the symptom is: the matrix A has a unitary part); (2) the reduced earlier space is larger than the later space.

It is not too hard to see that these conditions are necessary and sufficient. The following observations are in order: (1) The occurence of a defect is rather rare! It only happens when in the original reachability space, a later isotropic vector is generated by an earlier non-singular basis. (2) It can't happen when the Pick matrix is semi positive definite, because in that case all the augmented spaces must be completely reduced in all stages.

Example : Example 2 of the previous section shows a defect. We have 0 as in equation 9. Clearly,

1/v~] C2= l/v/~

will cut H2 down to size all right, but the first block in H1 is defectuous, and cannot supply the two basis vectors [10] and [01], while in example 1 no new vectors are needed, and the loading goes without problems. The example is actually generic.

In the defectuous case, the only way out is to weld stages together, so that the space causing the defect is welded to the space that is able to supply the necessary information. An algorithm to do that goes beyond the purposes of the present paper.

222 P. Dewilde

4 CONCLUDING REMARKS

Once more we see that results which originally belonged to the realm of complex function theory are purely of an algebraic nature, and can be generalized to matrices and operators. These can be viewed as 'classical' transfer functions whose coefficients belong to an algebra of diagonal operators. A generalization of systems theory provides the link. The reachability operator and a connected Pick matrix plays the central role.

Acknowledgements

The author gratefully acknowledges fruitful discussions on the topic with Harry Dym and Alle-Jan van der Veen. The research presented was partially supported by the Commission of the EU under the NANA BRA Project nr. 6632.

References

[1] I. Schur, "Ueber Potenzreihen, die im Innern des Einheitskreises beschr~nkt sind, I," J. Reine Angew. Math., vol. 147, pp. 205-232, 1917.

[2] V.M. Adamjan, D.Z. Arov and M.G. Krein, "Analytic Properties of Schmidt Pairs for a Hankel Operator and the Generalized Schur-Takagi Problem," Mat. USSR Sbornik, 15(1):31-73, 1971 (trans. of Iz. Akad. Nauk Armjan. SSR Ser. Mat. 6 (1971)).

[3] D. Alpay and P. Dewilde, "Time-varying Signal Approximation and Estimation", in Signal Processing, Scattering and Operator Theory, and Numerical Methods, vol. III, pp. 1-22, Birk~user Verlag, Basel, 1990.

[4] H. Dym, J Contractive Matrix Functions, Reproducing Kernel Hilbert Spaces and Inter- polation, 71, Conference Board of the Mathematical Sciences, American Mathematical Foundation, Providence, Rhode Island, 1989.

[5] D. Alpay, P. Dewilde and H. Dym, "Lossless Inverse Scattering and Reproducing Ker- nels for Upper Triangular Operators," 47 Operator Theory and its Applications, pp. 61-135, Birkh~user Verlag, Basel 1990.

[6] P. Dewilde and H. Dym, "Interpolation for Upper Triangular Operators," in Operator Theory: Advances and Applications, vol. 0T56, pp. 153-260, Birkh~user Verlag, Basel, 1992.

[7] P. Dewilde, "A lecture on Interpolation and Approximation of Matrices and Opera- tors", in Challenges of a Generalized System Theory, Royal Netherlands Academy of Arts and Sciences, 1992.

[8] P. Dewilde and A.-J. van der Veen, "On the Hankel-norm Approximation of Upper- Triangular Operators and Matrices," Integral Equations and Operator Theory, 1993.


[9] A.-J. van der Veen, Time-varying System Theory and Computational Modeling, Real- ization, Approximation and Factorization, Ph.D. Thesis, Delft University of Technol- ogy, 1993.

[10] A.-J. van der Veen and P. Dewilde, "On Low-Complexity Approximation of Matrices," subm. to Linear Algebra and its Applications, 1994.


P A R T 3

A R C H I T E C T U R E S A N D R E A L T I M E I M P L E M E N T A T I O N



227

S P H E R I C A L I Z E D S V D U P D A T I N G F O R S U B S P A C E T R A C K I N G

E.M. DOWLING, R.D. DEGROAT, D.A. LINEBARGER, H. YE University of Texas at Dallas, EC33 P.O. Box 830688 Richardson, TX 75083-0688 U.S.A. [email protected]

ABSTRACT. In this paper we study a low complexity square root algorithm that tracks the dominant or subdominant singular subspace associated with time-varying data matrices. The algorithm complexity is O(nr) where n is the data dimension and r the tracked subspace dimension. The algorithm sphericalizes most of the signal and noise eigencomponents to reduce computation via deflation while retaining enough eigen information to track the rank of the input process. The algorithm is shown to converge in the mean with probability one to the desired subspace and singular levels using the stochastic algorithm ordinary differential equation (ODE) approach. We develop a strongly consistent MDL-based detection scheme for rank determination. We also show how to significantly reduce computations when forward backward averaging is employed.

KEYWORDS. SVD updating, subspace tracking, algorithm ODEs.


Spherical subspace (SS) tracking algorithms average sets of eigenvalues and apply deflation to reduce computation [1, 2, 3, 4, 5, 6, 7]. The three SS algorithms that have proven most useful are the two level signal averaged (SA2), the four level signal averaged (SA4), and the signal eigenstructure (SE) updating methods. The SA2 algorithm separately averages the signal and noise eigenvalues to create two averaged eigenlevels. The sphericalized subspace bases are then arbitrarily reoriented to deflate the update computation. SA2 is the least expensive, but does not generate enough internal variables to be easily used for signal rank detection. The SE algorithm only averages the noise subspace eigenvalues, so therefore wastes effort updating each signal subspace eigenpair.

In this paper we will analyze the four level Single sided Signal Averaged (SSA4) subspace tracker presented in the companion paper in this volume [8]. This algorithm efficiently tracks the signal subspace and provides information needed to track the signal rank.

228 E.M. D o w l i n g et al.

2 SSA4 S U B S P A C E C O N V E R G E N C E ODE

The SSA4 update as presented in the companion paper [8] takes the form:

Q . (x)

Here the tildes denotes updated quantities, Jx and J2 are the deflation rotations, Q is the 4x4 right Jacobi sweep matrix, the columns of VaECn• span the sphericalized portion of the signal subspace, vr is the sub dominant signal subspace basis vector, vr+l is the dominant noise subspace basis vector, and vr+2 is the projection of the current data vector, x k E C n into the spherical portion of the noise subspace. The G transformation [8] that scales the v-columns to make the core SVD real is assumed to be applied prior to (1).

To understand the expected behavior of the algorithm in a statistical sense, we characterize adaptive subspace evolution in terms of projection operator trajectories over projection operator manifolds. The trajectory itself is characterized by an ODE derived by taking limits and expectations on the updating direction associated with the updating rule. To perform the analysis, we follow the basic stochastic algorithm ODE method of Ljung [9].

Let {Xk) be a zero mean stationary input process with a correlation matrix R = E[xkx~l. After the J1 and J2 deflation rotations are applied, the basis associated with spherical portion of the signal subspace is reoriented, but its span remains the same. Hence

~ J

right before the 4x4 Q rotation is applied, we can write the projection operator into this ( r - 1)-dimensional subspace as,

= + (2)

where we break out [Vs vr-1 ] = VsJ1J2 from (1). After the update described by (1) and with this definition, we have,

P(@= V , V H + ~r_l~H_l . (3)

For analysis purposes, we are interested in the projection operator update, hence we write,

P(')= P~_)I + Ak (4)

so that Ak = P(')- P~)~ ,or

~ k :- Vr-lvH-I--Vr-IVH-1. (5)

Now, following Ljung's ODE approach, we write the subspace update in a generalized form,

p~~ P~)~ +

( where Q' P(@, vr, vr+l, XkX = Ak, and the gain sequence satisfies g(k)>0 for all k, lira g(k) -- k--*~

0 , and ~ . ~ = o g ( k ) = cr Ljung [9] introduces the "fictitious time," r = ~tk= l g ( k ) as the algorithm ODE's independent variable.

The next step in Ljung's approach is to define the function (r depends on t)

Sphericalized SVD Updating for Subspace Tracking 229

which is the crux of the algorithm ODE that describes the expected convergence trajectory,

d (p(,)(~)). (s) drP(@(v) = f

The problem is really to find an expression for f (P(S)(v)) . Fortunately (7)involves e x -

p e c t a t i o n and limiting operations, so it is not too difficult to evaluate. Directly following Ljung's method, we take the limit in (7) above and note that as k--*c~ , the incremental step taken during the k th update approaches zero since g(k) - . O. Thus in the limit we look at an infinitesimal step of the algorithm. To do this we may write, g(k) = e = dr and

look at the perturbation dP(S)(v)= f (P (S) (v)) dv . Since Q' is the update direction, from

(5) and the definition of v we have,

We will use perturbation analysis to obtain the desired result. The 4x4 core Jacobi rotation, Q will update v r - l (v ) to Vr-l(v) according to

Vr-l(V) -- q11vr-i(v) -F q21vr(7") + q31vr+1(v)+ q41Vr+2(V). (I0) The expected perturbation after an infinitesimal update can be shown to be

! ! _ "~174 s ,

~_,(~) = ~-~(~) + ~ ~ ( ~ ) + ~ ~+1(~) + ~ ~ + ~ ) 0"1 - - 0"2 0"1 - - 0"3 " 1 - - " 4

(II)

where e is the infinitesimal step size, and the #'-terms are the core deflated input parameters that are used along with the a-values to compute the rotation parameters, qii in Q. Substituting (11) into (5), (5) into (9), replacing e with dr, and ignoring the e 2 terms that vanish in the limit of (9) we get,

dP(') v ( ) = E ~ - 7~7--- L i -- " i - r + 2

d~. (12)

To evaluate the expectation further, define,

a 1 ~, = ~?_ _ ~?_~+~_ i = r, . . . , r + 2. ( 1 3 )

Now since the v i ' s axe slowly time varying compared to the data we may assume them

to be statistically independent of the data. Hence defining Pi ~ E[ui] , write, r + 2

i--r

r + 2

i--r

r + 2

= ~ [', (P( ' ) ( r )R~v i ( r )~H(r )+ 9i ( r ) '~H(r )RP( ' ) ( r ) ) i=r

At this point define P i ( r ) = viv H for i = r, r + 1 and Pi(v) = P('~)(r) for i = r + 2. Here P(n)(v) is the projection operator into the subdominant ( n - r - 2)-dimensional subspace.

230 E.M. Dowling et al.

Then using (8), (9) and (14)the ODE becomes,

dP(')(r) ~ pi (p(s) (r)Rpi(r) + pi(r)Rp (,) (r)) ~r =

i=r

(15)

Note that if Pr = P~+I = P~+2 then this ODE simplifies to the SA2 ODE described in [3]. Sphericalizing the three sub dominant eigensubspaces produces an SA2 algorithm.

Now that we have an ODE for the dominant subspace of the SSA4 algorithm, we need to show that it converges to the true dominant subspace in the mean with probability one. To do this, we must show that the dominant subspace of R is the global stable attractor point of the ODE. To start note that (15) goes to zero when P(a)(r) = VsVa H where Vs is now the true dominant subspace. To see why this is the case, write R = V s D s V H + [Vr Vr%l Vn ]Dx[ Vr Vr+l Vn ]H where Dx is a diagonal matrix containing the (n - r) subdominant eigenvalues of R. Substitute this expression for R into (15) with P(S)(r) = VsVs H and note how all terms go to zero.

To show P(S)(r) = VsVs H to be the global attractor, define the Lyapunov function,

Next define the ( r - 1)-dimensional projection operator manifold,

Pr-l,,~ ~ {PIP6C '~• P= = P = pH, Rank (P)= r- I}. Note that for any positive definite matrix R with ordered eigenvalues,

r--1

P(~ i=1

(17)

(18)

and the maximum is unique if ),r-l>Ar �9 This maximum occurs when P(S)(r) = V s V s H

which shows that the stable point is the unique maximum point provided At-1 >A~. With finite amounts of data, Ar-x >At with probability one.

A potential problem with the above analysis is that the ODE might take P(S)(r) to a point off the projection operator manifold. The following lemma shows that trajectories that start on the projection operator manifold will stay on it.

L e m m a 1. The solution to (15) subject to initial condition p!s)EPr-x,~ is a function P(a)(r) defined over [v0, c~) that satisfies P( ' ) ( r ) e Pr-l,n for every [v0, c~).

Proof . For P ( ' ) ( r ) t o be in Pr-l,n a necessary condition is that P(S) ( r )= (P( ' ) ( r ) ) 2. Differentiating both sides of this static constraint gives the dynamic constraint,

P(')(r) = P(')(r)P(')(r)% P(')(r)P(')(r). (19)

It is easy to substitute the ODE of (15) for lh(S)(r) on the right hand side of (19) to verify that (15) satisfies (19). This shows the algorithm ODE will propagate a projection operator quantity. The ODE will also keep the trajectories in Pr-l,,~ because after dr "time" units,

the projection operator norm has moved .llP(S)(r)+ dPII F" = ....llP(S)(r)llF -- ~ / r - 1. The i | i J | | | |

following theorem shows that the SSA4 ODE trajectories lead to the unique maximum


point of the Lyapunov function.

T h e o r e m 1. The four singular subspaces of the SSA4 algorithm converge in the mean with probability one to four right sphericalized SVD-subspaces associated with the stationary input vector time series. For convergence in the mean with probability one, ?(k) must meet Ljung's conditions.

Proof . We start by showing that the Lyapunov function increases in the direction of any trajectory described by (15). This, together with the Lemma 1, shows that the algorithm dynamics are such that the stochastic rule brings the expected trajectory to the desired stable point, P(8)(r) = VsVs H. To show this, we take a derivative of the Lyapunov function along an arbitrary trajectory defined by (15). This turns out to be a straightforward calculation,

dr ~'r

= tr [ ~ P ( ' ) ( r ) R P ( ' ) ( r ) + P ( ' ) ( r ) R ~ p e s ) ( r ) ] (20)

r+2 = ~ P~tr[(P(')(rlRPi(r)+ Pi(r)RP(')) Rp0)

i=r--1

+P(')R (P(')(~)RP,(~)+ Pi(r)RP("))] >_ 0

This is so because each vi is positive and R is positive definite.

We have thus shown the dominant ( r - 1)-dimensional subspace estimate to converge. To show that the r th basis vector, vr( r ) converges to the r th eigenvector of R, write the signal subspace projection operator as

P(Sig)(r) = P(s)(T)+ vr(r)vH(r) = P(')(r)+ Pr(v) (21)

which is analogous to (2), (3), (4), and (5). If we let t~oc we have P(s)(T)--+vsvH, but now we are interested in showing P(Sig)(r)--+[Vs Vr][V, vr]H . So we are interested in deriving the ODE for P(Sig)(r). For large v, where P( ' ) ( r ) has converged to V , V H we can use the same approach as (6)-(9) to write,

dp('ig)(r) = lim E[~r(r)~H(r)--vr(r)vH(v)]dr. (22) t--+OO

Note that this is also the ODE for the r th dimension, Pr(r) = vr(v)vH(v). Using the same approach as (I0) and (II), we have,

I I I I ~2~3E 7~7~e ~2~'z s (T) -~- Vr(T ) + 2 2 EVr+ 1 (T) -11- 2 2 s (23) V~(r) = z - - ev~_~

0"2 O'z 0"2 -- 0"3 0"2 -- 0"4

so that a few manipulations similar to (14) give

d P r ( r ) ~+2 ~rr = ~ Pi(P~(r)RPi(r) + Pi(r)RP~(r)), (24)

i = r - l , i ~ r


and in the limit, for large v (where P( ')(v) has converged to VaV H ) we can write,

r+2 dP('ig)(r) = ~ P i ( P ( 8 0 ) ( r ) R P i ( r ) + Pi ( r )RP( ' ig ) ( r ) ) (25)

dr i= r+ l

This ODE shares the same convergence properties as (15). Since the signal subspace converges to the dominant r-dimensional subspace of i t , and the sphericalized component of the signal subspace converges to the dominant (r - 1)-dimensional subspace of R, we have that v r ( r ) ~ v r �9 Another similar induction step shows that Vr+I(V)--*Vr+I and thus the orthogonal complement satisfies P (n ) ( r )~P( '0 .

T h e o r e m 2. The four singular levels of the SSA4 algorithm converge in the mean with probability one to the four sphericalized SVD singular levels associated with the input vector time series. For a proof see [10].

3 S S A 4 - M D L R A N K T R A C K I N G S C H E M E

In order for SSA4 to track time-varying subspaces for high resolution algorithms such as MUSIC, Minimum Norm, and total least squares, it must also track the number of signals. SSA4-MDL uses the four singular levels generated by the SSA4 algorithm to decide if the algorithm should increase, decrease, or not change the subspace dimension. However, MDL [11] must be modified to work with SSA4. Since all of the singular value information is not retained by SSA4, the MDL criterion can not be directly applied. SSA4-MDL will not generate a signal rank estimate but will rather generate one of the following outcomes:

1. No S i g n a l ( a l 2~a22~a32~a42)

2. Decrease Dominant Subspace Size ( a2>o'22 .~ a~ ~ a24 )

3. Dominant Subspace Size Correct ( a2>a2>a23 ,.~ a 2 )

4. Increase Dominant Subspace Size ( a~>_a~>_a~>a~ )

Thus, it is only necessary to test for four possible outcomes, i.e., q = 0, r - 1, r, r + 1 . For the details and the strong consistency theorem, see [10].

4 S I M P L I F I C A T I O N S W I T H F O R W A R D - B A C K W A R D A V E R A G I N G

The data model of [8] assumes forward-only averaged data. Here we discuss how to reduce computations when forward backward averaging is applied. For a more detailed discussion of the topics covered here, see [12]. Write the forward-backward averaged correlation matrix,

1 j A T ] [ A k ] = AHBkAFB,k . (26) RrB,k= AN

The following theorem, proven in [12], shows how to transform this correlation matrix to a purely real matrix with purely real square root data factors.


Theorem. Let Z FB be defined according to

ZFB -- L H A F B K (27)

where K is given by

[i 0 1 [ I j I ] 1 0 j~/'2 0 (28)

K = ~ S - j S ' K = ~ J 0 - j J

for even and odd ordered matrices respectively, and AFB is the FB data matrix defined in (26), and L is a unitary transformation defined by

1 [I j I ] (29) s = ~ i - j i

then,

B = K H R F B K = K H A H B A F B K = ZTBZFB (30) and ZFB is real if A is complex or ZFB is real and block diagonal if A is real. Furthermore, if we evenly partition A as A = [A1 A= ], then (27) simplifies to

[ Re(A1 + JA2) I m ( A 1 - JA~)] ZFB -- -Im(A1 + JA2) Re(A, - JA2) " (31)

Assume the data is complex and let ZFB be defined by (27). Similarly, let AFB be defined by (26). If the SVD of ZFB is given by

v r (32) ZFB ----- U Z~Z Z

where Uz and Vz contain the left and right singular vectors, respectively, and ]Ez is a non-square diagonal matrix containing the singular values of ZFB , and the SVD of AFB -- U AN2AV/~ , then

UA ---- L U z , ~A = ~Z and V A = KV z. (33)

The SVD computational cost is thus reduced by approximately 75%.

SVD updating of AFB,k can be more efficiently accomplished by updating ZFB,k instead. First permute AFB,k as defined in (26) by adding a pair of forward backward data vectors. To compute a forward-backward update in less computations than a forward-only update,

/x 1. From the k th data vector, xH=ak = [al,k a2,k] , compute rows z2k-1 and Z2k

according to (31).

2. Append Z2k-1 and Z~k to ZFB,k_ 1 and use two SSA4 updates (or other subspace tracking method) to update the right singular vectors, Vz and the singular wlues.

3. As needed, compute VA,k = KVz, k �9 (Note: ~-,A,k = }3Z, k .)


5 CONCLUSION

In this paper we derived the algorithm ODE for the Single sided Signal Averaged 4-level (SSA4) subspace tracker. We used this ODE to prove convergence in the mean with probability one to the signal subspace in a stationary environment. The algorithm is inexpensive and useful in practical slowly time-varying sonar and radar tracking problems. SSA4-MDL monitors the signal rank. Forward-backward averaging adds structure that can be exploited to reduce computation and boost performance.

Acknowledgments

This work was supported in part by the National Science Foundation Grant MIP-9203296 and the Texas Advanced Itesearch Program Grant 009741-022.

References

[1] I. Karasalo, "Estimating the covariance matrix by signal subspace averaging," IEEE Trans. ASSP, vol. ASSP-34, pp. 8-12, Feb. 1986.

[2] It. D. DeGroat, "Non-iterative subspace tracking," IEEE Trans. Signal Processing, vol. 40, pp. 571-577, Mar. 1992.

[3] E. M. Dowling and It. D. DeGroat, "Adaptation dynamics of the spherical subspace tracker," IEEE Transactions on Signal Processing, pp. 2599-2602, Oct. 1992.

[4] It. D. DeGroat and E. M. Dowling, "Non-iterative subspace updating," in SPIE Adv.SP Alg., Arch. and Imp. II, (San Diego, California), pp. 376-387, July 1991.

[5] It. D. DeGroat and E. M. Dowling, "Spherical subspace tracking: analysis, convergence and detection schemes," 26th Asilomar Conference, pp. 561-565, Oct. 1992.

[6] E. M. Dowling and It. D. DeGroat, "A spherical subspace based adaptive filter," in ICASSP, pp. III:504-507, April 1993.

[7] It. D. DeGroat, H. Ye, and E. M. Dowling, "An asymptotic analysis of spherical subspace tracking," 27th Asilomar Conference, Nov. 1993.

[8] E. M. Dowling, It. D. DeGroat, D. A. Linebarger, and Z. Fu, "Iteal time architectures for sphericalized SVD updating," in this volume.

[9] L. Ljung and T. Soderstrom, Theory and prac. of rec. id. MIT Press, 1983.

[10] It. D. DeGroat, E. M. Dowling, H. Ye, and D. A. Linebarger, "Multilevel spherical subspace tracking," IEEE Trans. Signal Processing, Submitted August 1994.

[11] M. Wax and T. Kailath, "Detection of signals by information theoretic criteria," IEEE Trans. ASSP, vol. ASSP-33, pp. 387-392, Apr. 1985.

[12] D. A. Linebarger, It. D. DeGroat, and E. M. Dowling, "Efficient direction finding methods employing forward/backward averaging," IEEE Tr. on Sig. Proc., Aug. 1994.


235

R E A L T I M E A R C H I T E C T U R E S F O R S P H E R I C A L I Z E D S V D U P D A T - I N G

E.M. DOWLING, R.D. DEGROAT, D.A. LINEBARGER, Z. FU University of Texas at Dallas, EC33 P.O. Box 830688 Richardson, TX 75083-0688 U.S.A. [email protected]

ABSTRACT. In this paper we develop a low complexity square root algorithm and a pseudo-systolic array architecture to track the dominant or subdominant singular subspace associated with time-varying data matrices in real time. The pseudo-systolic architectures are ideally suited for implementation with custom CORDIC VLSI or networks of available parallel numeric processing chips such as iWarps or TMS320C40's. The real time update complexity is O(n) where n is the data dimension. Further parallelization is possible to scale complexity down linearly as more processors are added. We show how to track the dimension of the signal subspace and gear-shift the array when the subspace dimension changes.

KEYWORDS. SVD updating, subspace tracking, systolic arrays.


The background information about the Single sided Signal Averaged 4-level (SSA4) subspace tracking algorithm is discussed in the companion paper in this volume [1], so it will not be repeated here. The rest of the paper is organized as follows. In section II we will develop the SSA4 parallel subspace tracking algorithm. Section III describes the array algorithm that prealigns the spherical subspace basis, and section IV covers the operations associated with the core 5• SVD Jacobi sweep. The use of Jacobi sweeps in subspace updating algorithms has been employed by both Moonen and Proakis [2, 3]. In section V we show how the array implements SSA4-MDL and how it gear-shifts to increase/decrease the dimension of the tracked subspace during tracking. Section VI concludes the paper.


2 SSA4 A L G O R I T H M D E V E L O P M E N T

Consider a vector time series {xk} whose associated slowly time-varying correlation matrix may be estimated recursively according to

where 0 < a < l is an exponential window fading factor and l~k is the correlation matrix estimate. Suppose the vector time series is generated by the nonstationary narrowband model [4],

..L% xk = ~_~ a(Oj(k))sj , k + nk (2)

j--1

where a(0j(k))EC n is a signal vector, 8j(k) is the j th signal direction at the k th sample, sj,k is the k th sample of the j th modulating process, r is the number of sources and nkEC n is additive white noise. Here the set of signal vectors, { a ( O j ( k ) ) l O j ( k ) , j = 1, . . .r} defines the signal subspace. If we define the matrix r ( 0 ( k ) ) e c ~• to have ~(0s(k)) ~ its j th column, then we can write

xk = r ( 0 ( k ) ) ~ + nk (3)

where now the j th element of s k E C r is sd, k . Now consider the square root data matrix

formulation,

�9 .. �9 ~ v ~ x k (4) l xk 1 x H

where Wk = diag( ak ) ,..., a, 1 is an exponential weighting matrix. Note that for stationary

data, (1 / (1 - a ) ) R k = ~j=ok ak -JXkX H = AHAk so that the r dominant right singular vectors of Ak span the signal subspace, i.e., the range of r (8(k)) .

Suppose that at time k - 1 we have a 4-level sphericalized SVD estimate of the data matrix. A 4-level SVD estimate can be obtained from a full SVD by replacing the first r - 1 singular values by their root-square average value, and by replacing the last n - r - 2 singular values by their root-square average value. At time k = 0 we initialize the estimate to be in the 4-level form and from then on each rank one SVD update is resphericalized to produce a new 4-level sphericalized estimate. Hence, at time k - 1 we have the 4-level sphericalized SVD estimate,

which has the 4-level structure

I 0.1 I t - 1 0"2

A~4) x_ = U k - 1 0"3

(5)

a 4 I m - r - 1

v ~ ' V~

(~)

In (6), al and 0"4 represent the root-square average singular-levels and the columns of the sphericalized portions of the signal and noise subspaces, V, and Vn are the associated

Real Time Architectures for Sphericalized SVD Updating 237

spherical right singular subspace basis vectors. To avoid expensive and redundant computations, the algorithm will actually only update either the signal or noise subspace basis. Thus we will only carry along the smaller of V8 or V,~. Also, since we are only interested in the right singular subspace, we do not update U k.

Once (6) is available, we append a new row as follows:

So the question is, how do we restore the form of (6) to the form of (7)?

At this point we will take advantage of the spherical subspace structure induced on the problem to doubly deflate the computation to a core 5x4 SVD. Write the appended and

A H A rotated input, j3=x k Vk-1, and define G = diag(Z~'/]Zl]," "Z~/]Zn]) so that 7 = ZG is real. Then, since G is unitary, we can rewrite the decomposition as

o r ,

x H - =[Uk-10 01jL G][0 01][ C-~DTk-~]GHVHk-t" (9)

Note that the core matrix is diagonal except for the bottom row and that all elements are real.

For notational convenience, denote the two left most matrices on the right hand side of (9) as Ok-1 and G . Block embedded matrices will follow this hat-convention in all that follows.

Let J1ER nxn be an orthogonal transformation that, when multiplied onto 7, annihilates the first r - 2 elements. Let J2ER nx'~ be an orthogonal transformation that, when multiplied onto 7, annihilates the last n - r - 2 elements. These transformations can be selected as sequences of plane rotations. Next, using the hat-convention, apply these rotations as (note the inner J 's cancel due to the structure of D in (6))

x k 7

to doubly deflate the core matrix,

Xk 1 0...7'...0 k-1

where now 7'ER lx4 and falls under the middle portion of Dk-1 as outlined in (6).

238 E.M. Dowling at al.

At this point the problem is nearly solved. All that remains is to diagonalize the core deflated 5 x 4 matrix given by,

0.1

0.2

0" 4

This can be done in a number of ways [5]. One parallel approach is to apply Jacobi sweeps which we collect as the J'a and J3 transformations (left and right sides) needed to complete the diagonalization. It turns out that in practice, one Jacobi sweep is usually sufficient to make the off diagonal elements in (12) close enough to zero. Thus we apply one sweep and set the off diagonal elements to zero. Once done, we are left with,

Xk - 3 - ~ - 1 k - 1

which can be readily put back into the form of (6). We see that the updated subspace is given by

Vk = Vk-~ GJ1J~J3. (14)

Now we reduce computational complexity by implicitly computing certain quantities�9 Suppose we are tracking the signal subspace. Then the J1 rotations must be applied to annihilate the top portion of 7, and must also be applied to the first r - 1 columns of Vk-1. The J~ rotations are applied implicitly and we only carry along two noise subspace basis vectors. We compute

/

12 (I v~+~ ~+~ xll~ (15) ,I. -Iv. ][v. ]") and

1 (I [V. ][V~ ]H)x. (16) V r + 2 -~ _.--7"--" -- V r + l V r + l 7r+2

The J1 and the J~ rotations do not affect vr , vr+l , 7r nor 7r+1.

The algorithm may be summarized as follows:

1. Compute ~ = xH[V, vr+,]= [ft, ~r+,] and [x]~ .

C o m p u t e r ( I - [ V s Vr+,][Vs V r + l ] H ) x = x - - V s ~ s - Z r + l V r + l . 2.

3. Compute the diagonal elements of G = diag(Z~'/]~l],'"Z*+l/]Zr+ll) and compute 7 = / 9 G .

4. Scale the columns of [V8 vr+l ] according to [Vs, vr+l]G H.

5. Using "h, . . . ,7r-1, compute the sequence of rotations J( i , i + 1,0) for i = 1 , 2 , . . . , r - 2 to annihilate all but the last element This will produce �9 %-1 �9 Apply these rotations

N

down the columns of Vs .


, - j and compute %' 2 - ilxll 2 _ ,2 ) 2 ,2 6. Set 7r' = T r ,%+1 - %+1 + - 7 r - l - - T r --%+1 �9 Check each update to insure this quantity is real, and if not, set it to zero.

7. Compute vr+2 = 1__~._r r + 2

8. Construct the 5• core matrix (12), compute and apply a :Iacobi sweep, and apply the right rotations, J3 to Vr-1, Vr, Vr+l, Vr+a.

9. Re-average, i.e., re-sphericalize the signal and noise subspace singular values according

i 2~ i (m- r -2 )~+~g wherea~ and a~ are the modified to al = (r- a~+~{~ and a4 = - 1 m - r - 1

values after the core Jacobi sweep is applied.

3 I N P U T P R O C E S S I N G A N D BASIS P R E A L I G N M E N T

In this section we show how to pipeline the sequence of J1 plane rotations together with the implicit J2 rotations which are implemented via an innerproduct according to (16). These computations serve to compress the input vector and prealign the spherical subspace bases. Other topological variations of the array structure are outlined in [6, 7]. We will refer to the steps of the above algorithm summary as we proceed.

The first step of the plane-rotation based SSA4 tracker is to compute/3 = x H [Vsvr+l] = [/~s/~r+l] and [Ixl[~. The [/3s /3r+l ] vector can be computed systolically using the scheme depicted in Figure 1, where we assume r = 4. Here the elements of the current input vector, x stream in one at a time and move to the right. Meanwhile, the signal subspace basis vectors stream in from above staggered by one element in order for the heads of each of the v-streams to meet with the head of the x-stream. With this alignment, for j = 1, 2, ..., r + 1 the jth cell can accumulate/~j = xHvj in place. The (r + 2) nd cell accumulates IIx]] 2.

* * * V3, 4 V2,5 *1%2* [vs,3 Ivy4 jv~,, * v~s ' 4 ~ v~l Ivy2 IVy3

71,1

Figure I: Innerproduct array for input compression and noise vector computation

Figure 2: Bi-linear array architecture for SSA4 updating. The boxes represent innerproduct cells and the diamonds represent rotation cells.

A convenient way to specify the operation of the systolic array is through the use of cell programs. Collectively, the cell programs specify the operation, timing, and data flow of the array. In this paper we do not have the space to report the cell programs, but the cell programs of closely related two-sided EVD versions of this algorithm (SA2 and SA4) are found in [6, 7]. C-language cell programs that fork multiple Unix processes which communicate through Unix shared memory segments have been implemented and tested. These programs are based on code supplied by G.W. Stewart.


Now consider how to pipeline steps 1 and 2 of the SSA4 tracking algorithm. The problem is to pipeline the step 2 computation right behind the computations of step 1. Note that the ~--outputs from step 1 are inputs to step 2. Hence step two can start execution as soon as the ~-values become available. As soon as the cells P(1)-P(r + 1) finish the computations of Figure 1, they start executing step 2. To see how, write r as

r ---- X - Yl/~l -- Y2~2 - - ' ' ' - - Vr-{-1/~r+l (17)

where vj denotes the jth column of V. Here each of the terms in this expansion are systolic streams that combine into an accumulator stream. During step 1, cell P(j) stores a local copy of vj , for j = 1, 2, ..., r + 1 and P ( r + 2) stores a copy of the input vector, x. Hence each cell of Figure 1 next generates one of the terms in (17) as soon as the /~-values become available. Here P(1) starts the process by propagating the v1~1 stream to the right. Next P(j) for j = 2, .., r accepts as input the accumulator stream containing acc = v~x + . . . + v/3j_l and outputs the stream acc = v1~1 + . . . + v~/~j. The last cell, P(r + 2) finishes the computation described by (17) and routes the v~+2-stream into a vector variable for later use.

Next the array computes the sequence of plane rotations to annihilate the first r - 1 elements of the -},-vector. The array also applies these rotations to the columns of Vs to accumulate the basis according to (11). The basis accumulation array and its interconnec- tion with the previously described innerproduct array is depicted in Figure 2. The cells in the basis accumulation array are represented by diamonds while those of the innerproduct array are represented by boxes. The cells with different functions from the others are highlighted. In the figure cells P(1, r), P(1, r + 1) and P(1, r + 1) double as innerproduct array cells and basis accumulation array cells. Also, the figure identifies the input and output streams of the various cells as labeled by the communications arrows.

To understand the operation of the basis accumulation array, assume that each P ( 2 , j ) for j = 1, 2, ...r + 1 contains the vector vj. The first thing that these cells do is propagate their initial contents south. This provides input to the array of Figure 1 via the connections described by Figure 2. After the P ( 2 , j ) cells send their vj vectors south, they must each receive back their jth fl-value. As soon as /3j is available, P ( 2 , j ) can compute the jth diagonal element of G as described in step 3 of the SSA4 tracking algorithm. Hence cells P(2,1) - P(2, r + 1) in Figure 2 send down their vj-vector and receive back/~j and compute gj and 7j. Next the cells exchange information in an overlapped pattern from left to right to compute the sequence of plane rotations that annihilate the first r - 2 elements of columns of 7 and produce 7 ' -1 . These rotations are applied in an overlapped systolic fashion to successive pairs of columns of V s = VsGJ1 �9 For more details to include cell programs, see [6, 7].

4 5 • J A C O B I O P E R A T I O N S

The computations of the last section computed 7 'ER 1• and set up [Vr-lVrV~+lVr+2] for their final set of core rotations. The array next computes the applies the core 5• Jacobi sweep to (12) and applies the right-rotations to the columns of [Vr-lV~Vr+lVr+2] �9 This core Jacobi sweep exchanges information between the signal and noise subspaces.


Figure 3: Application of Jacobi rotations to update the boundary subspace basis vectors.

The subarray, {P(i , j) l i = 1,2; j = r, r + 1, r + 2}, computes and applies the Jacobi sweep to (12) and applies the right rotations to [Vr-lVrVr+lVr+2]. First it applies a sequence of left rotations to convert (12) into an upper triangular matrix whose bottom row is annihilated. These operations involve only length-4 or less quantities so the communications needed is local. Next it applies the Jacobi rotations to diagonalize the 4• upper triangular portion following the method described on page 455 of [8] or in [2, 3]. The main computational load involves applying these right-rotations to [V~-lV~Vr+lV~+2]. The rotation locations in the subarray and the data flow is depicted in figure 3.

5 S S A 4 - M D L A N D G E A R - S H I F T I N G

The SSA4 array uses SSA4-MDL to track the number of sources. SSA4-MDL is described in the companion paper in this volume [1]. The cell programs for the two-sided version, SA4-MDL, are presented in [7] and do not differ much from the cell programs needed to implement SSA4-MDL. SSA4-MDL uses the four singular levels retained by the SSA4 algorithm to direct the array to increase, decrease, or not change the subspace dimension.

After the completion of the Jacobi operations, the top part of the subarray will contain the singular level estimates used to make the SSA4-MDL gear shift decision. If the decision is to leave the rank the same, when the Jacobi rotations are applied as depicted in Figure 3, the streams [v~-lv~vr+lvr-2] will return to their host cells in the top portion of the basis accumulation array. If the decision is to increase the rank, an idle cell in the left portion of the array will become active and all the basis vector streams will shift left by one cell. Here the second computed noise vector, vr+2 , becomes vr+l, and the new Vr+ 2 is computed according to (16) during the next update. If the decision is to decrease the signal subspace dimension, the left-most active cell becomes inactive and all of the cells route their host basis vectors one cell to the right while v~+l is discarded.

6 C O N C L U S I O N

In this paper we derived the Single sided Signal Averaged 4-level (SSA4) subspace tracker. The algorithm uses stochastic approximation techniques to reduce computation and is quite useful in slowly time-varying signal scenarios as encountered in practical sonar and radar angle and frequency tracking problems. We designed a pseudo-systolic array for real time


processing applications. The algorithm was designed using innerproducts and plane rotations, so that it is inherently parallel and amenable to pipelining. The array uses SSA4- MDL to track the number of signals and automatically gear shifts when the number of tracked signals changes. The array presented herein can be modified to l-D, 2-D and 3-D mesh topologies as discussed in [6, 7].

Acknowledgments

This work was supported in part by the National Science Foundation Grant MIP-9203296 and the Texas Advanced Research Program Grant 009741-022.

References

[1] E. M. Dowling, R. D. DeGroat, D. A. Linebarger, and H. Ye, "Sphericalized SVD updating for subspace tracking," in SVD and Signal Processing: Algorithms, Applications and Architectures III, North Holland Publishing Co. 1995.

[2] M. Moonen, P. VanDooren, and J. Vanderwalle, "Updating singular value decompositions. A parallel implementation," in SPIE Advanced Algorithms and Architectures for Signal Processing, (San Diego, California), pp. 80-91, 1989.

[3] W. Ferzali and J. G. Proakis, "Adaptive SVD algorithm and applications," in SVD and signal processing II, (Elsevier), 1992.

[4] E. M. Dowling, L. P. Ammann, and R. D. DeGroat, "A TQR-iteration based SVD for real time angle and frequency tracking," IEEE Transactions on Signal Processing, pp. 914-925, April 1994.

[5] S. VanHuffel and H. Park, "Parallel reduction of bounded diagonal matrices," Army high performance computing research center, 1993.

[6] Z. Fu, E. M. Dowling, and R. DeGroat, "Spherical subspace tracking on systolic arrays," Journal of High Speed Electronics and Systems, Submitted may 1994.

[7] Z. Fu, E. M. Dowling, and R. D. DeGroat, "Systolic MIMD architectures for 4-level spherical subspace tracking," Journal of VLSI Signal Processing, Submitted Feb. 1994.

[8] G. H. Golub and C. F. VanLoan, Matrix Computations, 2nd Edition. Baltimore, Mary- land: Johns Hopkins University Press, 1989.


243

S Y S T O L I C A R R A Y S F O R S V D D O W N D A T I N G

F. LORENZELLI, K. YAO Electrical Engineering Dept., UCLA Los Angeles, CA 90024-159~, U.S.A. lorenz@ee, ucla.edu, yao@ee, ucla.edu

ABSTRACT. In many applications it is required to adaptively compute the singular value decomposition (SVD) of a data matrix whose size nominally increases with time. In situations of high nonstationarity, the window of choice is a constant amplitude sliding window. This implies that whenever new samples are appended to the data matrix, outdated samples must be dropped, raising the need for a downdating operation. Hyperbolic transformations can be used for downdating with only minor changes with respect to the updating algorithms, but they can be successfully used only on well conditioned matrices. Orthogonal downdating algorithms which exploit the information contained in the orthogonal matrices of the decomposition are less prone to instabilities. Here, we consider hyperbolic and orthogonal downdating procedures applied to two algorithms for SVD computation, and discuss the respective implementational issues.

KEYWORDS. SVD, orthogonal downdating, hyperbolic transformations.

1 M O T I V A T I O N

In numerous situations, such as adaptive filtering, bearing estimation, adaptive beamforming, etc., the received signals (appropriately sampled and digitized) are stored in a matrix form. Linear algebraic techniques are subsequently employed to extract from these matrices useful information for further processing. The quantities of interest are typically given by numerical rank, individual singular values and/or singular vectors, or global information about singular (signal/noise) subspaces. The estimation of these entities is then used for the calculation of filter weights, predictor coefficients, parametrical spectral estimators, etc.

The receiving signals are quite commonly nonstationary in nature, and so are all the quantities pertaining to the data matrices. Moreover, it can be analytically proved that some characteristics of the data would change in time even in a stationary environment, by the mere presence of noise. These are the reasons which motivate the interest in adaptive algorithms, i.e., algorithms which can track the time variability of the involved signals.

244 F. Lorenzelli and K. Yao

As data change, it is advisable to give prior importance to the latest samples and discard the outdated ones. It is thus common practice to "window" the data matrices prior to any computation. There are many different kinds of windowing techniques, but two have received the most attention, namely exponential forgetting and the sliding window approach. The former method involves the multiplication of the old data by a forgetting parameter 0 < fl _( 1, before the new samples get appended to the data matrix. As a consequence, each sample decays exponentially with time. In turn, the alternative sliding window approach retains the same number of samples at any given time, thus requiring one (block) downdate per each (block) update. Exponential forgetting is usually accomplished with only minor changes to the fundamental algorithms. This is in contrast to the more complicated operation of downdating, required in the sliding window approach. Hsieh shows in [6] that the sliding window approach is more effective than the exponential forgetting in highly nonstationary environments, even when the parameters are optimally chosen.

Many algorithms can be used to solve the problems listed above, with different tradeoffs of computational complexity, parallelism, up/downdating capabilities, etc. The theoretically best algorithms are usually based on the singular value decomposition (SVD) of the data matrix. Despite its numerous properties, the SVD was until recently considered too computationally intensive and exceedingly difficult to update. With the advent of parallel architectures, this view has been changing and various SVD parallel algorithms have been proposed. Of particular interest are the linear SVD array based on Hestenes' algorithm and proposed in [3] and the 2-D SVD array based on Jacobi rotations [10] and fully analyzed in [4], [7] and [8]. The former array performs block computations, whereas the latter is ideally suited for updating with exponential forgetting. Both algorithms can be extended to include up/downdating with sliding window, by making use of hyperbolic or orthogonal transformation.

2 T H E J A C O B I SVD A L G O R I T H M - H Y P E R B O L I C D O W N D A T I N G

The Jacobi SVD algorithm can be transformed to include hyperbolic downdating oper-

ations. Consider themat r ix ( A ) z_ , where z_ is the information to be downdated. Let

A = ~r ( ~ ~H be the SVD of A. Let the hyperbolic transformation H be chosen so that \ ] 0

0 - H R (~) 0 ~T~ (re+l) xCm+l)

and g is an n x n upper triangular matrix. In general, let H be hyperexchange with respect to ~, in the sense that

n m - n + l

0 -1 ' m - n + 1 0 ~2 '

where ~k = d iag (+ l , . . . , -4-1), k = 1, 2. If Jt and Jr are the Jacobi rotations that diagonalize R, i.e., JeAJ H = R, and Je is hyperexchange with respect to ~1, so that JH~I Je - ~3, we

Systolic Arrays for SVD Downdating 245

obtain

(a) AHA - xHz -- r -- ~," ( ~ (_X~r)S)(~ X_. X_

= v~?(RH o ) H H ~ H ( R ) v H - V A V H 0

where V = V Jr and A_. = A~3A. As suggested in [9], by using hyperexchange matrices instead of hypernormal matrices (for which (I) = (I)), possible ill-conditioning can be detected by the negative elements of A_. If the problem is well posed, we have of course A_ = ~.._2.

2.1 GENERATION OF THE HYPERBOLIC TRANSFORMATIONS

In recursive Jacobi SVD algorithms, the rotations involved in the triangularization of (1), as well as the Jacobi rotations ,It and Jr, are usually computed from sequences of 2 x 2 transformations. Consider for the time being the triangularization operation of (1).

There are three main kinds of hyperexchange 2 x 2 rotations, according to how the signature matrices change. Let /2 be the signature matrix identical to the 2 x 2 identity matrix, and

1 01 ~ . �9 ( 0 ) ( : 0 2 - ' 1 /

( ) (~ Let H2 = hll h12 the 2 x 2 transformation imposed on vector v = The three h21 h22 b "

possibilities are as follows: 1) /2 ~ I2: In this case, the rotation is orthogonal and well defined. We have hll = h22 = cos S, h12 = -h21 = sin 0, for some angle 0. 2) ~2 ~ ~2, or ~2 ---* ~2" The hyperbolic transformation H2 is well defined if la[ > Ibl. We have that hll = h22, h12 = h21, and h~l - h122 = 1. 3) ~2 --* ~2, or ~2 ~ ~2" In this case lal < Ibl. It follows that hll = h22, h12 = h21, and h21 - h122 = -1 .

Notice that the rotations listed above, corresponding to signatures different from the identity, are defined only when [a[ ~ Ib[. It is possible to implement the rescue device suggested in [9] for these cases in which the hyperbolic transformation cannot be defined. The triangularization of (1) is replaced by the following two-sided transformation:

0 = H R GH x_.V 0 '

where H is again hyperexchange with respect to (~, while G is an orthogonal rotation. In order to explain how the matrices G and H are computed, consider a generic diagonal processor of the algorithm in [8]. This processor has access to the 2 x 2 submatrix R2 =

0 d " During the triangularization step, the 2-element vector (v, w), derived from the

product x . V, is input into the processor. If Iv[ ~ [al, then a hyperbolic transformation can be generated which zeroes out the vector component v against the matrix entry a. If [v I = la[, then additional computation is required. In particular, left and right orthogonal rotations, respectively by angles r and r are generated and applied as follows (ci = cos r

246 F. Lorenzel l i and K. Yao

si = sin r i = 1, 2):

0 d" = - 8 2 c 2 0 -k d ~ = - 8 2 c2 0 0 d Cl - 8 1 .

v ~ w ~ 0 0 1 v ~ w ~ 0 0 1 v ~/) 81 CX

The rotation angle Cx is chosen so that la" I # Iv'l. The sub diagonal fill-in, here symbolized by a ' , ' , is zeroed out by the left rotation by angle r One can choose the rotation angle r which maximizes the difference (a") 2 - (v ' ) 2. It can be shown that angles r and r satisfy tan(2r ) = 2(ab - v w ) / (a 2 - b 2 - d 2 - v 2 -t- w 2 ), tan r = - d sin r / (a cos Cx + b sin r ). The rotation angle r is then propagated to the V array.

Consider now the rediagonalization step. In this case, left and right rotations are simultaneously generated. With a careful selection of rotation parameters, it is always possible to generate an orthogonal right rotation and a hypernormal left transformation which diagonalize a generic 2 • 2 triangular matrix R2, defined as before, i.e.,

s c 0 d a 7 ' -y

hyperbolic orthogonal

Feasible hyperbolic and orthogonal parameters are given by

(1 + r) t/2 + (1 - r) 1/2 (1 + r) 1/2 - (1 - r) 1/2 a (bc + ds)

c = 2 ( 1 - r ) 1 / 4 , s = 2 ( 1 - r ) l / 4 ' 7 ac '

where r - - 2 b d / ( a 2 + b 2 -t- d2). The above implies that the rotation J~ can always be made ^ hypernormal, and ~3 = ~x. Combining the triangularization and the rediagonalization steps, we obtain

A H A - xHx_. = V G J r A @ I A j H G H v H = V A V H,

where V = V G J r , and A = A ~ I A . Ill-conditioned situations are signalled by negative elements on the main diagonal of the signature ~1, which is determined during the triangularization step.

3 T H E J A C O B I SVD A L G O R I T H M - O R T H O G O N A L D O W N D A T I N G

The calculation of hyperexchange transformations, together with the tracking of the signatures, may be inconvenient in certain applications, due to the additional computational burden. On the other hand, regular hyperbolic transformations are known to be prone to instabilities when the matrix to be downdated is particularly ill-conditioned, or in presence of strong outliers. An alternative approach is to use orthogonal transformations throughout [5]. But even this approach does not seem to solve the problem of instability. Recent analyses by BjSrk, Park and Eld~n [2], and Bendtsen et al. [1] in the context of QR downdating and least squares problems, show that any method (hyperbolic or orthogonal) which is based on the triangular factor and the original data matrix may lead to a much more ill-conditioned problem than using the additional information contained in the orthogonal factor. Among the techniques compared in [2] is the Gram-Schmidt (GS) downdating algorithm, which is characterized by relative simplicity of implementation and accuracy of the


results. In all the experiments conducted, the GS downdating algorithm displays a behavior comparable to the sophisticated method of corrected seminormal equation and superior to the LINPACK downdating algorithm, which is an orthogonal downdating algorithm exclusively based on the triangular factor of the QR factorization. The storage requirement is the same as for the hyperbolic downdating (note that the storage of outdated data samples is here replaced by the need to keep track of the orthogonal matrix). On the basis of these considerations, we propose here an alternative algorithm for SVD downdating which make use of orthogonal transformations only and exploit the information contained in the left singular matrices.

The updating algorithm which constitutes the basis for the Jacobi SVD algorithm requires that three distinct operations be performed: a vector-by-matrix multiplication, a QR update, and a sequence of Jacobi rotations to restore diagonality. If an additional (n + 1) x (n + 1) array is available for storage of the left singular matrix U s, then during update all the left rotations involving rows i and j of R will also be propagated along the ith and j t h rows of U n. After update, the matrix U is (n + 1) x (n + 1). For the downdate

consider the following. Let A = ~ = ~ - ~ H be the SVD of .4. It is desired to compute

the SVD of the downdated matrix B = U 2 V H. We have that

1

The equations above suggest the following procedure:

for i = 1 to n, apply a rotation to the ith and (i + 1)th rows of ~rn in order to zero out U(1, i); apply the conjugate of the previous rotation to rows i and (i + 1) of ~; this operation generates fill-in in positions (i, i + 1) and (i + 1, i); zero out element (i + 1, i) using a right rotation on columns i and (i + 1) of ~; apply the same rotation to columns i and (i + 1) of V.

end

Now ~ is upper triangular. Moreover U(1, i) = 0 for i = 1 to n. Zero out the elements of above diagonal of rows 1 to n using a sequence of Jacobi rotations. The conjugate of the

left rotations are applied from the right to the columns 1 to n of U. The right rotations are also applied to the right of V. Eventually, we have

~ = u_ o ' = Z v ' ~ = y

Note that the downdated vector is now generated as an output. The storage of the data matrix for downdating is now replaced by the storage of the U matrix, of identical size. This approach pipelines with the updating operations.


3.1 SVD UP/DOWNDATING IN PARALLEL

The parallel algorithm that we present here is based on the scheme proposed in [8], and we assume that the reader is familiar with its operation. In particular, the E matrix is stored in a triangular array, composed of O(n 2) locally connected processing elements, each of which has access to four memory cells (entries of ~). All the processors are capable of both column and row rotations, and the processors on the main diagonal additionally perform 2 • 2 SVD's. The particular array for the updating of the V matrix is not of particular concern in this paper, and will be neglected in the following. In the sequel, we assume that each update is followed immediately by a downdate, so that the size of the U-matrix oscillates between (n + I) x (n + i) and n x n. For these reasons, we propose to use an additional n • 1 linear array of processors, each having access to two adjacent rows of U H. juxtaposed to the diagonal of the E-array, so that the ith column of U corresponds to the ith row of E (except for U(:, n + 1)).

In order to explain how the proposed array works, let us assume that the updating is completed and the (n + 1) 2 entries of the U-matrix are non-zero. Starting from the top pair of rows of U H, the rotations which sequentially zero out the first n entries of U(1, :) are generated and propagate through the matrix. If this operation starts at time slot 1, then the rotation parameters reach the diagonal elements of E at times 1, 3 , . . . , 2 n - 1. When this occurs, the same rotations are propagated row-wise through the E-matrix. As explained earlier, the left rotations produce fill-ins in the first sub diagonal of E, which can be immediately zeroed out by the diagonal processors by the use of right rotations. The rotation parameters associated with these column rotations are subsequently propagated to the columns of E and those of the V matrix.

A complication is due to the fact that at the end of the procedure just explained the contents of the square U-array are not properly arranged for the subsequent update. The required transformation is shown in the diagram below, for the case n = 3, where the • symbolize generally non-zero entries. The matrix on the left shows the result of the above operations on U s , while the matrix on the right shows the ideal configuration of the square array at the beginning of the updating step. 0)

0 x x x x x x 0

0 x x x x x x 0 "

1 0 0 0 0 0 0 1

This problem can be easily solved by reindexing the elements of each column of U. No physical data movement is required. At the end of the downdating operation, the three arrays (containing the elements of U s, ~, and V) are ready for the update. The incoming data vector is input into the V matrix and the z = z. V product is input into the top row of the triangular array. Simultaneously a row vector w initialized at ( 0 ,0 , . . . , 0 , 1 ) is input at one end of the linear U-array. The rotation parameters which annihilate the ith element of z are computed by the diagonal processors in the triangular array, and propagated into both the ~ and U arrays. The rotation is applied to vector pairs (z, ~(i, :)) and (w, vH(i, :)).

Following arguments similar to those put forward in [8], and keeping in mind that left (right) rotations involving disjoint pairs of rows (columns) commute, one can prove that the


rotations relative to the different steps of QR update, downdating, and rediagonalization can be pipelined and interleaved. The pipeline rate is basically unaltered with respect to the updating SVD array of [8]. Note that because of the downdating algorithm chosen, the matrix U must be represented explicitly and not in factorized form. The scheme so far presented may be viewed as an intermediate design towards a one-dimensional SVD array of O(n) elements where each memory cell stores an entire row of the matrices ~ and U H.

4 T H E H E S T E N E S SVD A L G O R I T H M - H Y P E R B O L I C D O W N D A T I N G

(A) Consider Brent & Luk's array performing the SVD of the (m + 1) x n matrix A0 =-

where x_ is the data row to be downdated. As suggested in [9], the eigenvalues and eigenvectors of the downdated correlation matrix A H A - xHx can be obtained by generating a sequence of unitary rotations V j such that for growing i,

A, = Ao l I v_j - - . __.v o ' j=o (m+l)•

(m-t.-1) X n

where ~ is an n • n diagonal matrix and U is hypernormal with respect to the signature

matrix ~ = (Imo -10 ) , (where Im is the m • identity matrix)in the sense that _uH~u_ =

~. After convergence, one has that

A H A - x H x = r = V ( ~ O) ~ U = V . x_. x__ . . . . .

The hypernormality of U can be ensured as follows. The sequence of matrices Ai+l = A i ~ is generated by using plane rotations, V~, which operate on pairs of columns of Ai. The rotation parameters are selected in such a way as to make the resulting column pairs hypernormal. The procedure is iteratively executed by selecting sweeps of all the possible different column pairs, and by repeating the sweeps a sufficient number of times (until convergence). There are numerous strategies for choosing the order in which column pairs are rotated. One efficient ordering, suitable for parallel implementations, is given in [3].

5 T H E H E S T E N E S SVD A L G O R I T H M - O R T H O G O N A L D O W N D A T I N G

As mentioned previously, hyperbolic transformations are susceptible to ill-conditioning and may generate numerical instabilities. Here, we present a possible procedure for orthogonal up/downdating and give a high level description of the required array of processors.

For simplicity of up- and downdate, consider the SVD of A H (for the sliding window approach, we can assume that m is not too much larger than n). Also, assume that m is an odd number. The linear array is composed of (m + 1)/2 elements, each containing a pair of columns of A s (in practice, only (n + 1)/2 elements are required). At the end of a sweep [3], the processing elements contain the n columns of W = V~, together with the n columns of U, and m - n + 1 columns of zeros. During update, the (m + 1)st new column ~S is appended to the rightmost element and the columns of (W, 0, ~H) are orthogonalized


to produce the updated matrices I~ and ~'. In fact, if Q is the orthogonal matrix used for reorthogonalization,

( w ~H) o 1 = ( W )Q o 1 = ( ~ v o) ~ . (n+l) x(m+l)

For the downdating, consider the following. Let ~H = (~H, ~-H)H, and let G be the orthogonal matrix that orthogonalizes the columns of ~H. Then we have

(~ 0)(~ @)=(# 0)cc'(~ Y Ztt

Up)= ( w o)(zQ) H = (~_ w uH),

where Q is the matrix that orthogonalizes Y and _U = M(2 �9 m % 1,1 �9 n), M - ZQ. Notice that downdating requires two reorthogonalization steps, one on matrix Us and one on matrix Y, and is thus twice as expensive as the updating step. The reason for this is that the information on the individual matrices Z and V is unavailable, and one needs to work on the matrix W = VZ.

Acknowledgments

This work was partially supported by the NSF grant NCR-8814407 and the NASA-Dryden grant 482520-23340.

References

[1] C. Bendtsen, P. C. Hansen, K. Madsen, H. B. Nielsen, and M. Pmar. "Implementation of QR Up- and Downdating on a Massively Parallel Computer". Report UNIC-93-13, UNI.C

[2] A. BjSrk, H. Park, and L. Eld4n. "Accurate Downdating of Least Squares Solutions". SIAM J. Mat. An.~ Appl., 15(1994), to appear.

[3] R. P. Brent and F. Luk. "The Solution of Singular-Value and Symmetric Eigenvalue Problems on Multiprocessor Arrays". SIAM J. Sci. Star. Comput., Vol. 6, No. 1, pp. 69-84, 1985.

[4] W. Ferzali and J. G. Proakis. "Adaptive SVD Algorithms for Covariance Matrix Eigenstructure Computation". In Proc. ICASSP, 1990.

[5] G. H. Golub and C. F. Van Loan. "Matrix Computations". John Hopkins, 2nd ed. 1989.

[6] S.-F. Hsieh. "On Recursive Least-Squares Filtering Algorithms and Implementations". PhD Thesis, UCLA, 1990.

[7] M. Moonen, P. Van Dooren, and J. Vandewalle. "A Singular Value Decomposition Updating Algorithm for Subspace Tracking". SIAM J. Mat. An.~ Appl., 13(4):1015-1038, October 1992.

[8] M. Moonen, P. Van Dooren, and J. Vandewalle. "A Systolic Array for SVD Updating". SIAM J. Mat. An. ~ Appl., 14(2):353-371, April 1993.

[9] R. Onn and A. O. Steinhardt. "The Hyperbolic Singular Value Decomposition and Applica- tions". IEEE Tr. SP, 39(7):1575-1588, July 1991.

[10] G. W. Stewart. "A Jacobi-like Algorithm for Computing the Schur Decomposition of a Non- hermitian Matrix". SIAM J. Sci. Star. Comp., 6(4):853-863, 1985.


251

S U B S P A C E S E P A R A T I O N B Y D I S C R E T I Z A T I O N S OF D O U B L E B R A C K E T F L O W S

K. HOPER, J. GOTZE, S. PAUL Technical University of Munich Institute of Network Theory and Circuit Design D-80290 Munich Germany knh u @n ws. e- tech n ik. tu- m ue nche n. de

ABSTRACT. A method for the separation of the signal and noise subspaces of a given data matrix is presented. The algorithm is derived by a problem adapted discretization process of an equivalent dynamical system. The dynamical system belongs to the class of isospectral matrix flow equations. A matrix valued differential equation, whose time evolution converges for t ~ oc to block diagonal form is considered, i.e., only the cross-terms, correlating signal and noise subspaces, are removed. The iterative scheme is performed by computing some highly regular orthogonal matrix-vector multiplications. The algorithm essentially works like a Jacobi-type method. An updating scheme is also discussed.

KEYWORDS. Double Bracket flow, gradient flow, block diagonalization, principal eigenspace, Jacobi-type methods.


Given a data matrix H E ~nxm (n > m), a frequently encountered problem in signal processing (DOA-estimation, harmonic retrieval, system identification) is the separation of the column space of H into signal and noise subspaces. The SVD of H is the most robust and reliable tool for this task. The signal subspace is defined by the right singular vectors corresponding to the large singular values. Computing the SVD of H, however, is computationally expensive and yields much more information than necessary to separate the signal and the noise subspaces. In order to avoid the computationally expensive SVD, various methods for determining the subspaces in a computationally less expensive way have been published. Among these methods are the Rank-Revealing QRD [2], the URV decomposition [11] and the SVD-updating algorithms [8, 9] which can be considered as

252 K. Haper et al.

approximate SVD's. Other methods not based on the SVD have also been proposed, e.g. the Schur-type method presented in [4]. Another quite obvious approach is the block diagonalization of H, i.e.

H = U 0 H22~ V T ,

0 0

where U E O(n), V E O(m), and Hllo~ E/R dxd, H22r E/R m-dxm-d have dimensions due to the dimensions of signal and noise subspaces, respectively. It is not straight forward, however, to extend known linear algebra algorithms such, that any block diagonalization of an arbitrary matrix is obtained. In this paper an algorithm for block diagonalizing a given matrix by orthogonal transformations is derived. This is achieved by a problem adapted discretization process of an equivalent dynamical system. The matrix flow is Brockett's double bracket flow [1] H = [H, [H, N]], choosing the matrix N appropriately with respect to the subspace separation problem. Essentially, after discretization, this results in a Jacobi- type method which only works on the cross terms H12 E 1R dxm-d (assuming a preparatory QRD). Therefore, only d ( m - d ) rotations are required per sweep, while the standard Jacobi- type methods apply m ( m - 1)/2 rotations. The algorithm can be considered as a method for maximizing an objective function of a continuous-time gradient flow. It is a specific example of a gradient ascent method using geodesic interpolation (see [5], [7], and [10] for related work). An updating scheme for this algorithm is very similar to the SVD-updating scheme of [8]. Again, the updating scheme only operates on the cross terms.

2 D Y N A M I C A L S Y S T E M F O R B L O C K D I A G O N A L I Z A T I O N

In this section a dynamical system for block diagonalizing symmetric matrices with distinct eigenvalues is briefly presented. We shall state the necessary propositions without proofs. For convenience, we have decided to present the results for symmetric matrices (covariance matrices). The extension to arbitrary n x m data matrices (SVD) is straightforward.

The isospectral matrix flow is governed by

= [H,[H,N]], (1)

and the associated flow on orthogonal matrices by

6) = O[H,N], (2)

with H = H T E IR re• O(t) E O(m), [X, Y] d*d X Y - Y X , and N = diag(Id, 0,~-d).

Proposition 2.1 Eq.(1) is the gradient flow H = grad f ( H ) of the function f ( H ) = - t r H n . Eq.(2) is the gradient flow 0 = grad r of the function r = - t r NOTHoO + tr H0 = f ( n ) .

R e m a r k The gradient flows in Proposition 2.1 are defined with respect to different metrics. See [5] for details.

P ropos i t i on 2.2 For t ~ oo the time dependent H(t) of (1) with initial H(O) converges

Discretizations o f Double Bracket flows 253

to block diagonal form:

[Hl1(0) H12(0) 1 t--*r162 [ H l 1 ~ 0 H(O)= H~(O) H~(O) ' 0 H ~

with Hll E 1R dxd, H12 E 1R dxm-d, and H22 E ~m-d• Assuming a(Hllcr a(H22or = {}, for all )q E a ( H ~ ) and for all )~j e a(H22~r holds )~i > )~j. For t ~ c~ the time dependent O(t) o./'(2) with initial 0(0) e O(m) converges to (|162162 02~r with 0Tr162162162 = Id and 0Tr162 = Im-d. It holds

span(OlcclHOxcr = 01r162162162 s p a n { q 1 , . . . , q d l H q i = qi)q, Ai E a(H11~r

span(O2cclH02r162 - 02r162162162 s p a n { q d + l , . . . , q m I H q i = qiAi, Ai e a(H22r162

Proof: see e.g. [7].

3 D I S C R E T I Z A T I O N S C H E M E F O R B L O C K D I A G O N A L I Z A T I O N

A discretization scheme for the above gradient flow is now presented. Taking into account that at each time step only an orthogonal transformation in a specific 2-dimensional plane of the matrix H is performed, a modification of the cyclic-by-row Jacobi method results. This method works with the full group SO(2) or equivalently, the rotation angles r e [-7r/2, ~r/2]. Excluding certain pathological initial conditions (saddle points) which in general do not occur for subspace separation problems, this algorithm is obviously globally convergent. Furthermore local quadratic convergence for the case of full diagonalization (including multiple eigenvalues) was recently proved using methods from global analysis [6] (see also the contribution by U. Helmke to this workshop). Here we borrowed from [3] the very readable MATLAB-like algorithmic language and notation.

Algor i thm 3.1 Given a symmetric H E j ~ m • and p, q E 1N that satisfy 1 < p < d and d + 1 <_ q <_ m, this algorithm computes a cosine-sine pair (c,s) such that if H I =

I I ~> I J(p,q)rHJ(p,q) th~,~ h~ = O a,~d h~ _ h~.

fu,,~tion: ( j~, j~ , j~, j~) = ~ym..~r (H, p, q) if hpq ~ 0

hqq - hpp sign r 1 r = 2hpq , t = I r l + v / l + r 2 ; c = v / l + t 2 ; s = t c

else c = 1; s = 0

end if hpp <_ hqq

jpp = - s ; jpq = c; jqp = c; jqq = s else

jpp = c; jpq = s; jqp = - s ; jqq = c end

end sym.sehur2 . sor ted

Algor i thm 3.2 (Block Diagonalizing Cyclic Jacobi) Given a symmetric H E 1R 'n• and a tolerance e >_ O, this algorithm overwrites H with block diagonal H I = OTHO where

254 K. Htiper et al.

0 E O(m) and A f = t r ( H ~ l - Hll) < e. It returns 0 = (01,02) with 01(02) being an orthogonal basis.for the principal d-dimensional (minor ( m - d ) - dimensional) eigenspace of H, respectively.

function: (0, H) = bloek.diag. jaeobi (0, H, e) O = I,~ trOld = 0 t r N e w = t r a c e ( H ( Z : d, 1: d)) while ( t r N e w - trOld) > e

for p = l : d for q = d + l : m

(jpp, jpq, jqp, jqq) - sym.schur2 .sor ted(H, p, q) H = J(p, q)THJ(p, q); | = | q)

end end t ~ O t d = t~ lve~; t ~ N e ~ = t ~ e ( H ( 1 : d, 1: d))

end end block.diag. jacobi

Obviously, Algorithm 2.2 is essentially a Jacobi-type method. The case of fully diagonalizing the matrix H corresponds to choosing the matrix N with distinct eigenvalues. Modifying Algorithm 2.2 such, that each sweep works on the whole off-diagonal part, the standard cyclic-by-row Jacobi method results, except that rotations sorting the diagonal entries are applied via Algorithm 2.1. Note, that instead of the off-diagonal norm the function trace(NH) is used as the optimization criterion.

4 U P D A T I N G P R O C E D U R E

In a time varying environment, the most common strategy for updating the covariance matrix H is the rank one update, which involves applying a forgetting factor # to H due to nonstationary processes. Suppose that at a certain time step, the matrix H is reduced to block diagonal form via Algorithm 3.2, with associated invariant subspaces span(| and span(02). Our update procedure is as follows (number of sweeps is predetermined):

A lgo r i t hm 4.1 (Upda te ) Given a symmetric H = diag(Hll,H22) and associated O = (01, Oz), a new data vector x and a possibly time-varying forgetting factor #, with Hll E j~dxd H22 E j~m-dxm-d oTI 01 _. Id, oT~)2 = In-d, and x E 11:I mxl

H - #H ~-OTxxTO

(0, H) = bloek.diag. jacobi(0, H, e)

5 S I M U L A T I O N S A N D D I S C U S S I O N

The performance of the different Jacobi methods is compared for the subspace separation problem using a frequency estimation scenario, n = 100 sample points of a signal composed

Discretizations o f Double Bracket flows 255

of 2 sinusoids is used, i.e.,

s(t) = 2 cos(2~an) + 2 cos(2~bn) + w(n),

where a = 0.15, b = 0.25 and w(n) is Gaussian white noise. Data vectors of dimension m = 8 are formed generating the n x m Toeplitz data matrix X.

Computing the signal (noise) subspace of X requires the separation of the row space of the data matrix X (the row space of the covariance matrix A = x T x , respectively). Once these subspaces are computed the unknown parameters can be estimated using methods like ESPRIT or MUSIC. Usually, the subspaces can be obtained by computing the SVD of the data matrix X or the EVD of the covariance matrix A. Assuming a certain signal to noise ratio SNR the row space of X ( A ) can be separated into signal and noise subspace according to the magnitude of the singular(eigen)values, respectively. The right singular(eigen)vectors corresponding to the large singular(eigen)values form the signal subspace. In our example the dimension of the signal subspace is d = 4 (two times the number of signals in case of working with real data).

In the following simulations we distinguish between three different Jacobi-type methods:

�9 SCJ (standard cyclic Jacobi): a diagonalization of A is executed applying rotations (8 _< ~r/4) which do not take into consideration the magnitude of the resulting diagonal matrix elements.

�9 SORTCJ (sorting cyclic Jacobi): a diagonalization of A is executed applying rotations (8 _< 7r/2), which sort the resulting diagonal elements according to their magnitude.

�9 SORTCJ-CT (SORTCJ working only on the cross terms): it works like SORTCJ but only the cross terms are annihilated in each sweep, i.e., one sweep consists of d ( m - d ) rotations. Note that for working only with the cross terms the sorted version of the Jacobi method is mandatory in order to guarantee convergence.

Also, two different off-diagonal quantities are considered in the following:

�9 sd = I I A - DIIF , where D = diag(aii), i.e., the square root of the sum of the squares of all off-diagonal elements (the usual off-diagonal norm).

�9 8c = IIA(I: d, d + 1: m)llF , i.e., the square root of the sum of the squares of the cross terms, only.

First, SCJ and SORTCJ are compared- both methods actually diagonalizing the matrix. In figure 1 the reduction of the off-diagonal quantities sd and sc is shown for SCJ and SORTCJ. Obviously, the diagonalization of the matrix (i.e. considering sd) works as well for SCJ as for SORTCJ. Reduction of the cross terms (i.e. 8c), however, is significantly faster for SORTCJ than for SCJ. Therefore, SORTCJ performs favorable with respect to the subspace separation task.

In figure 2 the reduction of the cross terms sc is shown for SORTCJ and SORTCJ-CT. Obviously, in this case convergence of SORTCJ-CT is only linear, such that a greater number of sweeps is required. Note also, that sc even increases in the beginning of SORTCJ-CT.

256 K. Htiper et al.

10 5

10 0

o 10-5 "1o

o

~e" 10 "1~

10 "15

10 .20 1

�9 ~ .,~...

.......... SCJ Norm sc """,,' ' "~i"~i~)' ~' """ "..

. . . . . . . SORTCJ Norm sd x ~,. "..

. .. \ ~ ..

"'.. ~ ",-

SORTCJ Norm sc ~ ".. ":

~ ". i i i i i 2 3 4 5 6 7

s w e e p s

Figure 1" Reduction of the off-diagonal norms sd and sc vs. sweeps for SCJ and SORTCJ.

10 5

10 0

o 10 .5

E =o, ~1= 0.10 O 1

10 "15

1 0 "2o

SORTCJ-CT

',,, ~ -SORTCJ

I \

" 'o ' 'o ' 5 1 15 2 25 sweeps

30

Figure 2: Reduction of the cross terms, i.e. sc, vs. sweeps for SORTCJ and S O R T C J - C T .

Discretizations of Double Bracket flows 257

10 0 , , , ,

/

i' , v ,,','",",,"

~ 10 -lo

1 0 " 1 6 ~

. 1 8 ~ 10 0 20 4=0 60 80 100

t

Figure 3: Distances between updated subspaces and exact instantaneous subspaces vs. time (solid line: SORTCJ s = 1; dashed line: SORTCJ-CT s = 1; dotted line: SORTCJ-CT s = 3 ) .

However, this does not contradict to the convergence of the algorithm, since the optimization criteria used for the derivation of SORTCJ-CT (SORTCJ) is not the off-diagonal norm but the trace of the matrix H weighted by the matrix N! Now, let fs denote the factor of the increase in the number of sweeps, i.e., fs = sweeps(SORWCJ-CW)/sweeps(SORWCJ) (in our example fa ~ 5). SORTCJ-CT only requires Nc = d ( m - d) rotations per sweep while SORTCJ requires Ns = m ( m - 1)/2. Therefore, SORTCJ-CT is advantageous for N c f a < Ns, i.e., for d << m.

Finally the tracking capabilities of the different algorithms are considered. Figure 3 shows the ensemble average (20 runs) of the distance between the signal subspaces ~'(t) computed by the updating procedure and the instantaneous exact subspaces V(t). The distance between these subspaces is measured by dist(V(t), Y(t)) = ( 1 - a2in(vT( t )Y( t ) ) 1/2 [3]. The updating by SORTCJ uses s = 1 sweep per updating (solid line). The updating using SORTCJ-CT is shown for s = 1 (dashed line) and s = 3 (dotted line) per update.

6 C O N C L U S I O N AND O U T L O O K

In this paper a Jacobi-like method is presented for block diagonalizing symmetric matrices with application to subspace separation problems. A generalization to SVD is straight forward. The algorithm is also well suited to neural network applications, where block diagonalization into more than two blocks is required.

258 K. Haper et al.

Acknowledgments

The authors are grateful to W. Mathis for drawing their attention to the topic of isospectral matrix flows. They would also like to thank U. Hehnke who spent considerable efforts for introducing them to the optimizational point of view and R.E. Mahony for making his thesis available. The authors are indebted to J.A. Nossek for his continuous support and many helpful discussions.

References

[1] R. W. Brockett. Dynamical systems that sort lists, diagonalize matrices, and solve linear programming problems. Lin. Algebra ~ Applic., 146:79-91, 1991.

[2] T.F. Chan. Rank revealing QR-factorization. Lin. Algebra 8~ Applic., 89:67-82, 1987.

[3] G. Golub and C. F. van Loan. Matrix Computations. The Johns Hopkins University Press, Baltimore, 2nd edition, 1989.

[4] J. GStze and A.-J. van der Veen. On-line subspace estimation using a Schur-type method. IEEE Transactions on Signal Processing, 1993. submitted.

[5] U. Helmke and J.B. Moore. Optimization and Dynamical Systems. CCES. Springer, London, 1994.

[6] K. Hiiper and U. Helmke. On the convergence and structure of Jacobi-type methods, 1994. preprint.

[7] R.E. Mahony. Optimization algorithms on homogeneous spaces. PhD thesis, Australian National University, 1994.

[8] M. Moonen, P. van Dooren, and J. Vandewalle. A singular value decomposition updating algorithm for subspace tracking. SIAM J. Matrix Anal. Appl., 13:1015-1038, 1992.

[9] M. Moonen, P. van Dooren, and F. Vanpoucke. On the QR algorithm and updating the SVD and the URV decomposition in parallel. Lin. Algebra 8~ Applic., 188:549-568, 1993.

S. T. Smith. Geometric optimization methods for adaptive filtering. PhD thesis, Har- vard University, Cambridge, May 1993.

G.W. Stewart. An updating algorithm for subspace tracking. IEEE Transactions on Signal Processing, 40:1535-1541, 1992.

[10]

[11]


259

A C O N T I N U O U S T I M E A P P R O A C H T O T H E A N A L Y S I S A N D D E - S I G N O F P A R A L L E L A L G O R I T H M S F O R S U B S P A C E T R A C K I N G 1

J. DEHAENE, M. MOONEN, J. VANDEWALLE Katholieke Universiteit Leuven Kard. Mercierlaan 9~, 3001 Leuven, Belgium Jeroen. Dehaene @esat.kuleu ven. ac. be

ABSTRACT. Continuous time equivalents of discrete time algorithms are often surprisingly simple, and allow for easy manipulation, analysis and parallelization. Moreover, they establish a connection between systolic arrays and adaptive neural networks. Up to now, the connection between discrete and continuous time algorithms for adaptive signal processing has been established mainly through Ljungs approach [1,2,3], using differential equations parametrized by an input correlation matrix, for the asymptotic analysis of iterative algorithms. Our approach works with a continuous time input, and uses exact integration for the case of a piecewise constant input signal to establish the connection with discrete time algorithms.

KEYWORDS. Systolic array, continuous time algorithms, adaptive neural networks, spherical subspace tracking, stochastic gradient ascent, reorthogonalization.


This paper has a double aim. We give a new efficient discrete time systolic algorithm for subspace tracking, which was designed using inspiration from continuous time algorithms and can be analyzed using the connection with a continuous time algorithm. Secondly, we want to emphasize the central role of the formulas given in lemma 1 in both the analysis and the design of algorithms.

1Jeroen Dehaene is a research assistant with the N.F.W.O. (Belgian National Fund for Scientific Re- search). Marc Moonen is a research associate of the N.F.W.O. The following text presents research results obtained within the framework of the Belgian programme on interuniversity attraction poles (IUAP-17 and IUAP-50) initiated by the Belgian State - Prime Minister's Office - Science Policy Programming. The scientific responsibility is assumed by its authors. In addition, this research was supported by the F.K.F.O. project G.0292.95.

260 J. Dehaene et al.

2 THE EVOLUTION OF MATRIX FACTORS

A central role is played by the Jacobian of some common non linear functions of matrix space, like the inverse of a matrix and some factors of classical decompositions of a matrix such as QRD, LU, EVD, SVD or Cholesky decomposition. These Jacobians link the evolution .~ of a matrix X to the evolution of a factor in some decomposition. Related formulas can be found at different places throughout literature, often in a context of perturbation theory, but they are seldom put together, and used for the design of algorithms as below. We first introduce the following notations for some triangular parts of a matrix.

Y = upph(X) iff Y ~ , j = 0 i f i > j , Yii, =~Xi,i,1 Y = lowh(X) iff t~,j = Xid if i > j, Yi,i = ~Xi,i,

L e m m a 1:

If Y = X -1, then

d Y �9 = - ~ - ~ ( Y ) = - Y / ( Y

If X = c T c is a Cholesky decomposition of a non singular matrix X, then

dC .](. u p p h ( C _ T ~ ( c _ l )C,

If X = QR is a QR decomposition of X, and X is non singular, then

Q, = ~r~(X) = Q w ( Q T X R -1) and

where co(Z) = lowh(Z) - ( lowh(Z))T and p(Z) = ( lowh(Z))T 4- upph(Z).

If X = V A V T is an EVD of a symmetric matrix X with distinct eigenvalues, then

A = ~ ( . X ) = d]ag(VT .Xy)and = r z ( ~ : ) = v ~ ( v r ~ : v )

where #(Z) is an elementwise scaling of the elements of Z, by #i, j(Z) = zi, j /(Aj - Ai) if i r j and/~i,i(zi,i) = O. If X has repeated eigenvalues Ai = Aj over a finite time interval, V is not uniquely defined, and the corresponding entries in I~ i , j (vT / (v ) , can be replaced by arbitrary values as long as # ( v T x v ) is kept skew symmetric. If X has repeated eigenvalues Ai = Aj at an isolated time instant to, and if the eigenvectors have a differentiable time evolution, #i,j(to) is found by continuous extension of #i,j(t).

Similar formulas exist for the LU decomposition. A similar formula for the SVD can be derived easily, applying the formula for the EVD to x T x and X X T. We only state the well known result for the evolution of the singular values: If X = U E V T is an SVD of a matrix X with distinct singular values, then

dE �9 = ~-~(x)= d~g(UrXV)

and l~,j = Xi,j if i < j, (1) and Y~,j = 0 if i < j.

Parallel Algorithms for Subspace Tracking 261

3 A C O N T I N U O U S T I M E S U B S P A C E T R A C K I N G A L G O R I T H M

Subspace tracking consists in the adaptive estimation of the column space of a slowly time- varying n x Jr matrix M, given a signal z, generated by x(t) = Ms( t ) + n(t), where s(t) is a source signal with non singular correlation matrix E{s(t)s( t)T}, and n(t ) is additive white noise, with variance a 2.

We study the following algorithm

fi = 7 (zz T A - 2A upph(ATzz T A)) (2) where z E R '~ is an input signal, the first tr columns of A E I~ nxn span an estimation of the signal subspace (the other columns are not needed for the adaptive computation of the first tr columns), and 7 is a (possibly time dependent) scalar. The algorithm is clearly a continuous time version of the neural stochastic gradient algorithm as described in discrete time by 0ja[2] as

A A ( k - 1) = 7 ( k ) { x ( k ) z ( k ) T A ( k - 1) - 2 A ( k - 1 ) u p p h ( A ( k - 1 ) T z ( k ) z ( k ) T A ( k - 1))} (3)

where 7(k) is a scalar. It can also be derived as a continuous time version of the spherical subspace tracker, a discrete time algorithm, related to the well known SVD approach but with reduced complexity, in which the signal singular values as well as the noise singular values are averaged at each step [3]. In continuous time one can work with (a generalization of) the SVD, or more con- veniently, with the EVD of N = fto~ z(r)z(r)Te-;~(t-")dr = V A V T. The tr first columns of V are an estimate for the signal subspace. Exact tracking of V, applying 1emma 1 to

= zz T - AN, leads to lk = V # ( v T z z T v ) . Parallel realization of this algorithm would require the number of links between different cells (storing matrix entries) to be proportional to the problem dimension n. But, as in the discrete case, constant averaging of the signal eigenvalues and the noise eigenvalues reduces the complexity.

Application of lemma 1 (with continuous averaging of the eigenvalues) now results in different algorithms, depending on the choice of the arbitrary components in #. These different possibilities correspond to representations by different bases of the signal and noise spaces. (Our choice does not correspond to the choice of [3]). A possible solution, with ~i,j(xi,j) = -#j,i(xi, j) = x i , j / ( A s - An) i f i < j and #i,i(xi,i) = 0, yields #(X) = 1 / ( ~ , - ~ . ) ( X - 2 upph(X)), and

ft = 7 ( x x T A - 2A upph(ATxxTA))

where 7 = A8-1 A,~ and ~s = ~1 llATxll2 - AAs and A,~ = ~1 liATxll2 - AA,~. (4)

This corresponds to (2) with an adaptive factor 7.

From lemma 1 the algorithm can also be regarded as a stochastic QR-flow. Whereas the QR flow, given by (~ = Qw(QTAQ), tracks the Q-factor of a matrix B, obeying B = N B , to find the eigenvalue decomposition of N, equation (2) tracks the Q-factor of a matrix B, obeying B = 7xxTB. The.algorithm also corresponds to a square root version of a stochastic double bracket flow Z = 72[Z, [Z, xxT]], where Z = AA T.


4 A N A L Y S I S

In this section we briefly show how the continuous time background of the algorithm can also shed new light on the analysis of its behavior. Lemma 1 provides an easy interpretation of the stability of the orthogonality of A and of the convergence of the estimated subspace to the right solution in the stationary case, and with asymptotically vanishing noise.

Application of lemma 1 to (2) to find out the evolution of the singular values of A yields

= ( ~ r ~ ) ~ ( ~ _ ~),

where u is the corresponding left singular vector. Checking the sign of d one easily sees that a converges to 1, keeping A orthogonal (under mild conditions for x).

Furthermore, applying the same lemma 1 for the singular values of .MTA, where M = or th(M) spans the (stationary) exact subspace, gives the evolution of the cosines of the canonical angles between the estimated and the exact subspace, in the absence of noise. A simple substitution cos 0 = a gives the evolution of the canonical angles.

= _ ( ~ r M r ~ )~ ~i~(20) ,

where ~ is the corresponding right singular vector of MTA. Generalizations of these formula for non stationary and noisy signals will be published elsewhere.

5 P A R A L L E L R E A L I Z A T I O N

An interesting aspect of many continuous time algorithms is that they have a simple parallel signal flow graph. In [4] we have shown how continuous time algorithms of the form A = ~i Ti, where Ti axe terms of the types given below, can be realized as an array of identical cells (uniform parallel realization).

terms of type I: T = ed T terms of type II: T = tri(eeT)A or T = At r i (d f T) terms of type III: T = tri(edTA-1)A or T = A t r i ( A - l e d T)

where A must be upper triangular for terms of type III, and tri stands for one of the triangular parts defined in (1), and must also be upper triangular for terms of type III. The vectors e, e E lt~ m can be of the form (AAT)kx or (AAT)kAy, and d, f e it( n can be of the form (ATA)ky or (ATA)kATx, where k >_ 0 is an integer, and x E lir and/or y E Ii~ m are external inputs. All three types can be interpreted as adaptive neural networks in the sense of [2].

For the parallel realization of these systems the matrix A is represented as an array of analog cells storing the entries of A. Vectors are realized as signals in the rows or the columns of the array. A component xi, ci or ei (i = 1 , . . . , m ) of a vector x, e or e is realized as a signal, available to all cells (i, k) in row i of the array. The vectors x, e, e, are said to be available in the rows. Similarly, vectors y, d, or f, are available in the columns. The inputs x and/or y are supplied externally. The other vectors are obtained through matrix vector multiplication in a straightforward way. The presence of upph amounts to the need of partial sums of these matrix vector products. For triangular matrices A one can


also easily compute A - I x or A-Tz by back-substitution [4]. As an example fig. 1 shows the realization of algorithm (2) with 3' = 1.

I i ~l,j =0

Oi,j dj

a i , j Xi Xi Xi

,1 - - 0 ~ (Xi -- ~i , j -- ~ i , j + l ) ,j +1 - :

/ ~i,j + a i j d j

r},+l,j =1 l rIi,j + aid xi dj

r I . + l , j dj = rln+l,j

i

Figure 1: cell ( i , j ) and boundary cells for the realization of ~{ = x x T A - 2 A upph(ATxxTA)

6 I N T E G R A T I O N

In the previous section, we desribed some continuous time systems with a straightforward parallelization. The next problem is to convert these algorithms into discrete time algorithms. We will do this by exact integration of the continuous algorithm for a piecewise constant input signal. Each new value of the input corresponds to a new discrete iteration step. The integration essentially amounts to using lemma 1 in the opposite direction. The discrete formula and its parallel realization can also be found by exploiting the analogy with an inverse updating algorithm [6] for recursive least squares estimation (RLS). For RLS, the Cholesky factor R or the inverse Cholesky factor S = R -1 of the matrix N = ft_~o z(r)x(r)Te-~(t-~)dr is to be tracked. Using lemma 1, one finds

k = upph(R-TzzTR-1)R - ~tg (5)

and again with lemma 1,

:i = - S u p p h ( S T z z T s ) + ~S. (6)


When supplied with piecewise constant inputs, these algorithms give the same solutions as discrete algorithms for RLS. The first term of algorithm (5) is of type III and the first term of algorithm (6) is of type II. The second term of both algorithms has a trivial realization. Algorithm (5) is a continuous time limit of the well known Gentleman Kung array for QR- updating [5]. Algorithm (6) is a continuous time limit of the systolic algorithm described in [6].

The resemblance of (6) and (2), suggests that a similar systolic array exists for a discrete version of (2). It turns out that a (discrete) systolic signal flow graph can be obtained, from which a linear pipelined array can be derived easily. The following theorem gives the exact integration of (2) for constant input.

Theorem 1 If A0 = A(0), and if z e ~ is constant and = ~(t) = ~ v ( ~ r ~ yo ~ ~(~)d~)

v - 1 , . _ T A_ B = B( t ) = Ao + x-,~x~ ,~o

v2-1ATxxTAo) where icf stands for inverse Cholesky factor S = S ( t ) = i c f ( I + ~ 0 A = A ( t ) = B S then

p(to) = 1 b = ~xTxv

B(to) = Ao B = 7 x T x B S(to) = I $ = - 2 7 S u p p h ( A T z x T A ) A(t0) = Ao fl = 7 ( z z T A - 2 A u p p h ( A T x z T A ) )

These formulas are easily verified, by derivation, using lemma 1, and the formulas z TB = uxTAo and x T A = uxTAoS.

The formula for u depends on an assumption of the evolution of 7. A simple formula is 1 (compare with (4)) resulting in u(t) 1 + xTxTt . obtained assuming ~ = zTx , =

Every step of the corresponding discrete algorithm consists in calculating a new matrix A' = A(Ta) (where Ta is a sampling period) from the old matrix A = A(0), with a new given input z. For parallel realization, many variations are possible. A two dimensional signal flow graph for updating A, with a signal flow similar to the continuous parallel realization is shown in figure 2. The array consists of n x ~ cells ( i , j ) storing the entries of A, an extra column to the left with cells (i, 0), and an extra row at the bottom with cells (n + 1, j) .

v-1 xxTAo should be added to A0. The vector First, to obtain B = B(Ta) from A = A0, yT _ xTAo is accumulated by the signals r/, as each cell ( i , j ) calculates ~i+l,j = ~i,j + z i a i j . The quantity xTx is accumulated by the signals & in the cells (i, 0). Using the external input

Y--1 7, u = 1 + xTxTt and a = ~ are calculated in cell (n + 1, 0) and a is passed to all cells (n + 1, j ) of the bottom row. There ~ = ay is calculated and sent upwards to the cens (i,j). With this information, each cell ( i , j ) can already add xi~/j to aid to obtain bi,j.

To obtain A' = A(Ts), B is to be multiplied by S. By analogy with the inverse updating array for RLS [6], this is done as follows. One can prove that

['? S = [II0] G 1 . . . G n -~ where Gi, i = 1 , . . . , n are Givens rotations, such that (7)

1 [ y T I - - - ~ ] G ~ . . . G ~ = [0.. .01y~+~... Y.l~+x]

v p


Xi

51 = 0

5~

Y

J (i,O)

]

~l,j = 0

flj cj, sj ai,j

> z i x~ > ( i j )

$=05

~+1 r/i+a,j ~)j cj, sj

i,j

Xi

~n+l 7]n+l,j yj C j , 8j

/ I

(n+l j )

Cj+l

Figure 2" signal flow of a systolic array for the discrete time algorithm

where ~ = v2-1 and each successive Gi affects only components i and n 4- 1 of the vector x-,/~x, it is acting on. That is, post-multiplying with S means adding an extra column of zeros, applying n successive Givens rotations that roll [ y T _ ~ ] into its last component and

dropping the extra column. The extra column is realized by the signals r in fig 2. To calculate the Givens rotations, cell (n 4- 1, 0) calculates el = - - ~ and every cell (n 4- 1, j)

u

2 2 1 and calculates cj sj and ej+l from yj and ej such that cj 4- sj =

[yj ej] cj sj = [0 er - s j cj

The quantities cj and sj are passed upwards such that each cell ( i , j ) can calculate the updated a~,j = ai,j(Ts) and ~i,j+l from bi,j - ai,j + xi,jfli,j and r by

[a~,j ~i,j+l] = [bi,j ~i,j] cj sj - s j cj


As their is no data flow from right to left, the algorithm can be pipelined in a straightforward way on a linear processor array storing the columns of A. Obtaining a pipelined two-dimensional array is more difficult. One way is to apply the same manipulations as used in [6] to obtain a pipelined array for RLS. This works for the Givens rotations but to calculate B from A in the retimed array, products of inputs x at different time instants are needed, which leads to a more complicated array. A second way is to store and update an LU-decomposition of A. The resulting continuous time algorithm can be derived, using lemma 1. And a discrete version is obtained using an approach similar as above. Unfortu- nately, the algorithm breaks down, when for some A, the LU-decomposition does not exist. However, the example illustrates that our approach to derive discrete parallel algorithms also works in other cases than the present one.

It follows from the correspondence with the continuous algorithm, that A(0) need not be orthogonal, as A automatically converges to an orthogonal matrix. Whereas explicit orthogonalizations would need O(n 2) operations, the multiplication with S (as n Givens rotations) can be regarded as a cheaper orthogonalization, which works slower if A is far from orthogonal, but is sufficient to restore orthogonality in every step if A(0) is orthogonal.

The complexity of the algorithm is the same as for the spherical subspace tracker. But whereas in [3] a different representation of the signal and noise subspaces was introduced to obtain this lower complexity, this is not necessary for the given algorithm. As a consequence, the columns of A can be interpreted as approximations of the eigenvectors of N. Using these approximations one can also estimate the eigenvalues, as an exponentially weighted average of xTA. j . These estimates can be used to estimate ~ if unknown and to calculate a time step 7 inspired by (4).

References

[1] L. Ljung, "Analysis of Recursive Stochastic Algorithms", IEEE Trans. on Automatic Control, AC-22 (1977) 551-575.

[2] E. Oja, "Principal Components, Minor Components, and Linear Neural Networks", Neural Networks 5 (1992) 927-935.

[3] R.D. DeGroat, "Non-iterative subspace tracking", IEEE Transaction on Signal Pro- cessing 40 no.3 (1992) 571-577.

[4] Dehaene J., Vandewalle J., "Uniform local implementation on networks and arrays of continuous time matrix algorithms for adaptive signal processing", report 93-34I, ESAT-SISTA, K.U.Leuven, and shorter version to appear in proceedings of MTNS 93, symposium on the Mathematical Theory of Networks and Systems, Regensburg, Germany, 2-6 August, 1993.

[5] W.M. Gentleman and H.T. Kung, "Matrix triangularization by systolic arrays", Real- Time Signal Processing IV, Proc. SPIE 298 (1981) 19-26.

[6] M. Moonen and J.G. McWhirter, "A systolic array for recursive least squares by inverse updating", Electronics Letters, 29 No. 13 (1993) 1217-1218.


267

S T A B L E J A C O B I S V D U P D A T I N G B Y F A C T O R I Z A T I O N O F T H E O R T H O G O N A L M A T R I X

F. VANPOUCKE, M. MOONEN Katholieke Universiteit Leuven Dept. of Electrical Engineering, ESA T Kard. Mercierlaan 9~, 3001 Leuven, Belgium { Filiep. Van, oucke, Marc.Moonen} @esat.kuleuven.ac. be

E. DEPRETTERE Delft University of Technology Dept. of Electrical Engineering Mekelweg 4, 2628 CD Delft, The Netherlands ed@dutentb, et. tudelft, nl

ABSTRACT. A novel algorithm is presented for updating the singular value decomposition in parallel. It is an improvement upon a Jacobi-type SVD updating algorithm, where now the exact orthogonality of the matrix of short singular vectors is guaranteed by means of a minimal factorization in terms of angles. Its orthogonality is known to be crucial for the numerical stability of the overall algorithm. The factored approach has also advantages with respect to parallel implementation. We derive a triangular array of rotation cells, performing an orthogonal mat r ix- vector multiplication, and a modified Jacobi array for SVD updating. Both arrays can be built with CORDIC processors since the algorithms make exclusive use of Givens rotations.

KEYWORDS. Adaptive singular value decomposition, systolic arrays.

1 I N T R O D U C T I O N A N D P R E L I M I N A R I E S

The singular value decomposition (SVD) of a matrix X E ~NxM is defined as

X = U. E. V T,

where U E ~NxM, V E ~MxM are orthogonal and E E ]~MxM is a diagonal matrix holding the singular values of X. This matrix decomposition is central in the theory of model based signal processing, systems identification and control. Because of the availability of robust

268 F. Vanpoucke et al.

Al$orithm 1

V[o] '-" IM Rio] ' - OMxM

f o r k = 1 , . . . ,o0 1. Input new observation vector z[k ]

~T T z[k ] ~- z[~].V[k_~] 2. QR updating

] ki l II "

+_ G M + I - i [ M + 1T

0 i= l 3. SVD steps

(3a) R[k] +--

(3b) V[k] *--

endfor

A. R~}_I]

M - 1 M - 1 (~ M - i i M + l - i ~r & i]i + l

H "t l �9 kt l" II -t 1 i---1 i'-I

M - 1 •ili+1

i -1

algorithms with excellent numerical properties, it is also an important algorithmic building block.

In real-time applications the data vectors Z[k ] E n~ M a r e processed as soon as they become available. The data matrix X[k] is often defined recursively

[ A'X[k-I] ] X[k] = x(~] '

k x M

The real scalar A < 1 is an exponential weighting factor which deemphasizes older data. In the above applications, the SVD of X[k] has to be tracked.

Unfortunately, an exact update the SVD at each time instant k requires O(M 3) operations. For many real-time applications this computational cost is a serious impediment. Therefore, approximate algorithms for SVD updating have been developed which trade accuracy for computational complexity.

A promising algorithm is the Jacobi-type SVD updating algorithm [1]. It is reprinted as Algorithm 1 and computes an orthogonal decomposition

xt j = vt j. RE j. where U[k] E nt k• V[k] E n~ M x M a r e orthogonal but now R[k] is an upper triangular matrix which is nearly diagonal (i.e., close to the true ~[k]). Since U[k] grows in size, the algorithm only keeps track of R[k] and V[k], which is sufficient for most applications.

First the incoming vector z[k] is multiplied by the orthogonal matrix V[k-1]. The second step is a QR decomposition updating step. The transformed vector Z[k] is worked into the weighted triangular matrix A �9 R[k_l] by means of a sequence of M Givens rotation

f2 i lM+l n~ ( M + I ) x ( M + I ) Such a Givens rotation matrix Qilj is a square matrices [2] '-'[k] E

Stable Jacobi SVD Updating 269

matrix embedding of a 2 x 2 rotation operating on rows/columns i and j , i.e., Qilj is identical to the identity matrix except for 4 elements QilJ(i,i) = QilJ(j , j ) = cos(ailj) and _QilJ( i , j ) = QilJ(j, i ) = sin(a ilj) for some angle a ilj. The QR updating step degrades the nearly diagonal structure of R[k]. Therefore the third step consists in applying a sequence

(~i[i+1 ~i[i+l ]~MxM) of M - i row and column rotations ~'-'[k] , ~[k] E to R[k] to reduce the size of

the off-diagonal elements again. Finally the column rotations ~i[i+l have to be applied to [k] V[k-1]. For details on how to compute the rotation angles, we refer to the original paper P].

A high level graphical representation (signal flow graph or SFG) of the algorithm is given in Figure 1. The upper square frame represents a memoryless operator which computes the matrix - vector product x[k]-.T = X[k]T �9 V[k-1] (step 1), and the matrix - matrix product ~k] = ~k-1] ' @[k] (step 3). The heavy dots (here and in subsequent figures) represent delay operators. The lower triangular frame represents an operator which performs the QR updating (step 2) and the SVD steps (step 3). A systolic Jacobi array implementing this SFG is described in [3].

Algorithm 1 has one numerical shortcoming. It suffers from round-off error accumulation in the orthogonal matrix V[k]. At each time instant V[k-1] is multiplied by a sequence of

�9 �9 i l i + l �9 M - 1 Givens rotations (I) k] Rounding errors in these multiplications will perturb V[k] in [ . a stochastic manner. These errors do not decay, but keep on accumulating. This can easily be verified experimentally. The orthogonality of V[k] is known to be crucial for the overall numerical stability and accuracy [1]. Therefore, an unbounded increase of the deviation of orthonormality is clearly unacceptable�9

The algorithm can be stabilized by including a reorthogonalization step based on symmetric Gram-Schmidt orthogonalization [1]. At each time the rows are V[k] are reorthogonalized by 2 • 2 transformations. However, this method does not guarantee orthonormality at each iteration. In combination with exponential weighting it only keeps the deviation sufficiently low and bounded. Secondly, the resulting systolic implementation is rather tricky and in- efficient.

An alternative to keep a matrix orthogonal is to describe it by means of a set of rotation angles. In applications where the matrix is constant, e.g., orthogonal filters, this parameterization has been used extensively. However, in adaptive signal processing, updating the angles is a non-trivial issue.

In section 2 it is reviewed how the orthogonality of the V[k ] matrix can be preserved by parameterizing the matrix as a finite sequence of Givens rotations. In addition, the corresponding triarray for orthogonal matrix-vector multiplication is presented. In section 3 a rotation method is described to update the rotation parameters of V[k] without explicit computation of V[k]. This method is the major contribution of this chapter. Section 4 combines these results into a modified systolic Jacobi array for SVD updating.


z[k]

Yt - vt l

~.T ~ _ t'[k-1] [~] ~[k] ~ - - - ~

R[k]

Figure 1: SFG of Algorithm 1 for SVD updating.

2 O R T H O G O N A L M A T R I X - V E C T O R P R O D U C T

In this section we show how to factor the matrix V[~] as a finite chain of Givens rotations, each determined by an angle a. If all computations are performed on the angles, the matrix V[k] can never leave the manifold of orthogonal matrices. Rounding errors will now perturb the stored rotation angles, but by construction the perturbed V[k] is still orthogonal. First we derive a unique factorization of an arbitrary orthogonal matrix V. Secondly we use this

T 'V[k-1] This leads to an elegant systolic array for factorization to compute the product x[k ] orthogonal matrix - vector multiplication.

L e m m a 1 Let V be a real orthogonal matrix (V T . V = IM). Then V can be factored uniquely into a product of M . ( M - 1)/2 Givens rotations Qilj and a signature matrix S, i.e.,

v --- I I o �9 s , \ i=a j=i+a

where S is equal to the identity matrix of size M, except that the last diagonal entry is 4-1.

Example

For a 3 x 3 orthogonal matrix, the factorization is given by

V - Q112. Q1[3. Q213. S.

To construct the factorization, it is sufficient to apply the weU-known Givens method for QR decomposition [2].

Stab le J a c o b i S V D U p d a t i n g 271

Ix x x ]~ Ix x x I ~ [1 ~ ~ ]Q23 [1 ~ ~ xx xx xx _ x 0 xx xx _ 00 xx xx _ 00 0 1 0 1 v s

At each stage we need to compute a ilj such that after rotation, v~i is zeroed. The angle

is non-negative. If both vii = 0 a ilj is unique if the convention is taken that vii v i i = , we define a i[j -- O.

After zeroing all off-diagonal elements in a column, the diagonal entry equals 1 since the columns of an orthogonal matrix have unit-norm. The same argument holds true for the rows. Finally, the sign of the (M, M)-th entry of 5' is not controlled by the algorithm. It is positive or negative depending on the sign of the determinant of V. n

Comments

1. One can always choose a matrix of short singular vectors with positive determinant such that the signature matrix can be omitted.

2. The ideal hardware component for fast computation of the angles, given a matrix V, is a CORDIC processor [4] in angle accumulation mode.

The signal flow graph (SFG) for an orthogonal matrix-vector product Z[k ]~T __ X[k]T "V[k-1] in factored form is now shown in Figure 2. The triangular graph consists of M . ( M - 1)/2 nodes, having local and regular interconnections. The functionality of each node is the same. It stores the rotation Qil j and applies it to its input pair coming in from the top and from the left. Its output data pair is propagated to the bottom and to the right respectively. Again the most efficient implementation of a node is a CORDIC processor in vector rotation mode.

Mapping the SFG into a systolic array is trivial, since the data flow is unidirectional. It suffices to insert a delay cell on all dependencies cut by the dashed lines in Figure 2. The pipelining period of the systolic array is one cycle.

This systolic array has already been derived in the context of computing the QR decomposition of an arbitrary square matrix [5]. There, on feeding in the matrix, the nodes first compute their rotation angles a ilj and the triangular matrix R is output. Once the angles are fixed, the array operates as in Figure 2. The array also bears resemblance to the well-known Gentleman-Kung array for QR updating. Note however that in Figure 2 the rotation angles are resident in the cells, whereas in the Gentleman-Kung array they are propagated through the array.

Except for guaranteed orthogonality, the factorization has the additional benefit that the scalar multiplications in step 1 are eliminated. The whole SVD updating algorithm now consists exclusively of Givens rotations. This regularity is important in view of a hardware realization. It allows to construct a systolic array for SVD updating only using CORDIC processors, provided that the updating of V[k-1] (step 3) can be done in factored form. This is the topic of the next section.


/ ' in

/__I_L-~ /'__I_E-~ / X out / /" i

1 / ," out

~ ' . . . , "1~ ' / ' 1 ~ / z~"' i (r -a~ / , ''-ff . / ~ - ~ L

Xl x2 x3 x4

2•qut 3

= QilJ T . i n ]

xi in z j

Figure 2" SFG of the factored orthogonal matrix-vector multiplication (M = 4).

3 U P D A T I N G T H E A N G L E S

In step 3 of Algorithm 1, the V[k_l] m a t r i x is post-multiplied by a sequence of M - 1 Givens �9 ili+x ,~ilj rotations r . In this section we present an O ( M 2) method to update the angles ~'[k-1]

directly, without explicit computation of the V-matrix.

The updating matrix 1 (~ is the product of rotations on neighboring columns,

M-1 (I) = H (I)i{i+l"

i=1

Each transformation of the form V ~ V. ~ili+1 will alter several rotation angles. Starting from the tail, a rotation ~ilj is worked backwards into the factorization. It interacts with the preceding rotations ~klt in a way which depends on the relative position of the coordinate planes defined by their indices. Three types of transformations have to be considered.

i. THE INDEX PAIRS (k,l) AND (i,j) ARE DISJOINT. In this case the rotation matrices Qklt and CqJ commute since they affect different rows or columns, i.e.,

~ilj . Qklt = Qklt . r

2. THE INDEX PAIRS (k , l ) AND (i,j) ARE EQUAL�9 Here the rotation angles of Qilj and r simply add together, i.e.,

3. THE INDEX PAIRS ( k , l ) AND ( i , j ) SHARE A COMMON INDEX.

1The time index k is omitted for notational convenience.


This is the complicated case. Let k = j . Generically, the matrices Qjlt and ~ilj do not commute and it is even impossible to calculate an equivalent pair of rotations such that ~i.lj .Qj.lt = Qjll. r However, reordering the indices becomes possible if a third rotation, Qilt, is taken into account. The sequence of 3 Givens rotations in the (i, l), (j,/), (i,j)-planes, defines a rotation in the 3-dimensional ( i, j, /)-space,

ViJt = Qilt. Qjlt. (~ilj.

The 3-dimensional rotation can also be represented by a different set of three Givens rotations by choosing another ordering of the coordinate planes.

ViJ t = (~ ~,lJ . Qi, lt . QJl , .

There is no simple trigonometric expression for the mapping from the former to the latter set of angles. A natural algorithm is to compute V ijt explicitly and refactor it. The computational complexity of this 3 x 3 core problem is relatively low and independent of the matrix dimension M.

1 0 0 ] 0 1 0 0 0 1

u._ J y _ ,

/3

x x 0 x x 0 0 0 1

Q3lZ )

xxxx xx xx 0]o Ex x x]o: xx xx xx

Ix x [1 ~ E1 ~ ~ xx xx - 00 xx xx 00 01 01 J

It is even sufficient to compute only two columns of V ijt. When selecting the first and last column, the operation count is optimized to 7 rotations over a given angle (vector rotation), and 3 rotations in which a coordinate is zeroed (angle accumulation). On a CORDIC processor, both operations have the same complexity.

Below the course of the computations in the 4 • 4 example is detailed for the first updating rotation r The numbers between braces refer to the respective transformation types above. In the first line, ~112 can be commuted with Q3[4 (type 1). To interchange r with Q214, Qll4 must be adjacent to Q214. Therefore, Qll4 is commuted with Q213. Then the equivalent set of rotations in the (1, 2, 4)-space is determined (type 3). The same operations axe then repeated in the (1,2,3)-space. On the last line the angles in the (1,2)-plane are summed (type 2).

V.r _ Q112.Q11a.Ql14.Q213.Q214.Q314.(~l12

= Ql12.Ql13. Qll4. Q213. Q214. ~112. Qal4

= Ql12. Qlla.Q213. Ql14. Q214. (I)112. Q314 (T1)

(T1)


" T

..., ..

........ ,O214 "'t ............ D314

<~112 r r

ili+z

= JAI ol I'+ l iji+l = Q[~+I . eili+l out

<•ili+l ou t Qilj ] 0!+11/ OU~ "v t'n.

= I L

JBV sn I "~ o u t

I)iJi+l oilJ 0i+1[/ o!lj i+llj. (i)i~+1 Q o u t " V.~ ou t " , ,~ ou t - - -~ sn " 1 , i n

Figure 3: SFG for the updating process of the rotation angles.

= Q112. Q113. Q213. (~1,[2. Q1,14. Q2,[4. Q314 (T3)

= Q112. (~1,[,2. Q1,[3. Q,213. Q1,[4. Q2,N. Q314 (T3)

= Q~,I2. Q~,Ia. Q1,14. Q2,13. Q2,14. Qal4 (T2, T1)

The SFG for updating the complete parameterization is shown in Figure 3. First the bottom-left node updates the rotations Ql14 and Q214. Next, the neighboring nodes (up and to the right) perform their computation in parallel. The computation gradually evolves towards the diagonal. Note that the commutations (type 1 transformations) follow naturally from the structure of the graph.

The horizontal contraflow in this SFG complicates its pipelining. To introduce delay elements on all edges, we need to apply two well-known retiming rules [6], i.e., time-scaling and delay-transfer. The time-scaling rule allows to double all delays in the graph, and consequently halve the rate at which new data are fed in into the array. In fact, we can slow down a graph by any positive integer n. The delay transfer rule permits to add a number of delays on all outbound edges crossing a cut, provided that the same number of delays is removed on all inbound edges. When applied to the cuts (dashed lines) in Figure 3, the outcome is a systolic array with a pipelining period of 2 cycles.

4 A M O D I F I E D A R R A Y FOR SVD U P D A T I N G

In the previous sections two separate systolic arrays were presented for orthogonal matrix- vector multiplication and for updating the parameterization. These arrays only partially implement the SVD updating algorithm. The complete SVD array consists of the triarray of section 2, placed on top of a triarray which performs QR-updating and generates the row and column transformations [3]. This is shown in the signal flow graph of Figure 4. The upper triarray stores the factorization Q[k] of the orthogonal matrix V[k]. Its nodes perform


Qtk-~s

z[k]

[ 1

R[k]

Figure 4: SFG for SVD updating.

the matrix-vector product as well as the updating of the angles. The lower triarray takes in new matrix-vector products and generates new column transformations.

Pipelining the combined graph is no longer straightforward. To pipeline a long vertical dependency loop, the same algorithmic transformations as in [3] are needed.


In this paper we have presented a modified algorithm for SVD updating. Two desirable properties are achieved. First the matrix V is kept orthogonal at all time by factoring V as a sequence of M �9 (M - 1)/2 Givens rotations. The factorization prevents V from drifting away from orthogonality due to linear error buildup. This is crucial for the numerical stability of the recursive algorithm. Secondly the algorithm consists exclusively of planar rotations, which increases the regularity of the operations in its signal flow graph. The overall updating algorithm has O(M 2) computational complexity.

The factored orthogonal matrix-vector multiplication leads to a triangular array of rotation nodes with regular local interconnections. This array is then incorporated into a modified array for SVD updating, which now consists exclusively of rotation nodes. There- fore, such an array may be built in hardware using CORDIC-based processors only. The regular implementation on a systolic array makes the algorithm a competitive candidate for real-time high-throughput SVD tracking applications.


References

[1] M. Moonen, P. Van Dooren, and J. Va~udewalle, "An SVD updating algorithm for subspace tracking," SIAM J. Matrix Anal. Appl., vol. 13, pp. 1015-1038, Oct. 1992.

[2] G. Golub and C. V. Loan, Matriz Computations. John Hopkins, 2nd ed., 1989.

[3] M. Moonen, P. Van Dooren, and J. Vandewalle, "A systolic array for SVD updating," SIAM J. Matriz Anal. Appl., vol. 14, pp. 353-371, Apr. 1993.

[4] J. Voider, "The cordic trigonometric computing technique," IRE Trans. on Electronic Computers, vol. 8, pp. 330-334, Sept. 1959.

[5] H. Ahmed, J.-M. Delosme, and M. Moff, "Highly concurrent computing structures for matrix arithmic and signal processing," Computer, pp. 65-82, Jan. 1982.

[6] S. Kung, VLSI array processors. Prentice-Hall, 1988.


277

T R A N S F O R M A T I O N A L R E A S O N I N G O N T I M E - A D A P T I V E J A C O B I T Y P E A L G O R I T H M S

H.W. VAN DIJK, E.F. DEPRETTERE Department of Electrical Engineering Delft University of Technology 2628 CD Delft, The Netherlands ed@dutento, et. tudelft, nl

ABSTRACT. The ordering of operations in the execution of Jacobi-type algorithms is not unique. Given a sequential imperative program specification of a Jacobi algorithm, there is a method of transformational reasoning to convert the program to any one of a set of input- output equivalent concurrent programs. The method explores associativity and (pseudo) commutativity properties in the algorithm to tune the program's critical path length to an optimal throughput in a desired parallel implementation. The method constructs a certain precedence graph in which vertices represent elementary transformation steps and edges expose step precedence relations. Every feasible cut-set of the precedence graph yields a dependence graph of a concurrent program which is input-output equivalent to the given one. Moreover, regular dependence graphs will be transformed into regular dependence graphs if the cut-set is chosen to keep that property invariant. The method has been successfully applied to time adaptive algorithms in which QR, inverse QR and SVD Jacobi algorithms play a crucial role. The time adaptive SVD algorithm will be used in this paper to illustrate the power of the method.

KEYWORDS. Jacobi-algorithms, algorithmic transformations, re-timing, critical path, adaptive algorithms, parallel processing.

1 INTRODUCTION

Signal processing applications often require a high data-throughput. A typical example is the adaptive processing of data received by an antenna array, where the number of operations can be as high as several Giga operations per second. Such a high throughput can usually only be sustained by exploiting parallelism and pipelining in the computations. However, applications are commonly specified in terms of sets of sequential imperative

278 H.W. van Dijk and E.F. Deprettere

subroutines and, in general, pipelined implementations are not easily obtained from them. Assuming that the given subroutines are nested loop programs, two approaches to arrive at convenient operation schedules are loop transformations and flow graph transformations. Loop transformations may help but most often they do not. Whereas flow graph transformations such as Leiserson-Saxe, systolization and scattered look-ahead techniques may fail because they do not affect the program's dependence graph although they do remove non-essential orderings in the given program.

In this paper we show that it is possible to define local transformations on a dependence graph (DG) of a given program - a set of nested loop subroutines - which have the property that they effectively shorten the critical path length of the computations by exploiting associativity and commutativity properties of these computations and hence by transforming - implicitly - the given program into a non-trivial equivalent one. Moreover, there is no need for an explicit derivation of the program's DG. The method introduces the slice description to keep track of the imperative sequential algorithm and we introduce two transformation steps on these slices. One is used to reveal the critical path and the operations on the critical path, the other transformation actually modifies the DG and implicitly 'rewrites' the given program. A remarkable result is that the latter transformation steps have precedence relations among themselves; they are the vertices in a precedence graph. A cut-set of this precedence graph yields an ordered set of transformation steps and when applied to the initial sequence of computations a new program is found that has different time behaviour. It is possible to choose the set of transformation steps in such a way that the resulting program is again a nested loop program. We thus arrive at a complete set of equivalent DGs and in some specific cases, we are able to retrieve known high-throughput algorithms which have been published in the literature and were derived in a heuristic manner.

We focus on a special class of nested loop programs which we call Jacobi-type programs and we find transformations that transform the dependence graph DG(0) - derived from the given nested loop (Jacobi) program- into DG(1), a dependence graph with reduced critical path. Typical examples of Jacobi programs are the matrix factorization programs QR, singular value decomposition (SVD) and Schur (eigenvalue) decomposition which can be found in Golub's book [2]. To illustrate the method we consider the time-adaptive SVD based on the Kogbetliantz algorithm specified in terms of a cyclic by rows program [5]. This algorithm involves a vector matrix product, adaptive QR steps and SVD steps. We aim at decreasing the critical path of the algorithm, however known parallel versions of the adaptive QR steps and SVD steps can not be combined. The SVD cyclic by rows algorithm though can easily be interleaved with the adaptive QR steps but has a low degree of inherent parallelism; it suffers from long critical paths. We show that it is possible to exploit associativity and commutativity between operators in the program and effectively decrease the critical path in the SVD cyclic by rows algorithm. In fact we come up with a whole set of input-output equivalent algorithms.

The rest of the paper is organized as follows. In section 2, we give preliminaries and some definitions. In section 3, we introduce transformations, some of which are dependence preserving and some of which are not. In section 4, we apply the method to the crucial part of the time adaptive Kogbetliantz SVD algorithm, specified in terms of a cyclic by rows sequential program.

7~me-adaptive Jacobi Type Algorithms 279

2 P R E L I M I N A R Y A N D D E F I N I T I O N S

Let X be in R 2• and define 2 x 2 elementary coordinate transforms (ECT) J(a) , G(a) and H (a) 1 as follows

j ( c~ )= [cosc~ -s inc~ G(c~)= 1 0 ] H(c~)= coshc~ s inh~ I (1) s ina cosa ' a 1 ' s inha cosha

In this paper we define an elementary Jacobi step to be the matrix multiplication

X ' ~ A(8) X BT(r (2)

where A(8) and B(r are any two of the ECTs. In (2), the arguments 8 and/or r may be either given or depending on the entries of X. We say that the ECT is unconditional if its argument is given and that it is conditional if its argument is to be determined from the entries of X in a pre-specified way.

We write A(i,j) when A is embedded, that is, when it is an identity matrix of appropriate order in which the entries (i, i), ( i , j ) , ( i , j ) and ( j , j ) are replaced by A's four entries (1, 1), (1,2), (2,1) and (2, 2), respectively.

Let k E Z , Irk E R ('~-1)• and x T a row vector. The following is a typical step in a Jacobi-type time-adaptive algorithm.

Yk ~-A(i 'J lXk B(P,q) r where X k = r (3) xk

For a non-specific Jacobi-type algorithm, it is not specified whether a matrix M (id) is conditional or unconditional and, therefore, it makes sense to speak in terms of operators rather than matrices. Thus if A (i'j) is the (left hand side) matrix, then we denote by A ilj its corresponding operator and

Y ~ A( id)X = Y ~ Ail j (X )

Similarly, if B(id) is the (right hand side) matrix, then we denote by B ilj its corresponding operator and

= X T = Y ~-- B iIj ( X )

Let s be a set of operators, s = {p l , . . . , pw} , and let an ordering of the set induce the chain (sequence) a of operators a = pw . . . Pl. Application of a to a matrix X yields a matrix Y; Y = a (X) = pw . . . pl (X ) = pw o . . . o p l ( Z ) . A rough conceptual outline of a time-adaptive Jacobi program is given in program 1.

Next, localization of operations is made possible by introduction of the dot product operator. To this end, let X be in ]~m• and let [ X l . . . x s . . . x p ] be a column partitioning of X, that

p is, x8 is in ]~m• and n = ~ s = l ns. The dot product is defined as

Y -- ( P l , . . . , P s , . . . , P p ) ~ X = ( P 1 , . . . , P s , . . . , P p ) ~ (Xl , . . . Xs . . . Xp) = [ p ~ ( ~ ) . . . p ~ ( ~ ) . . , p ~ ( ~ ) ] ' ' ' (4)

1The methods presented in the paper do not require matrices to be real valued. It is only for notational simplicity that we shall stick to real arithmetic.


f o r a l l k in { 0 , . . . , K } d o

L - I Yk ~- ]-I M~ " lj~ ( Xk )

$"-0

endfor

Program 1: Outline of a typical Jacobi-type time-adaptive algorithm

where Pa is an operator on x8 and Pr an (possibly different) operator on xr. Now, recall that A ilj was introduced to allow the mode of A(i,j) to be left open. The dot product allows localization of the operator A iij and come closer to what is the case with most of the Jacobi-type algorithms.

: ( x ) : .A',I 0 x Y

; [A'?(, , ) . . . A;,,(,,)]

where all operators A~ Ij as well as the operator A ~lj originate from the same matr ix A(i,J). Thus, if AqJ is unconditional, then so are all Ais [j. On the other hand, if AqJ is conditional, then one of the A~ Ij, say Aia [j, must be the one that actually determines the parameter a(i,j),

and all other Ai, Ij, s ~ a, borrow this parameter from it. Bearing in mind these dependences we give the chain form 2 of the dot product.

A~(x)=A~I~ A~,~ .,1~ a '1~ A '1~ (X) (6) (n) " ' " ( a + 1 ) ' ~ ( 1 ) " ' " ( a - l ) (a)

Time-adaptive algorithms typically have sliced dependence graphs, that is, they are chains of similar subgraphs. This slicing can also be recognized in the algorithm's nested loop specification, that is, the program is a concatenation of similar subprograms.

D e f i n i t i o n 2 . 1 ( s l i c e ) Let S k be the set of operators Mik pIjp in the body of the k-loop of program 1. The program imposes a sequential ordering of the operators in the set S k. The ordered set, as imposed by the program, will be denoted S~ and called a (program) slice. It is thus a chain (or sequence) of operators

S~ = '"kaAriz-1 IJz-1 . . . M~illJl M~olJo

The length of S~ is L, the number of operators in the chain. A subslice S~ ( i ' j ) of S~ is the chain of operators i, i + 1, . . . , j , (1 <_ i < j <_ L). Application of S~ to a matrix X yields a matrix Y

Yk ~- s?~ (xk)

With this definition, program 1 is a simple concatenation of similar slices S~

Yg ~ S~: . . . S? . . . S o ( Xo ) (7)

2Notice the slight difference in notation in (5) and (6). In (5), Ai~ ~ is the s-th operator and is assumed to be dimension compatible with the s-th partition x~ of X. In (6), the subscript (s) is a pointer to the dimension of the s-th partition, that is, (a) actually means that A iff operates on the partition x E R "~• (~) ~ .

Time-adaptive Jacobi Type Algorithms 281

Def in i t ion 2.2 (slice shif t) Let S~ be a slice of length L. The left-shifted slice, S~ -1 is recursively defined as, for p = O , - 1 , . . .

S~ -1 ---- S~+ 1 ( L ' L ) S~ (1" L -- 1)

The right-shifted slice, S p+I is defined recursively as, for p = O, 1 k ' ' ' '

s~+~k = sk~ ( 2 . z ) sLI ( I" 1) []

A direct consequence of the above definition is the following. If L is the length of slice S~ then

sP+L p-L k+x = S~ = Sk_ ~

The representation (7) of program 1 is a concatenation of slices S~ the use of shifted slices instead, is equally suitable e.g. let 0 < p < L, then

YK ~- SK p . . . S[ p . . . So p S-~ (Xo) (8)

where S~P and S-~ axe the appropriately clipped 3 versions of SK p and S_-~ respectively.

3 T R A N S F O R M A T I O N S

In this section we introduce two types of local transformation steps which are defined to operate on a pair of operators in a slice, they transform slices into slices. The transformation steps reorder the initial ordering of operators in a slice. The first type of transformation steps are referred to as dependence preserving transformation steps; they do not alter the underlying dependence graph (DG). These transformations are used to allow a critical path analysis in the slice and they reveal the possibility to apply a second type of transformation steps, algorithmic equivalence transformation steps. This latter type of transformation steps rely on associativity and commutativity properties of adjacent operators in a slice and they change dependences in the underlying DG and therefore may affect the DG's critical path.

3.1 DEPENDENCE PRESERVING TRANSFORMATIONS ON SLICES

Recall that the operations in a slice correspond to the nodes in a subgraph of the DG. The particular subgraph becomes apparent when looking to the DG through an appropriately chosen window. Sliding the window will reveal different subgraphs which in turn correspond to shifted slices. Therefore slice shifting is a dependence preserving transformation step. The usefulness of the DG is that it bespeaks which orderings in the program are due to dependencies, hence necessary, and which are arbitrary, hence unnecessarily restrictive.

Def in i t ion 3.1 ( i n d e p e n d e n c y ) Two adjacent operators M ilj and M plq in a slice are said to be mutually independent if they are essentially concurrent operators, that is if

M ilj ( Z ) U ~lj M plq (X ) - M plq U ilj (X ) = Mpjq (X)

SAn alternative would be to replace Xo by an appropriately defined X_I.


The operators are otherwise said to be dependent. If independent, then the application of the implication

MilJ Mplq =~ MPlq MilJ

is called a dependence preserving transformation step.

For any particular Jacobi-type algorithm, the behaviour of the operators in a slice are well defined and it is, then, easy to express where independent operations, if any, can be found.

The independency property makes explicit that the ordering of the operators in a slice for a certain DG is not unique. In fact we exploit this property to detect the critical path in the algorithm. The critical path detection is based upon the so-called earliest, TE, and latest, TL, possible execution times for every operator in the slice [3] with both times relative to the same reference point, Tsrc. For simplicity we assume that each operator 'executes' in unit time. The length of the critical path of the overall program is proportional to the length of the critical path of a slice, the minimum slice-latency, TM.

Definition 3.2 (critical pa th order ing) Let S~ be a slice of length L and let S~ (~*) be an alternate shifted and reordered version of S~. Assign to every operator in the slice its possible execution time interval M~,q]; TE = p and TL = q. S~ (i~ is said to be in critical path ordering if it obeys (TM being the minimum slice latency)

S ~ (i*) "- $ [ T M - - I , T M - I ] " ' " S[t,~-H-] $[t,~] " ' " 3[2,2] $[I,I-H-] 8[I,I]

where all operators M{t,t ] in subslice s[t,fl have a possible execution time interval [ t, t] and

all operators M{t,t++ ] in subslice s[t,t++] have a possible execution time interval [ t, t + c] (1 < c < T M - t), that is, TE = t and TL > t. []

Critical path ordering is a feasible, DG preserving, reordering of a slice. Notice that all operators in subslices s[t,fl are in a critical path of the slice.

3.2 ALGORITHMIC EQUIVALENCE TRANSFORMATIONS ON SLICES

Algorithmic equivalence transformation steps do change the program's DG but not its input-output relations. They may change the critical path of the program, especially when the operators involved are in the critical path.

Definition 3.3 (a lgor i thmic invariance) Let M ilj and M plq be two dependent adjacent operators in a program slice. The program is said to be exchange invariant with respect to these two operators if

M ilj M plq ( X ) = M plq M ~lj (X)

The application of the implication

MilJ MPlq =~ MPlq MilJ

is called an algorithmic equivalence transformation step. []

For any particular Jacobi-type algorithm, the behaviour of the operators in a slice are well defined and it is, then, easy to express the relationship between the pair of indices

l~me-adaptive Jacobi Type Algorithms 283

i , j and p,q of operators M i[j and M plq, respectively on which algorithmic equivalence transformation steps may be applied.

P r o p o s i t i o n 3.4 (cr i t ical p a t h handl ing) Let slice S~ be in critical path ordering

S ~ - S [ T M _ I , T M _ I ] . . . S[t+l,t+l ] Sit,t++] Sit,t] . . . 8[2,2] 8[1,1-H-] 811,1]

Let at be the set of all operators in subslice s[t,t ]. The set at+l x at is the set of all pairs

(Mit+l,t+l ] M{t,t]) with ' , M[t+l,t+l ] /n s[t+l,t+l] and M{t,t ] in s[t,t]. Denote by Y the set of all possible algorithmic equivalence transformation steps that can be applied in at+l • at, and by S~ the slice that results when all algorithmic equivalence transformation steps in T have been applied to S~.

If ITI = lat+l • atl then Critical path of S~ >_ Critical path of S~

PROOF Omitted due to lack of space.

3.2.1 PRECEDENCE GRAPH The operators involved in algorithmic equivalence transformation steps are dependent, so there is a precedence relation between subsequent algorithmic equivalence transformation steps. Let vi be the characterization of an algorithmic equivalence transformation step. We may form a precedence graph (PG) with vertices vi and edges that expose the precedence relations between subsequent algorithmic equivalent transformation steps. A feasible partition of the PG yields a feasible cut-set; a subgraph with vertices vi and no incoming edges. Application of all transformations steps in such a set to the initial program results in an input-output equivalent one but with a different DG and different critical path.

The new program does not necessarily admit a nested-loop type specification a/though the given one is in such a form. It is, however, possible to choose a cut-set for which the new program too does have a nested-loop type specification. This will be illustrated in the next section in which we illustrate the theory by means of an example.

4 A P P L I C A T I O N

We illustrate the approach by an example for which we take the Kogbetliantz Singular Value Decomposition algorithm organized in a cyclic by rows manner. This particular ordering is useful when the SVD steps have to be interleaved with, for instance, time adaptive QR steps as in the Kogbetliantz based time adaptive SVD algorithm [4].

Let 0 (i'i+l) and r be the embedding of the Jacobi matrices J(a) and J(~), respectively (1). The Kogbetliantz SVD algorithm for an infinite number of sweeps [2] is given in program 2 (the matrix R is upper triangular and of order n).

for k = 1 : oo, T R 4= 0 (n - l ' n ) . . " 0 (2,3). 0 (1'2) R ~)~1,2)" (I)~2,3)"" (~(n-l,n)

endfor

Program 2: Kogbetliantz SVD algorithm


We refine the body (single sweep) of program 2 further into program 3, which is a well known cyclic by rows Jacobi program. In program 3 the subsequent elementary Jacobi steps R r O (r'r+i) R @Fr,r+x) have been replaced by their dot product expansions.

1 r - 1 arl r+l t ' / -~ , q r l r q ' l ~' l '+~(k) O~l'+~(k) (R) = ~1~+~(k) �9 �9 �9 ~ l , + ~ ( ~ ) ~ . . . . ~+~ , ' , ~ , ,+~ (~) ( ~ )

fork=1 : c~ for r = i : n-l,

,0,'1,-+1 R ~ ,i,+1 (~1 (R) for s = r+2 : n,

R ~ 0;'~+x(k)(R) endfor

for I; = r-1 : 1 (-1), t

endfor endfor

endfor

....... y L ~ ' . l , + i (k )

""' . , l R

a-l-+i M M �9 _ -It+ 1 [gJ tgJ -- i

"'. i "..@

~ tl "'.@ : "'-.O i

%,. i r ",.O :

"", O i

Figure 1: local operators for sweep index Program 3: Cyclic by rows, slice(k = k), k refined version

The behaviour of the algorithm is visualized in figure 1 for an arbitrary sweep index k and row index r. The ordering of the operators as induced by the loops in the program determine the initial slice (for n = 4) S~

s~ = ~ l , (k )~ l , (k ) .~J ' (k) ~ )e~l~(k) ~l~ ~l~ val 4 13(k ~9213 (k) oll2(k) O~12(k) ~9112 (k) (9)

The following proposition determines whether two adjacent operators are independent (definition 3.1).

P ropos i t ion 4.1 ( independency) Let tonal(.) and wc~2(.) be two adjacent operators in a f ?~rlr ' l ' l r l r + l slice both taken from the set t ~i~+1 (.),O~ (.),~O:ic+l(.)}. Denote by Ri • Ci the set of

all possible pairs (ri, el) with ri from Ri and ci from Ci.

R2(.) are independent if and only if The operators r and wc2

o r l r + l [ 1 , . ~ ~ r l r + l ( / , ~ r + 2 k ~ / ~ k ' l r - 1 r l r + l

a~ . wa2 ~o i~+l(k) Zgrl~+ x (k) R,1 X C1 N R2 X C2 = O and r ( ) cz (') r arlr+l/b ~ tFlr+l/t.~

~-x k r k 4,+ ~ ( ) ~,I~+~ ( )

PROOF The proof follows from the specification of the operators, o

Proposition 4.1 provides the tool to reorder the sequence of operations in a slice and therefore it is, along with the slice shift step, a tool to detect the critical path moreover it reveals the opportunity to apply algorithmic equivalence transformation steps.

P ropos i t i on 4.2 (a lgor i thmic invariance) Let the triple < r, c, k - l > be a character-

l~me-adaptive Jacobi Type Algorithms 285

ization for the following subslice (operator) composition:

rlr+l(b~ rlr+l r ~ r+ l (1 a o+~ , . , , oo (k) ..,,.+~(t) ~l~+x,.,

If I < k then the following is an algorithmic equivalence transformation step

O~l~+x ,-I,-+1 ~ ~+1 ~+1 a~ l~+ l 0~1~+1 o+x (k) o~ (k) ~'olo+~(t) (t) ~ ~ (t) (1) (k) (k) For historical reasons we call the triple < r, c, k - l > a weave candidate and we refer the algorithmic equivalence transformation step as the weave step.

PROOF The weave candidate defines an elementary Jacobi-step X' , - J(O)X J(7~) T (2), which is associative because J(O) and J(7~) are unconditional matrices; the weave step exploits this property. []

Slice S~ is the, two times, right shifted slice of slice S~ (9). Application of a sequence of dependency preserving transformation steps reorder S~ into slice S~.

~g .~" (k) 4 k) O~'~(k) ~'~ I 1 o~'~(k ~ 1~1~ = r - 1)~3214(k (k) ( i 0 )

Here the first weave candidate announces itself (boxed). The critical path ordering of the slice shows that the critical path is of length 2 ( n - 1) and the operators in the weave candidate are all on a critical path of the slice.

PRECEDENCE GRAPH The occurrences and relationships between the weave candidates (r, c , k - l I in the DG define a precedence graph (PG). Figure 2 gives an example of the precedence graph of weave candidates for a matrix of order n, with n = 7. In this figure AK stands for relative sweep distance k - l.

c o

3 4

A K 3 2

Figure 2: Weave candidate PG for n = 7.

Any feasible cut-set of the PG defines a partial ordered set of weave candidates which includes the root candidate. The weave steps in this set, when applied to the given program (slice) will transform it into an input-output equivalent program (slice). The dependence graph of the initial cyclic by rows algorithm is regular [1]. The aim of 'weaving' is to decrease

286

the program's critical path and, moreover, to arrive, if possible, at an equivalent program that again is regular. The following proposition gives (without proof) the conditions.

P r o p o s i t i o n 4.3 Let prog be the cyclic by rows SVD algorithm of order n with operators 0 rlr+x (k), 8rlr+l(k) and ~lc+x(k) (cf. program 3). Let DG be the dependence graph ofprog

Let p be any point in the 3-D integral space with coordinate axes r, c and A K. Let x and y be integral scalars ( z ,y E Z ) a n d put )~T = [1, -1 , z] and a = x - y . Consider the hyperplane partition

hp(z, y) " )~Tp < a

This partition with 1 < z < n - 3 and 1 < y < n - 2 defines feasible cut-sets. Application of the weave steps in a partition hp(x, y) to DG yield an input-output equivalent dependence graph DG(z, y) which is, moreover, regular together with the original one. The critical path of DG(z, y) is at most of the same length of the critical path of DG; more precisely, if DG has a critical path of length 2(n - 1) then the critical path of DG(x, y) is 2d with 4 <_ d < n - 1.

0

Simulations show that there exists a path that gradually incorporates more and more weave steps of the PG whereby the corresponding programs have gradually decreasing critical paths. However when all weave steps in the PG are applied to the initial program then the resulting input-output equivalent program has a different DG but identical critical path length. The program, DG(x,y), that corresponds with a hyperplane partition hp(z ,y) = hp(2,1) has minimal critical path of length 2 • 4. This particular version has prior been derived by Moonen [5].

A c k n o w l e d g e m e n t

This research was supported in part by the commission of the EU under the ESPRIT BRA program 6632 (NANA-2), and partly by the Dutch Technology Foundation (STW) under contract DEL 00.2331. The authors wish to express their gratitude to having had many opportunities to discuss algorithmic issues with Marc Moonen from the Katholieke Universiteit Leuven, Belgium.

References

[1] Ed Deprettere, Peter Held, and Paul Wielage. Model and methods for regular array design. Int. J. of High Speed Electronics and systems, 4(2):133-201, june 1993. Special issue on Massively Parallel Computing-Part II.

[2] G.H. Golub and C.F. Van Loan. Matrix Computations, (2nd ed.). John Hopkins University Press, 1989.

[3] S.Y. Kung. VLSI Array Processors. Prentice Hall, 1988.

[4] M.Moonen, P. Van Dooren, and J. Vandewalle. An svd updating algorithm for subspace tracking. SIAM J. Matrix Anal. Appl., 13(4):1015-1038, 1992.

[5] Marc Moonen. Jacobi-type updating algorithms for signal processing, systems identification and control. PhD thesis, K.U. Leuven, dept. ESAT, november 1990.


287

A D A P T I V E D I R E C T I O N - O F - A R R I V A L E S T I M A T I O N B A S E D O N R A N K A N D S U B S P A C E T R A C K I N G

B. YANG, F. GERSEMSKY Department of Electrical Engineering Ruhr University Bochum ~ 780 Bochum Germany yng@sth, ruhr-uni-bochum, de

ABSTRACT. In this paper we present an adaptive algorithm for both rank and subspace tracking. It requires O(nr) operations every update where n is the input vector dimension and r is the number of desired eigencomponents. We use this algorithm in sensor array processing to estimate the direction-of-arrival adaptively. It will also be compared with the exact eigenvalue decomposition and the recently proposed URV updating.

KEYWORDS. Adaptive DOA estimation, rank and subspace tracking, PASTd, information theoretic criteria, URV.

1 INTRODUCTION

Subspace tracking has recently attracted much attention [1, 2, 3, 4, 5, 6, 7, 8, 9]. It has a lower computational complexity than an eigenvalue decomposition (ED) of the sample correlation matrix or singular value decomposition (SVD) of the data matrix. Its usefulness in adaptive direction-of-arrival (DOA) estimation and frequency retrieval has been demonstrated in a large number of works.

Unfortunately, most of the subspace tracking algorithms assume a fixed and known number of signals. This may be a valid assumption in certain applications. In many other situations, however, the number of signals or equivalently the rank of the signal part of the correlation matrix is unknown and/or time varying. In this case, we need both rank and subspace tracking.

Stewart [6] proposed an updating algorithm for tracking the signal and noise subspace. It is based on updating the rank revealing URV decomposition. This algorithm is one of the few exceptions which are designed to handle a variable number of signals. It requires O(n 2) operations every update.

288 B. Yang and F. Gersemsky

In this paper, we present another approach for tracking the rank and the signal subspace. This method consists of two parts. First we adopt a known algorithm to update the signal eigenvectors spanning the signal subspace and a fixed number of auxiliary eigenvectors. Additionally, we compute the corresponding eigenvalues and an averaged noise eigenvalue recursively. Then we apply information theoretic criteria to estimate the number of signals and to adjust, if the number of signals has been changed, the number of eigencomponents to be tracked. The total number of operations required by this algorithm is O(nr) per update.

The following notations are used in this paper. Matrices and vectors are represented by boldface and underline characters. The superscripts * and H denote the complex conjugation and Hermitian transposition, respectively. I1" I] is the Euclidean vector norm.

2 E I G E N V A L U E D E C O M P O S I T I O N A N D URV D E C O M P O S I T I O N

We consider an array with n sensor elements. Let x_(t) 6 C n be the vector of sensor outputs at the discrete time instant t >_ 1. A straightforward but computationally expensive method for adaptive D0A estimation updates the sample correlation matrix

c ( t ) = ~ c ( t - 1) + ~(t)~H(t) (1) where 0 < ~ _ 1 is the forgetting factor. Then the ED of C(t) is computed.

Let {vi(t),)~i(t)li = 1 , . . . , n} denote the eigenvectors and eigenvalues of C(t). We assume Ai(t) to be decreasingly ordered. If the number of signals r is known a priori, the first r eigenvectors vl( t ) . . . ,vr(t ) span the signal subspace and the last n - r eigenvectors Vr+l(t ) . . . , v_v_v_n(t ) span the noise subspace. In the case of an unknown r, information theoretic criteria like AIC and MDL [10] are popular tool for rank estimation. They are defined by

AIC(k) = ( n - k)L ln (a (k ) )+ k ( 2 n - k), (2)

t~(L) (3) MDL(k) = ( n - k)L ln(a(k)) + k (2n- k) 2 for k = 0 , 1 , . . . , n - 1 with

a(k)= (i=k~+l )~i(t)) ~(n-k) (i=k~+ 1 )~i(t)) l/(n-k) " (4)

In Eqs. (2) to (4), L denotes the number of sensor output vectors used to form the sample correlation matrix whose eigenvalues are employed in the AIC or MDL criterion. For the sample correlation matrix in (1), L is set equal to the effective length of the underlying exponential window

t Zt-i 1 - Z ' , > I 1 L(~ , t) = E - ~ ~ . (~)

~=~ 1 - ~ 1 - ~

The estimate of the number of signals r is given by the value of k for which the criteria are minimized.

Adaptive DOA Estimation 289

This approach is computationally cumbersome since it requires O(n 3) operations for the ED. Stewart [6] proposed a different scheme for subspace tracking. He introduced the rank revealing URV decomposition. For the t • n data matrix

X(t) = [X/~-lx_(1), . . . , V/'~x_(t- 1), x_(t)] H (6)

with the property C(t) = xH(t)X(t), the URV decomposition is given by

x(t) = u(t) R(t) F(t) o G(t) vH(t). (7)

V(t) G C ~• is an unitary matrix. U(t) e C t• has orthonormal columns. I t( t) E C r215 and G(t) E C (n-r)• are upper triangular. The smallest singular value of It(t) is

approximately equal to the r-th singular value of X(t), and the Frobenius norm of G(t)

is constrained to be at the same level as the white noise variance a 2. Accordingly, the number of columns of It(t) corresponds to the rank of interest, and the first r and last n - r columns of V(t) span the signal and noise subspace, respectively.

When updating the URV decomposition, we start with the known URV decomposition of X(t - 1). At the arrival of the new sensor output vector ~(t), a sequence of left and right Givens rotations is applied to

[ , / ;~x(t- 1) X(t) [ _~H(t)

[ [ ] ] = 0 G ( t - 1) v H ( t - 1) (8) 1 x_H(t)V(t- 1)

to bring X(t) back to the form (7). During this process, two important decisions have to F(t)

be made. First the Frobenius norm of G(t) is compared with the threshold [6, 11, 12]

toll = r /3 a. (9)

Remember that a is the square root of the white noise variance. If the Frobenius norm is larger than toll, the rank may have increased. In this case, Givens rotations are applied such that the small elements in all but the first column of F(t) and G(t) are preserved. Then r is increased by one to reflect the possible increase in rank. The second decision is necessary to detect rank degeneracy. This happens if some of the previously present signals disappear. For doing this, the smallest singular value of the triangular matrix It(t) (or the c~-norm of I t -x ( t ) ) i s estimated by condition estimators [13] and compared with a second threshold

1 (10) to12=r 1 " /3a"

If it is smaller than to12, the rank is decreased by one and R(t) is deflated by forcing the elements in the last column of It(t) as small as a.


Note that toll and to12 are two user supplied thresholds which depend on the noise variance parameter a. In all works known up to date [6, 11, 12], the exact value of a which serves as a design parameter in simulations is employed. This is not realistic because in real applications the noise level is unknown and has also to be estimated. In our simulations reported in section 4, we use the averaged value of all diagonal elements of G(t) instead of a / v ' l - ~ in Eqs. (9) and (10).

We found that the URV updating has two drawbacks. First its computational complexity is quadratic proportional to n. Second it is difficult to choose the user supplied, empirical, and signal dependent factors r and r in Eqs. (9) and (10). The performance of the URV updating is very sensitive to the choice of r and r In our experiments, for example, we had to tune these factors very carefully and to use different values of r for different signal scenarios.

3 A N E W A L G O R I T H M F O R B O T H R A N K A N D S U B S P A C E T R A C K I N G

In this section, we present a new algorithm for tracking rank and signal subspace. For the rank estimation, we need estimates of eigenvalues. Note that not all subspace tracking algorithms are able to provide eigenvalue estimates. The spherical ROSA tracker by De- Groat [5] and the PAST algorithm given in [8] are two such examples. They compute an arbitrary basis of the signal subspace instead of the eigencomponents. We found that one algorithm suits particularly for this task. It is the so called PASTd algorithm described in [8]. It updates both signal eigenvectors and exponentially weighted signal eigenvalues.

The basic idea of the PASTd algorithm is to minimize t

J(~(t)) = ~/~'-~11~_(i ) - ~(t)~H(t)~(i)ll ~ (11) i--1

with respect to v_(t). We have proven in [8] that this leads to the first dominant eigenvector of the sample correlation matrix C(t) in (1). In order to compute v(t) recursively, we approximate v_H(t)x(i)in (11), the unknown projection of x_(i) onto v(t), by the expression y_(i) = v_H(i- 1)x_(i) which can be calculated for all 1 < i _ t at the time instant t. This results in the following exponentially weighted least squares criterion

t

z'(~(t)) = ~ Z~-~El~(i) - ~(t)y(i)ll ~ (12) i--1

which is well studied in the theory of adaptive filtering [14]. The vector v_(t) which minimizes J'(v(t)) can be recursively computed by recursive least squares (RLS) scheme.

After the first dominant eigenvector of C(t) has been determined, we apply the same procedure to seek the second and other signal eigenvectors by deflation. When we remove the projection of the current data vector x.(t) onto the first eigenvector from x(t) itself, the second eigenvector becomes the most dominant one and can hence be extracted in the same way as before. Applying this procedure repeatedly, all signal eigenvectors are estimated sequentially.

Table 1 summarizes the PASTd algorithm. The vectors v~(t) and the quantities ,~i(t)

Adaptive DOA Estimation 291

represent estimates of signal eigenvectors and eigenvalues. The last equation in the table describes the deflation step. The total operation count of this algorithm is 4nr + O ( r ) where one operation is defined as one multiplication and one optional addition. For simplicity, we make no distinction between real and complex numbers.

�9 ~(t) = =(t) FOR i = 1 , . . . , r Do

A~(t) = ZA~(t- 1 ) + ly~(t)l 2 u~(t)

~ ( t ) = ~ ( t - 1) + [~( t ) - ~ ( t - 1 ) y ~ ( t ) ] ~ " " 6 \ - 2

x_i+ 1 (t)= x~(t)- v~(t)yi(t)

Table 1: The PASTd algorithm for tracking the signal subspace

Note that the eigenvector estimates vi(t ) are not perfectly orthonormal. If an orthonormal basis of the signal subspace is required as in the MUSIC or minimum-norm estimator, we must reorthonormalize v_i(t ) at an additional expense of O(nr 2) operations. If we use the ESPRIT method [15] to calculate DOA from the signal subspace estimate, no orthonor- malization is necessary.

The key issue of our approach for rank estimation is to estimate always one auxiliary eigencomponent more than the number of currently detected signals. Let r(t) and rs(t) denote the number of updated eigencomponents and the number of detected signals at the time instant t. We require

~(t) = ~ , ( t ) + 1. (13)

In addition, we track the white noise variance in terms of an averaged noise eigenvalue AN(t). Remember that the vector x~+z(t ) in Table 1 results from r deflation steps. It can be viewed as the projection of the data vector x_(t) into the orthogonal complement of the subspace spanned by v__x(t),...,~(t). Therefore we may update AN(t) as follows

AN(t) = #AN(t- 1) + II~+~(t)ll2/(n- r). (14)

The number of signals is then estimated from the n eigenvalue estimates

~ ( t ) , . . . , ~ ( t ) , ~ N ( t ) , . . . , ~N(t) (15)

n - r times

using the AIC or MDL criterion defined in (2) to (4).

Table 2 gives a brief summary of our algorithm for both rank and subspace tracking. We start with the averaged noise eigenvalue AN(t- 1) and r ( t - 1) eigencomponents {v~( t - 1), A i ( t - 1)1i = 1 , . . . , r ( t - 1)} at the time instant t - 1 from them the first rs( t - 1) = r ( t - 1 ) - 1 ones span the signal subspace. At the arrival of the sensor output vector x_(t), all these quantities are updated by the PASTd algorithm and Eq. (14). Then information theoretic criteria are applied to (15) to estimate the number of signals rs(t). If there is a change in the number of signals, the number of eigencomponents to tracked is corrected.


It is easy to verify that the total computational complexity of this algorithm is still at the order O(nr).

�9 Perform the update {v~(t - 1), Ai ( t - 1)1i= 1 , . . . , r ( t - 1)} ,AN(t - 1), x_(t) {v~(t),Ai(t)[i= 1 , . . . , r ( t - 1)}, AN(t)

via the PASTd algorithm and Eq. (14)

�9 Apply AIC or MDL to estimate the number of signals r~(t) from (15) and

set r( t )= r,(t) + 1 �9 IF r,(t) < r , ( t - 1)

Remove {v~(t), Ai(t)li = r(t) + i,..., r(t- I)} which are no longer needed ELSEIF r,(t) > ro( t - 1)

Generate a new auxiliary eigencomponent by computing

v_.,(t)(t)=z__,(,_~)+x(t)/ll~(,_x)+~(t)ll and A,(,)(t)= AN(t) END

Table 2: An adaptive algorithm for both rank and subspace tracking

We note that when we estimate rs(t), the last n - r(t - 1) noise eigenvalues are identical to AN(t). This makes AIC(k) and MDL(k) to be strictly increasing in k when k >_ r ( t - 1). Hence we only need to compute AIC(k) and MDL(k) for k <_ r ( t - 1) and we always obtain ra(t) <_ r ( t - 1) = r , ( t - 1) + 1. This limits the increase in rank at each update to maximally one. The reason is that we only allowed one auxiliary eigencomponent, see Eq. (13). If more than one signals are expected to appear simultaneously, we can use more than one, say ra~x > 1, auxiliary eigencomponents. An extension of the algorithm in Table 2 to the case r(t) - r,(t) + raux is straightforward.

4 S I M U L A T I O N S

In the following, we show some simulation results to demonstrate the applicability and the performance of our algorithm in adaptive DOA estimation. We used a linear uniform array with n = 9 sensors. At each time instant, the PASTd algorithm and the AIC criterion are used to track the signal subspace. After that, the TLS-ESPRIT method [15] is employed to calculate DOA of uncorrelated narrowband signals. For the startup, we do not use any a priori informations. We set rs(0) to 0 (no signals), v_l(0 ) to the unit vector el, and )q(0) and AN(0) to 1, respectively. For comparison, we also did the same experiments using the exact ED of the sample correlation matrix and the URV updating.

In the first experiment (Figure 1), three closely spaced signals have a DOA spacing which is one half of the beamwidth. In the second scenario (Figure 2), two signals cross at t = 500. Clearly, as they get closer, the algorithms cannot separate them and treat them as one signal. The last experiment (Figure 3) deals with sudden DOA changes.

We observe that the tracking performance of PASTd-AIC is comparable to that of the exact ED. The URV updating provides similar results at the expense of O(n ~) operations

Multiple Subspace ULV Algorithm 293

15 , a) ED-AIC , , b) URV 15 , c) PASTd-AIC,

Ii! _ -10 t , ~ , -10

0 100 200 300 400 0 100 200 300 400 0 100 200 300 400 Time t Time t Time t

Figure 1: Tracking closely spaced signals (SNR = 10 d B , ~ = 0.97, raux = 1, Cx = 1.06, r = 1.6)

Time t

a) ED-AIC b) URV c) PASTd-AIC

0P ~176 i t i o ,o .....

0 500 1000 0 500 1000 0 500 1000 Time t Time t

Figure 2' Tracking crossing signals (SNR = 5 dB for the crossing signals and 0 dB for the alone one,/5 = 0.97, raux = 1, r = 1.06, r = 1.6)

a) ED-AIC

50/~- ' , , o

-50

|

~-5o I

b) URV ,

200 400 600 800 0 200 400 600 800 Time t Time t

c) PASTd-AIC

5O t

0 " 'I!

-50

0 200 400 600 800 Time t

Figure 3: Tracking sudden DOA changes (SNR = 10 dB for the top signal and 0 dB for the other ones,/3 = 0.9, r~x = 2, r = 1.4, r = 1.6)

a) URV b) URV

t o ,. g -50 ~- ~ -50

[:.

0 500 1000 0 200 400 600 800 Time t Time t

Figure 4: Sensitivity of URV updating with respect to the choice of r a) as in Figure 1 except for the choice of r = 1.4 b) as in Figure 3 except for the choice of r = 1.06

294 S. Hosur et al.

and tends to overestimation of the rank in the initial time. Moreover, the performance of URV updating is sensitive to the choice of the signal dependent factors r and r In experiment 3, for example, we had to use a different value for r as in the first two experiments (see Figure 4). In comparison, our algorithm does not need any subjective judgment in the decision process and requires only O(nr) operations.

References

[10]

[11]

[12]

[la]

[14] [15]

[1] P. Comon and G. H. Golub, "Tracking a few extreme singular values and vectors in signal processing," Proc. IEEE, pp. 1327-1343, Aug. 1990.

[2] J. R. Bunch, C. P. Nielsen, and D. Sorenson, "Rank-one modification of the symmetric eigenproblem," Numerische Mathematik, vol. 31, pp. 31-48, 1978.

[3] J. Yang and M. Kaveh, "Adaptive eigensubspace algorithms for direction or frequency estimation and tracking," IEEE Trans. ASSP, vol. 36, pp. 241-251, 1988.

[4] M. Moonen, P. van Dooren, and J. Vandewalle, "Updating singular value decompositions: A parallel implementation," in Proc. SPIE Advanced Algorithms and Architec- tures for Signal Processing, vol. 1152, pp. 80-91, San Diego, Aug. 1989.

[5] R. D. DeGroat, "Noniterative subspace tracking," IEEE Trans. Signal Processing, vol. 40, pp. 571-577, 1992.

[6] G. W. Stewart, "An updating algorithm for subspace tracking," IEEE Trans. Signal Processing, vol. 40, pp. 1535-1541, 1992.

[7] E. Oja, "Principal components, minor components, and linear neural networks," Neural Networks, vol. 5, pp. 927-935, 1992.

[8] B. Yang, "Subspace tracking based on the projection approach and the recursive least squares method," in Proc. IEEE ICASSP, pp. IV145-IV148, Minneapolis, Apr. 1993.

[9] B. Yang and F. Gersemsky, "An adaptive algorithm of linear computational complexity for both rank and subspace tracking," in Proc. IEEE ICASSP, pp. IV33-IV36, Adelaide, Apr. 1994. M. Wax and T. Kailath, "Detection of signals by information theoretic criteria," IEEE Trans. ASSP, vol. 33, pp. 387-392, 1985. M. F. Griffin, E. C. Boman, and G. W. Stewart, "Minimum-norm updating with the rank-revealing URV decomposition," in Proc. IEEE ICASSP, pp. IV465-IV468, San Francisco, Mar. 1992. K. J. R. Liu, D. P. O'Leary, et al., "An adaptive ESPRIT based on URV decomposition," in Proc. IEEE ICASSP, pp. IV37-IV40, Minneapolis, Apr. 1993. G. H. Golub and C. F. Van Loan, Matrix Computations. Johns Hopkins Press, Balti- more, 2 ed., 1989. S. Haykin, Adaptive Filter Theorey. Prentice-Hall, Englewood Cliffs, 2 ed., 1991. R. Roy and T. Kailath, "ESPRIT - - estimation of signal parameters via rotational invariance techniques," IEEE Trans. ASSP, vol. 37, 1989.


295

M U L T I P L E S U B S P A C E ULV A L G O R I T H M A N D L M S T R A C K I N G

S. HOSUR, A. H. TEWFIK, D. BOLEY University of Minnesota 200 Union St. S.E. Minneapolis, MN 55~55 U.S.A ( hosur@ee, tewfik@ee, boley@cs}, umn. edu

ABSTRACT. The LMS adaptive algorithm is the most popular algorithm for adaptive filtering because of its simplicity and robustness. However, its main drawback is slow convergence whenever the adaptive filter input auto-correlation matrix is ill-conditioned i.e. the eigenvalue spread of this matrix is large [2, 4].

Our goal in this paper is to develop an adaptive signal transformation which can be used to speed up the convergence rate of the LMS algorithm, and at the same time provide a way of adapting only to the strong signal modes, in order to decrease the excess Mean Squared Error (MSE). It uses a data dependent signal transformation. The algorithm tracks the subspaces corresponding to clusters of eigenvalues of the auto-correlation matrix of the input to the adaptive filter, which have the same order of magnitude. The algorithm up-dates the projection of the tap weights of the adaptive filter onto each subspace using LMS algorithms with different step sizes. The technique also permits adaptation only in those subspaces, which contain strong signal components leading to a lower excess Mean Squared Error (MSE) as compared to traditional algorithms. The transform should also be able to track the signal behavior in a non-stationary environment. We develop such a data adaptive transform domain LMS algorithm, using a generalization of the rank revealing ULV decomposition, first introduced by Stewart [5]. We generalize the two-subspace ULV updating procedure to track subspaces corresponding to three or more singular value clusters.

KEYWORDS. Adaptive filtering, least mean squares, subspace tracking, singular value decomposition.

296 S. Hosur et al.


The LMS adaptive algorithm is the most popular algorithm for adaptive filtering because of its simplicity and robustness. However, its main drawback is slow convergence whenever the adaptive filter input auto-correlation matrix is ill-conditioned i.e. the eigenvalue spread of this matrix is large [2, 4]. A class of adaptive filters known as the transform domain filters have been developed for the purpose of convergence rate improvement [4]. All transform domain adaptive filters try to approximately de-correlate and scale the input to the adaptive filter in the transform domain, in order to obtain an autocorrelation matrix with zero eigen value spread in that domain.

The convergence rate of the Least Mean Squares (LMS) algorithm is poor whenever the adaptive filter input auto-correlation matrix is ill-conditioned. In this paper we propose a new LMS algorithm to alleviate this problem. It uses a data dependent signal transformation. The algorithm tracks the subspaces corresponding to clusters of eigenvalues of the auto-correlation matrix of the input to the adaptive filter, which have the same order of magnitude. The algorithm up-dates the projection of the tap weights of the adaptive filter onto each subspace using LMS algorithms with different step sizes. The technique also permits adaptation only in those subspaces, which contain strong signal components leading to a lower excess Mean Squared Error (MSE) as compared to traditional algorithms.

Our goal in this paper is to develop an adaptive signal transformation which can be used to speed up the convergence rate of the LMS algorithm, and at the same time provide a way of adapting only to the strong signal modes, in order to decrease the excess MSE. The transform should also be able to track the signal behavior in a non-stationary environment. We develop such a data adaptive transform domain LMS algorithm, using a generalization of the rank revealing ULV decomposition, first introduced by Stewart [5].

The ULV updating procedure [5] maintains and updates only two groups of singular values: the large ones and the "ones close to zero." This is suitable if the input autocorrelation matrix has eigenvalues which could be so classified.

In this paper, we generalize the two-subspace ULV updating procedure to track subspaces corresponding to more than two singular value clusters. Each step of the generalized procedure may be viewed as a recursive application of the ULV decomposition on the upper triangular matrix tt computed at the previous stage within the same step.

2 T H E ULV D E C O M P O S I T I O N

The SVD is tyiJically used to isolate the smallest singular values, and the success of any method based on the SVD depends critically on how that method decides which singular values are "small" enough to be isolated. The decision as to how many singular values to isolate may be based on a threshold value (find those values below the threshold), by a count (find the last k values), or by other considerations depending on the application. However, in extracting singular values one often wants to keep clusters of those values together as a unit. For example, if all values in a cluster are below a given threshold except one, which is slightly above the threshold, it is often preferable to change the threshold than split up the


cluster. In the SVD, this extraction is easy. Since all the singular values are "displayed", one can easily traverse the entire sequence of singular values to isolate whichever set is desired. In this section we present a set of primitive procedures to provide these same capabilities with the less computationally expensive ULV Decomposition.

2.1 DATA STRUCTURE

The ULV Decomposition of a real n x p matrix A (where n _ p) is a triple of 3 matrices U, L, V plus a rank index r, where A = ULV T, V is p • p and orthogonal, L is p • p and lower triangular, U has the same shape as A with orthonormal columns, and the leading r x r part of L has a Frobenius norm approximately equal to the norm of a vector of the r leading singular values of A. That is, A = ULV T with

L = E F

where IlclJ~ ~ ~,~(A) + . . . + ~,~(A) encapsulates the "large" singular values of L. This implies that ( E , F ) (the trailing p - r rows of L) approximately encapsulate the p - r smallest singular values, and the last p - r columns of V encapsulate the corresponding trailing right singular vectors.

In the data structure actually used for computation, L is needed to determine the rank index at each stage as new rows are appended, but the U is not needed to obtain the right singular vectors. Therefore, a given ULV Decomposition can be represented just by the triple [L, V, r].

2.2 PRIMITIVE PROCEDURES

We partition the ULV updating process into a five primitive procedures. The first three procedures are designed to allow easy updating of the ULV Decomposition as new rows are appended. Each basic procedure costs O(p 2) operations and consists of a sequence of plane (Givens) rotations [1]. By using a sequence of such rotations in a very special order, we can annihilate desired entries while filling in as few zero entries as possible, and then restoring the few zeroes that are filled in. We show the operations on L, partitioned as in (2.1). Each rotation applied from the right is also accumulated in V, to maintain the identity A = ULV T, where the U is not saved. The last two procedures use the first three to complete a ULV update.

�9 Absorb_0ne: Absorb a new row. The matrix A is augmented by one row, obtaining

A aTv ) vT"

Then the L, V are updated to restore the ULV structure, and the rank index r is incremented by 1. No determination is made if the rank has really increased by 1; this is done elsewhere. The process is sketched as follows, where C denotes large entries, e,f denote small entries in the ULV partitioning, R denotes an entry of the new row, + a temporary fill, and . a zero entry:

298 S. Hosur et al.

C . . . . apply C+. . . apply C . . . . CO... rotations CC§ rotations CC.. . CCC.. from CCC+. from CCC.. e e e f . right eee f+ left CCCC. e e e f f to get e e e f f to get e e e e f RRRRR . . . . > R . . . . - - - - > . . . . .

�9 Extract_Info: The following information is extracted from the ULV Decomposition: (a) the Frobenius norm of (E, F) (i.e., the last p - r rows of L), (b) an approximation of the last singular value of C (i.e., the leading r • r part of L), and (c) a left singular vector of C corresponding to this singular value. These are computed using a condition number estimator [3].

�9 Deflate_0ne: Deflate the ULV Decomposition by one (i.e., apply transformation and decrement the rank index by one so that the smallest singular value in the leading r • r part of L is "moved" to the trailing rows). Specifically, transformations are applied to isolate the smallest singular value in the leading r • r part of L into the last row of this leading part. The transformations are constructed using item (c) from Ext rae t_Info . Then the rank index is decremented by 1, effectively moving that smallest singular value from the leading part to the trailing part of L. This operation just moves the singular value without checking whether the singular value moved is close to zero or any other singular value.

�9 Deflate_To_Gap: This procedure uses a heuristic to try to move the rank boundary, represented by the rank index r, toward a gap among the singular values. Let s be the smallest singular value of C and let f be the Frobenius norm of [E, F]. Then we use the heuristic that a gap exists if s > dr, where d is a user chosen Spread. In order to allow for round-off or other small noise, we pretend that the trailing part has an extra p + 1-th singular value equal to a user chosen Zero_Tolerance b. Then the heuristic actually used is s 2 > d2(f 2 + b2). If this condition fails, Deflate_0ne is called repeatedly until this condition is satisfied. Hence, any singular value that is below b or within a duster of b will be treated as part of the trailing part. The on!y two user defined parameters needed for this heuristic are the Spread d and the Zero_Tolerance b.

�9 Update: This procedure encompasses the entire process. It takes an old ULV Decom- position and a new row to append, and incorporates the row into the ULV Decompo- sition. The new row is absorbed, and the rank is deflated if necessary to find a gap among the singular values.

3 G E N E R A L I Z E D ULV U P D A T E

The idea of a generalized ULV decomposition, which divides the singular values into more than two clusters can be introduced with the simple example where there are three groups of singular values. Now there are two singular value boundaries, ri & r2, which have to be maintained and updated properly. We have the following primitive procedures which are all implemented by calling the ordinary procedures discussed above with either the data structure [L, V, rx] or [L, V, r2], depending on which boundary must be updated.


�9 Generalized_Absorb_One. Add a new row and update the two boundaries. This procedure just calls Absorb_0ne using the second boundary, i.e. with the data structure [L, V, r2]. This has the effect of incrementing r2, But the resulting rotations have the effect of expanding the top group of singular values by one extra row, hence the first boundary, r l is incremented by one.

�9 Generalized_Deflate_One. This procedure deflates the lower singular value boundary using Defla te_0ne applied to [L, V, r2]. But as in Generalized_Absorb_0ne, the upper boundary must be incremented by one. In order to restore the separation between the first and second groups of singular values that existed before application of these update procedures, the upper boundary must be repeatedly deflated until a gap is found. This process is accomplished using Deflate_To_Gap on [L, V, rl], which does not affect the boundary r2 at all.

Using the generalized ULV decomposition, we can group the singular values of any matrix into an arbitrary number of groups. The number of groups or clusters is determined automatically by the largest condition number that can be tolerated in each cluster. This implies that if one chooses the clustering to be done in such a way that each cluster has singular values of the same order of magnitude, the condition number in each cluster is improved which in turn implies a faster convergence of the LMS filter applied to a projection of weights in the corresponding subspace. The largest condition number is the maximum of the ratio of the largest singular value in each cluster to its smallest singular value. This value depends on the Spread and Zero_Tolerance, specified by the user.

4 T H E U L V - L M S A L G O R I T H M

Let the input signal vector at time n be given as

x , = [ x ( n ) , x ( n - 1 ) , - . . , z ( n - N + 1)] T

and let the weight vector at this time be hn. The corresponding filter output is

zn = xThn,

and the output error en is given as the difference of the desired response d(n) and the output zn of the adaptive filter at time n:

er, = d ( n ) - zn.

The LMS algorithm tries to minimize the mean squared value of the output error with each new data sample received as

h,~+l = h,~ + 2#x,~e,~,

where 0 < # < 1 is the step size.

The convergence of the LMS algorithm depends on the condition number of the input autocorrelation matrix

Rx = XX T ~ E[x,~xT].

300 S. Hosur et al.

If the input vector xn is transformed to un = ETxn, where E is the unitary eigenvector matrix of ttx, then the output process zn would be de-correlated. However, this implies that we need to perform an eigen decomposition of the autocorrelation matrix or a singular value decomposition of the data matrix at every adaptation, implying a computational computational complexity of O(N 3) for every adaptation. One could replace E with V, where V is any unitary matrix which block diagonalizes Rx, separating the signal subspaces from the noise subspace, but this still takes O(N 3) operations to compute.

Instead of transforming the input using the eigen matrix, we could transform the input using the unitary matrix V obtained by the generalized ULV decomposition, which approximately block diagonalizes Rx. This would imply a savings in the computational costs as the ULV decomposition can be updated with each new data at a relatively low computational cost. We note that V almost block diagonalizes Rx in the sense that it exactly block diagonalizes a small perturbation of it. If X T = ULV T with L defined by (2.1)so that

Rx = VL T LV T,

then V exactly block diagonalizes R x - h as follows:

V ( R x - A ) V T = ( CTCO F TO ) w h e r e A = V T ( F TE ETF)

So [[AIIF < f2 is small, where f = [[[E,F]IIF defined above. For a more detailed analysis of the generalized ULV and the subspace tracking LMS algorithm refer to [6, 7].

The input data vector xn is transformed into the vector

Yn = yTxn �9 These transformed coefficients are then weighed using the subspace domain adaptive filter coefficient vector gn. The output signal zn is given as

T zn = gnYn,

and the LMS weight update equation is given by

gn+x = gn + 2Menyn,

where en is the corresponding output error and M is a diagonal matrix of the step sizes used. The diagonal elements of M can usually be clustered into values of equal step sizes, corresponding to the subspaces isolated using the generalized ULV. This clustering is due to the fact that each subspace is selected to minimize the condition number in that subspace. Hence adaptation of all the projected tap weights within each subspace has nearly the same convergence speed and one only needs to match the convergence speeds of the slow converging subspace projections of the tap weights to those of the fast converging subspace projections. This can be done by using larger step sizes for those subspace projections of the tap weights which converge slowly, to increase their convergence speed. Also the clusters obtained using the generalized ULV are very well organized, with the largest singular value duster first, making construction of M is very straightforward. The diagonal values of the upper triangular matrix generated in the generalized ULV decomposition reflect the average magnitude of the singular values in each cluster. This information can also be used in the selection of the step sizes and hence in the construction of M


~ -1{

~ .1 . ~

-2{

.72

' " ~ 1 ~

.1{

-1!

-2{

-2~

Figure 1: Learning curves with two values of W for the LMS algorithm (left) and for the ULV-LMS algorithm (right) (Curves are averages of 20 runs). ~ 1994 IEEE

An increase in step size usually implies an increase in the misadjustment error. The subspaces which belong to small singular values are dominated by noise and would tend to increase the noise in the solution. Thus by not adapting in those subspaces, we can reduce the misadjustment error. This can be simply done by setting those diagonal entries of M, which correspond to projections of the tap weights onto these subspaces, to zero.

5 S I M U L A T I O N RESULTS

We illustrate the performance of our procedure with a simple example in which a white noise random sequence a(n) that can take the values -1-1 with equal probability is filtered with a 3 tap FIR filter whose impulse response is a raised cosine h(n) = ( l+cos(27r(n-2)/W))/2, n = 1, 2, 3. White Gaussian noise is added to the output and an 11 tap equalizer is adaptively constructed using the LMS and ULV-LMS algorithms (Fig. 1). Note that whereas the speed of convergence of the traditional LMS algorithm depends heavily on the eigenvalue spread of the input covariance matrix as determined by W, the ULV-LMS algorithm has no problem adapting to the environment even when W is large (W = 3.5) and the condition number of the input covariance matrix is correspondingly large (Am~x/)~min = 47.4592).

An Adaptive Line Enhancer (ALE) experiment was also conducted to illustrate the performance of the algorithm when the adaptation is done only in the signal subspaces. The

7r 5 1 r input to the ALE was chosen to be 0.1 cos (i~n) + cos ( ~ n ) corrupted by white Gaussian noise of variance 0.0001. The autocorrelation matrix of the input to the ALE has only four significant eigenvahes, which could be grouped into two clusters. The ALE was adapted using both the LMS and the ULV-LMS algorithms. The ULV-LMS algorithm was adapted only in the subspaces corresponding to the two large singular value clusters. The superior performance of the ULV-LMS algorithm can be seen from the learning curves are plotted in Fig. 2.

302 S. Hosur et al.

.I

-7(

,i Ii !ili'!

of ~ ln~Om (D)

Figure 2: Learning curves for the ALE experiment (both methods averaged over 20 runs). ~1994 IEEE

Acknowledgements

This work was supported in part by ONR under grant N00014-92-J-1678, AFOSR under grant AF/F49620-93-1-0151DEF, DARPA under grant USDOC/60NANB2D1272, and NSF under grant CCR-9405380. Figures 1, 2 from [6] are used by permission.

References

[1] G.H. Golub, C.F. Van Loan. Matriz computations. Johns Hopkins Univ. Press, 1988.

[2] S. Haykin, Adaptive Filter Theory, 2nd ed., Prentice Hall, 1991.

[3] N. J. Higham, A survey of condition number estimators for triangular matrices, SIAM Rev. 29:575-596, 1987.

[4] D.F. Marshall, W.K. Jenkins, J.J. Murphy, "The Use of Orthogonal Transforms for Improving Performance of Adaptive Filters," IEEE Trans. Circ. & Sys. 36:474-483, 1989.

[5] G.W. Stewart, "An Updating Algorithm for Subspace Tracking," IEEE Trans. Signal Proc. 40:1535-1541, 1992.

[6] S. Hosur, A. H. Tewfik and D. Boley, "Generalized URV Subspace Tracking LMS Algo- rithm," ICASSP-94 III:409-412, Adelaide, Australia, 1994.

[7] S. Hosur, A. H. Tewfik and D. Boley, "Generalized ULV Subspace Tracking LMS Algo- rithm," Under Preparation.

P A R T 4

A P P L I C A T I O N S



305

SVD-BASED ANALYSIS OF IMAGE BOUNDARY DISTORTION

F. T. LUK, D. VANDEVOORDE Department of Computer Science Rensselaer Polytechnic Institute Troy, New York 12180, U.S.A. [email protected], [email protected]

ABSTRACT. We develop a systematic scheme to limit the effects of boundary distortion caused by the linear restoration of partially captured objects. Although reasonable arguments can be presented to interpret the empirical success of our method, no useful theoretical bound on restoration quality has yet been found. However, for specific cases, an analysis based on the SVD can be constructed to explain the strong attenuation of boundary effects for all images. In addition, the analysis enables us to make more credible claims on the reliability of our restoration method when applied to certain classes of image degradation.

KEYWORDS. Image restoration, image boundary, incomplete deblurring, regularization, SVD.

1 INTRODUCTION

Imaging systems attempt to build a digital or analog representation f of a desired scene. In practice, technological limitations restrict these systems so that they can only obtain an imperfect image g. In this paper we will assume that this imperfection is due to a linear blur and additive noise. Moreover, we will refer to a discrete scene f which is the ideal image we are trying to recover. Thus, we have:

g = H f + e, (1) where g measures the image and e models the noise. The columns of the imaging operator H represent the so- called point-spread function (PSF), i.e., the images resulting from the blur of unit-intensity pixels. In particular, H will exhibit a block-Toeplitz-with-Toeplitz-blocks (BTTB) structure when the PSF is space-invariant (i.e., the spread of each pixel is independent of its position up to the factor intensity). A BTTB-structured matrix H corresponds to an aperiodical convolution operator.

As inverse problems of this kind are known [3] to be ill-conditioned, considerable effort has been devoted to finding well-conditioned nearby problems that have nearby solutions. This process is known as regularization and has been applied primarily to deal with undesirable noise contributions. In this paper, we present and analyze a localized regularization scheme which successfully deals with unpredictable intrusive energy near the boundaries. Previous publications addressing the boundary distortion problem are few; we are aware of only the paper by Woods et al. [5] in which Kalman filters are used to restore the image.

306 F.T. Luk and D. Vandevoorde

This paper is organized as follows. Section 2 reviews common assumptions for modelling the boundary and explains how they may cause problems during restoration. Section 3 briefly describes regularization methods to reduce the effect of noise. Section 4 discusses approaches to handle boundary effects, including the incomplete deblurring method advocated by the authors in [1]. An SVD-based analysis of incomplete deblurring is given in Section 5, followed by conclusions in Section 6.

For ease of comparison, we use the same (f, g) pair in Sections 2 to 4. The image consists of 100x 100 pixels, so the H operator has about 10,000 rows and columns. From Figure 1 we obtained Figure 2 by applying a Gaussian blur with a standard deviation of two pixels and adding 2% uncorrelated Gaussian noise. To avoid the cost of a 10,000x 10,000 SVD, we change to a one-dimensional example in Section 5.

Figure 1 Ideal picture (scene f). Figure 2 Observed picture (image g).

2 RESTORING THE IMAGE BOUNDARY

From the introductory discussion it should be clear that the scene distortion causes every pixel of the image to incorporate a portion of the energy of several pixels in the scene (the contributed fractions can be read from the corresponding row in the matrix H). Figure 3 below illustrates this concept:

ii w

Figure 3 Spread of energy into and out of the image (left: scene f; right: image g).

As a result, the portionf of the scene that affects the image g contains more pixels than g. Phrased differently: system (1) is underdetermined (even though noise is ignored). In the following discussion, we will assume that the image consists of m rows of n pixels each (for a total of M = mn pixels in g) and that the nonzero elements of the PSF fit in a p-by-q template so thatfhas N = (m+p-1)• components.

SVD-based Anlaysis o f Image Boundary Distortion 307

The most commonly used methods for calculating solutions to underdetermined systems (which are also the methods with the lowest computational requirements) find the solution to (1) with the smallest 2-norm. Unfortunately, in the case of our restoration problem, these solutions usually contain many negative (i.e., physically meaningless) components.

A first set of approaches consists in solving the underdetermined system using a criterion to select a physically sound solution. In [1 ] we surveyed some of these criteria (maximum entropy, maximum a posteriori probability . . . . ) and observed that, although many of these methods often produce visually acceptable restorations, they are usually too computationally expensive to use. As a result, a second set of methods to deal with the underdetermined character of equation (1) reduces the number of unknowns by making certain assumptions about the boundary.

One of these methods simply sets the scene elements outside the range of the image to zero, i.e., black. Figure 4 shows the effect of this approach on the image reconstruction; the boundary artifacts result from ignoring what we call intrusive energy (the spread of a part of the scene outside the range of the image g into that image). In space imagery, however, the scene often consists of a few small bright objects on a nearly black background and so this approach is justified. For this case, our linear model reduces to:

g = H z f z + e, (2) where f z contains those elements o f f that are in the range of the image and H z consists of the corresponding columns of H.

Figure 4 Regularized restoration; the scene is assumed to be zero outside the imaging range (p = 0.02, 30 iterations).

Periodical boundary conditions are also commonly used: they are hard to physically justify, but in the case of a space invariant PSF they result in a system that can be solved very efficiently. Indeed, when the image is assumed to be periodical, the space-invariant imaging operator simplifies to a matrix which is block-circulant- with-circulant-blocks (BCCB), i.e., a periodical convolution. It is well known that such a system can be solved using a two-dimensional FFT and even when a simple noise model is taken into account, the resulting Wiener- filter can still be efficiently applied.


3 REDUCING NOISE

Although the noise term in the imaging equation (1) may be very small, it should not be overlooked. Indeed, i f f were estimated using no more than the pseudo-inversion of H:

fe,t= H+g = H+Hf + H+e, (3) the resulting restoration would most likely be unacceptable due to the ill-conditioning of H.

Regularization consists in solving a well-conditioned nearby problem to avoid the consequences of noise in the image. Although several regularization techniques exist, Tikhonov regularization [4] is often preferred because it can co-exist with other algorithms without hurting their computational characteristics. Applied to our imaging equation, Tikhonov regularization results in the following least-squares minimization problem:

- - - S f , , , 2 ) . (4) f P$ 2

The regularization parameter p determines the balance between satisfying the original imaging equation on the one hand (the first term in the right hand side of (4)) and limiting some aspect of f (the constraint operator S is usually the identity matrix or a gradient operator) on the other hand. Although theories exist on the optimal choice of p, they require some knowledge of the image-noise characteristics; consequently, p is usually chosen empirically (i.e., via trial and error).

Nagy et al. [2] show how a BTI'B-structured problem can still be efficiently solved in its regularized form, provided that the constraint operator has a BTTB structure as well. It is also noteworthy that their conjugate- gradient-based algorithm not only provides regularization using the Tikhonov formulation, but also allows additional control by limiting the number of iterations. We have found that this combination of parameters is very useful to handle noisy images; specifically, our two-dimensional examples were all obtained after 30 iterations.

4 REDUCING BOUNDARY DISTORTION

Although global regularization attenuates boundary distortion to a limited degree, it lacks effectiveness against a localized and correlated intrusion. In the following subsections we will use the correlation properties of the intrusive energy to achieve our goals.

4.1 EXTRAPOLATION METHODS

One idea that comes to mind when contemplating the intrusive energy that causes boundary distortion, is to estimate this intrusion and compensate the image g for it. Thus, we may propose the following restoration algorithm based on image extrapolation:

1. Letfx denote the portion of the scenefcorresponding to the image g (i.e., same size and position). 2. Estimate using extrapolation how fx would spread outside of g and build an augmented

image gx that incorporates this "outward spread."

3. Extrapolate the operator H similarly (e.g., assuming space-invariance) to get H,. 4. Work with the new imaging equation: gx = Hxfx + ex.

SVD-based Anlaysis of Image Boundary Distortion 309

This is certainly not the only extrapolation method that can be imagined; in [1 ] we proposed several more but this one seemed to produce the best results (see Figure 5). The extrapolation we talk about in step 2 can be as simple as taking the average of a few (e.g., two) pixels just inside the boundary of g and extending this value in a direction perpendicular to the image boundary (with appropriate scaling to take into account the number of pixels spreading out of g). At a cost of additional computational complexity, linear regression over small (e.g., 2-by-3 pixels) patches can incorporate edge detection into the procedure to further reduce the extrapolation error.

Figure 5 Regularized restoration using extrapolation (9 = 0.02, 30 iterations).

Because the fundamental (ill-conditioned) character of the imaging operator has not been changed, small extrapolation errors still cause noticeable artifacts as can be seen in Figure 5. The unreliability constitutes the main drawback of this approach.

Sometimes, images of adjacent pieces of a scene are available. This might be the case, for example, when a series of images are taken by a polar satellite as it travels along a meridian of the planet. Then the information which would be determined by extrapolation could be replaced by the actual measurement of adjacent images.

4.2 SCENE REDUCTION

Extrapolation attempts to extend the vectors (scene or image) so that we find a square approximation to the imaging operator. We can also reduce the scene vector so that if we blurred this reduced scene fR we would obtain an image gR which is exactly as large as the observed g. An estimator for this gR can be obtained by appropriately scaling down those pixels of g that are not totally determined by the reduced scene fR"

From our experiments with this approach we concluded that the initial loss in reconstructed area is not sufficiently compensated by the apparently improved reliability of the method as compared with extrapolation. We attribute this better reliability to the lack of differentiation in the process of setting up the approximate linear system (as opposed to extrapolation).


4.3 INCOMPLETE DEBLURRING

To improve on extrapolation, we proposed in [1] a new approximation technique to handle the restoration of partial objects without excessive boundary distortion. We call this procedure incomplete deblurring.

As one may expect, the boundary distortion increases as the PSF becomes flatter, i.e., the condition number of the imaging operator worsens. Rather than trying to perfectly recover the boundary, we attempt to "locally improve" the condition of the imaging operator. We do this by assuming a more peaked PSF than the measured one for the pixels near the image boundary and organizing a quick but progressive transition to the real PSF as we model pixels towards the center area. It is hoped that this way the boundary area will remain partially blurred and will absorb the intrusive energy that causes the undesired distortion.

The next question that arises is how to organize this transition. Our answer is to make the sum of each column of H z equal to one (forcing the PSF to be conservative) by increasing (if necessary) the value of the corresponding diagonal entries. This not only achieves our goal of making the PSF more peaked near the boundary of the image, but also restores a balance that was lost by ignoring the energy that spreads outside of the area of interest. Figure 6 illustrates the process for a one-dimensional boundary.

/ \

/ / \ raise peak // ~ (diagonal)

! \

!iiiiii~:: ......... 1

Figure 6 Diagonal compensation for incomplete deblurring.

Figure 7 Regularized restoration using incomplete deblurring (p = 0.02, 30 it.).

Thus our new (square) approximation becomes:

g = (Hz + D ) f t + e = H l f ~ + e , (5) where D = I - diag { column sums of H z }. Note that D can be computed by left-multiplying Hz r with a vector of all ones (and using any available fast algorithm to do that). The result of using this system with Tikhonov regularization on our "standard problem" is shown in Figure 7.

We observed in [1 ] that incomplete deblurring is more reliable than the other methods in this section in that the obtained restoration quality is less dependent on the measured image. Although we cannot prove this claim in general, we give an SVD-based analysis for a particular blurring operator in the next section.


5 SVD-BASED ANALYSIS

In this section we discuss some properties of incomplete deblurring using a one-dimensional example. More precisely, using the singular value decomposition of our original imaging operator H z and compensated operator//I:

Hz = UzSzVz and Hz = U~27zV~, (6) we will compute unregularized restorations:

f z = Hz-l g and f~ = H/l g (7) by applying the SVD factors one at a time. The blurring in this example will be Gaussian with a standard deviation equalling two pixels (Figure 8). The image g we have chosen for illustration will be the intrusive energy caused by a uniform unit-intensity scene immediately to the left of the imaged area (Figure 9).

0.2

0.18

0.16

0.14

0.12

0.1

0.08

0.06

0.04

0.02

5 10 15 20

Figure 8 Gaussian PSF (standard deviation = 2 pixels).

0.46

0.4

0.35

0.3

0.25

0.2

0.15

0.1

0.05

0\ 20 ,,.0 6'0 8'0 loo

Figure 9 Image g (f = 0).

Figures 10 and 11 show a comparison between the singular values (on a logarithmic scale) of our two operators and the angles between the corresponding right singular vectors. It is interesting to note that big differences are not found at the extremes of the SVD spectrum (which determine the condition number).

100

10 "1

10 .2

10 .3

10 .4

10 -s

10 .6

10 -r

10 .8

10 "e

l o , ;

Figure 10 Singular values of H z (dashed line) and H~ (solid line).

Y

Figure 11 Angles (in radians) between corresponding right singular vectors of Hzand Hr.


After applying U r to g we find the vectors (images) shown in Figures 12 and 13. These vectors do not have a direct spatial interpretation anymore, but instead are more closely related to the singular values. It is more instructive to plot abs(Urg) on a logarithmic scale (see Figure 14). We observe that the last few components of this vector are about two orders of magnitude smaller for the case U7 than for the uncompensated case. This is important because, in the next step, we apply the inverse of Z' and those last few components of Urg will be divided by very small numbers, leading to the large results shown in Figure 15. At this point, it is clear that the small amount of intrusive energy with which we started has been grossly magnified by the inversion process and much more so by the uncompensated operators. The final application of V only redistributes this energy in the spatial domain; see Figures 16 and 17.

0.05

0

-0.05

~ 2'o ~ ,b 8'0 1oo' ~o ,oo

Figure 12 Components of Uzrg.

1 2~ g0 ~ ~ 10o

Figure 14 Log. plot of abs(Uzrg) and abs(U~rg), dashed and solid line, resp.

100G 80C

6OO i 4OO

2OO ~ 0 I

-200

-40C

-60C

-800

- 1 ~ 2'o 4'0 8'o 8'0 loo

Figure 16 Components of Vz~Vz~Uzrg.

0"15 T

o.1

0.05

0

-0.05

SOIII I I 1 ' ' ~ -0.1 ~ To 60

Figure 13 Components of UFg.

lO 4

103 1 102

101

100

lO -I

lO -~

lO ~ ~ 6~ 8'0 loo

Figure 15 Log. plot of abs(,gz ~ UzTg) and abs(Z'[JUtrg), dashed and solid line, resp.

10

.~c ~ ~'o ~ ~o ~oo

Figure 17 Components of V1~v[Iu~rg.


In reality, regularization reduces the observed explosions significantly, but the difference remains. For the case of incomplete deblurring the resulting artifacts reduce to almost nothing, while for the uncompensated operator H z they remain noticeable; note the difference in axis scale in Figures 18 and 19. In fact, it is easy to verify that the left singular vectors are not affected when Tikhonov regularization is applied or when the SVD is truncated. In conclusion, we surmise that the restructuring of the left singular vectors is the main benefit of incomplete deblurring, even when combined with a regularization method.

6

4

o

-2

-4

-6

-8

- l o o

1

o.8

0.6

O.2

0

-

Figure 18 Regularized reconstruction with Hz (P = 0.001" S =/) .

Figure 19 Regularized reconstruction with/-/i (9 = 0.001" S = I).

6 CONCLUSIONS

We have presented an SVD-based analysis that provides some insight into the behavior of our new compensation algorithm. In particular, it appears that for a class of commonly occurring point-spread functions the change in the imaging operator will result in intrusive energy that will be constrained in a "less singular" subspace during the inversion process. Numerical experiments performed using dual-peaked PSFs seemed to result in similar improvements; however, for these cases, artifacts remained fairly obvious even when incomplete deblurring was applied.

More investigations remain to be done about the theoretical foundations and practical implementations of incomplete deblurring. For future work, we plan to examine the possibility of developing new (i.e., specialized) preconditioners to enhance the performance of the linear solver we apply to handle the new imaging operator.

Acknowledgments

This work was supported in part by the Office of Naval Research under contracts N00014-93-1-0268 and N00014-93-1-0396. The authors wish to thank Joyce Brock for her careful reading of the manuscript.

References

[1 ] F.T. Luk and D. Vandevoorde. Reducing boundary distortion in image restoration. Proc. SPIE, 2296, Advanced Signal Processing Algorithms, Architectures, and Implementations VI, 1994.

[2] J.G. Nagy, R.J. Plemmons and T.C. Torgersen. Fast restoration of atmospherically blurred images. Proc. SPIE, 2296, Adv. Signal Processing Algorithms, Architectures, and Implementations VI, 1994.

[3] C.K. Rushforth. Signal restoration, functional analysis, and Fredholm integral equations of the first kind. In: H. Stark (Ed.), Image Restoration: Theory andApplication, Academic Press, pp. 1-27, 1987.

[4] A.N. Tikhonov and V.Y. Arsenin. Solutions of Ill-Posed Problems. V.H. Winston and Sons, Washington, D.C., 1977.

[5] J.W. Wood, J. Biemond and A.M. Tekalp. Boundary value problem in image restoration. Proc. ICCASP, pp. 18.11.1-4, Tampa, 1985.



315

T H E S V D I N I M A G E R E S T O R A T I O N

D.P. O'LEARY Department of Computer Science and Institute for Advanced Computer Studies University of Maryland College Park, MD 207~2 U.S.A. [email protected]

ABSTRACT. This paper focuses on SVD-based algorithms for ill-posed problems in image restoration. We discuss the characteristics of solutions produced by various regularization methods, including truncated least squares, regularized least squares, regularized total least squares, and truncated total least squares. Economical ways are presented to compute each of these solutions using iterative methods.

KEYWORDS. SVD, image restoration, deblurring, total least squares, least squares, ill- posed problems.

1 S V D - B A S E D M E T H O D S F O R I M A G E R E S T O R A T I O N

In image processing applications, data are gathered by convolution of a noisy signal with a detector. A linear model of this process leads to an integral equation of the first kind. Since the measured data is available only at a finite number of sample points (pixels), the continuous model is replaced by a discrete linear model equation

K z = Yo + e = y, (1)

where y contains the measured data values, Y0 contains the (unknown) true data values, e represents noise, K is a matrix of dimension m x n, and we assume that m _ n. In all but trivial problems, the continuous problem is ill-posed in the sense that small changes in the data can cause arbitrarily large changes in the solution, and this is reflected in ill- conditioning of the matrix K of the discrete model, increasing as the dimension of the problem increases. Thus, at tempts to solve (1) directly are doomed to failure.

316 D.P. O'Leary

2 L E A S T S Q U A R E S M E T H O D S

Practical methods for solving the discretized problem (1) must diminish the effects of e and differ only in their filtering of this noise. The singular value decomposition (SVD) plays a fundamental role, both in the computation and in the analysis of these methods. Let the SVD of the matrix K be denoted by

g = ~ a, u, v T. (2) i--1

Here, the singular vectors are ui and vi and the singular values ai are ordered as al >_ a2 _> �9 .. >__ an _> 0. In discrete ill-posed problems, the matrix K has a cluster of singular values near zero, and the size of this cluster increases when the dimension is increased.

The least squares solution to (1) is the solution to the problem

m~n Ilgx - YlI2, (3)

and can be written as

XLSQ = E - - Vi, (4) i=10"i

where ai = uTy. In XLSQ, error in the directions corresponding to small singular values is greatly magnified and usually overwhelms the information contained in the directions corresponding to larger singular values. Regularization methods differ only in how they choose to filter out the effects of these errors. To explain these choices, we will follow the development of I-Iansen and O'Leary [9].

In T i k h o n o v r egu la r i za t ion [13] we solve the minimization problem

~ n { IIg~ - YII] + ~ll~ll~ }. (5)

The parameter A controls the weight given to minimization of []zl[ 2 relative to minimization of the residual norm. Sometimes a seminorm IlL xl12 is substituted, where L typically is a discrete approximation to some derivative operator; by transformation of coordinates, this problem can be reduced to one involving minimizing the standard 2-norm [1].

In the t r u n c a t e d SVD regularization method [8, 16], one truncates the summation in (4) at an upper limit k < n, before the small singular values start to dominate.

Certain iterative methods for solving the least squares problem (3) also have regularization properties. The con juga t e g rad ien t family of methods minimizes the least squares function over an expanding sequence of subspaces

~k = span{KTy, ( K T K ) K T y , ..., ( K T K ) k - I K T y } .

The LSQR implementation of Paige and Saunders [11] is well suited to ill-posed problems.

Like the least squares formulation (3), the solutions produced by these regularization methods are solutions to minimization problems. There are two complementary views" they minimize the size of the solution x subject to keeping the size of the residual r = K x - y less than some fixed value, and in a dual sense they minimize the size of r subject to keeping

The SVD in Image Restoration 317

the size of x less than some value M(A), monotonically nondecreasing as the regularization parameter A decreases. The way that "size" is measured varies from method to method, but many of these methods are defined in terms of the norms induced by the bases of the singular value decomposition. Although the 2-norm is invariant with respect to the choice of an orthonormal basis, other norms do not share this property. We will denote the p-norm in the SVD basis by I1" lip, defined by

Ilzllp_ = I1~11, if x = ~ivi; Ilrllp_ = 11711, + IIr• if r = ~ T i u i + ra.. i=1 i=1

Hansen and O'Leary [9] verify the following characterizations:

Method II minimizes ! domain ]

Tikhonov Ilrl12 Truncated SVD

LSQR

{~: 11~112 ~ M(A)} Ilrll~ {x: Ilxlk ~ M(A)} Ilrll~ {x: Ilxll~ ~ M(A)} Ilrl12 {z:x e X:k}

The "smoothness" of the solutions in the lp family for a particular value of M(A) decreases as p increases. The truncated SVD (11) solution has no components in directions corresponding to small singular values. The Tikhonov (12) solution has small components in these directions. The lp (p > 2) solutions have larger components, and the ler solution has components of size comparable to those in the directions corresponding to large singular values. These methods can also be generalized to weighted lp norms.

From this discussion we see that the choice of regularization method is a choice of a pair of functions, one measuring the size of the residual and the other measuring the size of the solution vector. The functions determine precisely how the effects of error are damped. The choice of the regularization parameter determines the size of the solution vector.

3 T O T A L L E A S T S Q U A R E S M E T H O D S

All of the regularization methods considered so far have the weakness that they ignore errors in the kernel matrix K. In reality, such errors may be large, due to discretization error, numerical quadrature error, and departure of the behavior of the true measuring device from its mathematical model.

We can account for errors in the kernel matrix as well as the right-hand side by using a total least squares criteria (or errors in variables method) [5, 15] for measuring error. In total least squares, we seek to solve the problem

rain II(K, y ) - (k , ~)IIF subject to ~ = ~" x . x,g

The solution to this problem can be expressed in terms of the singular value decomposition of the matrix (K , y)

n + l

( g , y ) = ~ #ifii~ T. i=1

318 D.P. O'Leary

We partition the matrix V, whose columns are vi, as

P q ." ; (

where p is the largest integer so that #p > #p+l. Then the total least squares solution vector is defined by

If the problem is ill-conditioned, then the total least squares solution can be unduly affected by noise, and some regularization is necessary. In analogy to least squares, we can add the constraint

rank((~', #)) ~ k + 1

to produce a t r u n c a t e d total least squares solution, or add the constraint

Ilxll < M(~) to produce a T ikhonov total least squares solution. Some properties of the truncated total least squares solution for ill-posed problems are discussed by Fierro, Golub, Hansen, and O'Leary [2], while the Tikhonov total least squares sohtion is being studied by Golub, Hansen, and 0'Leary [3].

4 C O M P U T A T I O N A L ISSUES

For large matrices K, the SVD is too expensive to compute, but it is well known that anything defined by the SVD can be approximated through quantities computed in Lanczos bidiagonalization in the LSQR algorithm [11]:

KV = UB, where VTV = Ikxk, UTU = l(k+i)x(k+1), B = bidiagonal(k+1)x k.

The approximation will be good if singular values drop off rapidly and only the large ones matter, as is common in ill-posed problems. The problem is projected onto the subspace spanned by the columns of V, and either the least squares or the total least squares solutions can be computed on this subspace. Regularization can be added as necessary; see, for example, [10].

The Lanczos bidiagonalization iteration is inexpensive if K is sparse or structured, since it depends only on matrix-vector products plus vector operations.

5 NUMERICAL EXPERIMENTS

We illustrate the effects of different regularization methods using a trivial test problem, the 16 • 16 image shown in the figure. The kernel matrix K was constructed to model Gaussian blurring: the value for pixel (i,k) was constructed by summing the 9 nearest neighbors (j, l) weighted by e -'I((i-j)~+(~-02). This matrix K is a spacially-invariant idealization of a physical process. We assume that the true physical process has some spacial dependence,


represented by error drawn from a uniform distribution on [ -am, am] added to each matrix element. The true image was convolved with K plus these errors to produce a blurred image, and then Gaussian noise e, with standard deviation ar times the largest element in the original image, was added to the blurred image. This resulting image, along with the idealized kernel matrix K, form the data for the problem.

Much work has been done on how to choose the regularization parameter [14] (or decide when to stop an iterative method) using methods such as the discrepancy principle [6, w generalized cross-validation [4], the L-curve [7], etc. We sidestep the important question of how to choose this parameter by using an utterly impractical but fair regularization criterion" set the regularization parameter, truncation parameter, or iteration count to make the norm of the answer equal to the norm of the image. The choice of norm is the one appropriate to the method: 11 for truncated SVD, 12 for Tikhonov least squares, etc.

Before plotting the answers in Matlab, we set all negative pixel values to zero. For each problem, the results of seven reconstructions were computed:

�9 The truncated SVD method, based on the 11 norm in the SVD basis.

�9 Tikhonov regularization, based on the 12 norm in the SVD basis.

�9 The lo~ method, based on the loo norm in the SVD basis.

�9 The least squares solution from Lanczos bidiagonalization of the matrix K.

�9 The total least squares solution from Lanczos bidiagonalization of the matrix K.

�9 The least squares solution from Lanczos bidiagonalization of K preconditioned by a Toeplitz approximation to K.

�9 The total least squares solution from Lanczos bidiagonalization of K preconditioned by a Toeplitz approximation to K.

Numerical experiments were run with a,~ = 10 -1, 10 -3, 10 -5, and ar = 10 -1, 10 -3, 10 -s. Typical results are shown in the figures. Several conclusions emerged:

�9 The truncated SVD solution is useful in high-noise situations, but has limited usefulness when noise is small.

�9 The Tikhonov and loo solutions are useful when the noise is small.

�9 For noise of 10 -1, the Lanczos algorithm quickly reproduces the Tikhonov solution (in fewer than 40 iterations), but for noise of 10 -3 or less, the cost is high (greater than 130 iterations).

�9 Preconditioning can significantly reduce the number of iterations.

�9 The Lanczos total least squares solutions are quite close to the Lanczos least squares solutions.

320 D.P. O'Leary

References

[1] L. Eld~n, Algorithms for regularization of ill-conditioned least squares problems, BIT, 17 (1977), pp. 134-145.

[2] It. Fierro, G. H. Golub, P. C. Hansen, and D. P. O'Leary, Regularization by Truncated Total Least Squares, in Proc. of the Fifth SIAM Conference on Applied Linear Algebra, J. G. Lewis, ed., SIAM Press, Philadelphia, 1994, pp. 250-254.

[3] G. H. Golub, P. C. Hansen, and D. P. O'Leary, Tikhonov Regularization and Total Least Squares, Tech. ttept., Computer Science Dept., University of Maryland, to appear.

[4] Gene H. Golub, Michael Heath, and Grace Wahba, Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter, Technometrics, 21 (1979), pp. 215-223.

[5] G. H. Golub and C. F. Van Loan, An Analysis of the Total Least Squares Problem, SIAM J. Numer. Anal., 17 (1980), pp. 883-893.

[6] C. W. Groetsch, The Theory of Tikhonov Regularization for Fredholm Integral Equa- tions of the First Kind, Pitman, Boston, 1984.

[7] P. C. Hansen, Analysis of Discrete Ill-Posed Problems by Means of the L-Curve, SIAM Review, 34 (1992), pp. 561-580.

[8] P. C. Hansen, The Truncated SVD as a Method for Regularization, BIT, 27 (1987), pp. 354-553.

[9] P. C. Hansen and D. P. O'Leary, The use of the L-curve in the regularization of discrete ill-posed problems, SIAM J. on Sci. Statist. Comp., 14 (1993), pp. 1487-1503.

[10] Dianne P. O'Leary and John A. Simmons, A Bidiagonalization-Regularization Proce- dure for Large Scale Discretizations of Ill-Posed Problems, SIAM J. on Sci. and Statist. Comp., 2 (1981), pp. 474-488.

[11] C. C. Paige and M. A. Saunders, LSQR: An algorithm for sparse equations and sparse least squares, ACM Transactions on Mathematical Software, 8 (1982), pp. 43-71.

[12] J. Skilling and S. F. Gull, Algorithms and applications, in Maximum-Entropy and Bayesian Methods in Inverse Problems, C. It. Smith and W. T. Grandy Jr., eds., D. tteidel Pub. Co., Boston, 1985, pp. 83-132.

[13] A. N. Tikhonov and V. Y. Arsenin, Solutions of Ill-Posed Problems, Wiley, NY, 1977.

[14] D. M. Titterington, Common structure of smoothing techniques in statistics, Interna- tional Statistics Review, 53, 1985, pp. 141-170.

[15] S. Van Huffel and J. Vanderwalle, The Total Least Squares Problem - Computational Aspects and Analysis, SIAM Press, Philadelphia, 1991.

[16] J. M. Varah, Pitfalls in the numerical solution of linear ill-posed problems, SIAM 3. on Scientific and Statistical Computing, 4 (1983), pp. 164-176.


Original image

i,,":,--'_':))ill!)i I I | l l m l l l l l l l | l i l l i i l n l l

Lanczos LS

LT E 4j ':]

Prec. Lanczos LS I n ] n i l ' I l l lW I I U l

.... . =...~:., :-," �9 �9 ~ ~ ~ # - . ......

.... . .., .

Blurred image ........ ~::=----=:!

:~ ~ ~.:.~mm:

m m l l ~ 1

~ ) m m m n m m m m ~ im ~ = m m u n ~ = ~ : u ~ .|~m~mm~ u m m m ~ ~ m m m m

l ~ m m m m m . : ~ . m m ~

Prec. Lanczos TLS

i ~.,i... ~,~ ~,m~, sigma_.m - I .~-I, sioma_r - I .~-5

Trunc. SVD image

;i):)iiiiill mmn~= ~.~m~mmm

m ~ = ~ r ~m=m, m~ =m~m~ =mmi

m ~ m m * m m ~ m m m ~

I-infinity image

Im m n m m m m m m l m m lnmm mmimmumml immmm mmmmmmmm

iNN nmm i

I r . . . . .

Lanczos LS

=::~:=m r:.,~;::] n m I m | .... = ~ : , , , , , ,}....~ i .

lmm~mmmm mm mm

mum ~ m m m m m ~m m ~mm

~ m m ~ m~ m i ~.. ~ ~ = I m : : m m ~ m : : u : ;

m m ~ m = m ~ m '

Blurred image ~ ~=mmnmnmmumnl

Lanczos TLS

Prec. Lanczos TLS

sigma_rn" 1 .e-5, sigma_.r, 1 .e-1

Trunc. SVD image

~ m m a :i ~ m m m ~ m ~ a mmm~ mm~mmmma

lmm~mmmmu~ ~ma =mm.~m .......... ~ ~a

llmi ~Imm

| u i : . m n m m m m ....

mmm ~ mmm mm m �9 m)~ ~nmmm~mm

mn~ m m ~ : m m ~ ,: ~m =mm m ~ m ~

mmm ~mm m

m~~

I I I U ~ �9 ~ l �9 I I I . ' . ~ . I . " = ~ I I i N I I , - I I I R ~ m l |. m m ~ ~mmm !

|m~ m~mm, ~ m m ~ l

322 D.P. O'Leary

Original image

"-:-:Ji i : : : : :": : : ~::,U_.liiiiiiil

Lanczos LS i u ~..:~l ! i i I ! L~:~r:::IV::::~.S I m ~ ~ - - ',',: ',

Lal ............. ::T"I �9 m--..;

~ 4 ~ T l [ ~ ( o ) l i

=smi~u mnmmm ~

Blurred image

Lanczos TLS

Prec. Lanczos TLS

[ , l r : l u i l~,:i: l ::: l i :: l :5.1 I i : ! : ! : ~ : : ~ i

s igma_m = 1.e-I, sigma_r = I .e-1

im~ummmmmm~mm(

im)~ ~mmmm~mml ~mm~::~,mmmmmmm~ ~ m m ~ ~ m m m m l l~ma~ ~ ..... m~mm(

mm~m..~:~s~mm~.l

~::~m~mmmmmmm~

Tikhonov image ~ : . ~ ~ / : .....

. ~ H _ ~ .... ~+.

: ~ = ~ : : i ~ : | : : u ~ m ~ m m i u ~ m l

: m m ~ i , ~ m i m , m m u

mummmm~m:-~.mm::l

m~mm mm.mm m~l

Original image

: ':| |!|E|||| |) n m m i a m u n i m m ! mmmm iimnmmnm mmmmm mmmmmmn mm mmm mmmm| mm mmmm mmmm mmmmmmmmm mmm mmmm mmmm am

mmmmmmmmmmmm mmmmmmmmmmmmm!

mmmmmmmmmmmmmu

m mmmmmmmmmmm mmm mmmmmmmm| mmmm mmmmmmm| mmmmm mmmmmml mm mmm mmmml ~mm mmmm mmm|

l i B | m a n | i l i u m i m i i n a n m m i B | m m l

Bluffed image . . . . '..~. ~ | l l l i n i l

: ) ?

Lanczos TLS :==:==========: i = m m u n m m = m m u t n n u n n u u n n n n , ~ u u u u m n u n n n n , i m m l l u m m m m u m

t m m m m m I , m m m m = m m m m m t m m m m a m m m a m m m t a m m m m m m m m m a m m ,

: F T ; N ~ 11~: I n u u n m n n n m u n n n

i n m n n n n n n n m u u u m n u m u m u m m m i m m m u m a m m m m l l u u n n m m u u m u u i m u a n n n m u n i m m R m n m m m m

sigma..m = I.e-5, sigma..r = 1.e-5

Trunc. SVD image

~ ~ _ : i /~..:.; -. ~ . / ~ . . . . ,,

Tikhonov image

m~~ | | m n m m m | m m | m m l

n n n n m u n n n n n l u n n n u m n m m m u l u n m u m m u m m l n n - m n n m m m m l

n n m n n n n n n n u n m m m m m m m m m m m n m !


323

T W O D I M E N S I O N A L Z E R O E R R O R M O D E L I N G F O R I M A G E C O M P R E S S I O N

J. SKOWRONSKI, I. DOLOGLOU Laboratoire des Signaux et Syst~mes, CNRS-ESE Plateau de Moulon, 91192 Gif-sur- Yvette France skowronski @lss. supelec, fr

ABSTRACT. A zero error modeling technique for two dimensional signals is presented. It is pointed out that a rank reduction algorithm applied to the signal renders the modeling physically meaningful in the sense that the estimated exponential sinusoides coincide with the real ones. Image compression is performed as an application of the present method.

KEYWORDS. Zero error modeling, rank reduction, spectral estimation, prony, image compression.


The estimation of one and two dimensional exponential sinusoids is an important problem in many signal processing applications. This paper presents an optimal representation of a discrete two dimensional signal using these functions, according to the following expression,

2 2

I(m, n) = ~ ~ Aija~Z? sin(12jm + win +r (1) i=1 j = l

where I(m, n) denotes a sample of the signal. The major problem concerning this representation is the accurate determination of the attenuations a i ,~ j and the frequencies wi, llj. The application of a conventional least square modeling approach may yield an approximation of these parameters. The amplitudes Aij and the phases r of the exponential sinusoids can then be obtained by Prony's method. Since, in general, the modeling error will not vanish completely, there will be an error made on the attenuation and frequency estimates, causing important distortions to the signal reconstructed according to (1).

As a remedy to this problem, a zero modeling method has been proposed in [1] for monodimensional signals, which can be found at the meeting point of LPC and SVD analysis. When choosing the model order p sufficiently large, that is p = N , where N is the

324 J. Skowronski and L Dologlou

number of samples in the signal, the covariance matrix C ~ + 1 of order N + 1 has always

a zero eigenvalue, meaning that the corresponding eigenvector provides a zero modeling error. Consequently, the exponential sinusoids, obtained from this eigenvector, allow a perfect modeling of the signal.

In order to obtain a physically meaningful representation of the signal, using the above approach, it is necessary to dispose of a reduced rank signal (rank(C~.+l) < N). Since, in

general, signals are not of reduced rank an algorithm was proposed in [3] [2] to optimally reduce the rank of one and two dimensional signals. However, in practice, this algorithm only provides an approximately reduced rank signal, implying a small but non zero modeling error. This, in turn, does not permit a conventional modeling approach to determine accurately the exponential factors and frequencies. Only the zero error modeling technique can do so.

In the following, the outlined modeling procedure is first detailed for 1D signals and then generalized for images. Section 2 describes the zero error modeling approach which leads to a signal representation given by equation (1). In section 3 the rank of a signal is defined and the importance of a rank reduction of the signal is explained. Such a rank reduction algorithm is presented in section 4. The paper is concluded with an application of the present method to image compression.

2 SIGNAL MODELING

The determination of the frequencies and the exponential factors is basically a spectral estimation task and much work has been done on this subject for monodimensional [5] and bidimensional signals [6]. These methods are all based on a linear prediction model. In the monodimensional case the model can be written as follows:

htSp = e t (2) where h ~ = [hi, h2,..., hp+l] is the vector of prediction coefficients, Sp the signal observation matrix and e t the modeling error. For a 1D signal l(n),n = 1,...,N, the signal observation matrix Sp, which is of the Hankel form, is given by:

I(1) 1(2) . . . I ( N - p ) 1(2) I(3) . . . I (N - p + 1)

sp = : : ".. : (3)

I(p + 1) x(p + 2) . . . X(N)

Let us consider the solution of the SVD problem, that is the minimization of e 2 in : t ~ A(hth i e 2=ete=hSpSph+ - ) (4)

This solution is given by the eigenvector hmi,~ of the signal covariance matrix Cp+1 = SpS~, which corresponds to the smallest eigenvalue Ami,~ = e2min of Cp+l [5]. In general, Amin is small but nonzero, which results in a modeling that is not exact. Consequently, there will be an error made on the estimation of the exponential factors and frequencies. This inaccuracy, especially the one on the attenuations, yields fairly important distortions to the estimates of the exponential sinusoids, which is the reason why Prony's method gives poor

2-D Zero Error Modeling for Image Compression 325

results�9

The only way of eliminating the modeling error is to increase the order of prediction p. In [1], it was pointed out, that for a prediction order p = N, N being the number of samples, C ~ + 1 = S ~ S ~ becomes singular. The eigenvector hmi,~ corresponding to the

zero eigenvalue yields therefore zeros that exactly describe the N samples of the signal l(n).

In the following, this zero error modeling approach is generalized to twodimensional signals I (m , n), m = 1 , . . . , M, n = 1 , . . . , N. For this purpose we consider the modeling of a multichannel signal (k channels) as proposed in [2]. Consequently, a unique prediction equation describes each one of the k channels, according to the following expression,

t i t Yi = 1 k (5) h S p = e i . . . where h t denotes the common vector of prediction coefficients, S~ the signal observation

t the associated modeling error . The above k equations matrix of the i-th channel and e i may be expressed as follows,

hiS v = [e t , e~ , . . . , e~] = e' with

I ( i , 1) I(1,2) S p - -

. . . 1 ( 1 , N - p ) I(1,2) I(1,3) . . . I ( 1 , N - p + l)

�9 �9 . , �9

I ( 1 , p + l ) I ( 1 , p + 2 ) . . . I ( 1 ,N)

i(k, 1) z(k, 2) i(k, 2) z(k, a)

�9

I ( k , p + 1) I ( k , p + 2 ) . . .

�9 �9 �9

�9 0 �9

. . �9

�9 �9 O

. . . i(k, N - p)

. . . I(k, N - p + 1)

I(k, N)

(6)

(7)

As in the monodimensional case, we only obtain a global prediction error e equal to zero, if Sp is singular, that is, if rank(Sp) <_ p. In general this is not true, but one may show that for a particular order of prediction Pmaz, Sp,~ox becomes singular�9 This order Pmax is equal to,

k ~ N (8)

P,,~=x = k + 1 Hence, the maximum order p~a~ provides a model which vanishes the modeling error.

Note that for k = 1, that is for monodimensional signals, P~=z = N, suggesting that equation (8) is a direct generalization of the one dimensional case.

As before, the exponential factors and frequencies of the signals sinusoids are obtained through the modeling vector h. While these parameters are the same for every channel, the amplitudes and phases have to be calculated separately for each channel, using Prony's method.

Let us now consider an image I of dimension M • N. Our aim is to describe this signal matrix as a linear combination of 2-dimensional exponential sinusoids as stated in equation (1). We determine the exponential factors and frequencies sequentially, using the above described multichannel zero error modeling scheme, first in the horizontal (h) and then in


M N it is possible to model the image as a the vertical (v) direction. Choosing Ph = multichannel signal of M lines. Obviously the model order Ph must be a positive integer. Furthermore, in order to allow the construction of a physically meaningful observation matrix, the model order has to be inferior to the number of samples per line (Ph < N). These constraints restrict the image dimensions to:

M <_ g - 1 M , N E R (9) The corresponding constraints in vertical direction are obtained by interchanging M and N,

N <_ M - 1 M , N E R (10) These two constraints contradict each other and therefore an exact modeling of images is theoretically not possible. However, in practice this is not a problem, since we can always neglect lines or columns during the calculation of the attenuations and frequencies. As an example the squared lena-image (512 • 512) is considered. In order to determine the horizontal parameters, one line has to be removed. Similarly, for the calculation of the vertical parameters one column has to be ignored.

Once exponential factors and frequencies are known the phases and amplitudes can be obtained using Prony's method, that is by projecting the image onto the 2D-functions constructed from all possible combinations of the horizontal and vertical exponential factors and frequencies. When the number of these latter parameters gets too large, this optimal procedure becomes computationally infeasible. One then has to progress sequentially. First the horizontal attenuations and frequencies are determined and subsequently the phases and amplitudes are calculated for each line, using a monodimensional Prony's method. The amplitudes (phases) of each exponential sinusoid can then be considered as functions of the vertical space variable, which together form a multichannel signal. By modeling this signal as explained above, we can obtain the 2D signal representation (1).

The above described method allows the exact representation of a 2D signal of M • N samples, as a sum of exponential sinusoids. Since, in general, a finite signal contains an infinite number of exponential sinusoids the obtained parameters cannot be physically meaningful. If, however, the signal contains a limited number of exponential sinusoids (MN D%'f~- or less in the horizontal direction and N :W-gy M or less in the vertical direction), the above method provides physically meaningful parameters.

Hence, in order to obtain a physically meaningful representation of the signal, it is necessary to reduce the number of sinusoids contained in the signal, that is to reduce the signal rank. Such a rank reduction algorithm is presented in section 4, but first the rank of one and two dimensional signals is defined.

3 S I G N A L R A N K

The rank of a one-dimensional signal I (n) of N samples can be defined as follows:

Def in i t ion 1 A signal I(n) , n = 1 , . . . , N has rank r , iff the rank of the signal observation matrix verifies rank(St ) = r. Sr is given by:

2-D Zero Error Modeling for Image Compression 327

1(1) I(2) .. . / ( N - r) z(2) • . . . / ( N - r + 1)

Sr = . . .. . (11) �9 �9 �9 ,

I ( r + l ) I ( r + 2 ) .. . I ( N ) w~ ,av that ~ ,ig~at ha, /a l l ,a~k i Z , ~ k ( S ~ ) = ~. This definition is readily generalized to two-dimensional signals I (m, n). Since the image is described by different parameters in horizontal and in vertical direction, it is obvious, that the rank could be different for the two orientations. Let us therefore define the horizontal rank as follows (the vertical rank being defined in an analogous manner):

Def in i t ion 2 A signal I ( m , n ) , m = 1 , . . . , M , n = 1 , . . . , N has rank rh, iff the rank of the signal observation matrix verifies rank(Srh) = rh and iff rh is the smallest value for which this is true. Srh is given by:

1(1,1) 1(1,2) . . . I ( 1 , N - r h ) . . . I(1,2) I(1,3) .. . I ( 1 , N - r h + l )

S ? 'h - - �9 ~ �9 �9 " ~ "

�9 �9 �9 ~ ~ 1 7 6

I ( 1 , r h + l ) I ( 1 , r h + 2 ) . . . I(1, N) .."

I (M, 1) I (M, 2) . . . I ( M , N - r h ) 1(M, 2) I (M, 3) . . . I ( M , N - rh + 1) (12)

I (M, rh + 1) I (M, rh + 2) . . . I ( M , N ) M We say that a signal has full rank iff rank(S~+~y ) = D,u

The signal rank gives us an information about the number of exponential sinusoids that r exponential are contained in the signal. More precisely, a 1D signal of rank r contains

sinusoids (assuming the absence of pure exponential functions). Since we are aiming at a compact signal representation (1) using few exponential sinsuoides, it is important to reduce the rank of the signal, that is to extract the most important sinusoidal components of the signal, while keeping the distortion low. Such a rank reduction algorithm is described in the following. Note that it only reduces the rank assymptotically. Therefore it is essential to determine the sinusoidal parameters using the zero error modeling approach to avoid the effects of modeling errors as it will be demonstated experimentally in section 5.

4 RANK REDUCTION

The most efficient method to reduce the rank of a matrix is to decompose it in terms of its singular values [5]:

r + l

s , = ~ ~ ' (13) i U i V i i = 1

where ai are the real nonnegative singular values and ui(v/) the left (right) side eigenvectors of St. We obtain a reduced rank approximation St (closest to St in the sense of the Frobenius norm) by setting to zero the smallest (or several smallest) singular values ai.


Since, in general, Sr is not an observation matrix, that is it does not exhibit the particular Hankel structure, the reduced rank matrix Sr has to be projected onto the subspace of Hankel matrices. One way of doing this, is to replace the elements of each antidiagonal by their mean value. Obviously the projected matrix is not a reduced rank matrix anymore, but it has been shown in [2] that it is closer to a reduced rank matrix than St. By repeating this procedure, we obtain an assymptotically reduced rank observation matrix which lies 'close' to the original matix. The convergence of this algorithm and its generalization to multichannel signals has been presented in [2]. The rank of two dimensional signals can be reduced by alternating an iteration of the presented algorithm in the horizontal direction and then in vertical the direction.

In [4] it has been shown, that each iteration of this rank reduction method is equivalent to zero-phase filtering the input signal. Since during filtering no frequency components can be created, it becomes obvious, that the rank reduction algorithm supresses gradually the less important sinusoidal components of the signal, while it retains the energetically most important ones. Note also that this rank reduction algorithm only yields an exactly reduced rank signal after an infinite number of iterations.

In the following the zero error modeling method presented here is compared to a conventional least square modeling approach and it is shown how images can be compressed by means of the present technique.

5 E X P E R I M E N T A L RESULTS

In order to show the performance of the zero error modeling method, the rank of a multichannel signal (8 lines taken from the Salesman image) is reduced, that is the number of exponential sinusoids composing the signal is reduced as well. Figure 1 and 2 allow a straight comparison between the reconstruction of the rank reduced multichannel signal (rank(Sp) ~ 180) applying both the conventional modeling method (p < Pmax) and the zero error modeling method presented in this paper. Using the same number of sinusoids (90), the obtained PSNR (33.9dB in the first case and 94.9dB in the latter case) shows dearly the superiority of the new approach.

In figure 3 the original and the compressed version of the lena image are shown. Com- pression is obtained by first reducing the horizontal and vertical rank of the image. Then, the important sinusoidal components are determined using zero error modeling. Finally the image is reconstructed by means of Prony's method. The indicated compression ratio of 8.3 takes into account the higher precision needed for the coding of the exponential factors. Note however, that the number of exponential factors and frequencies is very low (O(N)) compared to the number of amplitudes and phases (O(N2)). Furthermore, higher compression should be obtained based on an hierarchical quantification of the amplitudes and phases.

2-D Zero Error Modeling for Image Compression 3 2 9

original mult ichannel signal

8 250 300 350

50

reconstruction based on conventinal modell ing

50 150 200 250 300 350

reconstruction based on zero error modell ing

6 350 50

Figure 1: R e c o n s t r u c t i o n of a R e d u c e d R a n k M u l t i c h a n n e l Signal us ing a c o n v e n t i o n a l M o d e l i n g M e t h o d ( P S N R 3 3 . 9 d B ) and the Zero Error M o d e l i n g M e t h o d ( P S N R 9 4 . 9 d B )

original mul t ichannel s ignal

1 2 0 . . . . . . . . . . . . ! . . . . . . . . . . . . . ! . . . . . . . . . . . . ~ . . . . . . . . . . . . . ~ . . . . . . . . . . . . . ! . . . . . . . . . . . . . . ! - - : . . . . . . . . . ! -

o o . . . . . . . . . . . . i . . . . . . . . . . . . . ~ . . . . . . . . . . . . . i . . . . . . . . . . . . . :: . . . . . . . . . . . . . :- . . . . . . . . . . . . . :: . . . . . . . . . . i-

80 ........... " ........... i ............. ~ ............. :: ............. i ............. i . . . . . . . . !-

50 1 O0 1 50 200 250 300 350

reconst ruc t ion based on convent ina l model l ing

, ~ o I - . . . . . . . . . . . . I . . . . . . . . . . . . . ! . . . . . . . . . . . . ! . . . . . . . . . . . . ! . . . . . . . . . . . . . ': . . . . . . . . . . . . . . I . . . . . . . - /~ . . . . ~ - - i 1 o o . . . . . . . . . . . . ! . . . . . . . . . . . . . . ! . . . . . . . . . . . . . i . . . . . . . . . . . . . i . . . . . . . . . . . . . ! . . . . . . . . . . . . . . ! . . . . . . . . . . ! -

50 1 O0 1 50 200 250 300 350

reconst ruct ion based on zero er ror model l ing

120 100

80 60 40

50 1 O0 150 200 250 300 350

Figure 2: One C h a n n e l of the M u l t i c h a n n e l Signal


6 CONCLUSION

We have presented a zero error modeling technique for two dimensional signals. It was pointed out that a rank reduction algorithm applied to the signal renders the modeling physically meaningful in the sense that the estimated exponential sinusoids coincide with the real ones. Using this optimal representation, data compression and in particular image compression can be performed.

References

[1] I.Dologlou, G. Carayannis, "LPC/SVD analysis of signals zith zero modeling error", pp.293-298, Signal Processing, Vol.23, 1991

[2] I.Dologlou, J.C. Pesquet, G. Carayannis, "Analyse multidimensionelle s l'aide d'un nouveau module multicanal et d'un algorithme de projections successives. Application

l'analyse d'images.", 14 ~me Colloque Gretsi,' Juan les Pins, 13-16 Sept. 1993

[3] J.A. Cadzow, "Signal Enhancement - A composite property mapping algorithm", IEEE Tr. on ASSP, Vol.36, No.l, 1988

[4] I.Dologlou, G.Carajannis, "Physical Interpretation of Signal Reconstruction from Re- duced Rank Matrices", pp.1681-1682, IEEE Trans. on Signal Processing, Vol.39, No.7, July 1991

[5] S. Haykin, "Adaptive Filter Theory", Prentice-Hall

[6] J.J. Sacchini et al., "Two-Dimensional Prony Modeling and Parameter Estimation", pp.3127-3137, IEEE Trans.on Signal Processing, VoI.41, No.11, Nov.1993

2-D Zero Error Modeling for Image Compression

!:.'.~8i

:!iiiii?~ii

,x, -:(<)-',: r

,g:g. .

ii i ::!iiii:::iii ii: ii

331

(a} (b}

Figure 3' (a) Original Lena Image (512x 512); (b) Reconstructed Lena Image (Compression Ratio: 8.3, PSNR = 30.8dB)



333

R O B U S T I M A G E P R O C E S S I N G F O R R E M O T E S E N S I N G D A T A

L.P. AMMAN University of Texas at Dallas P.O. Box 830688 Richardson, TX U.S.A.

ABSTRACT. Remote sensing has become an important resource for a wide variety of disciplines. This paper describes the application of a statistically robust SVD (RSVD) to remote sensing data. This robust SVD characterizes the main features of the data but, unlike the SVD, is not subject to the distorting effects caused by out-lying subpopulations (outliers). It is also shown how RSVD can identify and visualize these outliers. Specific topics discussed include a brief description of RSVD, storage of its output, image processing and histogram equalization, and modifications to allow efficient processing of high-dimensional remote sensing data. Examples of the application of these methods to a SPOT satellite data set are presented.

KEYWORDS. Robust singular value decomposition, remote sensing, image processing.


Remote sensing data from satellites such as the Landsat and SPOT series have been, and continue to be, collected and archived. Furthermore, new imaging satellites have been launched, or are planned, by the French (SPOT) and Japanese (MOS, JERS), and U.S. plans include a new series (EOS, Landsat). This data plays an important role in areas such as energy and mineral exploration, environmental and land use studies, military surveillance, and archeology.

Landsats 4 and 5 were launched in July 1982 and March 1984, respectively, each carrying a multispectral scanner (MSS) and Thematic Mapper (TM) in a 16 day repeat cycle sun synchronous orbit. The Thematic mapper has 30m resolution and records intensities at 7 spectral bands: green, red, blue, two near infrared, one middle infrared, and a thermal infrared. Plans for future platforms specify sensors capable of recording intensities at 128 spectral bands with 5 meter resolution.

334 L.P. Ammann

The Systeme Probatoire d'Observation de la Terre (SPOT) satellite was launched on February 21, 1986 by France and provides three wavelength bands in the visible and near infrared at 20 meter resolution and 10 meter resolution in a panchromatic mode.

Remote sensing data sets typically consist of a large number of multivariate observations, the structure of which may change over time. In order to visualize such data, one can assign an RGB color channel to each of 3 bands and then create a false-color image. Additional images can be created by using various band ratios to generate the RGB values. A common problem with satellite data is that topographic features produce uneven illumination due to shadowing from any appreciable relief present in the scene. One way to overcome the potential confusion from this situation is to use band ratios, since a surface should receive the same proportion of energy across the spectrum without regard to its orientation to the sun, and should reflect in proportion to its spectral reflectance properties. Furthermore, particular band ratios can be selected to emphasize the differential response of various surface components to different wavelengths when these components are specified in advance, [3].

The reflectance of a feature at one wavelength is typically correlated with its reflectance at other wavelengths, and so some of the information contained in the individual bands and band ratios is redundant. A commonly used approach to overcome this problem is to use principal component analysis (PCA) to "decorrelate" such information.

2 P R I N C I P A L C O M P O N E N T A N A L Y S I S A N D R S V D

Let X denote an n • p matrix of n observations on p variables. In remote sensing applications, n would represent the number of pixels and p would represent the number of bands and band ratios. Suppose for convenience that the mean of each variable has been subtracted from each respective column, so that the column means can be taken to be 0. The sample covariance matrix for this data is then

S= n -1 IX'X" Principal component analysis is derived from the EVD of S, or equivalently, from the

SVD of X. The eigenvectors define a coordinate system in which the new variables are uncorrelated. These new variables, called the principal components, are obtained by rotating the original data matrix, Z = X V , so that

c o v ( Z ) = 1 Z'Z= 1 E, n - 1 n - 1

where E is the diagonal matrix of eigenvalues.

Principal component analysis is a useful tool in remote sensing for the following reasons.

1. The principal components represent uncorrelated blocks of information, and therefore can provide better separation of different features within a scene.

2. The first principal component has maximum variance and so an image constructed from this component has the highest contrast among all possible linear combinations

Robust Image Processing for Remote Sensing Data 335

o

0 W t I I I

Figure 1' Mixture data with SVD prin- dpal components and 95% confidence ellipsoid

i

Figure 2: Mixture data with 95% confidence ellipsoid rotated to the principal component axes

Figure 3: Mixture data with RSVD principal components and 95% confidence ellipsoid

Figure 4: Mixture data with 95% confidence ellipsoid rotated to the RSVD principal component axes

of the original variables.

3. Each eigenvector defines a hyperplane, and the corresponding eigenvalue represents the variance of the data values about that hyperplane. If any eigenvalue is small, then the corresponding eigenvector defines a strong linear relationship among the band values.

4. The EVD can be used to define a distance scale for points in p-dimensional space which can be used to flag pixels whose reflectance values are unusually far away from the center.

5. Discriminant analysis is directly related to PCA. It is used for classification of pixels into subgroups based on their multivariate response. However, discriminant analysis ordinarily requires training data, which is obtained either from a detailed study of ground-based data, or from laboratory-based experiments that characterize the spectral response properties of known minerals, vegetation, or other objects of interest.

PCA is quite well-behaved if the distribution of the data is a multivariate Gaussian distribution. However, if even a small subset of the observations comes from a different

336 L.P. Ammann

distribution, then the eigenvalues and eigenvectors can be greatly distorted, [5], [7]. Since a typical remote sensing data set contains a variety of different features possessing different spectral responses, then classical PCA can result in a significantly distorted view of this data. Figure 1 shows a data set that has a main population and two subpopulations with the ordinary principal components superimposed along with a 95% tolerance ellipse. Figure 2 displays this data set in the coordinate system defined by the eigenvectors. As can be seen in these figures, the subpopulations distort the eigenvectors associated with the main population. The subpopulations also mask their presence due to inflation of the eigenvalues of the main group. This greatly reduces the ability of PCA to identify these subpopulations.

Figures 3 and 4 show the same data set using RSVD instead of the SVD. Note that the eigenvectors and eigenvalues returned by RSVD are close to those associated with the main population. RSVD is thus able to identify the subpopulations as outliers and effectively isolate them from the main population.

A detailed description of the derivation and properties of I~SVD is given in [1], [2]. There it is shown that the symmetric QR algorithm is equivalent to an iteration of two steps consisting of least squares regression fits of each column of the data matrix on the columns to its left followed by a rotation of the coordinate system to the hyperplanes defined by these least squares fits. RSVD replaces the least squares regression step by a weighted QR decomposition in which the weights are functions of the residuals and leverage of the data in the current coordinate system. In addition to rotation, an updated location estimate is obtained after each weighted QRD, and the data is then re-centered before obtaining the next set of weights.

The output of RSVD includes the rotated and re-centered data matrix (the robust principal components) and a matrix of weights W. Each column of W describes the distance of each data point from the center along the coordinate defined by the corresponding robust eigenvector. These weights are derived from a weight function, r A commonly used weight function is the biweight,

(1 - (z)2)2 Izl _< c, r 0, ' Iz l>~,

where c is a constant, and z represents a random variable scaled to have variance equal to 1.

3 I M A G E P R O C E S S I N G A N D RSVD F O R R E M O T E S E N S I N G

The raw data from Landsat or Spot consists of an integer in the range 0-255 for each pixel and spectral band. These values are stored as 1-byte character values, so the file size of one data set equals npixels, nbands. Full-color images can be constructed from these data sets by assigning ttGB channels to three of the bands in the data set. Oftentimes such images are not very useful because the data values tend to be concentrated within a fairly narrow interval of values within 0-255. For this reason, contrast enhancement techniques are commonly employed to construct more useful images. One such technique is referred to as histogram equalization, [3], [6]. This method transforms the data values so that the


new values have a specified target histogram. If Fk is the histogram of band k and if G is the target histogram, then the data values are transformed by Yk = G-I(Fk(Xk)). This transformation can be implemented via a look-up table to reduce the computational load.

Although the preceding discussion has concentrated on remote sensing data, color images can also be constructed from any three dimensional data set by converting the data values for each variable into integers in the range 0-255 and then assigning RGB channels to the resulting variables. However, histogram equalization would usually be required to obtain effective images, and it should be performed before converting the values to 0-255. If most of the data values are concentrated in an interval that is much smaller than the overall range, then these values would get mapped to just a few distinct output values by this data conversion, and so much of the information represented by this portion of the data would be lost. This situation is often the case with RSVD and PCA. Because the output of RSVD is ordinarily floating point numbers and therefore look-up tables could not be used, the need for histogram equalization would result in a very high computational cost in addition to a high storage cost.

A compromise between the retention of full information of the principal components and the computational advantages of look-up tables can be obtained by transforming the floating point output of the RSVD principal components to 2-byte integers, and then performing histogram equalizations on the resulting integer values, which are in the interval 0-65535. This interval is sufficiently wide to retain essentially all of the information in the principal components. The storage requirements for the principal components are also reduced, requiring 2 , npixels, nband8 bytes, and look-up tables can be used for histogram equalization. There would be a slight increase in computational cost since histogram equalization would need to be applied to 65536 values rather than 256, but that is still small compared to the data sizes associated with remote sensing.

The output of RSVD also includes weights for each band, which are floating point values in the interval 0-1. These weights are useful for outlier detection. They can be transformed to 1-byte integers in the interval 0-255 without much loss of information, greatly reducing their storage cost.

4 I M A G E C O N S T R U C T I O N

As described above, histogram equalization is a useful method for expanding the dynamic range of the core values of the principal components. A number of target histograms have been used for image construction, most notably the uniform distribution and the Gaussian distribution. The uniform distribution expands the contrast of the extreme values relative to the middle values, whereas the Gaussian distribution emphasizes the middle values. For general purposes, the Gaussian distribution seems to give the best results, and a target Gaussian distribution with mean = 128 and s.d. = 40 provides reasonable contrast across the interval 0-255.

Because remote sensing data sets often contain mixtures of reflectance features, there are some situations where visualization of the major feature of a data set is required, and other situations in which the identification and imaging of smaller features may be important.

338 L.P. Ammann

The weight matrix output by RSVD can be used to identify outliers in each principal component as follows. Suppose that Z is a random variable from a standard Gaussian distribution with mean 0 and s.d. 1. Let a denote some suitably small probability, e.g., a = .05, and let ua be the solution to

v(r < ~=)= ~.

Then any pixel with a weight less than or equal to ua in column k of the weight matrix would be classified as an outlier in principal component k. For example, if r is the biweight function with c = 4.685, a = .05, then ua = 0.6806, which would correspond to a value of ascii(174) in the weight matrix.

Identification and classification of pixels that belong to relatively rare features can be performed by converting the set of weights to a binary number, bl- . .bp, where bk = 1 if r < u~, bk = 0 otherwise, and p represents the number of bands and band ratios. There are potentially 2 p groups that could be defined in this way, and the classification of pixels into these groups can be done with simple integer operations on the weight matrix. Note that this unsupervised classification can be performed without the need for ground-truth data.

Let X denote the matrix of principal components from RSVD and let W denote the weight matrix. Images that emphasize the major feature of a data set can be constructed using the following steps, written in Matlab.

[n,p] = size(W); A = zeros(n,p);

f o r k = l : p

Y = X ( W ( : , k ) > 174, k);

Z = normalize(Y, #, a) % histogram equalization

A(W(: , k ) > 174, k ) = Z;

end

This algorithm performs histogram equalization only on the pixels that are not outliers in a particular output band, while the values for the outliers are replaced by 0. A similar process can be used to emphasize the minor features of a data set, the only difference being that the pixels in an output band that undergo histogram transformation are those which have weights that are less than the cutoff. The values of the other pixels in that output band are replaced by 0. Images can then be constructed as usual from any three of the resulting output variables.

Images can also be constructed directly from the weight matrix to reveal the location and structure of outliers by assigning RGB channels to three of the columns of W. Pixels that are dark in one or more of the RGB channels correspond to outliers in the respective coordinates.


5 H I G H - D I M E N S I O N A L D A T A SETS

Applications that require the addition of band ratios or other functions of the original data can produce fairly high dimensional data sets. In addition, new sensors will undoubtedly have a much higher number of spectral bands than current platforms. Unfortunately, the computational cost of applying RSVD to such data would be too large to be practical at the present time. However, the basic goal of RSVD is to uncover low eigenvalue subspaces that appear to have relatively large eigenvalues due to outliers or mixtures of subpopulations (as in Figure 1). Therefore, if the ordinary SVD of the data matrix has a set of low singular values, then it is not likely that RSVD could reveal any further structure within the corresponding subspaces. For this reason, the SVD can be used as an initial dimension reduction step, and RSVD can then be applied to the data after deflating to the subspace spanned by the eigenvectors corresponding to the largest singular values.

To illustrate the use of these methods, they were applied to a data set taken from a SPOT image of a portion of the barrier reef off the coast of Belize in Central America. Most of this image is reflected energy from the ocean's surface, with a much smaller portion from the shallow reefs and exposed cays. The three bands in this data set were augmented with the addition of the following band ratios'

1 1 2 2 3 3 1 , 3 1 , 2 2 , 3 2' 3' 1' 3' 1' 2' 2 , 2 ' 3 , 3 ' 1 , 1

Because the variability of the band ratios in this data set is considerably less than that of the bands themselves, the band ratios were multiplied by 50. After centering the resulting data matrix by subtracting columns means, the SVD of the centered data matrix was obtained, X = U D V ~. The resulting singular values are:

20.83167, 16.19306, 4.37871, 2.47400, 1.00195, 0.55052,

0.26432, 0.17507, 0.09431, 0.04762, 0.02009, 0.01517.

The data was deflated to the six-dimensional subspace defined by the first six eigenvectors, Y = XV( : , 1"6) , and RSVD was then applied to Y. The robust singular values obtained by RSVD are:

7.4638, 3.6880, 1.4418, 0.0297, 0.0210, 0.0114.

These robust singular values are smaller than the original singular values, which indicates that the original values have been inflated by the presence of outliers.

An image constructed from the weight matrix of RSVD shows that pixels located at deeper ocean locations are bright white, whereas the reefs and exposed areas are dark in one or more bands. This indicates that the reefs are outliers relative to the major response, ocean surface reflectivity. To emphasize the reefs, an image was constructed from the RSVD principal components by performing histogram equalization on the pixels of each output band that had weights _< 174. Output band values that had weights > 174 were replaced by 0. The result is an image in which the ocean surface reflectivity is essentially removed, and which therefore gives a much sharper view of the underlying reefs.

The pixels in this data set can be classified into 2 p -- 64 subgroups using 174 as the cutoff

340 L.P. Ammann

for outliers. In this case 339209 (57.51%) of the pixels are classified into the main group, which consists of those pixels that are not outliers in any principal component. The second largest subgroup is the set of pixels that are outliers in all principal components. This group contains 39955 (6.77%) of the pixels. The next two largest subgroups are those pixels that are outliers just in the sixth and just in the third principal components, containing 29338 (4.97%) and 26701 (4.53%) pixels, respectively.

Images constructed using the techniques described here are available via anonymous ftp from ftp.utdallas.edu in the directory, /pub/satellite. These images are in PPM format, which can be viewed by a variety of imaging programs. Images in this format can be converted to other formats by the publicly available PBMPLUS tools.

6 E X T E N S I O N S

Because of the increasing popularity of Geographic Information System (GIS) packages among researchers in Geology, Social Science, and Environmental Science, a number of researchers are developing multivariate databases in raster coordinates to take advantage of the capabilities of these packages to organize and display such data. These databases have the same basic structure as remote sensing data sets: each record corresponds to a geographic location within a lattice and has associated with it a vector of observations. Although the values of these observations are not necessarily integers in the interval 0-255, RSVD can be applied to these databases, and the output of RSVD can be put into the same format as for remote sensing data. The image construction and classification techniques described above can be performed on this output, which can provide visualization and analytical tools to search for any structure that may be present in these databases.

References

[1]

[2]

[3] [4]

[5]

[6]

[7]

L.P. Ammann. Robust singular value decomposit ions- a new approach to projection pursuit. J.A.S.A. 27, pp. 579-594, 1993.

E.M. Dowling, L.P. Ammann, R.D. DeGroat. A TQR.-iteration based adaptive SVD for real time angle and frequency tracking. IEEE Trans. Signal Processing, 42, pp. 914-926, 1994.

S.A. Drury. Image Interpretation in Geology. Allen and Unwin, 1987.

G.H. Golub, C.F. Van Loan. Matrix computations. North Oxford Academic Publishing Co., Johns Hopkins Press, 1988.

F.I~. Hampel, E.M. Ronchetti, P.J. Rousseeuw, W.A. Stahel. Robust Statistics, The Approach Based on Influence Functions. J. Wiley and Sons, 1986.

J.J. Jensen. Introductory digital image processing, a remote sensing perspective. Prentice-Hall, 1986.

P.J. ttousseeuw, A.N. Leroy. Robust regression and outlier detection. John Wiley and Sons, 1987.


341

S V D F O R L I N E A R I N V E R S E P R O B L E M S

M. BERTERO University of Genoa, Department of Physics Via Dodecaneso, 33 16146 Genova, Italy bertero @ge nova. in fn. it

C. DE MOL University of Brussels, Department of Mathematics Campus Plaine C.P.217, Bd du Triomphe 1050 Brussels, Belgium [email protected]

ABSTRACT. Singular Value Decompositions have proved very useful in getting efficient solvers for linear inverse problems. Regularized solutions can be computed as filtered expansions on the singular system of the integral operator to be inverted. In a few special instances of interest in optical imaging, we show that the relevant singular systems exhibit a series of nice analytic properties. We first consider the inversion of a finite Laplace transform, i.e. of a Laplace transform where the solution or the data have finite support. The corresponding singular functions are shown to be also the eigenfunctions of a second-order differential operator closely related to the Legendre operator. This remarkable fact allows to derive further properties, which are similar to some well-known properties of the prolate spheroidal wave functions (derived by Slepian and Pollack for the case of the finite Fourier transform inversion). The second compact linear integral operator we consider describes the one-dimensional coherent imaging properties of a confocal microscope. Its singular functions have very simple analytic expressions and moreover, the determination of the generalized inverse of this integral operator reduces to the inversion of an infinite matrix whose inverse can also be derived analytically.

KEYWORDS. Inverse problems. Laplace transform inversion, optical imaging.


The interest for inversion methods is growing in parallel with the development of remote sensing techniques, sophisticated imaging devices and computer-assisted instruments, which

342 M. Bertero and C. De Mol

require clever processing and interpretation of the measurements. In most modern imaging or scattering experiments, indeed, the measured data provide only indirect information about the searched-for parameters or characteristics and one has then to solve a so-called inverse problem. In many situations, the model relating these parameters to the data is linear or can be linearized. Moreover, the linear inverse problem is often equivalent to the inversion of an integral operator, i.e. to the solution of an integral equation of the first kind:

K ( z , y ) dy = g(x) . (1) f(u)

The integral kernel K(x, y) describes the effect of the instrument, medium, propagation, etc. For example, in space-invariant optical instruments, g ( z , y ) = S(z - y) where S(z) is the impulse response of the instrument. The inverse problem consisting in the restoration of the object f from the image g is then a deconvolution problem. In another type of applications, it is required to resolve exponential relaxation rates. This happens e. g. when trying to discriminate the sizes of small particles undergoing Brownian motion by the technique of photon correlation spectroscopy [1]. If all the particles are identical, the output of the correlator is a decreasing exponential function with a relaxation constant inversely proportional to the particle size. Hence, for an inhomogeneous population, a superposition of such exponential patterns is observed, and the determination of the size distribution requires the inversion of a Laplace transform, or in other words, the solution of eq. (1) with K ( x , y ) = exp(-xy) . Notice that in this case the unknown function has clearly a finite support because some bounds on the particle sizes can be given a priori on the basis of physical grounds. In a similar way, in optical imaging, it is often possible to assess a priori the support of the observed specimen. To overcome the non-uniqueness and stability problems inherent in inverse problems, it is of uttermost importance to take explicitly into account all the available a priori knowledge about the solution and, in particular, of such finite-support constraint. This explains why many of the integral operators one has to invert for practical purposes are in fact compact operators. In the next section, we recall some basic features concerning the inversion of compact bounded linear operators, stressing in particular the usefulness of their singular systems. In the rest of the paper, we present some properties of the singular system of two particular operators we studied in view of solving practical inverse problems. In Section 3, we consider the previously mentioned Laplace transform on a finite interval and in Section 4, an integral operator describing the imaging capabilities of a confocal microscope. We focus here on some mathematical results which we believe to present some interest independently of the original applications we were considering.

R E G U L A R I Z E D S O L U T I O N S OF L I N E A R I N V E R S E P R O B L E M S B Y F I L T E R E D SVD

In agreement with the previous discussion, we assume that the relationship between the unknown characteristic f and the data g can be described by a known bounded linear operator A (e. g. an integral operator) so that the linear inverse problem we consider

SVD for Linear Inverse Problems 343

consists in solving the operator equation

A f =g (2)

where f and g belong to Hilbert spaces, say respectively F and G. We assume moreover that A is a compact operator. The linear inverse problem is then ill-posed, because either the inverse operator A -1 does not exist (the null space N(A) of A is not trivial) or A -1 exists but is unbounded, so that the solution does not depend continuously on the data. When N(A) # {0}, a common practice for restoring uniqueness is to select the so-called minimal norm or normal solution belonging to the orthogonal subspace N(A) • On the other hand, a solution of eq. (2) exists if and only if the data g belong to the range R(A) of the operator. This, however, is unlikely to happen for noisy data and hence, usually, one looks for a pseudo- or least-squares solution of eq. (2), i.e. for a function providing a best fit of the data, say minimizing I ]A f - gila. The least-squares solution that has minimal norm is called the generalized solution f t of eq. (2). The operator A t : G --. F, which maps the data g on the corresponding generalized solution f t , is called the generalized inverse of the operator A. The generalized inverse of a compact operator is either unbounded or finite-rank (see e. g. [9] or [7] for more details).

A natural framework for analyzing the inverse problem is provided by the singular system of the operator A. To fix the notations, let us recall that this singular system {ak; uk, vk}, k = 0, 1,2, . . . , solves the following coupled equations

A vk = ak Uk ; A* u k - ak Vk. (3)

The singular values O'k, real and positive by definition, will be ordered as follows: a0 _> al > a2 > "... For a compact operator with infinite rank, we have limk~oo O'k -- 0. The singular functions or vectors uk, eigenvectors of AA*, constitute an orthonormal basis in the closure of R(A) whereas the singular functions or vectors Vk, eigenvectors of A'A, form an orthonormal basis in N(A) • We can then write the following expansion

A f = ~ ak (f, vk)F uk (4) k

which generalizes the classical Singular Value Decomposition (SVD) of matrices. Accord- ingly, the generalized solution f t admits the following expansion in terms of the singular system:

1

k

The above expression is clearly meaningless for noisy data when At is unbounded. Moreover, even ill the finite-rank case, when the sum is finite, the determination of f t is likely to be unstable, the stability being governed by the ratio between the largest and the smallest of the singular values of A, i.e. by the so-called condition number. When this number is too large (ill-conditioning) or infinite (ill-posedness), one has to resort to some form of regularization to get numerically stable approximations of the generalized solution f t . Notice that finite-dimensional problems arising from the discretization of underlying ill- posed problems are likely to be ill-conditioned. A classical way of regularizing the problem is provided by spectral windowing, which amounts to take as approximate solution a filtered


version of the singular-system expansion:

.~ W~,k f - Z - ~ k (g' Uk)a Vk (6)

k

where the coefficients W~,k depend on a real positive number a, called the regularization parameter. They should lie between 0 and 1 and satisfy the following condition

r~m w . k = l (7) ~-~0+

to ensure that f converges to f t when a ---, 0 +. A simple choice for the filter coefficients corresponds to sharp truncation:

1 if k _ [l/a] (8) W~,k= 0 i f k > [ 1 / a ]

where [l/a] denotes the largest integer contained in 1/a. This method is also known as numerical filtering. The so-called regularized solution lives then in a finite-dimensional subspace and the number of terms in the truncated expansion (6) represents the number of independent "pieces of information" about the object which can be reliably (i.e. stably) retrieved from the data. Therefore, it is sometimes referred to as the number of degrees of freedom. There are many other possible choices for the filters Wa,k (see e.g. [7], [9]), such as a triangular filter, discrete equivalents of classical spectral windows (e.g. Hanning's window) or the classical Tikhonov filter:

~ (9) Wa,k -= 2 a k + a

To get a proper regularization method one has still to give a rule for choosing the regularization parameter a according to the noise level. Roughly speaking, one has to ensure the convergence of the regularized solution to the true f t when the noise vanishes. In practice, one needs a proper control of noise propagation, namely a good trade-off between fidelity (goodness of fit) and stability of the solution. Many different methods have been proposed for the choice of the regularization parameter (see e. g. [7], [9]). Let us just mention here that a good "rule of thumb" is to take a in (9) of the order of the "noise-to-signal" ratio. Such an empirical choice can be fully justified in the framework of stochastic regularization methods (see e. g. [7]). When in a practical application the number of significant terms in the expansion (6) is not too high, the computation of a regularized solution can be done simply by implementing formula (6) numerically. This requires of course the computation of the singular system, which may be quite time-consuming. For a specific application, however, (i.e. for a given operator A), this computation is done once for all and the implementation of (6) with a prestored singular system results in very fast reconstruction algorithms. Numerical approximations of the singular system are obtained by collocation methods. Let us still observe that in experiments, the detection scheme imposes a natural discretization in data space (the collocation points being just the measurement points) whereas the discretization in the solution space is more arbitrary and may be done only at the last stage, just for numerical purposes.

SVD for Linear Inverse Problems 345

3 T H E F I N I T E L A P L A C E T R A N S F O R M

We define the finite Laplace transform as follows

(L f ) (p ) = e - ' t f ( t ) dt ; c < p < d . (10)

If a > 0 (and/or c > 0), then s L2(a, b) --. L2(c, d) is compact and injective. As already recalled in Section 2, the inverse operator 12 -1 can be given in terms of the singular system of s {ak; uk, vk).

Some of the properties of s are similar to those of the finite Fourier transform defined by

e- i~ t f ( t ) dt ; -f~ < w < +12. ( i i ) (.T'f)(w) - -T

This operator is also compact and its singular functions are related in a simple way to the so-called linear prolate spheroidal wave functions (LPSWF) considered by Slepian and Pollack [11]. Indeed, after trivial rescaling, both operators .T'*.T" and .T'.T'* coincide with the operator studied in [11], namely

- y ) ] (~c f ) ( x ) = 1 r ( x - y) f ( y ) dy ; - 1 < x < +1 (12)

where c = 12T. Its eigenfunctions, the LPSWF, are denoted by Ck(C,X) and the corresponding eigenvalues by Ak(c). Hence, the singular values of the finite Fourier transform ~" are ~/~k(C). As shown in [11], the eigenfunctions Ck(C, x) are also the eigenfunctions of the following second-order linear self-adjoint differential operator:

(Dcf)(x) = -[(1 - x2)f ' (x)] ' + c2x2 f ( x ) . (13)

As emphasized in [10] and [13], this is a remarkable fact, which allows to establish a series of nice properties of the singular functions of 9 v, as for example, that they have exactly k zeroes in their respective intervals of definition. Such results have been extended to the case where (U f ) (w) is given only in a discrete set of equidistant points inside the interval [-12, +12]. The set of corresponding eigenfunctions are then called the discrete linear prolate spheroidal functions (DLPSWF) [12] (for more details about the singular system of .T" and its link with the LPSWF, see e. g. [7] and for a review on the prolate functions, see [13]).

Similar results hold true also for the finite Laplace transform when c = 0 and d = oc. Then, if a > 0, Z: is a compact operator from L2(a, b)into L2(0, ce). It is easy to show [1] that its singular values depend only on the ratio 7 = b/a. It is therefore convenient, by changing variables, to transform the interval [a, b] into [1, 7] and to study the finite Laplace transform in the following standardized form

/; (s = e -p t f ( t ) dt ; 0 _< p < c~ . (14)

The operator s = s163 is a "finite Stieltjes transform" given by

^ Jl ~ f ( s ) (s = ds ; 1 < t < 7 . (15) t + s - -


A It has been shown [3] that s commutes with the following differential operator

(D.~f)($) - - [ ( t 2 - 1 ) ( " / 2 - t 2 ) f t ( t ) ] ! + 2 ( t 2 - 1)f(t) . (16)

The appropriate self-adjoint extension of this operator is obtained by requiring that the functions in the domain of D.~ are bounded at the boundary points t = 1 and t = 3' (notice

A that the differential equation (D~f)(z) = # f (z) has five regular singular points in the complex plane, one being the point at infinity). In the limit 7 ~ 1, the operator (16) reduces, after a change of variable, to the Legendre operator [3]. From the commutation property, it follows ^ that any singular function vk(t) is simultaneously an eigenfunction of s = s163 and of D~:

It is then possible to prove that [3] all the eigenvalues a~ have multiplicity 1 and that the ordering of the vk corresponding to increasing values of #k coincides with their ordering

2 As a corollary, vk(t) has exactly k zeroes inside corresponding to decreasing values of o" k . the interval [1, 7] (the boundary points cannot be zeroes); moreover, the usual interlacing property of the zeroes of the eigenfunctions of a differential operator holds true for the singular functions Vk. Finally, it is seen that the vk are alternately "even" and "odd" functions in the following sense:

,,~(~) = (_ l )k t ~k(t). ( i s )

In particular, when k is odd, then t = ~ is a zero of vk(t). As concerns the singular functions uk(p) it is possible to prove that they are eigenfunctions of the following fourth- order differential operator

( ~ g ) ( p ) = [p~g"(p)]"- (1 + ~) [#9' (p)] ' + (~p~ - 2)g(p) (19)

which commutes with the operator s = s The above properties have been exploited in [4] to compute the singular system of the finite Laplace transform. The case of a solution with infinite support (0, oo) and data in a finite interval (c, d), c > 0, can be treated in a similar way, by exchanging the operators s and s

4 A N I M A G I N G O P E R A T O R IN C O N F O C A L M I C R O S C O P Y

In a very simplified case, the imaging properties of a confocal microscope are described by the following integral operator

(Af)(x) = f / 5 sinc(x - y) sinc(y) f(y) dy ; x e T~ (20)

where sinc(x) = s in (rx ) / ( rx ) . Notice that the object is first multiplied by the sinc-like light pattern of the illumination lens and then convolved by the sinc-like impulse response of the imaging lens. The operator A is compact in L2(T~). Its nun space N(A) is not trivial and can be completely characterized as follows. Let PW2~r(Tt) be the Paley-Wiener space of the L2-functions whose Fourier transform is zero outside the interval [ - 2 r , 2r]. This is a dosed subspace of L2(7~) whose elements are entire functions which can be represented

347

by means of the Whittaker-Shannon expansion

Let us denote now by PW(2~ the subspace of PW2,~(n) containing all the functions r~TXT(+) vanishing at the sampling points x = 0 and x = 4-).,4-3,+~.,...,1 5 and by r,,2~. (Td,) the

subspace of all the functions that are zero at the sampling points x = 4-1, 4-2, 4-3,.... We have PW2,~(Tz)= PW(~ PW(+)(Tz). Then the following result holds true [2]

N(A) = PWi~(7"v,.) (~PW~~162 ; N(A) • =/-'~,':~.(+) (TO). (22)

The functions belonging to N(A) "t are represented by the following sampling expansion +co

f(x) = ~ f(xm) sinc[2(x- Xm)] (23) m - - - c r

with x0 = 0, Xm -- sgn(m) ( I m l - �89 m -- 4-1, 4-2,..., where sgn(m) denotes the sign of m. The null space of the adjoint operator is characterized as follows

N(A*) = PW~(7"r ; N(A*) l = PW,~(7"r . (24)

Hence the singular functions vk(x) belong to PW(+)(Td.) whereas the singular functions Uk(X) belong to PW,~(Tt). h very nice result proved by Gori and Guattari [8] is that these singular functions, as well as the corresponding singular values, have very simple analytic expressions. When k is odd (k = 21 + 1; l = 0, 1, 2, . . . ) , we have

l = o, ~ , 2 , . . . (25) a2z+l = r ( 2 l + 1) '

u2l+z(x) = ~ sinc x 2 2 '

v2,+l(X)=sinc[2(x 21+12 ) ] - s i n c [ 2 ( x + 2 1 + 1 2 ) ] " (27)

On the other hand, when k is even (k = 2l; l = 0, 1,2, . . . ) , denoting by ~t the positive solutions of the transcendental equation t a n ( ~ / 2 ) = 2//3 we have

v~ ~:~ = ~ - , (28)

u 2 / ( x ) = N i v ~ 2 [ s i n c ( x - 2 ~ - ) - s i n c ( x + 2 ~ ) ] ' (29)

1 ~z(~ ) = - - si~c(~) u~ (~ ) (a0)

O'2l

where N~ = (4 +/3~)/~r(8 +/3~).

Another nice analytic result concerning the integral operator (20) is the following. Con- . . . . (+)

sider the restriction of A to the subspace /~w~ (7~), i.e. to the subspace of the functions represented by the sampling expansion (23). This restriction is isomorphic to the following


infinite-dimensional matrix 1

Ano = ~ 6n0, n = 0, 4-1, : t :2, . . . (31)

1 ( -1 ) "+~ A,,m = 2r 2 x m ( n - x,~) ' n = 0,4-1,4-2 , . . . ; m = 4 -1 ,+2 , . . . (32)

with xm as defined above. Then this matrix is invertible and its inverse is given by [6]

(A-l)0,, = 2 ( - 1 ) n, n = 0, 4-1,:t=2,... (33)

(A_1)m., (_l)n+ I 2n - , n = 0 , 4 - 1 , • m = 4 - 1 , 4 - 2 , . . . (34) n - x m

This analytic result has provided us with a useful insight for analyzing related inverse problem in confocal microscopy and for estimating the resolution enhancement one could obtain with a multi-detector scheme (see [2], [5] and [6]).

References

[1] M. Bertero, P. Boccacci and E. R. Pike. On the recovery and resolution of exponential relaxation rates from experimental data. Proc. R. Soc. Zond. A 383, pp 15-29, 1982.

[2] M. Bertero, C. De Mol, E. R. Pike and J. G. Walker. Resolution in diffraction-limited imaging, a singular value analysis. IV. Optica Acta 31, pp 923-946, 1984.

[3] M. Bertero and F. A. Grfinbaum. Commuting differential operators for the finite Laplace transform. Inverse Problems 1, pp 181-192, 1985.

[4] M. Bertero, F. A. Grfinbaum and L. Rebolia. Spectral properties of a differential operator related to the inversion of the finite Laplace transform. Inverse Problems 2, pp 131-139, 1986.

[5] M. Bertero, P. Brianzi and E. R. Pike. Superresolution in confocal scanning microscopy. Inverse Problems 3, pp 195-212, 1987.

[6] M. Bertero, C. De Mol and E. R. Pike. Analytic inversion formula for confocal scanning microscopy. J. Opt. Soc. Am. A 4, pp 1748-1750, 1987.

[7] M. Bertero. Linear inverse and ill-posed problems. In : P. W. Hawkes (Ed.), Advances in Electronics and Electron Physics 75, Academic Press, New York, pp 1-120, 1989.

[8] F. Gori and G. Guattari. Signal restoration for linear systems with weighted inputs. Singular value analysis for two cases of low-pass filtering. Inverse Problems 1, pp 67-85, 1985.

[9] C. W. Groetsch. The theory of Tikhonov regularization for Fredholm equations of the first kind. Pitman, Boston, 1984.

[10] F. A. Griinbaum. Some mathematical problems motivated by medical imaging. In : G. Talenti (Ed.), Inverse Problems, Lecture Notes in Mathematics vol. 1225, Springer-Verlag, Berlin, pp 113-141, 1986.

[11] D. Slepian and H. O. Pollack. Prolate spheroidal wave functions, Fourier analysis and uncertainty, I. Bell Syst. Tech. J. 40, pp 43-64, 1961.

[12] D. Slepian. Prolate spheroidal wave functions, Fourier analysis and uncertainty, V: The discrete case. Bell Syst. Tech. J. 57, pp 1371-1430, 1978.

[13] D. Slepian. Some comments on Fourier analysis, uncertainty and modeling. SIAM Review 25, pp 379-393, 1983.


349

F I T T I N G O F C I R C L E S A N D E L L I P S E S L E A S T S Q U A R E S S O L U T I O N

W. GANDER, R. STREBEL Institut fSr Wissenschaftliches Rechnen EidgenSssische Technische Hochschule CH-8092 Ziirich Switzerland [email protected], [email protected]

G.H. GOLUB Computer Science Departement Stanford University Stanford, California 94305, U.S.A. golub @sccm. stanford, edu

ABSTRACT. Fitting ellipses to given points in the plane is a problem that arises in many application areas, e.g. computer graphics, coordinate metrology, petroleum engineering, statistics. In the past, algorithms have been given which fit circles and ellipses in some least squares sense without minimizing the geometric distance to the given points. In this article, we first present algorithms that compute the ellipse, for which the sum of the squares of the distances to the given points is minimal. Note that the solution of this non-linear least squares problem is generally expensive. Thus, in the second part, we give an overview of linear least squares solutions which minimize the distance in some algebraic sense. Given only a few points, we can see that the geometric solution often differs significantly from algebraic solutions. Third, we refine the algebraic method by iteratively solving weighted linear least squares. A criterion based on the singular value decomposition is shown to be essential for the quality of the approximation to the exact geometric solution.

KEYWORDS. Least squares, curve fitting, singular value decomposition.

1 PRELIMINARIES

Ellipses may be represented in algebraic form

F ( x ) = x W A x + b W x + c - - 0 (1)

350 W. Gander et al.

with A symmetric and positive definite. Alternatively, ellipses may be represented in parametric form

x ( r = Q(~) ~in + ~ for r e [0,2~[, (2)

where z is the center and a, b are the axes of the ellipse. The orthogonal matrix Q rotates the figure by some angle ~.

Ellipses, for which the sum of the squares of the distances to the given points is minimal will be referred to as "best fit" or "geometric fit", and the algorithms will be called "geometric". Determining the parameters of the algebraic equation F(x) = 0 in the least squares sense will be denoted by "algebraic fit" and the algorithms will be called "algebraic". We will further look at "iterative algebraic" solutions, which determine a sequence (uk)k=0..r162 of parameter vectors by solving weighted or constrained algebraic equations F(x; uk) = 0 in the least squares sense.

We define the following notation: The 2-norm ]1" ]]2 of vectors and matrices will simply be denoted by ]l" II.

2 G E O M E T R I C S O L U T I O N

Given points (xi)i=l...m, and representing the ellipse by some parameter vector u, we then seek to solve

m

d~(u) ~ = n~n , (3) i--1

where dl is the geometric distance of xi from the ellipse. We may solve this problem using either the parametric or algebraic form.

2.1 PARAMETRIC FORM

Using the parametric form (2), di may be expressed

di = ~in [Ixi - x(r u)]l,

and thus the minimization problem (3) may be written

I l x i - x( r u)ll ~ - rain . i=1 r ...r

This is equivalent to solving the nonlinear least squares problem

g~ = x~ - x(r u) ~ 0 for i = 1 . . . m . (4)

Thus we have 2m nonlinear equations for m + 5 unknowns (r . . .r zl, z2). This problem may be solved using the well-known Gauss-Newton method [6], which solves a

Fitting of Circles and Ellipses 351

linear least squares problem with the Jacobian of the nonlinear problem in each iteration step.

We may exploit the special structure of the Jacobian for (4) by multiplication from the left with the block diagonal matrix -d iag(Q(a) ) , and then reordering the equations (gll ,gl2. . .gml,gm2) to (gll . . .gml,gl2. . .gin2). The coefficient matrix J then looks

- a sin r -b sin r cos r cos a sin a �9 � 9 . . . .

�9 . .

- a sin Cm -b sin Cm cos r cos a sin a b cos r a COS r sin r - sin a cos a "

�9 . .

" ' . . . .

b cos r a cos r sin r - sin a cos a

We may efficiently compute the Q R decomposition of ] by m Givens rotations and a Q R decomposition of the last 5 columns.

2.2 ALGEBRAIC FORM

Using the algebraic form (1), the problem may be stated as follows

m

I l d i l l 2 - min where i--1

F(Xi + di; u) = 0 for i = 1 . . . m .

(5)

(6)

The vector u contains the parameters A, b and c, and di is the distance vector of xi to the ellipse. The Orthogonal Distance Regression (odr) algorithm provides an elegant solution to this problem using Levenberg-Marquardt. See [2] for a description of the algorithm, and [1] for an implementation in FORTRAN�9 Like in (4), the number of unknowns involved is the number of model parameters plus the number of data points. Although odr is generally applicable, the algorithm has a per step computational effort similar to the method applied in the parametric form.

2.3 CONCLUSION

Several methods for the solution of (3) exist, including for instance the Newton-method applied to (4). Unfortunately, all these algorithms are prohibitively expensive compared to simple algebraic solutions. We will examine in the next sections if we get sufficiently close to the geometric solution with algebraic fits.

3 A L G E B R A I C S O L U T I O N

Let us consider the algebraic representation (1) with the parameter vector

u = (a11, a12, a22, bi, b2, r149 (7)


To fit an ellipse, we need to compute u from given points (xi)i=l...m. We obtain a linear system of equations

Z21 2ZllX12 X22 Zll Z12 1) B u = �9 : : : : �9 u ~ 0 .

2 2XmlXm 2 2 1 X m l T'm2 ;r, m l Xm2

To obtain a non-trivial solution, we have to impose some constraint on u. For the constraint Ilull = 1, the solution vector ~ is the right singular vector associated with the smallest singular value of B.

The disadvantage of the constraint Ilull = 1 is its non-invariance for Euclidian coordinate transformations. For this reason Bookstein [3] recommended using the constraint

A12 4" A22 : a121 4- 2a122 + a22 = 1, (8)

where ~I and )~2 are the eigenvalues of A. While [3] describes a solution based on eigenvalue decomposition, we may solve the same problem more efficiently and accurately with a singular value decomposition as described in [5]. If we define vectors

V = ( b l , b 2 , c) w

W = ( a l l , V / ' 2 a 1 2 , a 2 2 ) T ,

the constraint (8) may be writ ten Ilwll- 1. The reordered system then looks

(xll x121 XllXl2 x 2) ( : ) : ) : . . . . . . ' Xml Xm2 1 X21 V/2XmlXm2 X22

~ 0 .

The QR decomposition of S leads to the equivalent system

which may be solved in following steps. First solve for w

R22 w ~ 0 where [[w[[ = 1

using the singular value decomposition of R22 , and then

v = - R l l -1R12w.

Note that the problem

where,,u,,- 1


is equivalent to the generalized total least squares problem finding a m a t r i x $2 such that

rank ( Sl $2 ) <: 5

[l(S1 $2) -- (S l S2)II = inf r~k (S~ ~2)<5

II(Sl $2 ) - (S~ S2)ll,

In other words, find a best rank 5 approximation to S that leaves $1 fixed. A description of this problem may be found in [7].

Since A12 + A22 # 0 for ellipses, hyperbolas and parabolas, the Bookstein constraint (8)is appropriate to fit any of these. But all we need is an invariant I ~ 0 for ellipses--and one of them is A1 -t- A2. Thus we may invariantly fit an ellipse with the constraint

AI + A2 -" a l l ~" a22 = 1,

which results in the astonishingly simple linear least squares problem

(2XllXl Xll x121)(Xl l) �9 : : : " u ~ " .

2 2 1 '2 2XmlXm2 Xm2 -- Xml Xml Xm2 --Xml

To demonstrate the influence of different coordinate systems, we have computed the ellipse fit for the following set of points

x 1 2 5 7 9 3 6 8 y 7 6 8 7 5 7 2 4 '

(9)

and for the same points shifted by ( -5 , 0). See figure 1 for the fitted ellipses.

CONCLUSION

Non-trivial solutions to the problem Bu ~ 0 depend on the additional constraint we impose on u. Choosing c = I in (1) excludes conics through the center, whereas ]lull = 1 makes the solution highly coordinate-system dependent. Although we might mitigate this deficiencies by standardizing the data (e.g. translation to center-of-mass), there are few reasons not to use constraints with the desired invariance properties.

4 I T E R A T I V E A L G E B R A I C S O L U T I O N

Especially for eccentric ellipses, simple algebraic solutions do not approximate well the geometric solution. If we define

Q ( x ) = x TAx + b Tx + c,

then a simple algebraic method minimizes Q for the given points in the least squares sense. The geometric meaning of Q is the following: Let r(x) be the distance from the center z to


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . ' . . . . . . . . . . . . . . . . . . . . . . . . . . . ' . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10 . . . . . . . . . . . . . . . . . . . . ~"~" . . . . . . . . ! . . . . . . . . . . . . . . . . "~'~"" . . . . . . . . . . i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

9 . . . . . . . . . . . . . . . ~'~ i " ~ ... . i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ~." . . . . " , , i

5 ................................ ! ........... o ..... ~ 5 f .......... i ~~ i i i i i i i i i i i i i i i i i i i i i i i i : i l t , l ~ ! ' . o ..... ! ................................ !! ' ................................ '. ......... 'T :~ ' . . . . . . . . . ~ i ................................ i

, ............... : o - - ~ - ~ . - : " , L iiiii i i ~ ~ ' : : .... i ................................ i

, 'I i i i ........ : 1 1 ............................... ................................ i ................................ i o o . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

i i " i - 5 ~ 0 5 10 -5 0 5 10

C o n s t r a i n t I1~11 = 1 Constraint A~ + A~ = 1 Constraint A 1 + A 2 = 1

Figure 1: Influence of coordinate transformations

x, and determine piby intersecting the ray from z to xi and the ellipse. Then, as pointed out in [3]

Q ( x i ) = a((r(xi)/r(pi)) 2 - 1) ( 1 0 )

,,, 2 r(xi) - r (pi) if xi ~ Pi (11) ~(p,) '

for some constant ~. This explains why the simple algebraic solution tends to neglect points far from the center. If we prefer to approximate the geometric solution, we may solve the weighted problem with weights proportional to

wi = d ( x i ) / Q ( x i ) , (12)

where d(x) is the geometric distance of x from the currently est imated ellipse. Since the computat ion of d(x) is costly, we will only approximate (12). Two possibilities are

~, = ~(x~) (13)

wi = r(xi) s in0 i , (14)

where 0i approximates the angle enclosed by the line from z to xi and the ellipse. Sampson describes in [8] the more general weighting scheme

~ = 1 / l l V Q ( x i ) l l . ( 1 5 )

Figure 2 shows the results for two weighting schemes applied to the algebraic method with constraint A1 + A2 = 1. The data points are given by

xi = a cos r + cos 2 r

yi = b s i n r 1 6 2 , where r E {kTr/6}k=O..al.


Note that the simple algebraic solution is close to the other solutions in terms of the relative distance--as the figure shows, but they differ significantly if we look at absolute distances.

5 -

- 2 -

- 3 -

- 4 -

o

o

-% 4'o 3'o 2'o ,'o o ~'o 2'o 3'o 4'o ~'o

Geometric solution Simple algebraic solution Iterative solution, wi = 1/IIVQI] Iterative solution, wi = r sin 0

Figure 2: Iterative algorithms after 4 steps, constraint )~1 "~- )~2 = 1

Even if the weights wi are chosen to fulfill (12) and thus represent the current geometric distances, the solution of the iterative algorithm will not generally minimize the geometric distance. To show this, we restate the problem for the algebraic method using the constraint ]lull = 1. Let G(x) be the coefficient matrix for the weighted problem--where the weights are determined by the parameter vector x. Then the geometric algorithm solves

I[G(x)x[]2 = min where ]lxll = 1.

The iterative algorithm determines a sequence (yi), where Yk+l is the solution of

I]G(yk)Yll2= rain where IlYll = 1. (16)

We may shortly examine the solution of (16) for small perturbations to G. Denote with a l , . . . , an the singular values of G in descending order. Let C1 = G + dG be the perturbed matrix, and denote with v, ~" the respective solutions of (16). Straightforward analysis (see [4]) then shows that following holds

/ 2[]dGil - vii < ~/

V O'n_ 1 - - O" n


If an-1 '~ an, the solution is poorly determined, and the iterative algorithm may converge slowly or even diverge.

CONCLUSION

Iterative algebraic algorithms find only an approximation of the optimal solution (in terms of geometric distance). The difference can be large especially if only few points are given or ff the optimum is poorly determined. On the other hand, iterative algebraic methods often give sufficiently good results after 2 or 3 steps. Thus, we are faster this way by a factor of about 10-20 than the exact geometric algorithm.

R e f e r e n c e s

[1] Paul T. Boggs, Richard H. Byrd, Janet E. Rogers, and Robert B. Schnabel. User's Ref- erence Guide for ODRPA CK Version 2.01--Software for Weighted Orthogonal Distance Regression. Gaithersburg, June 1992.

[2] Paul T. Boggs, Richard H. Byrd, and Robert B. Schnabel. A stable and efficient algorithm for nonlinear orthogonal distance regression. SIAM Journal Sci. and Stat. Computing, 8(6):1052-1078, November 1987.

[3] Fred L. Bookstein. Fitting conic sections to scattered data. Computer Graphics and Image Processing, 9:56-71, 1979.

[4] Walter Gander, Gene H. Golub, and Rolf Strebel. Fitting of circles and ellipses-- least squares solution. Technical Report 217, Institut fiir Wissenschaftliches Rechnen, ETH Ziirich, June 1994. Available via anonymous ftp from f t p . in f . e thz. ch as doc/ t e c h - r ep o r t s / 1 9 9 4 / 2 1 7 . ps.

[5] Walter Gander and Ji~f H~ebi~ek, editors. Solving Problems in Scientific Computing Using Maple and Matlab, chapter 6 (Walter Gander and Urs von Matt: Some Least Squares Problems), pages 69-87. Springer-Verlag, Berlin, Germany / Heidelberg, Ger- many / London, UK / etc., 1993.

[6] Philip E. Gill, Walter Murray, and Margaret H. Wright. Practical Optimization. Aca- demic Press, New York, NY, USA, 1981.

[7] Gene H. Golub, Alan Hoffmann, and G. W. Stewart. A generalization of the Eckart- Young-Mirsky matrix approximation theorem. Linear Algebra and its Appl., 88/89:317- 327, 1987.

[8] Paul D. Sampson. Fitting conic sections to "very scattered" data: an iterative refinement of the bookstein algorithm. Technical report, Department of Statistics, The University of Chicago, March 1980.


357

THE USE OF SVD FOR THE STUDY OF MULTIVARIATE NOISE AND VIBRATION PROBLEMS

D. OTI'E LMS International Interleuvenlaan 68 B-3001 Leuven, Belgium [email protected]

ABSTRACT. This paper deals with a global and consistent application of singular value analysis methodologies for multivariate noise and vibration problems of mechanical structures. By attributing a generic and consistent physical significance to the parameters forthcoming from the SVD of operating response data or frequency response functions, singular value analysis is developed as a robust diagnostic approach in experimental structural dynamics engineering. A global SVD method, referred to as principal response analysis, for multivariate operating response analysis leads to a number of diagnostic tools, such as virtual coherence analysis and principal operating field shape analysis, that are both altemative and complementary to existing approaches. A common perspective on the use of SVD for nonparametric impedance modelling, valid in the medium frequency range and based on multireference frequency response functions is explored. Using identical principles as in operating response analysis, an alternative for modeshapes, referred to as principal field shapes is developed. Case studies on the dynamic behaviour of a sports car and a propeller aircraft illustrate the real world applicability of the discussed methodologies.

KEYWORDS. Principal component analysis, data reduction, non-parametric modelling, structural dynamics.

1. INTRODUCTION

Modem methodologies in experimental structural dynamics engineering essentially rely on the analysis of multivariate data sequences. The involved multivariate analysis procedures primarily aim at simplification (data reduction) and explanation (modelling) of the dynamic phenomena, observed through these data sets. Out of a wide variety of multivariate analysis approaches, singular value analysis, based on the SVD of the data sequences is emerging as the foremost global and robust tool. In experimental structural dynamics however, singular value analysis still is only occasionally and certainly not consistently introduced.

This paper deals with the application of singular value analysis methodologies for multivariate noise and vibration problems of mechanical structures. Reaching far beyond already existing applications of the mere SVD in structure dynamics and/or vibroacoustics, which were essentially exploiting its mere algebraic properties (least squares solutions), the focus in this paper is placed on the aspect analysis.

358 D. Otte

By attributing a generic and consistent physical significance to the parameters forthcoming from the SVD of operating response data or frequency response functions, singular value analysis is developed as a robust diagnostic tool in experimental structural dynamics engineering. The analysis can aim at a recognition of pattems in the data set, a least squares based data reduction and/or a non-parametric modelling approach. The paper consists of two parts : a first part is dealing with singular value analysis as a diagnostic tool for operating response analysis in multi-source environments, the second part introduces the principal field shape approach, for medium-frequency system analysis of complex vibroacoustic systems, based on frequency response function (b-Rb3 measurements.

Both parts am implicitly linked to each other through a consistent physical significance. Both multivariate operating response analysis and multivariate FRF analysis can then be considered in a generic framework, also enhancing the interpretation and insight in the structure's dynamic behaviour.

2. PRELIMINARY CONCEPTS

In the most general sense, a mechanical structure can be considered as a complex vibroacoustic system, which is an assembly of coupled substructures, cavities and transmission paths. Such a system is characterised by multiple vibration modes, acoustic cavity modes and energy transmission paths. Under operating conditions, the structure is dynamically excited by a number of dynamic loads. The operating dynamic loads generally consist of multiple vibration and acoustic sources, kinematic forces and other extemal effects. They are referred to as the inputs of the vibroacoustic system. The dynamic characteristics of the structure define what its response to the inputs is like. The outputs, or the responses of the (continuous) system to the dynamic loads can be expressed in a theoretically infinite number of degrees of freedom (DOFs) and in any measurable parameter such as acceleration, strain, sound pressure, etc..

The aim of applying the SVD is then to detect a dominant structure in such sets of multivariate observations, grouped as vector sequences. The observed vector sequences are generally analysed for their complexity. Related problems are the estimation of the rank, as well as the definition of the corresponding subspace.

In case stationary loading conditions are considered, the dynamic behaviour of a mechanical structure is most likely represented in the frequency domain. For the general situation of the multiple input multiple output model of a system, the experimental definition of the mechanical system may be generated from a frequency response function (FRF) matrix. The size of this matrix is a function of the locations where forces are applied to the mechanical system and of the locations where responses or outputs are measured. These locations are generally referred to as measurement DOFs.

By a genetic and simplified notation, the multiple input multiple output relation in the frequency domain can be expressed as follows"

{X(f)} = [H(f)]{F(f)} (1)

with f : frequency, {X(f)} an n-dimensional column vector consisting of a set of observed response spectra, {F(f)} an s-dimensional vector consisting of the spectra of the acting loads [H(f)], the (n,s) FRF matrix that expresses the linear magnitude/phase relationship between the observed response DOFs and the DOFs where a load is exercised.

Multivariate Noise and Vibration Problems 359

3. SVD AND OPERATING RESPONSE ANALYSIS

3.1 THEORY

Operating response analysis refers to the analysis of the system's vibroacoustic response(s), observed under operating conditions. Only data that make up the left term of equation (1) are then obtained. Consequently, these data reflect the combined effects of the system characteristics and the loading conditions

Suppose n signals xi (t) are measured simultaneously. The observed spectra can be ordered in a

series of non-dimensional column vectors"

[X(f)]=[...{Xk(f)}.. ] (k=l,...,m) (2)

Generally, the number of averages, m, will be much larger than the number of signals. Due to the specific acquisition methodologies and computer memory constraints, the data are, in case of stationarity, made available for postprocessing in the form of auto- and crosspower spectra, rather than as individual sampled spectra, even if this results in a certain loss of accuracy for some of the multivariate analysis techniques. The much more manageable matrix of cross-spectral densities IS= (f)] can then be constructed. This matrix is Hermitian and non-negative semi-definite.

(3)

The application of singular value analysis, in this context to be seen as principal component analysis, to sets of spectra may be referred to as principal spectral analysis. Principal spectral analysis is then essentially based on the eigenvalue decomposition of the crosspower matrix.

IS= (f)]= [U(f)Isx,x,(f)Xu(f)] n (4)

[Sx,~,(f)] is a diagonal matrix, containing the eigenvalues of [S~x(f)] in descending order.

These eigenvalues can be considered as the autopower spectra Si,i,(f ) of the principal spectra

[X'(f)], which are mutually totally uncorrelated (crosspower spectra are zero).

The principal autopower spectra can be sorted in descending order and plotted as a function of frequency to obtain a graphical representation of the rank r of [S=(f)], which indicates the

number of incoherent phenomena, observed in the signal set at every frequency. Such a plot also indicates the frequencies where the response is largest.

Principal autopower spectra therefore may give direct and generic information on the correlation pattems that exist between the signals. Indeed, one diagram, showing n functions, now represents

the same information as what can be obtained through interpreting n{,n+l)/~ 2 diagrams with

coherence functions. Since the number of principal spectra with a significant amplitude level determines the number of dominant independent phenomena present in the signal data set, the study of a complex noise and vibration problem can be reduced to the study of independent, uncorrelated signal sources.

Since, for a large number of measurement signals, the full matrix of possible interdependences is not always readily measurable, and not always needed in fact, a distinction can be made between reference signals and target signals. Reference signals are measured at locations,

360 D. Otte

representative for the phenomena that build up the structure's dynamic behaviour. Target locations on the other hand are defined to describe the dynamic response as it is observed e.g. by the operator/driver/passenger of the structure/vehicle under test. Coherence analysis then typically involves the assessment of the causal relationship between the reference signal spectra, followed by an estimation of the relation between the references and the target location(s). An auxiliary set of "slave" spectra may be useful in getting more detailed understanding in this relation between target and reference measurements, e.g. in the form of vibration transmission paths. When these slave spectra represent geometric locations within a sufficient spatial resolution, operating field shapes can be estimated.

In the current context, operating field shapes can be operating deflection shapes or operating acoustic sound field shapes. Operating deflection shapes designate the periodic motion pattern by which a structure vibrates at a specific frequency, under specific stationary operating conditions. They are often referred to as running modes and can be expressed in terms of displacement. Sound field shapes denote the frequency-dependent pattern of sound waves within a cavity and are based on sound pressure spectra. On a graphic display, such a field shape can be visualised by animating the amplitude and phase relationship between the different DOFs as a periodic motion or sound pressure pattem.

The data are acquired in a number of separate cross-spectral matrices, with each matrix containing spectra from the same fixed references (x), and spectra from a subset of the slave DOFs

In a subsequent phase, the crosspower matrix [S,x (f)] referring to all slave DOFs can then be (y).

built up by these matrices. Care should be taken to assure the repeatibility and consistency of the reference spectra throughout the measurement passes. In practice, each measurement pass may also include a fixed set of target transducers. They may be used to assess the stationarity of the operating conditions. The peaks in the target spectra also indicate the frequencies where the animation of the operating field shapes will be of particular interest.

An eigenvalue decomposition of [Sx~(f)] leads to the assessment of a set of r significant

principal references. The virtual crosspower spectra with the slave spectra are then computed:

Is.. o')]- [S,x (5) However, it makes more sense to compute the referenced and properly scaled crosspower. The

principal field shapes are then given by (j' = 1 . . . . . r).

{Syj, (f)} (6) {u

3.2 APPLICATION

A case study was carried out, dealing with a structure-bome road noise problem in a Porsche 928, situated in the 220-250 Hz band.

The objective was to obtain a detailed -spatial-insight into the dynamic response of the rear suspension, related to the wheel inputs. This insight Was anticipated to be obtained from (principal) operating deflection shape analysis.


The testing was performed in an anechoic room. The car's rear wheels were driven by a roller bench, while the engine was turned off. In order to reduce the roller bench' noise, a speed of 25 km/h had to be maintained. The road tests had however already indicated an acceptable independence of the road noise contribution from the actual speed.

For the roller bench test, a set of 9 reference signals was used. Two 3-D transducers were mounted at the rear wheel suspensions. Additionally, one 3-D accelerometer was placed in the middle of the transaxle support beam. Four 'target' channels (microphones in the passenger cabin) and 21 (7 3-D) 'roving' accelerometers to characterize the rear suspension were used. A set of 11 subsequent M-channel measurements was then carded out, yielding a spatial vibration description of the rear suspension system by means of 77 locations.

A principal spectral analysis was then carded out on the set of reference spectra (Fig. 1). Although there is apparently a single coherent source, consisting of the driving roller bench, two uncorrelated phenomena are revealed in a band near 250 Hz. This leads to the conclusion that a two-dimensional resonance problem of the tire/suspension system is encountered. Identical but independent phenomena occur at each rear wheel, due to resonance phenomena in the tires.

The principal operating deflection shapes at the problem frequency (234 Hz) were then examined. When concentrating on the rear suspension support beam (Fig. 2), two independent deformation patterns are seen (Fig. 3). For reasons of crash proof design, this transaxle support beam is connected to the car body by means of screw mountings at the rear seats (wire frame nodes "stzL" and "stzR"). These connection points apparently are subject to high deformation amplitudes. Hence, important dynamic moments in the car body are induced due to the unexpected high levels for the out of plane vibrations, as revealed by the first principal deflection shape, caused by the swinging of the support beam. From decoupling testing it had been indicated earlier that near 240 Hz, the body is also acoustically sensitive to the excitation and reinforces iL

L .... _.

.... -/ ...V.. .,..., -J

15~o ~ 200 Hz 270 3S1~ .

Log

Fig. 1 : Principal component autopower spectra from roller bench test

362 D. Otte

~ iiii~ ~;ii~iiii~: iiiii i iii~ ~'

............... iiiiiiiiiiiiiiiiiiiiiii' ( .............................................

Fig. 2 : Roller bench test : geometric wire frame model of rear wheel support beam

/

Fig. 3 : Roller bench test : first and second principal operating deflection shape of support beam (front view and upper view)


4. SVD AND DYNAMIC SYSTEM IDENTIFICATION

4.1 THEORY

System identification aims at assessing a mathematical model for the description of the dynamic characteristics of a mechanical system. In the experimental approach, both inputs and outputs are observed under controlled excitation in laboratory conditions. Generally, a (sub)matrix of FRFs , relating the considered DOFs, is then estimated. This matrix may then constitute the basis for multivariate analysis procedures that derive a more global dynamic model from the m e a a u ~ FRFs, such as modal parameter estimation techniques. Experimental modal analysis techniques typically are applicable at low frequencies, where the amount of damping is limited and the structure complexity low. These techniques derive a set of modal parameters, such as resonance frequencies, mode shapes and damping factors. At higher frequencies however, a non-parametric approach must be considered.

Multireference FRF sets are forming a set of vectors, the elements of which are related to physical locations and DOFs. In fact, these vectors can be seen as field shapes caused by a unit excitation in the related reference DOF. It is the purpose of the following paragraphs to establish that through finding a structure in such a set of vectors by means of singular value analysis, insight in the system's global vibroacoustic behaviour can be obtained, even when no modal decomposition is possible.

At each spectral line, one can perform a singular value decomposition of [H(f)] �9

[-(:)]= (7)

At a specific frequency, one can assume that the response of the structure is described by a finite number of modes. A number of modes will be dominant and will describe the column space of [H(f)] . As the number of singular values different from zero, (or, in practice, larger than a

certain threshold ) denotes the rank r of the matrix, it also denotes the dimension of the column space (and of the row space) of [H(f)] . Plotting the singular values (~, as a function of frequency

gives a global idea of the dominant frequencies and the number of dominant modes (independent phenomena) at each spectral line. At each spectral line, the number of effective modes, controllable at the reference DOFs, is revealed by the number of dominant singular values r (if r<s). If one defines

In;, (:)]= (8)

then [H'v (f)] can be considered as the FRF matrix referring to a set of r principal reference

DOFs, related to the s physical input DOFs by the unitary transformation matrix [V(f)] (right

singular vectors).

364 D. Otte

4.2 APPLICATION

A fully trimmed twin propeller aircraft (Saab 340) was instrumented with accelerometers on a number of fuselage frames and with microphones in several cabin cavity sections. In total, 276 structural DOFs on the fuselage and 208 acoustical DOFs were measured.

A principal field analysis was performed, based on a 6-input broadband test (structure excitation). The shakers were positioned on one fuselage section, near the propeller plane. Fig. 4 shows the singular values of the FRF-matrix as a function of frequency, between 70 and 120 Hz. Estimating the number of dominant singular values, and hence the number of effective modes, is not obvious; neither is the determination of resonance frequencies. Fig. 5 represents the first principal deflection field and clearly shows which frame was excited. It is revealed that, due to the high damping, the fuselage response shows a forced nature. Except for their amplitude, the individual frame responses are similar, but shifted in phase, which jeopardises the success of any modal parameter fitting method.

Further analysis was focused on the first two principal deflection shapes of the excited frame and the corresponding acoustic field shapes in two neighbouring cabin sections. To visualise the acoustic fields, the microphone grids were presented as wire frames, and the pressures were visualised as vectors, orthogonal to the sections. This allows for an animated display of the response field, making it possible to interpret the amplitude/phase information as sotmd field shapes or (combinations of) acoustic modes. Fig. 6 shows the results at 102.5 Hz, where the first singular value slightly peaks. The left part represents the first principal shape, the fight part the second principal shape.

Some real modes of the fuselage frame are recognized, each one clearly coupled to specific acoustic field shapes. The second deflection shape shows, at ear height, out of phase vibrations at each side, favorizing an acoustic side-side mode. At the first field shape, these sides vibrate in phase and rather favorize longitudinal modes; the top of the frame however moves out of phase and corresponds with a coupled acoustic top-down mode.

((,~,s),,~

~.8%.

~ - ~ �9 I~.aO

Fig. 4 :Propeller aircraft ground test : singular values (70-120 Hz)


Fig.5 �9 Propeller aircraft ground test'first principal field shape (structure), 102.5 Hz

l Fig. 6a" Propeller aircraft ground test" first (L) and second (R) field shape (structure), 102.5 Hz

Fig. 6b" Propeller aircraft ground test" first (L) and second (R) field shape (cavity), 102.5 Hz

366 D. Otte

5. CONCLUSIONS

The concept of data reduction is directly linked to oriented energy repartition, to obtain insight in the spatial structure of a set of vectors that represent multivariate data. These concepts are translated to the physical significance of (operating) principal field shapes, corresponding to a set of measured response DOFs:

The research has been concentrating on the analysis of multivariate sets of experimental data, transformed into the frequency domain. Both operating response related data, such as auto- and cmsspower spectra, and system input-output related data, such as frequency response functions were processed and placed in the same framework by means of singular value analysis. The benefits of singular value analysis for operating response (spectral) analysis, as well as for dynamic system identification purposes (frequency response function analysis) are explored within a unique and consistent physical significance.

Further research in the cUrrent direction is envisaged. It is anticipated that concepts such as the generalised SVD and higher order statistics can be succesfiflly implemented in the similar framework for multivariate noise and vibration studies.

Reference

[1] D. Otte. Development and evaluations of singular value analysis methodologies for studying multivariate noise and vibration problems, Ph.D. Dissertation, Katholieke Universiteit Leuven, Dept. of Mech. Eng., 1994.


367

ON A P P L I C A T I O N S OF SVD A N D S E V D F O R N U R B S I D E N T I F I C A T I O N

W. MA, J. P. KRUTH Katholieke Universiteit Leuven Department of Mechanical Engineering Celestijnenlaan 300B, 3001 Heverlee Belgium [email protected], ac. be kruth @mech. kule u ve n. ac. be

ABSTRACT. NURBS stands for Non-Uniform Rational B-Splines, the popular mathematical tool used for computer-aided geometric design (CAGD) of free-form curves and surfaces. The geometrical shape of NURBS curves and surfaces is mainly controlled by their control points and the corresponding weights. This paper concerns the identification of NURBS curves and surfaces from discrete points measured on coordinate measuring machines (CMMs), laser scanners or other digitizing equipments. The complete process is divided into two steps. The weights of the control points are first identified with SVD (Sin- gular Value Decomposition) or SEVD (Symmetric EigenValue Decomposition) techniques through a linear homogeneous system. The control points are then solved with linear least squares fitting techniques. Both interpolation (exact) and fitting (approximate) solutions are studied. The emphasis of the present paper is on SVD and SEVD applications during the first step for the identification of the weights.

KEYWORDS. Singular value decomposition, symmetric eigenValue decomposition, NURBS curves and surfaces, fitting and interpolation, CAGD.


Non-Uniform Rational B-Spline (NURBS) curves and surfaces [3, 4, 14] are defined by the following equation

p(.) = E~=I B~(.)wivi E~=~ B~(.)~ ' (1)

where p(.) = Ix(.), y(.), z(.)] T is a point on the curve or surface, {vi = [xi, yi, zi]T}~ are the control points in 3D space, {wi}~ are the related weights of the control points, and

368 W. Ma and J.P. Kruth

{Bi(.)}~ are the so-called B-splines. {Bi(.)}~ are uniquely defined by an order k, number of control points n and a knot sequence t = {ti}'~ +k for NURBS curves, and by the orders ku and kv, number of control points n = nu x nv and knot sequences tu = { t u i } ~ +k~ and tv = {tvi}~- +k~ for NURBS surfaces. {Bi(.)}~ can be evaluated with the de Boor algorithm [1, 2]. In this context, ( . ) = (u) for NURBS curves and ( . ) = (u, v)for NURBS surfaces are location parameters locating a point on a curve or a surface.

This paper introduces how Singular Value Decomposition (SVD) and Symmetric Eigen- Value Decomposition (SEVD) 1 techniques are applied for NURBS curve and surface identification from discrete points measured on coordinate measuring machines (CMMs), laser scanners or other digitizing equipments [10].

2 A G E N E R A L A P P R O A C H F O R N U R B S I D E N T I F I C A T I O N

In order to develop linear techniques for NURBS identification, we first write equation (1) in matrix form. After moving the denominator of the right side to the left and then switching both sides, we further obtain the following non-rational equation

bT(.) x = x ( . ) . b T ( . ) . w b T ( . ) . Y = y ( . ) . b T ( . ) . w bT(.) .Z = z ( . ) . b T ( . ) . w

(2)

where

b(.) = [B1 ('), B2('), ..., B~(')] T (a)

is a collection of the B-splines used in equation (1) and

I x = [x~, x : , ..., x , ] r = [ ~ , ~ , . . . , ~ , ~ , ] r Y = [Y1, II2, ..., y , ]T = [wlYl, w2Y2, ..., w,Y,] T z = [z~, z2 , . . . , z~] r = [ ~ z ~ , ~ , . . . , ~ z ~ ] r w = [ ~ x , ~ , . . . , ~ ] r

(4)

are collections of homogeneous coordinates. Starting from equation (2) a general two-step linear approach can be further developed for NURBS curve and surface identification from the measured points.

Let m := {Pi = [~i, Yi,-~i]T)~ ' be a set of such points representing a free-form curve or surface. We assume that the location parameters u = {u i }~ for curve points and u := { u i ) ~ and v := { v i ) ~ for surface points are allocated to m by some means [9, 13] and a set of knots t or tu and tv are also fixed [13]. By introducing the discrete points m together with their location parameters u or u and v into equation (2), we obtain the following observation equations

bT(.i) x = ~ i ' b T ( ' i ) "w bT('i) "Y = Y i ' b T ( ' i ) "w bT(.i) .Z = ~ i ' bT( ' i ) "w

f o r i = 1,2, . . . ,m, (5)

1In [5], SEVD is also called the Symmetric Real Schur Decomposition.

SVD and SEVD for NURBS Identification 369

or in compact matrix form

I x iBoo xB] y O B O - Y . B _ " Z

O O B - Z . B 3 m • W

= [ o ] 4 ~ • (6)

where

X = diag{z1,-z2, ..., ~m ) Y = diag{y~, y2, ..., ym} (7)

are diagonal matrices whose diagonal elements contain the individual x, y and z coordinates of the measured points respectively, and

BI( ' I) B 2 ( ' l ) B 3 ( ' l ) . . . Bn('l) B~( .~) B ~ ( . ~ ) B ~ ( . ~ ) . . . B~( .~ )

B = . . . . (8)

B ~ ( . ~ ) B ~ ( . ~ ) B ~ ( . ~ ) . . . B ~ ( . ~ ) ~ •

is the observation matrix which is uniquely defined from the given knots and the location parameters. The unknowns of equation (5) are the homogeneous coordinates X, Y, Z and the weights w. By applying block matrix manipulation to equation (6), it has been proved in [12] that the weights can be separated from the control points as the solution of the following homogeneous equation

M,~x,~ .w,~• = [0],~• (9)

where

M = M~ + My + Mz (10)

is a symmetric and non-negative matrix [10] with

M~ = BTX2B - ( B T X B ) ( B T B ) - I ( B T X B ) My = B T y 2 B - ( B T y B ) ( B T B ) - I ( B T y B ) (11) Mz = BTZ2B - (BTZB)(BTB)-I(BTZB).

This leads to a two-step linear approach for NURBS identification from discrete points [12]. All the weights of the control points {wi}~ can be first identified from equation (9). The control points can then be solved with the identified weights as known parameters from equation (5) [7]. In the following sections, we shall apply SVD and SEVD techniques for achieving the general solutions {wi E ]R}~=~ and positive solutions {wi E ]R[wi > 0}~= 1 from equation (9).

3 W E I G H T S I D E N T I F I C A T I O N T H R O U G H SVD A N D SEVD

As equation (9) is homogeneous, we immediately have the following facts

370 W. Ma and J. P. Kruth

�9 w = 0 is obviously a solution of (9) but this is not what we want 2.

�9 Non-zero solutions exist if and only if the rank of matrix M is less than n.

�9 If w e ~ n is a solution of (9), Vc~ e JR, c~w is also a solution of (9).

In case that non-zero solutions exist, i.e., rank{M} < n, one interpolation NURBS curve or surface is guaranteed (but possibly with negative weights). Otherwise we can still expect a best non-zero solution which fits the measured points with some kind of criterion. Due to the last fact, we can simply consider the following criterion

min II M . w 112, (12) Ib,112--1

i.e., to minimize II M . w I] 2 from all w located on the unit circle in the n-dimensional Euclidean space. It is easy to prove that following equation is an equivalent of (12) for w

wTQw minR(w) = w e ~ II w 112# 0 (13) w w T w

where

Q = MTM. (14)

R(w) is called the Rayleigh quotient in linear algebra [6, 15]. Both R(w) and 7 = x/R(w) are called the rationality of the observation system (9) in [10].

The properties of R(w) are well understood. Among others, the relationship between the stationary points of R(w) and the SEVD of Q, and the minimax theorem [5, 6, 8, 10, 11] are very important. As an application, both the general solutions and solutions with positive weights of equation (9) can be easily represented as a linear combination of some eigenvectors of Q corresponding to smaller eigenvalues. Details wiU be elaborated in the following sections.

Due to the important relationship between the SVD of M and the SEVD of Q [5], one can alternatively use the SVD of M for NURBS identification as reported in [12]. Moreover, as M is a symmetric and non-negative matrix, the SEVD of M exists. Following the definition of SVD and observing the symmetric and non-negative nature of M, the SEVD of M is actually a SVD of M. Thus we can simply use the SEVD of M instead of the SEVD of Q. This fact can also be directly proved. Let

M = P D P T, (15)

be the SEVD of M, where

D = diag{dx, d2,...,dn} (16)

is a diagonal matrix whose diagonal elements are the eigenvalues of M in decreasing order with di >_ di+l >_ 0.0, and P is an orthogonal matrix whose columns Pi for i = 1, 2, ..., n are eigenvectors of M corresponding to di. The SEVD of Q is then given by

O = p D 2 p T. (lW)

2In order to avoid singularities in equation (1), positive weights are preferred in engineering applications.


4 S E T T I N G U P S O L U T I O N S

When the SEVD of M is available, we are then ready to identify the weights. Let (15) be the Symmetric EigenValue Decomposition (SEVD) of M , IB i = span{pn , Pn-~, ..., pn-i+l } be an invariant subspace of ]R ~ and Wl = [1,1, ..., 1] T E lR '~ be the uniform weights that define B-splines. Let us further define ]R n+ as a subspace of ~n such that Vw E ]R n, all of its elements are positive, i.e., wi > 0.0 for i = 1, 2, ...,n. We have then the following solutions for equation (9).

4.1 GENERAL SOLUTIONS

The general solutions of the weights w E ~n with II w 112~: 0 are given by r$

imp+l

where p is an index such that dp > dp+l . . . . . dn, {Pi}~=p+l are eigenvectors corresponding to the smallest eigenvalues {di}~=p+l of M, and {ai}n=p+l are arbitrary coefficients. The corresponding rationality of equation (18) is given by

-- ~ / R ( w ) - - I I M . w 0 I1=-- dp+l . . . . . dn, (19)

i.e., the smallest eigenvalues of M. Equation (18) gives the interpolation or exact solutions of equation (9) if the rank r of M satisfies r < n, i.e., di = 0 for i = p + 1, p + 2, ..., n. It gives the best fitting solutions if the rank r of M satisfies r = n, i.e., dn ~ 0.

4.2 SOLUTIONS FOR POSITIVE WEIGHTS

Equation (18) provides general solutions with possibly negative weights. As in practical applications, negative weights may introduce singularities and are not expected, we study practical algorithms for a set of positive weights in the eigenspace of M.

We first check if the general solutions contain positive alternatives. If there exists w E IE n-p such that all the elements of w are positive, w is then a set of positive interpolation or best fitting weights, w can be computed from the following minimization algorithm,

{ min~ II w - w~ II~ (20) subject to: wz _< wi _< w~,

where w = ~n=p+l ~iPi and w~, > wz > 0.0 are positive upper and lower bounds for the weights. The objective function of equation (20) guarantees a set of stable solutions. If the positive interpolation or best fitting solutions do not exist, one can still achieve a set of best fitting positive weights. The basic strategy is the following. We first look for the best subspace of ~ n that contains positive weights and a set of feasible solutions in this subspace. Starting from this feasible solution, we try to optimize the weights in this subspace in the sense of least value of R(w). The meaning of the best is twofold. On the one hand, the maximum rationality inside this subspace is the smallest compared with other subspaces containing positive weights. On the other hand, this subspace is the largest one compared with others for the same maximum rationality.


To find the best subspace containing positive weights and a set of feasible solutions in this subspace, let q be an index such that dq > dq+l = ... = dp. Furthermore, let l = n - q and w = pn. We try to move w into ]Et f3 JR. n+. If it is successful, following the minimax theorem [5], ]E ! is then the best subspace containing positive weights and w is a feasible solution in this subspace. Otherwise, I is incremented to include a new group of eigen~,ectors corresponding to the next larger eigenvalues, i.e. to include a new eigen subspace, and the searching process is continued till the objective is satisfied. The algorithm used to move w into ]Et N R n+ is linear programming. The positive constraints used are Wl < wi < wu for

rt i = 1, 2, ..., n with w = ~i=~-t+i/3ipi.

When we have the best subspace ]E l and a feasible solution in it, the following minimization problem can be performed to find a set of best fitting solutions in this subspace.

~y,n B2d2 R ( ~ ) = min~ z..,,.._,+, ~ -,

E;'. ._,+, ~ ~ (21) Subject to: w# < wi < w~,,

where w = ~ = r , - i + i 13iPi. The objective function of equation (21)is derived by introducing w into equation (13). According to the maximal theorem of R(w) [6, 10, 11], the rationality 7 of w is bounded by

/x",n (42d2 ~/ z.~i~_~-l+i H'i 2 i 7 =II M . w o 112 = V ~ I ~ < d~-t+x. (22)

5 SOME EXAMPLES OF NURBS IDENTIFICATION

The techniques developed for the identification of NURBS curves and surfaces from discrete points using SVD and SEVD have been applied to a number of industrial applications. They are typical examples in mechanical engineering for reverse engineering where a CAD model has to be created from free-form and hand-made physical parts [10]. The physical parts are first measured by coordinate measuring machines, laser scanners or other digitizing equipments. A CAD model is then created from the measured points of the physical models. Fig. 1 illustrates the process for transforming some measured points into a single NURBS surface. Fig. 1 (a) shows the measured points for an individual surface and Fig. 1 (b) shows the created surface model in wire frame. Fig. 2 illustrates some other examples of NURBS

XXXX ^ X X

. - ' ~ , , X . X ^ ~, X ,,, "%, X )'( X x "" �9 _ . ~ " . ~, X - - X X " ^ " X X X

a: some measured points for a single surface b: the fitted NURBS surface and fitting errors

Figure 1: NURBS surface identification from measured points

identification from measured points. In this figure, only surfaces are displayed and they are


a: a closed NURBS surface b: an open NURBS surface (3 duplicates)

c: a rotational NURBS surface d: some NURBS surfaces for a car wheel

Figure 2: Some other examples for NURBS identification

shown in shaded images. Fig. 2 (a)-(c) show respectively a general closed NURBS surface, a general open NURBS surface in three duplicates, and a rotational NURBS surface. Fig. 2 (d) shows a partial CAD model of a car wheel defined by NURBS surfaces.

6 CONCLUSIONS

This paper presents SVD and SEVD applications in computer-aided geometric design (CAGD) for the identification of NURBS curves and surfaces from discrete points. Both general solutions and positive solutions are studied. Some practical algorithms and industrial examples are presented.

Acknowledgements

This research is sponsored in part by the European Union through a Brite-Euram project under contract number BREU-CT91-0542, and by the Katholieke Universiteit Leuven through a doctoral scholarship.


References

[1] C. de Boor. On calculating with B-splines. Journal of Approximation Theory, 6, pp 50-62, 1972.

[2] M. G. Cox. The Numerical Evaluation of B-Splines. Journal of the Institute of Math- ematics and its Applications, 10, pp 134-149, 1972.

[3] G. Farin. From Conics to NURBS. IEEE Computer Graphics 8J Applications, 12, pp 78-86, September 1992.

[4] G. Farin. Curves and Surfaces for Computer-Aided Geometric Design, A Practical Guide, Second Edition. Academic Press Inc., 1990.

[5] G. H. Golub and C. F. Van Loan. Matrix Computations. The Johns Hopkins University Press, Baltimore and London, 1989.

[6] E. X. Jiang, K. M. Gao and J. K. Wu. Linear Algebra. People's Educational Publishers, Shanghai, 1978.

[7] J. P. Kruth and W. Ma. CAD Modelling of Sculptured Surfaces From Digitized Data of Coordinate Measuring Machines. Proc. of the 4th Int. Syrup. on Dimensional Metrology in Production and Quality Control. pp 371-387, Tampere, Finland, June 22-25, 1992.

[8] S. Lang. Linear Algebra. Addison-Wesley Publishing Company, Reading, Mas- sachusetts, 1977.

[9] E. T. Y. Lee. Choosing Nodes in Parametric Curve Interpolation. Computer-Aided Design. Vol. 21, Nr. 6, pp 363-370, August 1989.

[10] W. Ma. NURBS-Based CAD Modelling from Measured Points of Physical Models. Ph.D. Dissertation, Katholieke Universiteit Leuven, Belgium, 1994.

[11] W. Ma and J. P. Kruth. NURBS Curve and Surface Fitting and Interpolation. To appear in: M. Daehlen, T. Lyche and L. L. Schumaker (eds), Mathematical Methods in Computer Aided Geometric Design. Academic Press, Ltd., Boston, 1995.

[12] W. Ma and J. P. Kruth. Mathematical Modelling of Free-Form Curves and Surfaces from Discrete Points with NURBS. In: P. J. Laurent, A. Le M4haut4 and L. L. Schu- maker, (eds), Curves and Surfaces in Geometric Design. A. K. Peters, Ltd., Wellesley, Mass., 1994.

[13] W. Ma and J. P. Kruth. Parametrization of Randomly Measured Points for the Least Squares Fitting of B-spline Curves and Surfaces. Accepted for publication in Computer- Aided Design, 1994.

[14] L. Piegl. On NURBS: A Survey. IEEE Computer Graphics ~ Applications. 11, pp 55-71, January 1991.

[15] J. H. Wilkinson. The Algebraic Eigenvalue Problem. Oxford University Press, 1965.


375

A T E T R A D I C D E C O M P O S I T I O N OF 4 T H - O R D E R T E N S O R S : A P P L I C A T I O N T O T H E S O U R C E S E P A R A T I O N P R O B L E M

J.-F. CARDOSO Tdldcom Paris / URA 820 / Gdr TdSI Tdldcom Paris, 46 rue Barrault 7563~ Paris, France. cardoso@sig, enst. fr

ABSTRACT. Two results are presented on a SVD-like decomposition of 4th-order tensors. This is motivated by an array processing problem: consider an array of m sensors listening at n independent narrow band sources; the 4th-order cumulants of the array output form a 4th-order rank-deficient symmetric tensor which has a tetradic structure. Finding a tetradic decomposition of this tensor is equivalent to identify the spatial transfert function of the system which is a matr ix whose knowledge allows to recover the source signals.

We first show that when a 4th-order tensor is a sum of independant tetrads, this tetradic structure is essentially unique. This is to be contrasted with the second order case, where it is weel known that .dyadic decompositions are not unique unless some constraints are put on the dyads (like orthogonality, for instance). Hence this first result is equivalent to an identifiability property.

Our second result is that (under a 'locality' condition), symmetric and rank-n 4th-order tensors necessarily are a sum of n tetrads. This result is needed because the sample cumulant tensor being only an approximation to the true cumulant tensor, is not exactly a sum of tetrads. Our result implies that the sample cumulants can be 'enhanced' to the closest tetradic cumulants by alternatively forcing their rank-deficiency and symmetry. A simple algorithm is described to this effect. Its output is an enhanced statistic, from which blind identification is obtained deterministically.

This leads to a source separation algorithm based only on the 4th-order cumulants, which is equivalent to robust statistic matching without the need for an explicit optimization procedure.

KEYWORDS.Tet radic decomposition, super-symmetric tensors, al ternate projections (POCS), signal enhancement, cumulants, high order statistics, source separation.

376 J. -F. Cardoso

1 F O U R T H - O R D E R T E N S O R S A N D T H E T E T R A D I C S T R U C T U R E

1.1 DEFINITIONS AND NOTATIONS

In this paper, we consider tensors on an m-dimensional complex vector space ~'. For the sake of brievity, we abusively call 'matrices' the once covariant once contravariant tensors and simply ' tensors ' the twice covariant twice contravariant tensors. Unless explicitly stated, all vectors, matrices, tensors are related to the vector space ~'. We use the following convention: in some or thonormal basis, a generic vector v has components vi for 1 _< i _< m, the ( i , j ) - t h

J for 1 < i, j < m. Similarly a generic tensor entry of a generic matr ix R has components r i _ _ jt -J the components of the Q has components qik for 1 < i , j , k , l <_ m. We also denote a i

- j adjoint matr ix A H, so that a i = (a~)* for any i and j .

jl For any ' tensor ' Q = { Q i k } , there is a linear matr ix- to-matr ix mapping associated to it

in the following canonical way:

J = qJlml k 1 < i , j < m. (1) N = Q ( M ) ~ n i ~_, ik - - l<k, l<m

Accordingly, the rank of a tensor is defined as the dimension of range:

Range(Q) = { N [ N = Q(M); M e s x E*} rank(Q) = Dim(Range(Q)) (2)

which is one way of generalizing the notion of rank to ' tensors' . We note tha t other definitions are obtained by chooosing different indices in the contraction operation (1).

1.2 p-ADIC DECOMPOSITIONS

In order to contrast the 2nd-order and 4th-order case, we first state well known facts about decomposition of matrices. If R be a m • m hermitian matr ix with rank n < m, it can be decomposed as

R = B A B H (3)

where B is a m x n matr ix and A is a n x n diagonal matr ix with real elements. A key point is that such a decomposition is allowed by the rank and symmetry properties but is no t

un ique unless some additional requirements are put on B or A. An interesting constraint is for instance that B should verify B H B = I , in which case eq. (3) is an eigendecomposition.

In the array processing application considered in section 5, an m x m covariance matr ix R appears which depends on an m x n matr ix A (the 'array matr ix ' ) and on an n x n real diagonal matr ix S (the source covariance matrix) according to

R = A S A H. (4)

This is just in the form of eq. (3) but it cannot be used to identify the parameter of interest, i.e. matr ix A, since the decomposition (3) is not unique. Before turning to the 4th-order case, we rewrite the above quantities in indexed notations. Denoting ai the i-th diagonal

element of S, equation (3) reads

j P - j (5) = o'pa i ap. p=l,n

A Tetradic Decomposition of 4th-order Tensors 377

The use of higher-order statistics in addition or in place of second-order statistics leads to the introduction of cumulant tensors besides the familiar covariance matrices. In particular, in the array processing application considered below, the 4th-order cumulants of the array output form a 'tensor' (see section 5) with the following structure

qjl k . p z j . , p z l i < i j , k l< m (6) ik -- Z P~176 "p~k"*p -- ' ~ --

p = l ~n

where a i j is as in eq. (5) and where kl, �9 �9 kn are n real numbers, non necessarily positive (in the following they are assumed to be non zero). This is clearly the 4th-order analog of the second-order structure (5). It is a sum of n terms, each proportionnal to an elementary

p-j p-t which is a four-fold tensor product of the p-th column of A. Since each of tensor a i a p a k a p

these terms is made from a unique vector (the p-th column vector of A), they are called 'tetrads'. Similarly, decomposition (5) shows r i as a sum of n 'dyads', the p-th dyad being a ip-jap i.e. a two-fold tensor product based in the p-th column of A. The decomposition (6)

jl of the 4th-order cumulant tensor qik is called a n-tetradic decomposition. For the sake of simplicity, we restrict ourselves to the case where the columns of A are linearly independent, even though one may consider less restricitve situations with n > m where it is only required that the dyads constructed from the columns of A are linealry independent matrices [2]. In the following, we call 'n-tetradic tensors' these tensors which are a sum of n tetrads constructed from linearly independent vectors.

For later use, we note that, if a tensor Q has the tetradic decomposition (6), then, a direct substitution of (6) in (1) yields the image by Q of any matrix M as

Q(M) = A A M A H with AM = Diag(d l , . . . , dn) ml. (7) dp def ~ a~ ap

kl

In particular, if M is hermitian, the diagonal terms of A M a r e real and Q(M) then also is hermitian.

2 U N I C I T Y A N D C O M P U T A T I O N OF A T E T R A D I C D E C O M P O S I T I O N

At this stage, we may state a major difference between the 2nd- and 4th-order cases. While the decomposition of a 2nd-order tensor in the form (5) is not unique (as noted above), the decomposition of a 4th-order tensor having the form (6) is, under mild conditions, essentially unique. By 'essentially unique', we mean that the columns of A are detremined up to a permutation and that each column of A is determined up to complex scalar factor. Note that this is just the same degree of indetermination observed in the case of the eigendecomposition of a normal matrix with distinct eigenvalues. In this case, the eigenvectors are not ordered (unless the eigenvalues are sorted according to some additional convention) and they are determined up to a phase term if they are normed to unity. We have the following property.

P r o p o s i t i o n 1 The tetradic structure (6) is essentially unique if matrix A has full column rank and kp ~ 0 for 1 <_ p <_ n.

Proof. A simple proof is as follows. The full rank condition is equivalent to the linear independence of the columns of A. By an appropriate choice of M, the elements of A M in

378 J.-F. Cardoso

(7) can be given any values because A is full column rank and the kp's are non zero. It follows that any matrix P in the range of Q may be written as P = ~p aplip for some complex coefficients a l , . . . , an where lip denotes the orthogonal projector on the p-th column of A. Since the rank of P exactly is the number of such non-zero coefficients, any rank one matrix in the range of Q is necessarily proportional to some lip. It follows that the IIp's can be uniquely determined from the rank-one matrices in Range(Q). These, in turn, determine the columns of A up to a scalar factor. QED.

We do not wish to elaborate too much on the issue of computing a tetradic decomposition of a tensor having exactly structure (6). We content ourselves with the following remark: if any two arbitrary hermitian matrices M and N are selected and their images by Q are computed, one gets, according to (7), a pair of equations:

Q ( M ) = A A M AH and Q ( N ) = A A N A H (8)

which form an hermitian pencil. It follows that matrix A can be determined from Q ( M ) and Q ( N ) as solution of a generalized eigenvalue problem. This determination is 'almost surely' unique because the cases of degeneracy in this problem have zero probability if M and N are randomly drawn from continuous probability distributions. Such a solution is very simple but the 'almost sure' determination should be rigorously stated. This kind of probabilistic argument was avoided in the proof of proposition 1 and can also be avoided in computing the tetradic decomposition by resorting to 'deterministic' algorithms as in [2].

The reason why we do not emphasize the issue of determining A from Q and the possible numerical stability problems associated to it is that, in computing A from an est imate of Q obtained from a finite number of samples, the question of statistical robustness dominates the question of numerical robustness. Anticipating a bit, this is because our approach is two steps: i) find the closest n-tetradic tensor to the sample cumulant tensor, ii) extract matrix A form this n-tetradic tensor. Step (ii) is equivalent to a tetradic decomposition while step (i) is a projection onto the set of admissible statistics and handles the statistical part of the estimation problem. If step (i) leads a poorly conditionned tensor (that is a tensor with rank n but with one or several tetrads with a very small relative norm) then there may be numerical stability problems but they are not worth solving because the poor conditionning is a hint that step (i) was performed on sample cumulants carrying very unreliable information on the small tetrads. In others words, the problem becomes statistically ill-conditionned before even being numerically ill-conditionned.

3 R A N K , S U P E R - S Y M M E T R Y A N D T E T R A D I C S T R U C T U R E

This section exhibits a link between the tetradic structure on one hand and symmetries and rank properties on the other hand.

Tensors in the form (6) have rank n in the sense of section 1 i.e. the linear mapping Q defined via eq. (1) has rank n. Also, tensors in the form (6) are called 'super-symmetric' in the sense that they are invariant under all the canonical permutation symmetries. In the complex case, this amounts to

qjt lj jl tj , i k , , ki) , ik), qki), ik = qik = qki = qki = (qjt) = (qjl = (qtj = ( j 1 <_ i , j , k , l <_ m. (9)

A Tetradic Decomposition o f 4th-order Tensors 379

Note that the properties of supersymmetry and rank are defined independently of the specific 'internal' structure of the fourth-order tensor under consideration. They are a priori unrelated to the n-tetradic structure.

Clearly, any n-tetradic tensor is supersymmetric and has rank n. Therefore, denoting as S the set of all supersymmetric tensors, as T~ the set of all rank-n tensors and as 7" the set of all n-tetradic tensors, we have that

T c S n ~ . (i0) This raises the question of knowing if S N 7~ C 7" which would mean that any rank-n

supersymmetric tensor necessarily is a sum of tetrads. We have only a 'local' answer to this question, obtained by linearizing the sets S, 7~ and 7" in the vicinity of a n-tetradic tensor, where they are smooth manifolds. Let Q be a n-tetradic tensor and respectively denote Ps, Pr, Pt the tangent planes at Q to the smooth manifolds S, 7~, 7". The following property is our best result.

P r o p o s i t i o n 2 Pt = Ps n Pr.

It means that any rank-n supersymmetric tensor which is close enough to an n-tetradic tensor also is n-tetradic. Hence the property 7" = S N 7~ is true in some neighboorhood of 7-. The proof is by computing the intersection of the tangent spaces Ps and Pr and is given in [3].

The interest of this result is that, by defintion, consistent estimates of cumulants are, for large enough sample size, arbitrarily close to their ' true' value which is n-tetradic for the problem under consideration. Next section describes how this result allows the idea of 'signal enhancement' to be turned into an identification technique based on sample cumulants.

4 STRUCTURE FORCING

Provided a cumulant tensor Q = {qi~} has exactly the structure (6), procedures for com-

puting, up to a scalar factor, each column of A = {ai} from Q are easily devised (see section 2). In practice, though, the cumulants are computed from a finite set of data so that only

^ ~jl estimates Q = {qik} of Q are available. We propose the following estimation technique for the determination of A from Q.

1. Find ~) the tensor which is the closest to sample cumulant tensor Q and is the sum of n tetrads like the ' true' cumulant tensor, i.e. is in the form of (6).

2. Extract the tetrads from Q, i.e. find the (essentially) unique solution in A of the identification equation (6) based on the n-tetradic tensor Q.

We stress again that algorithms for step 2 do not need to be robust in a statistical sense because the statistical part of the estimation is integrally handled by step 1 as implied by the essential unicity of the tetradic decomposition.

380 J.-F. Cardoso

4.1 CUMULANT ENHANCEMENT

Step 1 can be implemented without resorting to an explicit minimization procedure by using the idea of 'signal enhancement' [1]. Since the true cumulants are known to be supersymmetric and rank n, enhancement is achieved by alternate projections of the sample cumulant tensor onto T~ and onto S:

A

1. Start with the sample cumulant tensor Q.

2. Project it onto 7~.

3. Project the result onto S

4. Go to step 2 until convergence is reached.

An iterative procedure is needed because rank forcing destroys symmetry while symmetrization destroys the rank-n property. Reaching convergence means that none of these operations destroy neither supersymmetry nor rank deficiency. Hence, it is reached by a cumulant tensor which is both rank-n and supersymmetric.

Signal enhancement is a general idea to 'dean up' statistics. A common example is the case of sample covariance matrices which can be enhanced by alternatively forcing properties like definiteness, Toeplitz structure, equality of the smallest eigenvalues, etc. In our case, the key result is provided by propositoin 2 which guarantees that the above algorithm if initialized with a sufficiently good estimates of the cumulant tensor will ultimately yield an n-tetradic tensor. The sample cumulant tensor being then enhanced into a statistic which has exactly the structure (6); it can be passed next to any algorithm for inverting this equation. As stressed above this is an algebraic, not a statistical issue.

When the sample cumulants are close enough to their true values, the sets T~ and S can be assimilated to their tangent planes and alternate orthogonal projections onto them is then asymptotically equivalent to orthogonal projection onto their intersection which is locally the set of tetradic tensors. Hence, cumulant tensor enhancement is asymptotically equivalent to orthogonal projection onto the n-tetradic set 7".

5 A P P L I C A T I O N A N D I M P L E M E N T A T I O N

5.1 THE SOURCE SEPARATION PROBLEM

The standard linear model in narrow band array processing is x(t) = As(t) + v(t) where x(t) is the m-dimensional array output, s(t) is the n-dimensional source signal vector, A is the m • n array response matrix and v(t) represent an additive noise. In standard array processing, each column of A is assumed to depend, in a known fashion, on a small set of location parameters. In contrast, source s e p a r a t i o n does not assume any a priori information on matrix A: this matrix and the source signals have to be estimated from the observations Ix ( l ) , . . . , x (T ) ] only. This is a very desirable feature since it makes the blind approach independent of array modelling. It is then, by essence, insensitive to errors in the array manifold.

A Tetradic Decomposition of 4th-order Tensors 381

Various approaches to blind array processing have already been proposed on the basis of 4th-order cumulants (see for instance [5, 4, 6, 7]). Assume that the source signals are stationnary and mutually independent. Denote kp the kurtosis of the p-th source signal

kp = Cum(sp(t),s;(t),sp(t),sp(t)), (11)

and assume that the additive noise is normally ditributed and independent form the source signals. The 4th-order cumulant tensor of the array output defined as

then have exactly the structure (6). This is easily found from the multilinear property of the cumulants, and from the fact that they are additive for independent variables and cancel at order higher than 2 for independent Gaussian components.

5.2 IMPLEMENTING ALTERNATE PROJECTIONS

Orthogonal projection onto T~, i.e. finding the closest rank-n approximation, is achieved by truncating an eigendecomposition to the n most significant components. Orthogonal projection onto S is achieved by a mere symmetrization (averaging through index permutations). These two steps may be implemented as described below.

A supersymmetric 4th-order tensor can be decomposed into orthogonal hermitian eigenmatrices. This is a simple extension of eigendecomposition of hermitian matrices which is implemented by arranging the elements of the m x m x m x m tensor into a m 2 x m 2 hermitian matrix and then calling a standard eigendecomposition routine. In index form,

~jl m 2 the eigendecomposition of qik into components reads

~jt J-' (13) = # e e i ek . e - l , m 2

J denotes the ( i , j ) - th entry of the e-th eigen-matrix (this notation is intended to where e i avoid adding an extra index) and where #e is the associated real eigenvalue. Orthogonal projection onto ~ amounts to the truncation of this eigen-decomposition to the n most significant components. Further projection onto S is obtained by symmetrization through all the symmetry operations listed in eq. (9). However, the fact that the eigenmatrices can

~jt ~t be chosen hermitian reduces the truncation/symmetrization of qik into qik to computing

_~t 1 t j qik = ~ ~ #e(eietk + eiek)" (14)

e=l,n

In summary, cumulant enhancement may be implemented by iterating through a decompostion in hermitian eigen-matrices (13) and a recomposition from the eigenmatrices after trunction and symmetrization (14). Note that S is a linear space; a result of [i] guarantees that alternating projections onto linear spaces and eigen-truncation is a convergent procedure.

382 J.-F. Cardoso


Blind source separation is motivated by the need for recovering source signals when only a mixture of these, obtained at the output of an array of sensors, is available. Blind source separation is necessary when no reliable information on the transfer function (matrix A) is available. If some additive Gaussian noise is also present, source separation may be based on 4th-order cumulants with the advantage that modelling the noise spatial structure is unnecessary since it does not contribute (asymptotically) to the cumulant structure.

However, processing of a limited subset of cumulants leads to poor statistical performance. Processing the whole set of cumulants (the cumulant tensor) increases the robustness but requires to deal with largely overdetermined equations. We have proposed a two-step solution: first enhancing the sample cumulants by projecting them onto the set of n-tetradic tensors and next solving the identification based on the enhanced cumulants. The second step is made algebraically simple after the first step is successfully completed. We suggest implementation of the projection in the first step by alternatively projecting the sample cumulant tensors onto the set of rank-n tensors and onto the set of supersymmetric tensors. This procedure is known to converge. In this paper, we have shown that it converges to an 'enhanced' tensor which is n-tetradic provided the errors in the sample cumulants are small enough. Numerical experiments (not included) show that this condition is fulfilled with a limited number of samples in the case of sources with negative kurtosis.

The Matlab code implementing cumulant enhancement and a demo of blind source separation based on 4th-order cumulants can be obtained from the author at cardoso~sig.enst.fr.

References

[1] James A. Cadzow. Signal enhancement - A composite property mapping algorithm. IEEE Tr. on ASSP, 36(1):49-62, January 1988.

[2] Jean-Francois Cardoso. Super-symmetric decomposition of the fourth-order cumulant tensor. Blind identification of more sources than sensors. In Proc. ICASSP, pages 3109- 3112, 1991.

[3] Jean-Fran~;ois Cardoso. Fourth-order cumulant structure forcing. Application to blind array processing. In Proc. 6th SSAP workshop on statistical signal and array processing, pages 136-139, October 1992.

[4] Jean-Frangois Cardoso and Antoine Souloumiac. Blind beamforming for non Gaussian signals. IEE Proceedings-F, 140(6):362-370, December 1993.

[5] Pierre Comon. Independent component analysis. In Proc. Int. Workshop on Higher- Order Stat., Chamrousse, France, pages 111-120, 1991.

[6] Michel Gaeta and Jean-Louis Lacoume. Source separation without a priori knowledge: the maximum likelihood solution. In Proc. EUSIPCO, pages 621-624, 1990.

[7] L. Tong, R. Liu, V.C. Soon, and Y. Huang. Indeterminacy and identifiability of blind identification. IEEE Tr. on CS, 38(5):499-509, May 1991.


383

T H E A P P L I C A T I O N OF H I G H E R O R D E R S I N G U L A R V A L U E D E C O M P O S I T I O N TO I N D E P E N D E N T C O M P O N E N T A N A L Y S I S

L. DE LATHAUWER 1, B. DE MOOR 2, J. VANDEWALLE

Katholieke Universitei t Leuven

Kard. Mercierlaan 9~

3001 Leuven Belgium L ie ven. D elatha u wer@esat, kule u ven. ac. be

Bart. Demoor@esat. kule u ven. ac. be

Joos. [email protected]. be

ABSTRACT. Due to the scientific boom in higher-order signal processing, the interest in algebraic manipulations of tensors is rapidly increasing. In this paper we will discuss a tensor decomposition that can be interpreted as a generalization of the Singular Value Decomposition of matrices. For an Nth-order decomposition, this "Higher-Order Singular Value Decomposition" involves N orthogonal matrices of singular vectors. The diagonal matr ix that contains the singular values in the second-order case, is replaced by" an all-orthogonal Nth-order tensor, in which all-orthogonality turns out to be the appropriate higher-order equivalent of diagonality. The Frobenius-norms of particular ( N - 1)th-order subtensors can be considered as singular values. We will show that this decomposition can be used to solve the blind source separation problem in Higher-Order Statistics. The derivation of the algorithm is established under noise-free conditions. The approach offers considerable conceptual insight, e.g. it allows a further interpretat ion of Independent Component Analysis as the higher-order refinement of Principal Component Analysis.

KEYWORDS. Singular value decomposition, tensor algebra, higher-order statistics, independent component analysis.

1Lieven De Lathauwer is a research assistant with the I.W.O.N.L. (Belgian Institute for Scientific Research in Industry and Agriculture). The research reported in this paper was partially supported by the Belgian Programme on Interuniversity Poles of Attraction (IUAP-17, IUAP-50), initiated by the Belgian State, Prime Minister's Office for Science, Technology and Culture. It was also supported by the European Community Research program ESPRIT, Basic Research Working Group hr. 6620 (ATHOS). The scientific responsibility rests with its authors.

2Bart De Moor is a research associate with the N.F.W.O. (Belgian National Fund for Scientific Research).

384 L. De Lathauwer et al.


In recent years the use of multilinear algebra for signal processing applications has con- stantly been increasing. To a large extent this is due to the important progress in the field of Higher-Order Statistics (see [1], [2], [3], [4], . . . ) , for the higher-order statistical quantities of a stochastic vector signal are higher-order tensors. More generally, multilinear algebra is the framework of choice when the quantities to be dealt with, can be considered as higher-order arrays.

The term "higher-order tensor" is used in this paper to denote higher-order tables of numerical values, i.e. the natural extension of vectors (first order) and matrices (second order). The classical definition, stated in e.g. [12], refers to the particular way in which tensors behave under coordinate transformations of an underlying vector space. This definition remains valid here but will not explicitly be exploited.

Because of the key role the Singular Value Decomposition (SVD) of matrices plays in (numerical) linear algebra and its applications (see [18], [8], [17], [7], . . . ) , the higher-order counterpart is a promising result. A first attempt to generalize the decomposition can be found in [15]. The author outlines some main ideas to approximate a given tensor by an expansion of orthogonal rank-1 components, computed consecutively. In this approach it is not taken into account that the elements of the expansion are intensively coupled when one's aiming at the optimal approximation for a fixed number of components. On the other hand, it is hard to give a physical meaning to the obtained results.

In this paper an other approach is highlighted. For the third-order case, the basics have been developed in the field of Psychometrics [16]. We have investigated the proposed way of data analysis from an algebraic point of view and proved that it yields a convincing generalization of the Singular Value Decomposition (SVD) to the case of higher-order tensors. Many properties have already been generalized. They all show a strong analogy between the matrix and the higher-order case ([5], [6]).

The second part of the paper shows how the new decomposition can be used to perform blind source separation. This problem can be stated as follows. Consider the linear transfer of a zero-mean stochastic "source vector" X to a zero-mean stochastic "output vector" Y when additive noise N is present:

Y = M X + N (1)

The matrix M has linearly independent columns and is (for convenience) supposed to be square. The goal of blind source separation now consists of the determination of M and the corresponding realizations of X, given only realizations of Y, and assuming statistical independence of the components of X. Sometimes the problem is referred to as "Independent Component Analysis" (ICA), especially when the solution is based on explicit minimization of the statistical dependence of the estimated source signals [4].

Our text is organized as follows. Section 2 introduces the concept of Higher-Order Singular Value Decomposition (HO SVD). We will show the analogy between the tensor decomposition and the classical matrix decomposition, and demonstrate how the HO SVD can be computed. In Section 3 we derive a new approach to ICA, based on HO SVD. It will

The Applications of Higher-order SVD to ICA 385

bring up an interesting relationship with the second-order problem of "Principal Component Analysis" (PCA)[10].

For clarity, the whole discussion will be restricted to third-order tensors with real elements. Moreover, the HO SVD-way to blind source separation will be derived under noise-free conditions. [6] contains perturbation expressions for the HO SVD, in case the data were affected by additive noise.

2 T H E H I G H E R - O R D E R S I N G U L A R V A L U E D E C O M P O S I T I O N

2.1 MULTIPLICATION OF A HIGHER-ORDER TENSOR BY A MATRIX

Definit ion 1 The mode-1 product of an (I x J • K)- tensor A with a (P x I)-matrix U, denoted by A x l U, is a (P x J • K)-tensor of which the entries are given by

(A x~ v)~jk aoj ~ ~jk v~ (2) i

The mode-2 and mode-3 product are defined accordingly.

The mode-number corresponds to the index of A that is varied in the summation. The mode- n product of a higher-order tensor and a matrix is a special case of the inner product in multilinear algebra [12] and, more generally, tensor analysis [11]. In literature it is mostly denoted using the Einstein summation convention, i.e. the summation sign is dropped for the index that is repeated. Especially in the field of tensor analysis this approach is advantageous, since an Einstein summation can be proved to have a basis-independent meaning. For our purpose however, the use of the • n-symbol will more clearly demonstrate the analogy between matrix and tensor SVD.

E x a m p l e The matrix product U . F . V t can be written as F • U x2 V.

2.2 THE HO SVD-MODEL

T h e o r e m Every real ( I x J • K)-tensor A can be written as the product

&----~X 1 U x 2 V x 3 w

in which:

(3)

�9 the "core tensor" S is a real ( I x J x K)-tensor of which the second-order submatrices Si=,~, Sj=,~, Sk=a, obtained by fixing an index to a, have the properties of:

- all-orthogonality: two matrices Si=~ and Si=/3 are mutually orthogonal, with respect to the standard matrix inner product, for all possible values of a and subject to a ~ ~ :

(Si=a, Si=/3) def trace{S~=z Si=a} = 0 when a ~ (4)

When i is replaced by j or k, the analog holds.


- ordering, with respect to the Frobenius-norm:

IIs~-xll > IIS~-211 > . . . > IIs~=ill > II s~_--i+xll . . . . . II s~-III = o (5)

When i is replaced by j or k, the analog holds.

�9 U, V and W are an orthogonal (I x I)-matrix, resp. (J x J)-matrix and (K x K)- matrix.

The F~obeniu~-no~m~ IIS~--~ll, IlSj=~ll, IlSk--~ll are the mode-l, resp. mode-2 and mode-3 singular values of A. The columns of U, V, W are the mode-l, resp. mode-2 and mode-3 singular vectors. The decomposition is visualized in Fig. 1.

Proof: due to space limitations, we refer to [6]. []

l Figure 1: Visualization of the HO SVD for a third-order tensor.

2.3 INTERPRETATION

Eq. (3) should be compared to the expression for the SVD of a real ( I x J)-matr ix F, which in our notation reads:

F = S • U • (6)

in which the matrices U, V are orthogonal and the "core matrix" S is diagonal and contains r = I = J strict positive elements, put in non-increasing order (see also Fig. 2).

Figure 2: Visualization of the matrix SVD.

Clearly Eq. (3) is a formal generalization of Eq. (6). Moreover, it can be proved that the HO SVD of a second-order array boils down to its matrix SVD [6].


We remark that the core tensor is generally not diagonal: counting the degrees of freedom reveals that diagonality of the core tensor would be a too strong condition on Eq. (3). This objection doesn't apply to the property of all-orthogonality, which is a straightforward generalization of diagonality.

2.4 CALCULATION

Reorganisation of Eq. (3) in a matrix format shows that the matrices U, V and W can be calculated as the left singular matrices of the (I x JR)-, (K x I J)- and (J x KI)-matr ix unfoldings of A, defined in accordance with Fig. 3.

>

[ I 1 I

Figure 3: Unfolding of the (I • J • K)-tensor A to an (I • JK)-matr ix A._.A(IxJK).

The core tensor is obtained by bringing the matrices in Eq. (3) to the other side:

= A xl U t x2 V t x3W t (7)

The way of cMculation and the ordering constraint on the core tensor show that the HO SVD obeys analog unicity properties as its matrix equivalent: in the generic case, the singular vectors are determined up to the sign. When the sign of a singular vector is changed, the sign of the corresponding submatrix in ~ alters too.

3 A P P L I C A T I O N TO I N D E P E N D E N T C O M P O N E N T A N A L Y S I S

3.1 OUTLINE OF THE PROCEDURE

We consider the noise-free version of Eq. (i):

Y = MX (8)

The separation problem will be solved by factorisation of the transfer matrix:

M = TQ (9)

in which T is regular and Q is orthogonal.

In a first step T will be determined from the second-order statistics of the output Y. The resulting degree of freedom, the orthogonal factor Q, is recovered from the higher-order statistics of Y.


3.2 STEP i: DETERMINATION OF T FROM THE SECOND-ORDER STATISTICS OFY

The covariance C Y of Y is given by

C2 y = M C X M t (10)

in which the covariance C x of X is diagonal, since we claim that the source signals are uncorrelated. Assuming that the source signals have unity variance, we get:

C2 Y = MMt (I 1)

This assumption means just a scaling of the columns of M and is not detrimental to the method's generality: it is clear that M can at most be determined up to a scaling and a permutation of its columns.

We can conclude from Eq. (11) that M can be determined, up to an orthogonal factor Q, from a congruence transformation of C Y"

C2 Y = M M t = (TQ) (TQ) t = T T t (12)

This equation shows that only the column space of M can be identified when just second- order statistical information on Y is used and no extra constraints are added. In order to solve the initial problem one has to resort to the higher-order statistics of Y.

When one sticks to a mere second-order approach, it is common to make the solution essentially unique by selecting a matrix with orthonormal columns - in which the extra constraint generally has no physical meaning. This corresponds to the well-known concept of PCA [10].

In the framework of ICA, the PCA-procedure can be considered as one alternative to perform the pre-whitening. Algebraically, this is realized by computing the EVD of C Y"

C~ = ED2E t = (ED)(ED) t (13)

When the output covariance is estimated following C Y = AFAr, in which Ay is an ( I x N)- dimensional dataset containing N realizations of Y, then the factor (ED) can be obtained in a numerically more reliable way from the SVD of Ay [9].

3.3 STEP 2: DETERMINATION OF Q FROM THE HIGHER-ORDER STATISTICS OFY

The third-order cumulant C Y of Y, defined by the element-wise expectation

c~k = E{l~l~Yk} (14)

is related to the third-order cumulant C x of the source vector X in the following way:

C Y = C x" • 2 1 5 2 1 5 (15)

as can easily be verified by combining Eqs. (14) and (8). In Eq. (15) C X is diagonal, since we daim that the source signals are also higher-order independent ([13], [14]). Substitution


of Eq. (9) in Eq. (15) yields

I~ = C x x l Q x2 q x3 Q (16)

in which the tensor ~ is defined as:

]~ def= cy Xl T -1 • T -I X3 T -I (17)

Hence, due to the unicity property in Section 3.3, Q can be obtained from the HO SVD of I~. (Eq. (16) is for the tensor B the third-order equivalent of the Eigenvalue Decomposition of a symmetric matrix.)

The transfer matrix M is finally given by Eq. (9).

3.4 DISCUSSION

We want to stress the conceptual importance of the new approach. It reveals an important symmetry when comparing the problems of PCA and ICA. In "classical" second-order statistics, the problem of interest is to remove the correlation from data measured after linear transfer of independent source signals. The key tool to realize this, comes from "classical" linear algebra: it is the matrix SVD.

More recently, researchers also aimed at the removal of higher-order dependence, which is a problem of Higher-Order Statistics. We proved that one can resort to a tool from multilinear algebra, which is precisely the generalization of the SVD for higher-order tensors.


We generalized the Singular Value Decomposition of matrices to the higher-order case. It was shown that this decomposition provides a new conceptual approach to solve the blind source separation problem in Higher-Order Statistics.

R e f e r e n c e s

[1] J.-F. Cardoso, P. Comon. Tensor-based independent component analysis. Signal Pro- cessing V : Theories and Applications, pp 673-676, 1990.

[2] J.-F. Cardoso. A tetradic decomposition of 4th-order tensors. Application to the source separation problem. In : B. De Moor, M. Moonen (editors). SVD and signal processing, III : algorithms, applications and architectures. Elsevier Science Publishers, North Holland, Amsterdam, 1995.

[3] J.-F. Cardoso, A. Souloumiac. An efficient technique for blind separation of complex sources. Proc. IEEE SP workshop on higher-order statistics, Lake Tahoe, U.S.A. pp 275-279, 1993.

[4] P. Comon. Independent component analysis, A New Concept? Signal Processing, special issue on higher-order statistics. 36 (3), pp 287-314, 1994.


[5]

[~]

[7]

Is]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[is]

L. De Lathauwer, B. De Moor, J. Vandewalle. A singular value decomposition for higher-order tensors. Proc. ATHOS workshop on system identification and high-order statistics, Nice, France, September 1993.

L. De Lathauwer, B. De Moor, J. Vandewalle. The higher-order singular value decomposition. To be submitted to: SIAM Journal on Matrix Analysis and Applications.

B. De Moor, M. Moonen (editors). SVD and signal processing, I I I : algorithms, applications and architectures. Elsevier Science Publishers, North Holland, Amsterdam, 1995.

E.F. Deprettere (editor). SVD and signal processing : algorithms, applications and architectures. Elsevier Science Publishers, North Holland, Amsterdam, 1988.

G.H. Golub, C.F. Van Loan. Matrix computations. North Oxford Academic Publishing Co., Johns Hopkins Press, 1988.

I.T. Jolliffe. Principal component analysis. Springer, New York, 1986.

D.C. Kay. Theory and problems of tensor calculus. McGraw-Hill, 1988.

M. Marcus. Finite dimensional multilinear algebra. Dekker, New York, 1975.

J.M. Mendel. Tutorial on higher-order statistics (spectra) in signal processing and system theory: theoretical results and some applications. Proceedings of the IEEE. 79 (3), pp 278-305, 1991.

C.L. Nikias, J.M. Mendel. Signal processing with higher-order spectra. IEEE Signal Processing Magazine. July 1993, pp 10-37.

M. Schmutz. Optimal and suboptimal separable expansions for 3D-signal processing. Pattern Recognition Letters. 8, pp 217-220, 1988.

L.tt. Tucker. The extension of factor analysis to three-dimensional matrices. In : H. Gulliksen, N. Frederiksen (editors). Contributions to mathematical psychology. Holt, Rinehart & Winston, pp 109-127, 1964.

It. Vaccaro (editor). SVD and signal processing, H : algorithms, applications and architectures. Elsevier Science Publishers, North Holland, Amsterdam, 1991.

J. Vandewalle, B. De Moor. On the use of the singular value decomposition in identification and signal processing. In : G. Golub, P. Van Dooren (editors). Numerical linear algebra, digital signal processing and parallel algorithms. NATO ASI Series, F70, pp 321-360, 1991.


391

B A N D P A S S F I L T E R I N G F O R T H E H T L S E S T I M A T I O N A L G O R I T H M : D E S I G N , E V A L U A T I O N A N D S V D A N A L Y S I S

H. CHEN, S. VAN HUFFEL, J. VANDEWALLE Katholieke Universiteit Leuven

Kard. Mercierlaan 9~ 3001 Leuven Belgium hua. chen @esat. kule u ven. ac. be

ABSTRACT. The research on the parameter estimation of a sum of K exponentially damped sinusoids has led to the development of many estimation algorithms. In some applications, however, it is desired to prefilter the input data in order to reduce the noise and enhance the parameters of interest. In this paper, we present a prefiltering technique in which a filter matrix is multiplied with the original data matrix prior to applying a subspace and SVD-based method. Two filter matrices are proposed, respectively, for the FIR and IIR prefiltering. A theoretical analysis on a special case of two exponentially damped sinusoids is given, which reveals the relationship between the singular values/vectors of the prefiltered and original data matrices.

KEYWORDS. Exponentially damped sinusoids, parameter estimation, frequency-selective, bandpass filter, SVD, total least squares, state space.


The estimation of the parameters of a sum of K exponentially damped sinusoids from noisy data has led to the development of many algorithms, such as the linear prediction (LP) method [1], the matrix pencil (MP) method [2] and Kung et al.'s method and its variant (HSVD and HTLS) [3, 4, 5]. The essence of these subspace and SVD-based methods lies in the SVD truncation, sometimes called SVD filtering since it filters out part of the noise by truncating the SVD of a Hankel/Toeplitz data matrix to rank K and discarding the non-significant singular values and vectors. However, it is not always satisfactory especially at low signal-to-noise ratios (SNR). Moreover, in some applications only a few (less than K) sinusoids are of interest. To further combat noise and/or enhance sinusoids of interest,

392 H. Chen et al.

it is frequently desired to prefilter the input data in order to improve the resolution and accuracy of the estimated parameters of interest. It should be noted that the SVD filtering of the subspace-oriented estimation methods cited above is based on the rank-K property of a Hankel/Toeplitz data matrix in the absence of noise. As such, a special prefiltering technique is required since a Hankel/Toeplitz matrix constructed from the filtered data, instead of the original noiseless data, does not possess the rank-K property anymore[6].

In the following sections, a prefiltering technique is first described in which a full-rank filter matrix is multiplied with the Hankel/Toeplitz data matrix�9 The prefiltered data matrix, or the product obtained thereof, keeps the rank-K property and can readily be processed by an SVD-based estimation algorithm, e.g. HTLS in this paper. Next, we present two filter matrices that implement the finite impulse response (FIR) and infinite impulse response (IIR) filters, respectively. Subsequently, a theoretical analysis on a special case of two exponential]y damped sinusoids is given revealing the relationship between the singular values/vectors of the prefiltered and original data matrices�9

2 P R E F I L T E R E D HTLS

The estimation algorithm used here is HTLS, which assumes that the data samples xn are modeled as follows:

K K

z= .~ ~ ckexp[--dknAt + j(2~r fknAt)] = ~ CkZ'~ n = 0, 1,." . , Y - 1 (1) k - ' l k - 1

where ck is the complex-valued linear parameter, dk (damping factor) and fk (frequency) are the nonlinear parameters of the kth peak and j = ~2"f, zk = exp[-dknAt+j(2~rhnAt)] is the pole of the signal and At is the constant sample interval. Obviously, fs = 1/zXt is the sampling frequency. HTLS is a subspace based method, which first arranges the data points in a Hankel matrix XLxM a s f o l l o w s

X0 Xl X2 � 9 1 4 9 1 4 9 X M - 1

Xl X2 . . . . .

x = ~ . . . . . . (2) �9 . �9 �9 �9

�9 �9 �9 . �9

X L _ 1 . . . . . X N _ 1

where L > K, M > K, N = L + M - 1, and then performs an SVD filtering

X = U K ~ K V ~ (3)

where UK and VK axe the first K left and right singular vectors, and ]EK is the diagonal matrix composed of the first K singular values. Finally zk is retrieved using the rotational invariance property of UK or VK and the TLS technique

= = ( 4 )

where the up (down) arrow stands for deleting the top (bottom) row, the superscript H denotes Hermitian conjugate, and the eigenvalues of both Z1 and Z2 are the signal poles Zk. This method gives better accuracy compared to Kung's method as shown in [5]. Once

Bandpass Filtering f o r the HTLS Estimation Algorithm 393

the poles and hence the nonlinear parameters are determined, the linear parameters can be found by the least squares method�9

By prefiltered HTLS we mean that a full-rank filter matrix H is multiplied with the Hankel data matr ix before applying the estimation algorithm HTLS. As formulated below, a right (resp., left) multiplication of a full-rank matrix Hr (resp., Ht) of appropriate dimensions retains the rank-K and rotational-invariant properties of UK (resp., VK) in the absence of noise.

= X H r = r d g ~ , g V H = EK=I dkffkVk g =V fdTg = T J K j Z 1

resp., 2 = HIX = TJK~aK ~TH "- EK=I ~kil.kVk H ~ ~rT K = ~VrK~Z H

The eigenvalues of Zl (resp. Z2) are the poles of signal. The tilde denotes the prefiltered counterpart throughout the paper. Since a right (resp., left) multiplication of a full-rank square matrix does not change the column (resp., row) space of X, there exists a unitary matrix 12 such that I5 = U ~ (resp., V = V ~ ) . Furthermore, it can be proven that

1 2 _ [ 1 2 1 0 1 2 2 0 1 w i t h 1 2 1 E c K •

3 F I R A N D I I R F I L T E R M A T R I C E S

For a linear filtering process, a filter matrix H can be found such that y = H x E C p• is the filtered data vector if x E C p• is the original data vector�9 A full-rank Toeplitz filter matrix HE shown below is set up from the impulse response of an FIR filter hi, h 2 , . . . , hq to implement the linear convolution with zero-padding�9

�9 "" hi 0 . . . . . . 0

h2 "'. "'. �9 . �9 �9 hi '. .

"'. : h2 " . 0

"'. hq : hi �9 , �9149

hT

hq

H F ~- 0

gl g2 0 gl

H I =

� 9 1 4 9 1 4 9

with the elements gx = br/a~,

E C p• w h e r e r = [q +2 l j

( b r - i + l - a r - l g i - 1 - a r - 2 g i - 2 . . . . . a r - i + l g l ) / a r , gi = ( - a r - l g i - 1 - ar-2gi -2 . . . . . aog i - r ) /ar ,

f o r 2 _ i < r + l for r + 2 <_i <_p

0 "" . . . . 0 hq . . . hm Given the backward recursion of an r th order IIR filter

aryn = brxn + br- lxn+l + . . . + boxn+r - ar-lYn+l . . . . . alyn+r-1 - a0Yn+r, ar ~ 0 with the initial conditions: Xp+l = xp+2 . . . . . xp+,. = O, Yp+l = Yp+2 . . . . . yp+r = 0

we can construct a full-rank upper-triangular filter matrix Hx �9 � 9 g p

�9 �9149 gp-1 . . . ~ c p• ( 5 )

�9 �9

0 gl

394 H. Chen et al.

The verification of HF , H I is simply writing out y = HFX and y = H i x .

In the prefiltered HTLS method, p = M and Hr = H T (resp., p = L and Ht = H) are taken for the prefiltering via right (resp., left) multiplication.

4 S V D R E L A T I O N S H I P S B E T W E E N F I L T E R E D A N D U N F I L T E R E D D A T A M A T R I C E S

In the absence of noise, the singular values and vectors of a Hankel data matr ix X depends on the model parameters and the filter matr ix as well. Consider only the case of a noiseless sum of two exponentially damped sinusoids prefiltered via right multiplication of a filter matr ix Hr . The relationships between the SVD of X, the SVD of X and the model parameters are investigated in this section in order to give some insight in the effects of filtering.

4.1 DEFINITIONS AND IMPLICATIONS

D e f i n i t i o n 1 Sk and tk are the L-tuple and M- tup le vectors of the k th exponential ly damped s inusoid in model (1), i.e.,

a a z~ , . . , z M sk = [1, zk , z~, , ~ L - x , r [1 zk -~ r �9 .. z k ] , tk = , , �9 ]

A H A T . A H H A D e f i n i t i o n 2 Some factors are defined as O~ki = s k si, flki -- t k ti , (~ki = s H H/si , ~ki =

tkTHrHrHti , , ~ki --A "k " H H r ~ r~I'lv'~, where the super sc r ip t , denotes conjugation.

Obviously, the factors aki and flki are the correlation of the kth and i th exponentially damped sinusoids. Since they are very similar (aki = flki for L = M) , we only comment on one of them, say, flki, as the prefiltering via a right multiplication of Hr is considered. Generally, [flki[/X/flkkflii _ 1 according to Cauchy's inequality. With the concept of a normalized correlation [flki[/X/flkkflii, two extreme cases of signals can be described using the t ime-domain data model (1).

�9 When the two peaks coincide, i.e., Zk = zi, then [flki[/~/flkkflii reaches its maximum, 1.

�9 When the two peaks are ideally nonoverlapping, i.e., dk = di = 0 and ( f i - f k ) / f a = -4-0.5, then [flki[/X/flkkflii reaches its minimum, 0 if M is even, or 1 / M if M is odd.

The degree of overlapping can also be analyzed in terms of the discrete Fourier transform (DFT) since flki T * M - 1 = tkt~ ~ 7~ ~ ( i ) i~ , = ~m=o ( m ) ~ * ( m ) where the M-point D F T of tk and has the form ak/[dk + j 2 ~ r ( i / M - f k / f s ) f s ] for exponentials. The smaller dk, the narrower peak k in the spectrum. When the two peaks, k and i, are approximately nonoverlapping, that is, they are sufficiently far away and have sufficiently small damping factors, their correlation is small as can be deduced intuitively from the frequency domain.

&ki and flki are simply the f l tered counterpart of aki and flki. Similarly, [&ki[/X/&kk&ii

<_ 1, [flki[/~/flkkflii <_ 1, and &ki = ~ki for L = M. The relationship between &ki and aki, flki and flki depends mainly on the filter matrix. Here we are going to use an ideal rectangular

Bandpass Filtering for the HTLS Estimation Algorithm 395

bandpass filter. By rectangular we mean that the frequency response of the bandpass filter is 1 inside the passband and 0 outside the passband, of which the lower and upper cut-off frequencies are ft and f~,. This is a useful simplification in that it delivers neat expressions which provide insights in the prefiltering method. 1 Using D FT properties, it can be proven that the 2-norm of such an rectangular filter matrix is 1. [6] Hence, ~kk = T H �9 tk HrHr tk <-[[Hr[[2tTt~ = flkk.

4.2 SVD EXPRESSIONS OF X AND X IN TERMS OF THE SIGNAL PARAMETERS

The above definitions are useful in that they simplify the expression of X and its singular values/vectors as a function of the signal parameters and the filter matrix. For example, the Vandermonde decomposition of X can be written in the form

K

X = [Sl, S 2 , ' ' ", SKI d i a g ( c l , c 2 , ' ' ' , OK) [ t l , t 2 , ' ' ", tg] T "- Z CksktTk k=l

To express the SVD in terms of the signal parameters, we invoke the following elementary properties relating eigenvalue decompositions and SVD's. For k = 1 , . . . , K , the first K nonzero eigenvalues, ~k, of ~ H and ~ H ~ are b~ The corresponding eigenvectors of ~:~H and ~ g ~ [ are fik and ~'k, respectively. When K = 2, the prefiltered data matrix is simply

= (clsl t T + c2s2tT)Hr -" (O'lUl vH + o'2u2vH)Hr Based on this description and our earlier definitions, the following relationships between the SVD and the signal parameters can be proven [6]. The left and right singular vectors fik and ~'k (_k = 1, 2)_can be written as the linear combinations of sl, s2 and t~, t~:

fik = buklSl + buk2s2, ~'k = bvklt~ + b,k2t~ The eigenvalues ~k -2 (k 1 2) are the roots of the quadratic equation: -- O- k = ,

~ 2 [(a11~n[c112 + a22~221c212 + 2Re(a~2~12ClC~)] ~

- 0

If no filter matrix is applied, then ~ki = ~ki, the above quadratic equation reduces to that of Ak, and uk = b~klSl + buk2s2, Vk = bvkl t~ + bvk2t~, as given in [7]. Closed-form solutions of o'k and bk (k = 1,2) are always obtainable from the general root expressions of the quadratic equations.

4.3 SVD RELATIONSHIPS BETWEEN X AND

In the previous section, we have derived the explicit expressions of ak, ak etc. in function of the signal parameters. Here, we are going to compare directly the singular values/vectors of the filtered and unfiltered data matrix.

Recall t h a t V g -" UK121 . For K = 2, we define the 2 x 2 matrix 121 as [Wki], where wki is the element on the kth row and ith column of 121. After a series manipulations, we get the general formulae below, which clearly reveal the relationships between the SVD of the prefiltered data matrix and the original one:

1To realize a rectangular bandpa~s filter, a filter matrix of infinite dimension should be used. This is impossible in practice and is used here to simplify the theoretical analysis. This is the reason why the filter is called ideal.

396 H. Chen et al.

filter characteristics [[ a. select both peaks b. select the 1st peak c. select the 2nd peak

/ ~II =

~22 ----- ~ 12 --

/~21 --"

7711 ----

T~22 =

?']12 ----

~21 --

r ~r2=

I~111 = I~221= I~121 = I~211=

/~22

/~12

~21

ffl 0" 2

0 0 0

ffl 0

0

/~22 0 0

~r 2 0

Table 1: Three ideal cases of filtering using an ideal rectangular filter.

] ~ = ~ ( ~ ~ + ~ ~ ) + I~1 + - ~2~)~

i ai2~ii - a22r]22 [W11[ 2 = I~J22[ 2 = 2 +

2 ~ / 4 4 ~ 1 ~ 1 1 ~ + (4~11 - ~ ) ~

[~2112 _ 1~212 _ i__ 0"27"]11- 0"27"]22 (6) 2 2~/4a2a22[r]2112 + (o.12~]11_ 0.2T]22)2

H HHvi depends on both the signal parameters and the In general, the factor ~?ki = V k Hr fdter matrix. Expressing vk as a linear combination of t~, we get:

~Tki - b;kl bvil~ll + b:k2bvi2~22 "t" b:kl bvi2~12 W b;k2bvn~21

For simplification, we consider three ideal cases of filtering:

a. both sinusoids (peak 1 and 2) are successfully selected.

b. the first sinusoid (peak 1) is successfully selected and the second sinusoid (peak 2) is completely suppressed.

c. the second sinusoid (peak 2) is successfully selected and the first sinusoid (peak 1) is completely suppressed.

By successful selection and complete suppression we mean that prefiltering retains the wanted peak without distortion. This can be realized under the assumption that an ideal rectangular filter is designed and the two peaks of the original signal have both zero damping factors such that infinitely long tails in the spectrum are avoided. The values of the

Bandpass Filtering for the HTLS Estimation Algorithm 397

factors/3ki and thus r]ki for the three ideal cases of filtering are listed in Table 1. Substitut- ing the values of r]ki into (6) yields neat expressions of the SVD relationship between :K and X as listed in the same table. These relationships clearly indicate that the wanted peaks are emphasized while the unwanted peaks are suppressed successfully after prefiltering via matrix multiplication.

3 O

I (7- I o __ 25 ii

20 Jl 0"I + -"

1 0 E m m ~ m l ~ . . ~ NH-Xm ~ m~l-lm m u ~ m m ~ K - ~ i ~ J | .x-q m x *

5 1 - : i ! :: ~ . x / i i~ - u 2 '"

_ x' xxxx,~ '<x 1..~ ~i • . . . . x,(x ,( o l ~~^ . . . . i~ . . . . -• , ,~ .~ , , o , o 0 , , ~

-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 f2 (Hz)

Figure 1: al , a2, al and a2 of a two-peak signal with fl = -0 .2 Hz, f2 variable, dl = 0.03 rad/s, d2 - 0.05 rad/s, and Cl = c2 = e i~ = 1. ft = f l - 0.1 Hz, fu = fl + 0.1 Hz, q = 99, L = 62, M = 60, N = 121.

Of course the results listed in Table 1 are derived under the following assumptions: (1) the filter is an ideal rectangular bandpass filter preserving the wanted peaks and ruling out the unwanted peaks in the spectrum; (2) the damping factors of the sinusoids are zero which enables a successful selection. In practice, the filter passband is not exactly rectangular and the signal to be processed is damped. But a nice feature of SVD is its robustness with respect to mild violations of these conditions. If both peaks are to be selected and the two peaks are sufficiently narrow in the spectrum, which means that dl and d2 are close to 0, then the two peaks can be enhanced considerably by a proper design of the filter preserving the two peaks and suppressing the noise outside the passband. If only one of the two peaks is wanted, the selected peak can also be emphasized, provided that the wanted and unwanted peaks are sufficiently narrow and well separated. To give an intuitive idea of these results in general cases, we show in Fig. 1 the variations of ak and bk when one of the two peaks varies along the frequency axis and peak 1 is selected. It can clearly be seen that al ~ a l , a2 ~ a2 when f2 is around fl = -0 .2 Hz or inside the filter passband, that is, when both peaks are selected; while al ~ al, a2 ~ 0 when f2 is around fl + 0.5 Hz = 0.3 Hz, which corresponds to the case that the two peaks are nonoverlapping and only the first peak is selected.


A prefiltering technique has been presented that performs filtering via multiplication with a full-rank FIR filter matrix HE or IIR filter matrix HI. Without increasing the rank of the

398 H. Chen et al.

Hankel data matrix, this prefiltering technique successfully reduces the noise outside the filter passband as demonstrated by the theoretical analysis. The proposed filter matrices HE and Hx also significantly improve the resulting resolution at comparable parameter accuracy, as confirmed by simulations [6] (omitted due to space limitation).

Acknowledgments

This paper presents research results of the Belgian Programme on Interuniversity Poles of Attraction(IUAP-nr.50 and 17), initiated by the Belgian State, Prime Minister's Office for Science, Technology and Culture. The scientific responsibility rests with its authors. The first author is a Ph.D. student funded by the Research Council of Katholieke Universiteit Leuven. The second author is a Research Associate with the N.F.W.O. (Belgian National Fund for Scientific Research).

References

[1] R. Kumaresan and D. W. Tufts, "Estimating the parameters of exponentially damped sinusoids and pole-zero modeling in noise", IEEE Trans. Acoust. Speech Signal Pro- cess., vol ASSP-30, No. 6, Dec. 1982, pp. 833-840.

[2] Y. Hua and T. P. Sarkar, "Matrix pencil method for estimating parameters of exponentially damped/undamped sinusoids in noise", IEEE Trans. Acoust. Speech Signal Process., vol ASSP-38, No. 5, May 1990, pp. 814-824.

[3] S. Y. Kung, K.S. Arun, and D.V. Bhaskar Rao, "State-space and SVD-based approximation methods for the harmonic retrieval problem", J. Opt. Soc. Amer., Vol.73, No.12, Dec. 1983, pp. 1799-1811.

[4] R. De Beer, D. Van Ormondt, W. W. F. Pijnappel, and J. W. C. van der Veen, "Quantitative Analysis of Magnetic Resonance Signals in the Time Domain", Isr. J. Chem. 28, 1988, pp. 249-261.

[5] S. Van Huffel, H. Chen, C. Decanniere and P. Van Hecke, "Algorithm for Time-Domain NMR Data Fitting Based on Total Least Squares", J. Magn. Reson. A, Sept. 1994 (in press).

[6] H. Chen, S. Van Huffel, and J. Vandewalle, "Bandpass Prefiltering for Exponential Data Fitting", ESAT-SISTA Report TR-94-32, ESAT Lab., K. U. Leuven, Belgium, 1994.

[7] W. W. F. Pijnappel, "Quantification of 1-D and 2-D magnetic resonance spectroscopic data", Ph.D. thesis, Feb. 1991, Technical University of Delft, The Netherlands. Hol- land.


399

S T R U C T U R E P R E S E R V I N G T O T A L L E A S T S Q U A R E S M E T H O D A N D I T S A P P L I C A T I O N T O P A R A M E T E R E S T I M A T I O N

H. PARK Computer Science Dept. Univ. of Minnesota, Minneapolis, MN 55455, U.S.A.

J. B. ROSEN Computer Science Dept. Univ. of Minnesota, Minneapolis, MN 55~55, U.S.A. and Dept. of Computer Science and Engineering Univ. of California, San Diego La Jolla, CA 92093, U.S.A.

S. VAN HUFFEL ESA T Laboratory Katholieke Universiteit Leuven 3001-Heverlee, Belgium.

ABSTRACT. The Total Least Squares (TLS) method is a generalization of the least squares (LS) method, and it minimizes II[EI r]llF so that (b+ r) e Range(A + E), given A e C mxn, with m _ n and b E C m• The most popular TLS algorithm is based on the singular value decomposition (SVD) of [AIb ]. However, in applications where the matrix A has a special structure, the SVD based methods may not always be appropriate, since they do not preserve the structure. Recently, a new formulation, called Total Least Norm (TLN), and algorithm for computing the TLN solution have been developed. The TLN preserves special structure of A or [AIb], and can minimize a measure of error in the discrete Lp norm, where p = 1, 2 or oo. In this paper, we study the application of the TLN method to various parameter estimation problems in which the perturbation matrix E or [EIr ] keeps the Toeplitz structure like the data matrix A or [AIb ]. In particular, the L2 norm TLN method is compared with the ordinary LS and TLS method in deconvolution, transfer function modeling and linear prediction problems, and shown to improve the accuracy of the parameter estimates by a factor 2 to 40 at any signal-to-noise ratio.

KEYWORDS. Deconvolution, Hankel, least squares, linear prediction, transfer function, Toeplitz, parameter estimation, total least norm, total least squares.

400 H. Park et al.

1 F O R M U L A T I O N OF T O T A L L E A S T N O R M ( T L N ) P R O B L E M S

An important data fitting technique frequently adopted in signal processing applications for solving an overdetermined system of linear equations is the Total Least Squares (TLS) method [2, 5]. The TLS problem can be stated as that of finding E and x, such that

rain II[EI r]lIF, (1) E,x

where r = b - ( A + E ) x , given A E C mx'~, with m_> n, and b E C m• This problem allows the possibility of error in the elements of a given matrix A, so that the modified matrix is given by A + E, where E is an error matrix to be determined. The generally used computational method for solving TLS is based on the singular value decomposition (SVD) of [A[b] [5]. In applications where the matrix A has a special structure, the SVD based methods may not always be appropriate, since they do not preserve the special structure. In fact the matrix E obtained from the SVD will typically be dense, with no structure, even when A is structured.

In many applications in signal processing and system identification, the matrix A has Toeplitz or Hankel structure. Recently, a new formulation, called Total Least Norm (TLN), and algorithm for computing the TLN solution have been developed [4]. The TLN preserves special structure of A and can minimize the error in the discrete Lp norm, where p = 1, 2 or oo [4]. A theoretical justification and computational testing of the TLN algorithm confirm that it is an efficient method for solving problems where A or [Alb ] has a special structure, or where errors can occur only in some of the elements of A or [A [ b].

In this paper, the TLN method is applied to various problems in system identification and signal processing in which the data matrix is Toeplitz and the perturbation matrix E or [E[r] preserves the same structure. The presented results also hold for Hankel structures, since Hankel matrices simply transform to Toeplitz matrices by permutations. In Toeplitz matrices q (< m q- n - 1) elements of A are subject to error, and a vector a E Ca is used to represent the corresponding elements of the error matrix E. If many elements of E must have the same value, then q is the number of different such elements. The vector a and the matrix E are equivalent, in the sense that given E, a is known, and vice versa. Now, the residual vector r = b - (A + E ) x , is a function of (a, x). Let D �9 R qxq be a diagonal matrix that accounts for the repetition of elements of a in the matrix E. Then the TLN problem can be stated as"

p

where II. I1~ is the vector/)-norm, for p = 1, 2, or e~. For p = 2, and a suitable choice for D, problem (2) is equivalent to the TLS problem (1), with the additional requirement that the structure of A must be preserved by A + E.

2 T L N A L G O R I T H M

In the iterative algorithm for solving the TLN problem [4], the vector E x is represented in terms of c~. This is accomplished by defining a matrix X 6 C mxq such that X a = E x .

H. Park et al. 401

A l g o r i t h m T L N Input - A TLN problem (2), with specified matrices A, D, vector b, and tolerance e Output - Error matr ix E and vector x

1. Set E = 0, a = 0, compute x from (5) and X from x, and set r = b - Ax.

2. r e p e a t

(a) minimize ( X A + E ) ( A O ~ ) ( - r ) l I ,,,~,A~ D 0 Ax + Da "

P (b) Set x : = x + A x , a : = a + A a .

(c) Construct X from x and E from a. Compute r = b - (A + E)x.

until (IIA~II, IILX~II ~ ~)

The matr ix X consists of the elements of x, with suitable repetition, giving X a special structure. If E is Toeplitz, then X is Toeplitz too. In fact, E and X have exactly the same number of nonzero elements.

In the minimization (2), a linear approximation to r(a, x) can be used. Let Ax represent a small change in x, and A E represent a small change in the variable elements of E. Then we have X A a = (AE)x , where A a represents the corresponding small change in the elements of a. Then, neglecting the second order terms in IIAall and IIAxll,

r ( a + Aa , x + Ax) = r(c~, x ) - X A a - (A + E)Ax. (3)

The linearization of (2) now becomes:

min M + M = (4) A~,Ax Ax Da ' D 0 " p To start the iterative algorithm, the initial values of E = 0, and the least norm value of x = xt,~ can be used, where xl~ is given by

m~n lib- Axllp. (5) x

The Total Least Norm (TLN) algorithm is summarized in Algorithm TLN.

For p = 2, the LS problem (4) can be solved efficiently by a QR factorization of the matr ix M when A + E has full column rank, since M has full column rank in this case. Also for p = 2, the TLN problem (2) can be stated in terms of minimizing the differentiable function

i H lo~HD2a (6) ~ ( ~ , z ) = ~r r + 2

where, r = b - (A + E)x. The first-order optimality conditions for a local opt imum of ~p(a, x) are the vanishing of the gradients V ~ and Vx~. Using the above relations, these conditions become"

Vacfl - - X H r 4- D2a = 0 and V~; = - ( A + E)Hr = O. (7)

402 H. Park et al.

Now consider the LS problem in Step 2a of Algorithm TLN. The corresponding normal equations are:

M H M Az - D ~ = - V=~ ' (8)

where the last equality follows directly from (7). When the matr ix M has full rank, (8) has a unique solution for (A~ T AzT) T, which will be zero if, and only if, the r ight-hand side of (8) vanishes. This means that convergence of the TLN algorithm is equivalent to satisfying the optimality conditions (7). Additionally, it is shown in [4] that Step 2a is essentially a Gauss-Newton method. For solving TLN for p = 1 and p = e~, see [4].

3 T L N F O R S T R U C T U R E D V E C T O R B

Frequently, s tructure is imposed not only on A but also on the right-hand side vector b or on [AIb ]. For example, in the LS linear prediction problem [3], we need to solve

mln IIAx - bll 2 (9) x

where A 6 C re• m _ n and either [AIb ] or [ b l A ] is Toeplitz . We now show how to modify the TLN algorithm so that it can treat possible errors in some or all elements of b.

We introduce a vector/~ representing possible errors in selected elements of b. This is similar to c~ representing errors in A. Suppose different errors can occur in l( _ m) elements of b. The error vector ~ 6 C z represents the error in b. The relation between ~ and b is given by a matr ix F 6 R re• so that the error in b is the same as F/3. The ( i , j ) t h element of F is one if flj is the error in bi, otherwise, it is zero. Initially, E , c~ and/3 are all zero, and the residual ~ = r = b - Ax. In general,

= ~(a , ~ , x ) = (b + F~) - (A + E ) z = r + F~.

In ideal situations, we can require that ~ = 0 since F ~ can play the role of the residual vector, r = ~ - F~ = b - (A + E)x . However, this is not always possible, due to the special structure tha t is imposed on F~. For the linear prediction problem with [AIb ] or [b lA ] Toepl i tz , it is always possible to find a Toeplitz perturbat ion [EIFI~ ] such that

b + F~ 6 Range(A + E).

Therefore, we can expect ~ to become zero when the solution is obtained and can formulate the problem as the following weighted least squares problem

II | (10) ~'~'= \ D2~

where w is a large number [i, 6] and the diagonal matrices DI and D2 account for the repetition of the elements of ~ and/3 in E and F~, respectively.

Now, we discuss backward prediction (the same results hold for forward prediction as well) where we need to impose the Toeplitz structure on [AIb ]. Note that the per turbat ion in b can be represented using the perturbat ion in A except for its first component. Specifically, if all the elements on the different diagonals of [AIb ] are different and subject to error

Structure Preserving TLS Method 403

A l g o r i t h m T L N - L P I n p u t - A Total Least Norm problem (2), with specified matrices A, D, vector b, and tolerance e, such that [ A I b ] is Toeplitz. O u t p u t - Error matr ix E, error/31, and vector x, such that [A -F E Ib +/3] is Toeplitz.

1. Choose a large number w Set E = 0, c~ = 0, ;31 = 0, compute x from (5) and X from x, and set ~ = b - Ax .

2. r e p e a t

(a) minimize II D 0 0 /~fll + D lip

(b) Set x "-- x -FAx, c~ := c~ H-/kc~, # := ~, #1 "= #1 -F A#I .

(c) Construct E from c~, and X from x. Compute ~ = (b + fl) - (A + E ) x .

until (llZx~ll, I1~11, IlZX#lll _< ~)

and ~ = (OZl...OZn.Fm_l) T, E is Toeplitz with first row = (c~,~...C~l) and first column -- ( c ~ . . "c~,~+m-1)T, F = I , ~ = (~1"" "~m) T, then since j3i = cq-1, i - 2 , . . . , m , we have

: + : o . . . o) + ( oi• ) I(m-1) x (m-l) 0(m-1)xn

Also, from (11), we have

x ~ - F~# + (A + S ) ~ = (X - Y ) ~ + (X + S ) ~ - ~#1~.

Algori thm TLN-LP summarizes the TLN computations for solving a Toeplitz structure preserving linear prediction problem where D 2 = diag(2, 3 , . . . , n + 1, n + 1 , . . . , 3, 2, 1) E R (m+n-1)• , provided all m -F n - 1 elements of A are different and subject to error. For solving TLN problems with an arbitrarily structured right-hand side vector, see [4].

4 C O M P U T A T I O N A L R E S U L T S

The accuracy of the TLN algorithm (implemented in MATLAB) is compared with that of the LS and TLS methods in several signal processing and system identification problems.

Test procedure. We start from m -F n - I data samples zi, arranged in an m • n Toeplitz matr ix Ac and set up the zero residual set of equations Acxc = bc by choosing a specific exact solution xc or right-hand side vector be. Random errors, normally distributed with

2 mean zero and variance a~, are then added to the data zi and be, generating the perturbed set A x ,~ b. We denote the LS, TLS and TLN solution of this per turbed set by xts, x~zs, xtz,~ respectively. The relative errors 64 = - with c~ = ls, t ls or t ln are averaged over

- J J ~ c l J 100 runs per a~. These tests are repeated for progressively larger a~. The L2 norm TLN solution is computed in 10 iterations and M always has full column rank.

404 Structure Preserving TLS Method

~ ggr ei ,~_r el ,~_r e l dr v v~tI~ va;tln .g e-10 7.12e-8 7.12e-8 2.04e-9 e-9 6.61e-5 6.61e-5 1.87e-6 e-8 7.19e-4 7.19e-4 1.98e-5 e-7 7.12e-3 7.12e-3 1.97e-4 e-6 6.95e-2 6.94e-2 1.91e-3 e-5 8.03e-1 6.53e-1 2.120e-2 e-4 3.66e+1 8.29e+0 1.86e-1

6 d [ f i ~ j r el ,~ j r ei ~ t l s ~ t l r t

1.34e-7 1.34e-7 4.02e-9 1.18e-4 1.18e-4 3.87e-6 1.30e- 3 1.30e- 3 3.54e- 5 1.35e-2 1.35e-2 3.74e-4 1.27e-1 1.27e-1 3.72e-3 1.29e+0 1.15e+0 3.86e-2 9.27e+1 9.62e+0 3.53e-1

'.'J~l~ "Jtfn 1.98e-8 1.98e-8 6.8e-10 1.98e-5 1.98e-5 6.06e-7 2.05e-4 2.05e-4 6.40e-6 1.99e-3 1.99e-3 6.87e-5 2.00e-2 2.01e-2 6.63e-4 2.32e-1 1.95e-1 6.94e-3 6.19e+1 3.06e+1 6.54e-2

(a) (b) (c)

Table 1: Relative accuracy of the (a) LP solution, (b) damping factor, (c) frequency in the following LP problem: m = 42, n = 8, zt = ~=1 e(-dk+2r~J'L-fyk)t, 1 _< t _< 50, the first column of Ae is (z42"" z2zl) T and bc = (zs0 "" z9) T, so [Aelbe] is Toeplitz. The 8 pairs (dk, fk) are (0.1, 0.5), (0.2, 0.4), (0.3, 0.3), (0.35, 0.1), (0.4, 0.2), (0.5, 0.45), (0.05, 0.25), (0.45, 0.05).

Test problems. We first consider a linear prediction (LP) problem, as described in Table 1. In addition, we show the relative accuracy of damping factor and frequency estimates obtained by rooting the computed LP polynomial in each run, as in Prony's method. Ob- serve t ha t TLN improves the accuracy of the TLS estimates with a factor 30 to 40 for noise standard deviations up to 10 -5, while the differences in accuracy between TLS and LS remain negligible! For a~ = 10 -4, the frequency accuracy improves even more than 400 times. Algorithm TLN-LP can also be used for computing the rank ( n - 1) Toeplitz approximation of any m • n Toeplitz matrix T, which is obtained by solving Az ,~ b, where A consists of the first n - 1 columns of T and b is the last column of T. Then the TLN solution yields the minimum perturbation [EI~ ] that makes [A + E Ib + ~] rank deficient.

Next, we explore the use of TLN in time-domain system identification. If the process can be modeled as a linear, time invariant, causal, finite-dimensional system with zero-initial state, then an impulse response model may be used:

y(t) = ~ h(k)~(t- k). (12) k - 0

The system is identified if its impulse response h(k), k = 1 , . . . , n can be estimated from observations of the inputs u(t) and outputs y(t) over a certain interval of time t = - n , . . . , m - 1. This so-called deconvolution problem is essentially reduced to a problem of solving a set of linear equations Ax ,~ b by writing (12) for t = 0 , . . . , m - 1, i.e., Aij = u ( i - j ) = zn+i-j, bi = y ( i - 1) and xj = h(j - 1) for 1 _< i _< m and 1 _< j _ n. Taking into account the Toeplitz structure of the data matrix A, as done by TLN, improves the accuracy of the TLS and LS impulse response estimates by a factor 3, as shown in Table 2(a). If u(t) is zero for t < 0 then the strictly upper triangular part of A is zero. The TLN is able to take also this information into account, thereby improving the TLS and LS estimates with a factor 2 (see Table 2(b)).

Another frequently used model to describe a system behavior is the transfer function model which essentially expresses an autoregressive moving average (ARMA) relation be-

Structure Preserving TLS Method 405

~ ~z~ l ~ x ~ ~r~l ~ '~ t l n

e-10 1.6e-10 1.6e-10 5.5e-ll e-8 1.76e-8 1.76e-8 6.41e-9 e-6 1.81e-6 1.81e-6 5.85e-7 e-4 1.51e-4 1.51e-4 5.98e-5 e-3 1.77e-3 1.77e-3 6.22e-4 e-2 1.71e-2 1.71e-2 5.67e-3 e-1 1.85e-1 1.83e-1 6.11e-2

2.7e-10 2.7e-10 1.4e-10 3.10e-8 3.10e-8 1.67e-8 3.28e-6 3.28e-6 1.66e-6 3.32e-4 3.32e-4 1.70e-4 2.98e-3 2.98e-3 1.50e-3 3.04e-2 3.04e-2 1.56e-2 3.34e-1 3.81e-1 1.66e-1

(~) (b)

Table 2: Relative accuracy of the impulse response in the following deconvolution problem: m = 30, n = 20; h = xe =(1.9 3.3 4.4 5.4 5.9 6.2 6.3 6.1 5.8 5.6 5.3 5.0 4.85 4.6 4.0 3.4 1.8 1.0 0.2 0.0) T is the exact impulse response. The inputs u(t) , t = - 2 0 , . . . , 29 are normally distributed zero-mean noise of unit variance: (a) u(t) # 0 Yt, (b) u(t) = 0 Yt < O.

tween the inputs u(t) and outputs y(t) fed to the system. In polynomial form, we have:

(1 + alq -1 + . . . + an, q - n ' ) y ( t ) - (blq -1 + . . . + bnbq-nb)u(t) (13)

with q-1 the backward shift operator. Since q - l y ( t ) = y ( t - 1), (13) reduces to :

y( t ) + a l y ( t - 1) + . . . + an~y(t - na) = blu( t - 1) + . . . + bnbu(t - nb) (14)

Writing out (14) for t = to , . . . , to + m - 1 with to a given offset, one obtains an overdetermined m x (ha + rib) set of equations: y = []I1 I U1]x with y = ( y ( t o ) , . . . , y ( t o + m - 1)) T, x = ( a l , . . . , a n ~ , b l , . . . , b n b ) T a n d t h e t t h r o w o f [ Y l l U 1 ] i s [ - y ( t - 1 ) , , - y ( t - n a ) , u ( t - 1) , . . . , u ( t - rib)]. This matrix is structured as a [Woeplitz I Toeplitz] matrix. For simplicity, we consider only models with a pure Toeplitz data matrix, i.e.: (a) pure MA models with na = 0 and modeling equations: y = Ulx. (b) pure AR models with nb = 0 and modeling equations: y = Y2x. x = ( - a n ~ , . . . , - h i ) T.

The tth row of Y2 is [y(t - n ~ ) , . . . , y( t - 1)] such that [Y2 I Y] is Toeplitz (the same as the LP equations).

(c) ARMA models with n b = 1 and modeling equations: [ y I - Y ~ ] x ~ = u with u = (u(to - 1) , . . . , u(to + m - 2)) T and x 1 " - (1, a l , . . . , a n a ) T / b l .

In Table 3, an example of each class is considered. The beneficial effect of taking the structure into account is clearly shown at any signal-to-noise ratio (SNR). Indeed, for all noise variances considered the TLN estimates of the coefficients hi, bi are 2 to 4 times more accurate than the corresponding TLS estimates. Note also that the gain in accuracy obtained by ordinary TLS compared to LS is negligible for high SNR. Only at very low SNR, the gain in accuracy can improve up to a factor 8.


H. Park is supported in part by the National Science Foundation grant CCR-9209726. B. Rosen is supported in part by the Air Force Office of Scientific Research grant AFOSR-91- 0147 and the Minnesota Supercomputer Institute. S. Van Huffel, Research Associate with the N.F.W.O. (Belgian National Fund for Scientific Research), is supported in part by the

406 H. Park et al.

Xxra x~rel x~ret O'u v .~ t l s v .~ t l n

e-10 6.1e-10 6.1e-10 1.9e-10 e-8 6.91e-8 6.91e-8 2.43e-8 e-6 7.31e-6 7.31e-6 1.97e-6 e-4 6.37e-4 6.37e-4 2.01e-4 5e-4 3.34e-3 3.30e-3 1.05e-3 e-3 2.02e-3 5e-3 1.06e-2 e-2 2.25e-2 e-1

~ ~ , re l ,~, , ,rel ,~ , , ,rel "~ ~ ' ~ t l s ~

3.9e-ll 3.9e-ll 2.3e-ll 4.65e-9 4.65e-9 2.56e-9 4.08e-7 4.08e-7 2.27e-7 3.78e-5 3.78e-5 2.45e-5 2.02e-4 2.02e-4 1.21e-4

v ,~ t l n

6.38e-3 6.39e-3 4.53e-2 3.70e-2 1.04e-1 6.67e-2

3.66e-4 3.58e-4 2.33e-4 2.79e-3 2.10e-3 1.18e-3 7.12e-3 4.23e-3 2.22e-3 3.99e-1 5.59e-2 2.70e-2

8.4e-10 8.4e-10 3.5e-10 8.88e-8 8.88e-8 3.31e-8 7.00e-6 7.00e-6 2.94e-6 8.19e-4 8.15e-4 3.42e-4 4.33e-3 4.20e-3 1.75e-3 8.26e-3 8.13e-3 3.32e-3 7.57e-2 4.10e-2 1.71e-2 2.29e-1 8.02e-2 3.48e-2

(~) (b) (~)

Table 3: Relative accuracy of the parameters of the following transfer function models: (a) 9~Y(t ) = (q-1+.99q-2 +.98q-3 +.97q-4 +.96q-S +.92q-6 +.87q-r)u( t ) , m = 30, n =nb = 7 and u(t) = ~Tk= 1 ak cos(21rfk( t- 1)A), t = 1 , . . . , 36 and A = .01. The 7 pairs (ak, fk) are (~-o ~, 42), (~-o.o~, ~2), (~-o.~, 52.1), (~-o,, 60), (~-o ~, 90), (.1~ -~ 92), (.01~ -o.o~, s~); (b) (1 + .98q -z + .95q -2 + .92q -3 + .9q -4 + .85q -s + .8q -6 + .76q-r)y(t) = O, m = 60, n = na = 7 and y(t) normally distributed zero-mean noise of unit variance; (c) (1 + .99q -1 + .98q -2 + .97q -3 + .96q -4 + .92q -s + .9q-6)y(t) = .063u(t), m = 30, n = n= +nb = 6 + 1 = 7 and u(t) normally distributed zero-mean noise of unit variance.

Belgian Programme on Interuniversity Poles of Attraction (IUAP-nr.50 & 17), initiated by the Belgian State, Prime Minister's Office for Science, Technology and Culture.

References

[1] A.A. Anda and H. Park. Self-scaling fast rotations for stiff least squares problems, to appear in Linear Algebra and its Applications.

[2] G. H. Golub and C. F. Van Loan. Matrix Computations, second edition. Johns Hopkins University Press, Baltimore, 1989.

[3] S. Haykin. Adaptive Filter Theory, second edition. Prentice Hall, Englewood Cliffs, NJ, 1991.

[4] J. B. Rosen, H. Park and J. Glick. Total least norm formulation and solution for structured problems, Supercomputer Institute Research Report UMSI 93/223, University of Minnesota, November, 1993, revised July, 1994.

[5] S. Van Huffel and J. Vandewalle. The Total Least Squares Problem: Computational Aspects and Analysis. SIAM, Philadelphia, 1991.

[6] C.F. Van Loan. On the method of weighting for equality constrained least squares problems. S I A M J. Numer. Anal., 22:851-864, 1985.


407

P A R A M E T E R E S T I M A T I O N A N D O R D E R D E T E R M I N A T I O N I N T H E L O W - R A N K L I N E A R S T A T I S T I C A L M O D E L

R.J. VACCARO, D.W. TUFTS, A.A. SHAH Department of Electrical Engineering The University of Rhode Island Kingston, RI 02881, U.S.A. vaccaro@ele, uri. edu

ABSTRACT. In this paper we consider the problem of estimating the min-norm solution to a low-rank, linear statistical model. We calculate the statistics of the solution as a function of the statistical characterization of the matrix containing observation noise. This result is obtained using a perturbation expansion of the SVD of a finite-sized matrix, and not on the basis of asymptotics which require the data record to become large. This approach can handle arbitrary correlation of the elements of the noise matrix. In performing this analysis, we assume that the rank r of the underlying noise-free matrix is known. In practice, the rank r of the underlying matrix may not be known a priori and will have to be estimated from the given noisy matrix. We present a new method for order determination. A significant feature of this method is that it allows the user to trade off probability of detection for probability of false alarm.

KEYWORDS. Linear statistical model, perturbation expansion, order determination.


We consider the problem of estimating parameters from noisy data. Our modeling procedures are based on the following principles:

P1 We do not try to model the data itself. Rather we only try to model the signal portion of the data. In many applications this leads to low-rank models.

P2 If we have some statistical knowledge of the observation noise, then we look for the minimum complexity signal model such that the residual unmodeled data is consistent with our knowledge of the background noise. This idea provides a foundation for order determination.

In this paper we apply the above principles to the problem of estimating the min-norm solution 0 to a low-rank, linear statistical model, which is described below. We first calculate

408 R.J. Vaccaro et al.

the statistics of the solution 0 as a function of the statistical characterization of N, the matrix containing observation noise. This result is obtained using a perturbation expansion of the SVD of a finite-sized matrix (described below), and not on the basis of asymptotics which require the data record to become large. In addition, this approach can handle arbitrary correlation of the elements of N. In performing this analysis, we assume that the rank r of the underlying noise-free matrix is known. In practice, the rank r of the underlying matrix may not be known a priori and will have to be estimated from the given noisy matrix. We present a new method for order determination which is outlined below. A significant feature of this method is that it allows the user to trade off probability of detection for probability of false alarm.

To give a mathematical formulation of the problem, consider the following linear statistical model

y = H m x n O , m > n. (1)

The parameter vector 0 is to be estimated from data contained in H and y. In the absence of observation noise on the data, we assume that the rank of H is r < n. Thus we refer to (1) as a low-rank model, and we are interested in the minimum-norm solution for O.

In practice, the data will contain perturbations (noise), and we assume that both H and y are perturbed as follows

[H ~]=[H y]+N (2) where [I~I ~] is a matrix containing observed data, and N is a matrix of noise samples. We assume that the elements of N are Gaussian random variables with an arbitrary, but known, correlation. Since the modal matrix H is low-rank, another decomposition of the data which reflects principle P1 mentioned above is

[H Y]=CH ~]+W (3) where [I:I St]is a rank-r approximation to [I=I ~r] and W is the residual error. This low-rank approximation is obtained from the SVD as follows

[ffI y] ~---[C 1 02 ] [~I o , 2 1 [ * r ~ VT ] (4)

where ~1 is r x r. Then

[I~ ~ ] = O~ ~gT. (5)

For future reference, the SVD of the noise-free matrix is

o o v r = u~ (6)

The estimator of interest is the min-norm solution to the low-rank model equation I:I0 = St. The min-norm solution is expressed in terms of the projection matrix P2 d.~ ~2~r T as follows:

= -IIP2en+l (7) r f, en+l 2en+l

The Low-rank Linear Statistical Model 409

where I T is an identity matrix with the last row removed, and en+l is the standard basis vector with a 1 in the (n-t- 1)st element. To our knowledge, the first use of 0 in the context of signal parameter estimation was by Kumaresan and Tufts in [1]. The problem we consider is a special kind of total least squares problem in which the data matrix is nearly rank deficient. A comprehensive survey of total least squares problems can be found in [2].

2 B I A S A N D V A R I A N C E OF

Our results are based on a perturbation expansion for the SVD that was developed in a series of papers [3, 4]. We have previously applied these matrix perturbation ideas to adaptive hulling of interference [5], adaptive detection [6], and performance analysis of array signal processing algorithms [7, 8].

In order to be useful, the perturbed subspaces must not be "too far" from the unperturbed subspaces. This will be true if the noise matrix N is "small enough." We have quantified the concepts of "too far" and "small enough" in our previous analysis of the threshold effect [9]. We define the signal subspace S and the orthogonal subspace S • as follows

S ~ ~ span(V1) ,and S • ~ span(V2). (8)

The perturbed subspaces are defined in a similar way using V1 and V2. Basis vectors for the perturbed signal and orthogonal subspaces can be found by appropriately combining the unperturbed singular vectors. In particular,

= span(V1 + V2R), and S -• = span(V2 + VIQ), (9)

where Q and R are matrices whose norms are of the order of Ilgll. The proof of this result can be found in [10].

Because the perturbed signal and orthogonal subspaces are orthogonal to each other, it must be true that the basis vectors for these subspaces are orthogonal. Thus,

(V T + Q T v T ) ( v 1 + V2R) = R + QT = 0 or R = _QT.

We only need to compute Q to some desired accuracy, and then use the above equation to get R. First and second-order expressions for Q, denoted as Q1 and Q2 respectively, were derived in [11] and are shown below

q l = - - ~ l l U T N V 2 , (10)

Q2 = Q1 + ( - -2712vTNTu2uT + N 1 1 u T N v 1 2 7 i - l U T ) N v 2 . (11)

Using the first-order expansion, we can express perturbed projection matrix which appears in (7) as 152 = P2 + Ap2, where

A p 2 L V~QTvT + V1Q1V T, (12)

and 1 = means "equal to first order." Next we expand 0 to first order in AP2 as follows:

ITP2en+IAP2(n + 1, n + 1) ITAP2en+I 1-- OMN + P2(n + 1, n + 1) - P2(n + 1, n + 1) (13)

where M(i, j) denotes the ij element of the matrix M, and OMN is the noise-free min-norm


x lO -3

21 ': ,: : :

1 . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 . . . . . . . . . i . . . . . . . . . . ! . . . . . . . . . . i . . . . i . . . . . . . . . . ! . . . . . . . . . . i . . . . . . . . . . i ........

1 1 . 5

ii Z ! ! ! i ,o Element Number

Figure 1" Bias and variance for the ten elements of 0. Theoretical bias (dashed line) and theoretical variance (solid line). Simulation results are shown with discrete symbols.

solution. Substituting (12) into (13) yields an expansion for 0 which is first order in N.

Using this expansion we can derive the covariance matrix C ~J E l ( 0 - OMN)(O- OMN)T]. In order to calculate C we need to evaluate expressions such as E[NvvTNT], where v is a fixed vector. In many cases, some elements of N are repeated. For example, N might have Hankel or Toeplitz structure. Let n be a vector containing the distinct noise samples in N. We assume that Rn ~ E[nn T] is known. We define the matrix M1 to satisfy vec(N) - M l n , where vec(N) is a vector consisting of all the columns of N. For any vector v we can write

N v = Mv vec(N) = M v M l n , My = v T | I, (14)

where | denotes Kronecker product, and the size of the identity matrix is the number of row of N. Then we have

E[NvvTN T] = M v M 1 R ~ M T M T. (15)

If the elements of N are i.i.d, random variables with variance a 2 this expression reduces to a2[[v[[ 2. If we expand 0 to second order in AP 2 and use the second-order expansion for P2 derived in [11], we can calculate E[0] and evaluate the bias.

To verify our expressions for bias and variance of {}, we formed a 10 x 11 rank-2 matrix [H y] whose nonzero singular values were 12.4 and 1.02. The additive noise matrix N for each trial contained i.i.d Gaussian random variables with zero mean and variance 0.001. We ran 100,000 trials and computed the bias and variance of the ten elements of 0. These are compared with the theoretical bias and variance in Fig. 1 and the simulation results verify the accuracy of the theoretical expressions.

3 O R D E R D E T E R M I N A T I O N

In the estimation problem discussed above, a low-rank approximation to the data matrix is utilized to draw an inference. In doing so, it is implicitly assumed that the underlying true


rank of the data matrix is known. In practice, this is seldom the case and the underlying true rank is unknown and needs to be determined.

Several automated procedures do exist for rank determination. But most of them suf- fer from a drawback that their performance is poor for short data records. We propose an automated rank-determination procedure that is effective over short data records. The method is nonparametric, thus, widely applicable. The method consists of sequential Con- stant False Alarm Rate (CFAR) tests on the sums of squares of singular values of the data matrix. The thresholds are based on approximate distributions of the sums. The approximations are derived using matrix perturbation ideas, and hold true over short data records and a wide range of SNR, resulting in improved performance for short data records. More details can be found in [12, 13].

Under high SNR conditions, the Perturbed Signal Subspace, spanned by columns of V1 is stable and is more or less determined by the Signal Subspace, the column-space of V1. This in turn stabilizes the Perturbed Orthogonal Subspace, i.e., the space spanned by V2. Hence in different realizations, the singular vectors of V2 may change erratically but the space

~

spanned by V2 remains relatively unchanged, and is closely related to the the column-space of V2, the orthogonal subspace. Thus the energy in the perturbed orthogonal subspace given by

s , = = II [ H (16) i = r + l

is also well defined. It is closely related to the noise energy in the subspace spanned by V2, the orthogonal subspace. The noise energy is given by (n - r ) m a 2. Using matrix

1 S perturbation approximations we quantify this idea and evaluate the distribution of ~ , to be 2 X(m-r)(n-r)' a central X 2 with ( m - r ) ( n - r) degrees of freedom. (For complex data

whose real and imaginary parts are i.i.d, with variance a2/2, the distribution of a--~2 Sr is

Based on this distribution, we can set a threshold T, such that S, < T, with a probability 1 - c~ where c~ is a small positive number. In other words, if rank is r then Sr can be well explained by the noise energy alone and will be below this threshold with high probability. If rank is r + 1 or greater then, due to the additional signal energy, S~ will exceed the threshold with high probability. Based on this idea, we develop a recursive procedure on the set of sums of squares of singular values of data matrix [I=I :~], that is essentially a signal energy detection procedure in enlarging subspaces.

The calculation of Tk requires the knowledge of noise level a 2. The noise level is assumed to be known or estimated independently. If the noise level is unknown, an estimate can be obtained from intervals of time or space or frequency which are unlikely to contain signal energy. If from the physics of the problem, it is known that for some rmax we have r < rmax, then one can estimate the noise level from { 2 n ai )i=rm~.+l" We define Hk as the hypothesis

r S I n - I that the rank of [H y] is k. We define l k~k=o as the cumulative sums of squared singular values of the data matrix [I:I :~]. The k-th sum is defined as Sk = ~=k+l a~. Thus, the first element So in the sequence is the sum of squares of all singular values, and the last element Sn-1 in the sequence is the square of the smallest singular value. We test


n - 1 o n {Sk}k-_O sequentially, starting from the last element Sn-1. The first test in the nested sequence of tests is as follows,

18ttest" H ~ Choose H0 UH1 t3 . . . t 3Hn_ l if S~-1 < T,~-I H I " Choose H , if Sn-1 _ Tn-1 (17)

If we choose H , then we stop and decide that the rank is n. Otherwise, we conduct the next test on Sn-2 and so on. The k th test in the procedure will be

Uhtest �9 H~ ' Choose Ho th H1 U . . .O H~-k if S~-k < T,~-k (18) H~" Choose H~-k+l if Sn-k > Tn-k

We keep testing until we choose H~ for some k and decide that the rank is n - k + 1. We select the set of thresholds n-1 {Tk}k=O according to a level a such that

P(So >_ TolHo) = P(S1 >_ TIlH1) . . . . . P ( S , - I >_ Tn-IlHn-1) = a (19)

At any given test, we interpret the level a as the corresponding individual false alarm level for that test, because P(Sk >_ TklHk) is the probability of choosing Hk+l when Hk is true. As said earlier, under hypothesis Hk (the number of signals or the signal-only matrix rank is k), Sk corresponds to the energy in the perturbed orthogonal subspace.

In order determination there are two basic types of error probabilities, error due to overestimation and error due to underestimation. The implications and importances of these two errors can be radically different in different applications. The proposed method allows the user to set a bound on error due to overestimation which we call the false alarm probability. We refer to an estimate of the model order which is smaller than the actual model order as a miss. The user can determine a value of SNR for which a prescribed value of probability of detection or probability of net error can be obtained. Thus, the user can specify the conditions under which performance goals, specified by error probabilities, can be obtained.

Here, we apply the proposed method to predict the model order for short data records. For each trial we formed a 10 x 15 matrix [H y] whose columns were zero mean, complex Gaussian random vectors with a rank-4 covariance matrix (eigenvalues = 36, 16, 9, 1) and added a noise matrix consisting of i.i.d, random variables with variance a 2. We define SNR

as 10 loglo(~ ). For comparing the performance of the proposed method with AIC, MDL and other

methods we plot the error probability of the proposed method against false alarm rate. In this case, AIC (Pe "~ 0.5), MDL (Pc "~ 0.1), and MDL-W (Pc "~ 0.04) perform poorly and a more important fact is that their performance does not improve much with increase in SNR. The graph of the proposed method is to be interpreted as follows: Given a false alarm rate on the abscissa, the ordinate of the graph gives the resulting probability of error for that particular SNR. As long as the false alarm rate is higher than the minimum of the graph, the user can lower the false alarm rate and achieve a better performance. Note that for very high false alarm levels (very low threshold values), the main contribution to Pc is from error due to overestimation (Pe .~ PEA). The plot of Pc vs PFA is linear and approximately follows the line Pc = PEA. As we lower the false alarm level (increase the threshold values), the contribution to Pc from error due to underestimation increases and


10 0 . . . . . . . . , . . . . . . . . , . . . . . . . . . , . . . . . . .

10 1 " ~ 8 dB

M o , - w !

.~ lO= lOdB ~

_ - - -2 - -~ II

12-14 dB 10'1.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1o 10 "3 10 "2 10 4 0 o

PFA

Figure 2: Comparison of the proposed method with AIC, MDL and MDL-W criterion.

the plot deviates upwards from the diagonal line (Pe - PEA). Here, Pe "- P F A "Jr- Pmis s .

For very low false alarm levels (very high threshold values) most of the error is due to underestimation and P~ ~ Pmiss. Thus, the graph of PevsPFA starts at (PEA, Pe) = (1, 1), follows line Pe = PEA deviating slightly upwards, reaches a certain minimum and then moves upwards with negative slope and ends at (PEA, P~) -- (0, 1), see Fig. 2. The graph of proposed method is to be interpreted as follows: Given a False alarm rate on the abscissa, the ordinate of the graph gives the resulting probability of error for that particular SNR. As long as the false alarm rate is higher than the abscissa of the minimum, the user can lower the false alarm rate and achieve a better performance. AIC and MDL are minimization criteria and thus give a single probability of error and false alarm, and user does not have any control over the false alarm rate, they will be represented by a single symbol for each SNR. Since the probability of error of AIC, MDL and MDL-W does not improve with increase in SNR, in the figure the symbols representing the performance overlap considerably 1 in the figure.

For 10,000 trials, SNR's above 11 dB $4 and $5 are well separated and there is no occurrence of an underestimation (a miss), and P~ = PEA. As the SNR falls below 11 dB, a miss event occurs more frequently and we can clearly see the deviation from the line Pe = PFA. As per the application, the user can choose an appropriate value of false alarm and from the graph can obtain the corresponding value of probability of error for the proposed method. In this example, the performance of the proposed method is superior to the performance of AIC, MDL and MDL-W [14] over a wide range of False Alarm Levels. The performance of AIC, MDL and MDL-W does not improve much as the SNR is increased. But in the proposed method for high SNR cases very low false alarm levels (orders of magnitudes lower than other criterion) can be chosen and still obtain a very low probability of error.

1The criteria used in simulations do not assume the knowledge of noise level. AIC and MDL criteria can be modified for the case of known noise level.


References

[10]

[11]

[12]

[13]

[14]

[1] R. Kumaresan and D. Tufts, "Accurate parameter estimation of noisy speech-like signals," in Proc. IEEE ICASSP'82, pp. 1357-1361, April 1982.

[2] S. Van Huffel and J. Vandewalle, The Total Least Squares Problem: Computational Aspects and Analysis. Philadelphia, PA: SIAM, 1991.

[3] D. W. Tufts, R. J. Vaccaro, and A. C. Kot, "Analysis of estimation of signal parameters by linear prediction at high SNR using matrix approximation," in Proc. IEEE ICASSP'89, (Glasgow, UK), May 1989.

[4] F. Li and R. J. Vaccaro, "Analysis of MUSIC and Min-Norm for arbitrary array geometry," IEEE Transactions on Aerosp., Electron. Syst., vol. AES-26, pp. 976-985, November 1990.

[5] I. P. Kirsteins and D. W. Tufts, "Rapidly adaptive nulling of interference," in High Resolution Methods in Underwater Acoustics, (M. Bouvet and G. Bienvenu, eds.), pp. 217-249, New York: Springer-Verlag, 1991.

[6] I. Kirsteins and D. W. Tufts, "Adaptive detection using low rank approximation to a data matrix," IEEE Transactions on Aerospace and Electronic Systems, vol. 30, pp. 55-67, January 1994.

[7] F. Li and R. J. Vaccaro, "Unified analysis for DOA estimation algorithms in array signal processing," Signal Processing, vol. 22, pp. 147-169, November 1991.

[8] F. Li, H. Liu, and R. J. Vaccaro, "Performance analysis for DOA estimation algorithms: Further unification, simplification, and observations," IEEE Transactions on Aerosp., Electron. Syst., vol. 29, pp. 1170-1184, October 1993.

[9] D. W. Tufts, A. C. Kot, and R. J. Vaccaro, SVD and Signal Processing, II, ch. The Threshold Effect in Signal Processing Algorithms Which Use an Estimated Subspace, pp. 300-321. R.J. Vaccaro, ed., Elsevier Science Publishers, 1991.

G. W. Stewart, "Error and perturbation bounds for subspaces associated with certain eigenvalue problems," SIAM Review, vol. 15-33, pp. 727-764, 1973.

R. Vaccaro, "A second-order perturbation expansion for the SVD," SIAM J. Matrix Anal. Appl., vol. 15, pp. 661-671, 1994.

A. Shah and D. Tufts, "Determination of the dimension of a signal subspace from short data records," IEEE Transactions on Signal Processing, vol. 42, September 1994.

A. Shah, Fast and Effective Algorithms for Order Determination and Signal Enhance- ment. PhD thesis, University of Rhode Island, Kingston, RI, 1994.

D. B. Williams, "Comparison of AIC and MDL to the minimum probability of error criterion," in Proc. of Sixth SSAP Workshop in Statistical Signal 8~ Array Processing, (Victoria, Canada), pp. 114-117, IEEE, Oct. 1992.


415

A D A P T I V E D E T E C T I O N U S I N G L O W R A N K A P P R O X I M A T I O N T O A D A T A M A T R I X

I.P. KIRSTEINS, D.W. TUFTS SACLANT Undersea Research Centre Viale San Bartolomeo 400 1-19138 La Spezia, Italy

University of Rhode Island Dept. of Electrical Eng. Kelley Hall Kingston, RI 02881 USA

ABSTRACT. Using an accurate formula for the error in approximating a low rank component, we calculate the performance of adaptive detection based on reduced-rank nulling. In this principal component inverse (PCI) method, one temporarily regards the interference as a strong signal to be enhanced. The resulting estimate of the interference waveform is subtracted from the observed data, and matched filtering is used to detect signal components in the residual.

A major accomplishment of the work is our calculation of the statistics of the output of the matched filter for the case in which interference cancellation and signal detection are carried out on the same observed data matrix. That is, no separate data is used for adaptation. An example is presented using simulated data where we calculate the mean and standard deviation of the PCI and sample matrix inversion (SMI) method test statistics.

KEYWORDS. Adaptive detection, interference cancellation, reduced-rank, PCI.


In [4, 11, 5] we proposed the Principal Component Inverse (PCI) method for rapidly adaptive nulling of interference that exploited the low rank structure of the interference, whether it occurred naturally or it is induced by a transformation of the data. The PCI method [4, 11, 5] of adaptive detection is based on reduced-rank nulling of interference. A significant advantage of the PCI method is that it achieves much more rapid adaptation [4, 5]

416 LP. Kirsteins and D. W. Tufts

than conventional methods such as adaptive loops [1] and the Sample Matrix Inverse (SMI) method [7].

In the PCI method, the interference component to be cancelled is first regarded as a strong signal to be enhanced. The interference component is estimated using our reduced-rank signal enhancement algorithm [10, 8]. The resulting estimate of the interference waveform is subtracted from the observed data and matched filtering is used to detect the signal components in the residual waveform.

We assume that the observed or constructed data matrix, under the signal absent hypothesis, is of the form

H 0 : Y , , , • (1)

and under the signal present hypothesis is

H I " Y,~• = Y § S § N, (2)

where matrices Y, S, and N represent the low rank interference, signal and background noise components respectively. Data matrices of this structure can arise, for example, in array processing. In array processing, the dements of the kth column of Y could represent the output of the sensors sampled simultaneously, say at time tk. By low rank interference we mean that rank[Y] = r << m i n ( n , K ) or rank[Y] ,~ r << m i n ( n , K ) . See [2] for examples how low rank interference components can arise. We now present the steps of PCI method:

2 P R I N C I P A L C O M P O N E N T I N V E R S E ( P C I ) M E T H O D OF I N T E R F E R - E N C E S U P P R E S S I O N A N D S I G N A L D E T E C T I O N

Let us assume that the matrix Y of formula (1) or (2) is composed of interference values that we wish to suppress, and that Y is strongly low rank, say rank r. In addition, it is assumed that the signal and ambient noise are much weaker than the interference component. Then, more specifically the steps of the PCI method for detecting a signal in any column of the data matrix are as follows:

1) E s t i m a t e t he I n t e r f e r e n c e Component An estimate of the interference contribution to the k th column of the data matrix I2 is

Y~ = ~,~Ty~, (3)

where columns of the matrix Us are the principal left singular vectors of the data matrix

? = [~l~21.. . I~K]. (4)

The columns of ]~ are defined by formula (1) or formula (2) depending on which hypothesis is correct. The columns of Us can be interpreted as perturbed versions of

Adaptive Detection Using Low Rank Approximation 417

the basis vectors that generate the column space of Y. Then the projection matrix UaU T in (4) can be thought of as a matrix filter that enhances the interference part of the data. We also note that the matrix Yr = Us U T ~ is the best rank r approximation to Y in the least square sense.

2) Remove the Interference Component

The interference estimate vector for each column of Y is subtracted from that column to form a residual matrix R, the mth column of which is

r.~ = ~ - ~ . (5)

3) P roces s each Res idua l vec to r rm by forming the inner product of rm and the signal vector S,n

z~ = ~r~r~. (~)

If the signal phase were a random variable, uniformly distributed on (0, 2~r), we would form the magnitude of the projection of rm onto a two dimensional signal subspace.

We also note that the test statistic can be rewritten as

Zm = WpcIYm -" (7)

where

(8)

The filter weights of formula 8 can be intrepreted as a generalized null-steerer, ie., nulls are steered in the direction of the estimated incoming interference.

4) Compare zm Against Threshold

To test for the presence of this signal component, we compare the value of the test statistic z,~ for the mth column test with a threshold value which has been set for a specified probability of false alarm.

In [2, 5] we provide some theoretical justification for the PCI method. We show that in the case of strong, low rank Gaussian interference plus white noise, the optimum weight

vector has the form

WoPT ~ ( I - usUT)sm (9)

where the columns of U8 are the r principal eigenvectors of the interference covariance matrix. The similarity between formulas (9) and (8)should be evident.


2.1 EXTENSION OF THE PCI METHOD TO THE STRONG SIGNAL CASE: THE REDUCED-RANK GLRT

When the low rank interference becomes weaker or the signal becomes stronger, PCI adaptive detection becomes a less accurate approximation to likelihood ratio detection [2, 3]. For such cases the PCI method can be modified. Kirsteins [3] proposed a generalized likelihood- ratio test (GLRT) that is similar in spirit to the PCI method and can be used when there are strong signal components in the data or the low rank interference is weak.

3 P E R F O R M A N C E A N A L Y S I S OF T H E P C I M E T H O D

The goal of this section is to present an analysis of the performance of our Principal Com- ponent Inverse (PCI) method [4, 11, 5] of rapidly adaptive nulling of interference when one has to use the same data set for detection as used for adaptation and also when adaptation and detection are done using separate data sets. Our previous analyses of the PCI method [4, 5] were restricted to the case where detection is done on a data set that is independent of the one used for adaptation. However, in practice, there are many instances where one would like to perform detection on the same data set that is used for adaptation. Naturally, a signal may be present in the data. Some examples include situations where only limited data is available due to, say, non-stationarity of the interference or to efficiently utilize the observed data.

The analysis requires the statistical properties of Us or equivalently, the orthogonal complement of the column space of Us, the basis for which will be denoted here as the columns of the orthonormal matrix [Io. It is difficult to determine the statistics of 0o exactly because the SVD is a highly non-linear operation. The approach we take is to use a first-order perturbation expansion for 0o derived by Tufts, Li, and Vaccaro [9, 6] to obtain the perturbation expansions for the projection operator [~o [[~ and then the PCI method test statistic.

From [9, 6],

Oo ~ Uo - U.E:IvTNTUo, (i0)

where Ua, Uo and Z~s arise from the SVD of Y, which is given by

[ r~. o [V.lVo]~• (11) Y,.,• [u.Ivo],,• 0 Eo nxK

where [U,[Uo], [ValVo] are unitary matrices and E. = diag(al, a 2 , . . . , at), P~o = diag(ar+l,ar+2,. . . ,amin(r, ,g)) with al >_ a2 >_ . . . >_ amin(n,g). It follows from the above approximation for Uo that the projection operator UoUo T can be approximated as

[IoOTo ~ P• - P • E-1U T - UoE'~XvTNTp• (12)

where P = U~,U T and P• = I - P. Formula (12)is gotten by substituting formula (10) into Uo0o T and then dropping second-order terms.

The first and second moments of the PCI test statistic can now be evaluated by noting that ( I - O a O T) = 0oUo T and then substituting formula (12)in place of 0oU T in the previous expression and then (7). The moment calculations have been worked out in [2] assuming


Y is fixed and rank r, and that the elements of N are IID Gaussian with zero-mean and variance a s. Since they are quite tedious and lengthy, we will only present the results. The reader is referred to [2] for details on the specific steps in calculating the moments.

3.1 DETECTION ON THE SAME DATA SET AS USED FOR ADAPTATION

In this case detection is performed on the m th column of l~. Let z be the PCI detection statistic. The moments of z for different cases are given below

3.1.1 Noise Only Case

g [z] = - a : ( n - r)eTb (13)

[z2] : ll oll llfll +

L-5- -+" (n - r) [lleil~ + ( n - r + 1)(bTe) 2] (14)

where b is a K x 1 vector with the m' th element set to one and zeros elsewhere, e T - sTus~-~IV T , cTo = sTuo, and f = VoVoTD. The operator g [ . ] represents the expected value.

3.1.2 Signal Plus Noise Case

g[z] = sTyob - a 2 ( n - r)eTb (15)

+ ~'(~ - ~) [ileil}+ k -1 ffk

( n - r + 1)(bTe) 2] - 2a2eTfcTg + cr2i]eli~l[gl]~ (16) where g = UTYob. Note that all the above calculations are based on the SVD of Y + S.

d

3.2 DETECTION ON A VECTOR NOT CONTAINED IN THE DATA SET USED FOR ADAPTATION

In this subsection, we present the moments of PCI test statistic, z, when detection is done on a vector not used for adaptation.

3.2.1 Noise Only Case Here we assume that the data vector :YK+I is of the form

~'g+1 = Y + n (17)

where y E Range[Y] and the elements of n are lID Gaussian with zero mean and variance a 2. In addition, it is assumed that the elements of the perturbation matrix N are

independent from those in n. The moments are:

g [z] = 0 (18)


(0"2) 2 []d[[~ 0"k

where a - VmE~ "1UTy, c = UTs and d = Ufs.

(19)

3.2.2 Signal Plus Noise Case Here we assume that the data vector ~rK+ 1 is of the form

YK+~ = y + s + n (20)

The moments are:

C[z] - Ildll~ c [(z- IId,, ) ]

(21)

__ ~2 (lldll~ll-II~ + Ildll~ + 411dll~-TvorTXe + 411dl12eTr72e)

) +(0"2) 2 IIUrsll~, ~ + ( n - r)llV~;;XclJ~. (22)

The formulas for the moments of the detection test statistic for cases (1) and (2) above can be used to compare the differences in performance between the two cases.

4 C O M P A R I S O N OF T H E P C I A N D SMI M E T H O D S

We now compare the PCI and SMI [7] methods on the basis of the theoretically calculated means and variances of the test statistic. We consider a scenario where a twenty element array is used to detect a monochromatic plane wave signal of amplitude .08 in the presence of interference from two, unit amplitude, monochromatic point source interferers plus white Gaussian noise. The monochromatic interferers are modeled as having Raleigh distributed amplitudes and uniformly distributed phases. The interferer spatial frequencies were chosen so they are 1 DFT bin away from the signal spatial frequency. Twenty-five independent snapshots of data are collected and arranged into the data matrix Y. The signal is only received in yl0 and thus signal detection is to be done on ~'10.

The SMI method test statistic is implemented as (see [7] for further detail)

Z ---- s T R - I ~ r l 0 , (23)

where 25 q

k = 2-~ E ~'k3"T" (24) k=l,k~lO

The PCI test statistic is

z = sTo~'oblo. (25)

The moment calculations for the PCI method are done assuming that the data matrix is constructed using the same realization of Y, but with independent realizations of N in each observation. However, the SMI test statistic moment calculations are done under the assumption that independent realizations of both Y and N are used to construct the data matrix in each observation. This assumption is needed in order to make the SMI statistical calculations tractable. The first and second moments of the SMI method test statistic are


calculated in [2].

The performance of the methods are judged by the separation of the regions defined by plus and minus one standard deviation about the mean of the test statistic for the noise- only and signal plus noise cases. In figures la and lb the regions defined by plus and minus one standard deviation about the mean of the test statistic are plotted for the PCI and SMI methods. The meanand standard deviation are calculated using the theoretically derived moments. From these plots it can be that the separation of these regions for the PCI method are much greater than for the SMI method. Although the PCI and SMI methods are compared under different assumptions (the same realization of Y is used to generate the data matrix for the PCI method while the data matrix for the SMI method is generated using independent realizations of Y and N in each observation), the analysis should nevertheless be indicative of the differences in performance.

U

a25

U

0.15

ILl

0

-0.l

/ f "

ak W m ~ W ~ W .k W LI

Fig. la. Fig. lb.

Fig. 1. High probability regions for the (a) PCI test statistic and (b) SMI test statistic with signal present (between solid lines) and with signal absent (between dashed lines) versus variance of background noise.

5 C O N C L U D I N G R E M A R K S

We have derived accurate formulas for the error in approximating the signal component of a data matrix by a low-rank approximation to the data matrix. These formulas can be used to analyze the performance and properties (eg. sensitivity to noise and rounding errors) of any application that uses a low rank approximation to a matrix. In particular, we have derived formulas for the first and second moments of the PCI detector test statistic for the cases where detection is done on the same data set as used for adaptation and also when detection is done using data not contained in the data set used for adaptation.

422 L P. Kirsteins and D.W. Tufts

Acknowledgment

We thank Abhijit Shah and S. Umesh for their assistance in programming and debugging the computer simulation examples that were presented in this paper.

References

[1] L. E. Brennan and I. S. Reed. Theory of adaptive radar. IEEE Trans. Aeros. Electron. Syst., AES-9(2):237-251, March 1973.

[2] I. P. Kirsteins. Analysis of Reduced Rank Interference Cancellation. PhD thesis, Uni- versity of Rhode Island, Kingston, RI, 1991.

[3] I. P. Kirsteins. A reduced-rank generalized likelihood-ratio test. In NATO ASI Conf. Acoust. Sig. Proc. for Ocean Explor., pages 572-575, Madeira, Portugal, July 1992. IEEE.

[4] I. P. Kirsteins and D. W. Tufts. On the probability density of signal-to-noise ratio in an improved detector. In Proc. ICASSP 85, pages 572-575, Tampa, FL, March 1985. IEEE.

[5] I. P. Kirsteins and D. W. Tufts. Rapidly adaptive nulling of interference. In M. Bouvet and G. Bienvenu, editors, Lecture Notes in Control and Information Sciences. Springer- Verlag, 1991.

[6] Fu Li, R. J. Vaccaro, and D. W. Tufts. Unified performance analysis of subspace-based estimation algorithms. In Proc. ICASSP 90, pages 2575-2578, Albuquerque, NM, April 1990. IEEE.

[7] I. S. Reed, J. D. Mallet, and L. E. Brennan. Rapid convergence rate in adaptive arrays. IEEE Trans. Aeros. Electron. Syst., AES-10(6):885-863, November 1974.

[8] A. A. Shah and D. W. Tufts. Estimation of the signal component of a data vector. In Proc. ICASSP 92, volume 5, pages 393-396, San Francisco, CA, March 1992. IEEE.

[9] D.W. Tufts. The effects of perturbations on matrix-based signal processing. In Proc. Fifth ASSP Workshop on Spectrum Estimation and Modeling, pages 159-162, 1990.

[10] D.W. Tufts, R. Kumaresan, and I. P. Kirsteins. Data adaptive signal estimation by singular value decomposition of a data matrix. Proc. IEEE, 7:684-685, 1982.

[11] R. J. Vaccaro, D. W. Tufts, and G. F. Boudreaux-Bartels. Advances in principal component signal processing. In E.F. Deprettere, editor, SVD and Signal Processing, pages 115-146. Elsevier Science Publishers B.V., Amsterdam, 1988.


423

REALIZATION OF DISCRETE-TIME PERIODIC SYSTEMS FROM INPUT-OUTPUT DATA

E.I. VERRIEST, J.A. KULLSTAM Georgia Tech Lorraine Technopole Metz 2000 2 et 3 rue Marconi F-57070 Metz, France erie verriest@ee, gatech, edu

ABSTRACT. A method is presented for the direct realization of a linear periodically time varying state-space model from input-output data. The algorithm is based upon an implicit reformulation into several time-invariant monodromy systems, derived in an operator theoretic framework. The SVD-exploiting subspace algorithm in [2,3] is extended to the periodic case. Reconstruction of the system from noiseless data is exact and fortuitously, the algorithm performs well in moderately noisy conditions.

KEYWORDS. Periodic system, input output data, subspace realization.

1 INTRODUCTION

The problem of modeling linear time-periodic (LTP) systems from input-output data is treated in this paper. Many natural (biological and ecological systems) and man-made phenomena (economy) are indeed inherently periodic, and the importance of obtaining accurate models is obvious. In [5] and [8] a method for the realization of a deterministic discrete LTP state-space system from generalized impulse response and periodic transfer function representations is discussed. While this work has important theoretical aspects-defining and clarifying canonical forms for LTP systems, the algorithm remains unfortunately, complex. In [7], the authors considered a polynomial approach. Transfer functions and impulse responses are idealized representations, and are often unavailable in practice (e.g., inputs may actually not be fully assignable in ecological systems). A more pragmatic approach is needed, treating the modeling problem directly from observed input and output data.

424 E.L Verriest and J.A. Kullstam

In this paper, an algorithm is presented to construct an LTP state-space realization relating discrete input and output data. Its main contribution is the decompostion, elucidated from an operator point of view, of the discrete LTP identification problem into several LTI identification subproblems, for which the SVD-based subspace realization algorithms are well suited.

In the next section the problem and notation is identified. In section 3, the stage is set by casting the state space model in an operator framework, and deriving the LTI monodromy systems. The LTP extension of the subspace algorithm is constructed in section 4, and experimental results are discussed in section 5.

2 FORMULATION OF THE PROBLEM

Let a sequence of inputs {u(k)}, and outputs {y(k)}, of a particular system be given. It is assumed that these observed data (m inputs and p outputs) have been generated according to an underlying LTP system of some order n and period N given by a system l~per

{ ~(k + 1) = %]~(k)+ Btk]~(k) (~) y(k) = Ctk]~(k) + Otk]~(k),

where [k] = k rood N. The free parameters in the model are the model order n as well as the Ak, Bk, Ck, mad Dk.

The goal is to derive from the given data, an underlying state space model. This amounts to a data reduction, exposing the structure while discarding uninteresting details. The model order n can be inferred at an intermediate point in the calculations. We assume however that the period N is known. This is essential in order to exploit the periodicity of the original system. This does not usually present a problem since the same reasons to suspect that a certain system is periodic almost always indicate the period length as well. It is at this point that the context and outside information are used in the modeling process.

3 PERIODIC SYSTEM REPRESENTATION

3.1 SEQUENCE ALGEBRA

Consider a signal, taking values in a finite dimensional vector space V, a Cartesian power of lit. A discrete time V-signal is a semi-infinite sequence of vectors in V. Denoting by IN the set of natural numbers (including 0), then signals live in the space of non-uniform sequences V I~ = 1;. As Cartesian product of vector spaces, ~his set of sequences inherits the addition and scalar multiplication of V in a natural way, and is therefore itself a vector space, ~, the sequence space of V. An arbitrary sequence in ~ will be denoted by {v} or {vk} (and more explicitly {v0, vl, v2. . .} if we want to describe it by its elements). We shall refer to vk as the indez-k element (it actually occupies the k + 1 position in the sequence).

It is useful to have an operator, ~, to construct a sequence from its entries (see also [6]):

,: v - , v : ~ -~ ~(~)= {~, 0, 0,. . .) (2)

By extension, define ~k as the operator V ~ V, placing its argument at index k in the

Realization of Discrete-time Periodic Systems 425

sequence, and making all other entries zero. We call these operators the constructors. Their adjoints,

~ : v - . v : {~} - , ~{~} = vk, (3)

axe the evaluators. Note that e~Lk(a)= a, Va e V, and ~.ketr LkL~{V} = {V}, V{V} e Y.

If V is endowed with an inner product, < .,. >y , then Y is also an inner product space, allowing sequences with exponential growth, by defining for 7 > 0,

< {~}, {~} > v = ~ ~-~k < ~k, vk > v . (4) kEW

This inner product induces a norm in F, and the space of sequences with finite norm is denoted by i v. This space is a separable Hilbert space.

Let ~) and W be sequence spaces. By a transfer operator we shall understand any linear operator T : Y ---, 14;. It represents a system that maps input signals in Y to output signals in )4;. Denote by B(F, )4;) the space of bounded operators from i v to I w. An operator T belongs to B(F, )4;)if V{v} e i v, it follows that T{v} e I w. The induced operator norm of T, is bounded in this case. Both ~ and ~* are bounded operators. A bounded operator defined everywhere on a separable Hilbert space, and mapping into a separable Hilbert space admits a matrix representation which uniquely determines the operator.

Extremely important for the study of dynamics are the shift operators defined on F: ~ : v - . v : {~} = {v0, v~, ~ , . . . } - , {~, ~, v3...} (5)

~*: v - . v : {~} = {~0, ~ , ~ , . . . } -* {0, ~0, ~ , . . . } (6) Together with ~ and ~*, they generate the algebra for describing a dynamical system. Note that aa* = 1 , while a*a is information destroying (a*a = 1 - ~*).

The extension of an operator A : V --* W is defined by a 'global' matrix multiplication:

Ao: V --* W : {v} ~ A o {v} = {Av}. (7)

Its norm is the largest singular value of the matrix A. In fact, any time invariant system or recursion in the form

X k + l - - Axk + Buk yk = Czk (8)

may be represented over sequence spaces by

a{x} = A o { x } + B o { u } (9) {y} = C o { ~ }

leading to the operational transfer inclusion {y} e C ( a I - A ) - I B o {u}, also (see [9]) oo (x)

{y} e ~ r o {~} + ~ r o ,V, (10) p=0 p=0

i.e., the familiar representation in terms of input and initial condition (a selection from V).

From these basic operators, other important periodic system specific operators are constructed (for period N): the projector 7r : {v0, v l , . . . , VN,...} ~ {v0, 0 , . . . , 0, VN, 0, . . .} , the sampler s : {v0, v l , . . . , VN,...} ~ {V0, VN, V2N...}, and its dual, the extensor s* :


{Vo, v l , . . .} ~ {Vo, 0 , . . . , 0, vl, 0 , . . . , 0, v 2 . . . } , respectively: oo oo oo

= ~ ~ ' ~ N ( X - ~ ' ~ ) ~ ' N , , = ~ ~ ' ' ( X - ~ * ~ ) * ~N , , ' = ~ * ' ~ N ( I - ~ ' * ) ~ . ( 1 1 ) p--0 p = 0 p = 0

More generally, we define also all the i-phasor, i E { 0 , 1 , . . . , N - 1}, by ~r Is] = a*i~ra i, and extend that symbol for all integers as ~r [~] = lr [~m~ It is also useful to introduce sequences of interleaved samples,

Va e IN: {v}a = sa"{v} = {vc,, vg+a, V2N+a, . . . } . (12) Many relations between s, s*, the phasors r[ i], and the constructors and evaluators, ~k and ~ respectively, ~ t (see [9]).

3.2 LIFTING OF PERIODIC SYSTEM

The sequence algebra is used to derive time invariant representations for periodic systems of the form (1) of period N. First, for each phase i E { 0 , . . . , N - 1}, (1) is equivalent to

~[~+~]{~} - A, o ~[,l{~} _ B, o ~[,l{~} = 0, (13)

or, by shifting the whole representation over p = 0 , . . . , N - 1 units,

aTr[i+ll{aP{x}} - A[i+p] o lr[il{aP{x}} - B[i+p] o ~r[i]{aP{u}} = 0. (14)

Operating with sa ~ on the left, and using the identity sa N - or s, and (12) one obtains for i e { 0 , 1 , . . . , N - 2},

{aP{x}} i+l = A[i+p] o {aP{z}}i + B[i+p] o {aP{u}} i a { a P { x } } o = A[N-I+p] o {X}N-1 + B[N-l+p] o {aP{U}}N_l . (15)

This corresponds to a singular system representation in y g with state vector {aP{x}} = [ { a P { x } } ~ o , { a P { x } } ~ , . . . , { a P { x } } ~ _ l ] ', and the vector input and output, {ap{u}} and {ap{y}}, defined similarly. The details are in [9]. Elimination of the sequences {aP{z}}l , . . . , {aP{Z}}N_, yields a (regular) representation in the 'state' {aP{x}}o = {x}p, referred to as the phase p monodromy system,

Defining the the generalized powers (transition matrices)

{ A(p,p) = I (17) A (p'q) = Ap_xA (p-l'q) = A(P'q+I)Aq for p > q,

then, ff~p = A (N+p,I~), is the state transition matrix over one period for phase p, 7~p is the phase-p teachability matrix,

T~p = [A(p+N'p+I)Bp, " " " , BIb+N-I]], (18)


(.O r the phase-p observability matrix,

c~ Or C[p+llA(r+l,r) = . (19)

C[p+N_I]A (r+N-l,r)

and jt4p a lower triangular matrix of Markov parameter blocks, for i, j = 1, 2 , . . . , N - 1

0 i < j [Mr]i j = Dip+i] , i= j (20)

C[p+i]A(r+',r+J+X)B[p + j] i > j.

Any LTI system realization or identification scheme may be used to find the monodromy systems (16), i.e., the Op, TCr, O r and M r matrices. However this misses the point, as then products of the matrices Ai, Bi, and Ci (i = 0 , . . . , N - 1) are identified. A difficult factorization problem would still have to be solved. This problem can be circumvented by exploiting the structure of the subspace algorithm. As described in [2,3] it generates a state trajectory realization first and then the coefficients are found to fit the state with the input and output . Since there is no need for the monodromy system other than generation of the state, we go directly to solving for the time-periodic coefficients.

4 E X T E N S I O N OF T H E S U B S P A C E A L G O R I T H M

Let us assume that the true system is represented by the state space description in equation (1) with period N. With the sequence aP{y}, define a vector of T = kN sequences

Y~ = [{y}~, { y h + ~ , . . . , {Y}~+kN-~]'. (21)

and define hop similarly. The induced input-output relation is

y~ = V~x~ + ~ u ~ (22)

for each p = t (mod N). The matrices ~p and ~ ' r are respectively the observability and Toeplitz matrix of the Markov parameters of the lifted system (16). To shift to a finite segment of real input-output data, the truncated (after l-th entry) sequences {u}t and {Y}t are replaced by the m x 1, respectively p x l matrices u[t] and y[t]. Similarly/gt and Yt are replaced by the data matrices in [2] and [3]. Denote these respectively by L/It] and y[t]. Finally, x[t] replaces {x}t. Only y[t] and/A[t] are used in the state realization scheme to generate x[t]. In fact, neither O"'p, M'p nor the system of equations in (3) are ever explicitly generated. They are, however, implicitly used in the state realization technique.

4.1 SINGLE PHASE STATE RECONSTRUCTION

Consider (22), and let us concentrate on one phase of the system. As there is no risk of confusion, we shall suppress the explicit dependency on the phase designator p. We shall assume that full row rank is enjoyed by/4, x (persistent excitation), and ~ (observability).


Let H denote the concatenation of/4 and y

As shown in [2,3], concatenation of the outputs contributes n new dimensions to the space spanned by the inputs i.e. rank (H) = rank(/4)+ rank(x). These n new dimensions are attributed to the effects of the state and are used to infer a state realization.

Now let

U[t] /41 H2 = = . (24) / / 1 -~ y [ t ] = Y l ' y [ t + kN] Y2

The state realization is generated from the relation

rowspan(x2) = rowspan(H1) N rowspan(H2). (25)

Therefore, the main computational task of the subspace algorithm in generating a state trajectory is to find this rowspace intersection, for which we refer to [2] and [3].

4.2 IMPLEMENTATION

We now discuss some aspects of a practical nature relating to the implementation of the

algorithm. We start with the data record of the input-output data d(t) = u(t) u(t) , for t =

0 ,1 , . . . , M < cr We proceed as follows. Let d[t] = [d(t), d(t + N ) , . . . , d(t + iN)]. Use the notation

d~[p] = dip] (26) d~[p] = dip + kN] (2T)

For each phase p form the data matr ices/ t l and H2 by interleaving the input and output vectors, i.e.

dl[p] d2[p] ~ [ p ] = d~[p.+ 1] , H~[p] = d2[p.+ 1] . (2S)

dl[p + kN - 1] d2[p + kN - 1] This is a rearrangement of the rows from our previous definition of Hx and H2. Clearly, all of the arguments made with the previous definition still hold�9 Next, find x2[p] by the method described in [2] and [3], i.e.,

(29)

Note that

x2[p + N] = Ur ] (

d[p + N] )

�9 o

d [ p + ( k + ~ l N - 11

(30)

this trick will become important in the reconstruction step and motivates the interleaving�9

While we will not go into any details, we state that H should contain many more columns than rows. It needs to contain enough information about the system to reconstruct a state�9


The periodicity will shrink the width by a factor of N so a moderately long sequence of data is required in order for the method to work properly (the data required for grows by a factor of N 2 since we must insure t h a t / t will be at least square).

At this point we have generated a state trajectory realization :for each phase from the input-output data. The states, inputs and outputs are still not interleaved but as it will turn out, this is the natural representation for finding the system matrices.

4.3 RECONSTRUCTION OF THE SYSTEM EQUATIONS

We arrive at the last step: the creation of a linear time periodic state space model. We seek the matrix coefficients Ap, Bp, Cp and Dp. At our disposal are the input, output and state trajectories. The u2[p] and Y2[P] are, of course, given at the outset, x2[p], for p = 1 , 2 , . . . , N - 1 are defined by formula (29). Also, x[N] comes from equation (30) with p=O.

For each p = 0 ,1 , . . . , N - 1 solve the over-determined system of equations

Y2[P] = C, D, u2[p]

in the total least squares sense [1]. Since known quantities on the right hand side are just as reliable as those on the left the use of standard least squares may not be well motivated.

5 S I M U L A T I O N R E S U L T S

We checked if the noise insensitivity of the subspace algorithm in the time-invariant case is also enjoyed by the periodic extension. To explore sensitivity, we used a 4-th order linear system with period 3, which was such that its response to a unit step at every phase was a unit step. While this is artificial, it makes interpretation of the results easier. Unit variance white Gaussian plant and measurement noise were added to obscure the true signal. To ensure excitation of all modes, a Bernoulli(�89 sequence of {-1 , + 1} was chosen as an input. Based on 800 data points (M), a 4th order model was then identified using the periodic extension to the sub-space model presented in this paper.

The figures compare the step response of the exact and the identified model. As can be seen from figure 1, the noise level is substantial. The model identified presents a reasonable approximation of the dynamics as seen in figure 2. As noise is reduced, the identified model quickly nears the true model.


We have described a simple strategy for state-space identification of periodic systems. The data record must contain several full cycles in order to exploit the periodic structure. Therefore, only short to moderate period lengths are feasible. The main development is the restructuring of the problem using the monodromy system formulations.

Empirical tests show that even when moderate amounts of noise are present the algorithm


produces a good approximation. The subspace method is much easier to implement than a polynomial matrix fraction to state-space realization technique. Therefore, the subspace algorithm can be used to obtain a reduced order model or it can be used as an identification scheme on data generated by real world systems. Note that the state trajectory realization defines N individual monodromy matrices. While the periodicity of the underlying system forces a correlation on these monodromy matrices (e.g. they all have the same eigenvalues), this information is disregarded in the creation of a state. It is, however, implicit in the construction of the LTP system matrices. In the case of high noise, especially when inputs are highly correlated with past outputs e.g. feedback situations, the subspace algorithm may not perform well. Also, in the high noise case a large data record will be needed to extract accurate information. The subspace algorithm is numerically expensive. As an alternative the stochastic identification method described in [4] can be used. From correlation estimates of past and future input-output data a state trajectory estimate can be derived by:canonical correlation analysis. This state estimate can then be used in the second part of the algorithm to determine the state-space equation parameters.

References

[1] S. Van Huffel, J. Vandewalle. The total least squares problem: computation aspects and analysis. SIAM 1991.

[2] B. De Moor, M. Moonen, L. Vandenberghe, and J. Vandewalle. The Application of the Canonical Correlation Concept to the Identification of Linear State Space Models. Analysis and Optimization of Systems, A. Bensoussan, J. Lions (Eds.), Springer-Verlag, Lecture Notes in Control and Information Sciences, Vol. 111, 1988.

[3] M. Moonen, B. De Moor, L. Vandenberghe, and J. Vandewalle. On- and Off-line identification of linear state-space models. Int. Journal of Control 49, No. 1, 1989, pp. 219-232.

[4] W. Larimore. System Identification, Reduced-Order Filtering and Modelling Via Canonical Variate Analysis. Proc. American Control Conference, H.S. Rao and T. Dorato, Eds., 1983, pp. 445-451. New York: IEEE.

[5] B. Park and E. Verriest. Canonical Forms for Discrete Linear Periodically Time- Varying System and Applications to Control. Proceedings of the 28th IEEE Conference on Decision and Control, Tampa, Florida, 1989 pp. 1220-1225.

[6] D. Alpay, P. Dewilde, and H. Dym. Lossless Inverse Scattering and reproducing kernels for upper triangular operators. In: I. Gohberg (Ed.), Extension and Interpolation of Linear Operators and Matrix Functions, Birkhs Verlag, 1990.

[7] O.M. Grasselli, A. Tornambe and S. Longhi. A Polynomial Approach to Deriving a State-Space Model of a Periodic Process Described by Difference Equations. Proceed- ings of the 2-nd International Symposium on Implicit and Robust Systems, Warsau, Poland, Aug. 1991.


[8] P.B. Park and E.I. Verriest. Time-Frequency Transfer Function and Realization Algo- rithm for Discrete Periodic Linear Systems. Proceedings of the 32nd IEEE Conf. Dec. Control, San Antonio, TX, Dec. 1993, pp. 2401-2402.

[9] E.I. Verriest and P.B. Park. Periodic Systems Realization Theory with Applications. Proc. 1st IFAC Workshop on New Trends in Design of Control Systems, Sept. 7-10, 1994, Smolenice, Slovak Republic.

0.2

0 - - i "0"20~ -0.4

slep rcspon.r of true system . . . . .

20 40 60 80 100 120

Figure 1" Noisy and Noiseless Step Responses Typical step response of the noisy system (solid line) and the step response of the noiseless system (dashed line).

slep response of eslitnatcd sy.~tetn 1 . 4 - -

1.2

I

0.8

0.6! IV, [

0.4

0.2

0 -

60 80 1 (X) 120

Figure 2: Identified and True Step Responses The step response of the 4th order system identified from noisy data (solid line) and the step response of the noiseless (dashed line).



433

C A N O N I C A L C O R R E L A T I O N A N A L Y S I S O F T H E D E T E R M I N I S T I C R E A L I Z A T I O N P R O B L E M

J. A. RAMOS 11200 N.W. 43 rd Street Coral Springs, F1 33065 zilouchi @acc. fau. edu

E. I. VERRIEST Georgia Tech Lorraine, Technop61e Metz 2000 2 et 3, rue Marconi, 57070 Metz France verriest @ 33. gatech, ed u

ABSTRACT. Recently, a fairly simple state-space system identification algorithm has been introduced, which uses I/O data directly. This new algorithm falls in the class of subspace methods and resembles a well known Markov parameter based realization algorithm. The heart of the subspace algorithm is the computation of a state vector sequence from the intersection of two subspaces. Once this state vector sequence is obtained, the system matrices can be easily obtained by solving an over-determined linear system of equations. It has been shown that this new subspace algorithm is equivalent to a canonical correlation analysis on certain data matrices. In this paper we apply the classical theory of canonical correlation analysis to the deterministic identification problem. It is shown that the first n pairs of canonical eigenvectors contain all the information necessary to compute the state vector sequence in the original algorithm. We further show the connections with forward- backwards models, a duality property common to canonical correlation based identification algorithms. This is useful for computing balanced realizations (an optional step). It is shown that, unlike the stochastic realization problem, certain dual covariance constraints have to be satisfied in order for the state sequence to be admissible. Thus, forcing the optimal solution to be of a canonical correlation analysis type. Algorithms that do not satisfy all the constraints are only approximate. Finally, we derive deterministic realization algorithms using ideas from their stochastic realization counterparts.

KEYWORDS. Deterministic realization theory, canonical correlations, balanced realizations, GSVD (QPQ-SVD, PQ-SVD, P-SVD).

434 J.A. Ramos and E.L Verriest


Consider the linear, discrete, time invariant, multi-variable system with forward state space representation

:r.k+l = Axk-k Buk (1)

Yk = C zk + Duk (2) where uk E ~,n, Yk E ~ , and xk E ~R" denote, respectively, the input, output, and state vectors at time k, and n corresponds to the minimal system order. Furthermore, [A, B, C, D! are the unknown system matrices of appropriate dimensions, to be identified (up to a similarity transformation) by means of a recorded I/O sequence N {Uk}k=l and {Yk}g=l.

Let the past (Hp) and future (H!) I/O data Hankel matrices be as defined in [3, 6]

[ U, ] [ Hankel{uk, uk+l,...,Uk+i+j_2} Hankel{yk, Yk+l,...,yk+i+j-2} Hp - .---- -

and

1 t i =

�9 ~

where {U~,, U!) e ~rn, x~, {yp,y!) E ~R ~ixj , and j ~> i. Let also the state vector sequences Xp and X / b e defined as

I ], I ] Then, it is shown in [3, 6] that there exists matrices U12 E ~(mi+~i)• U22 E ~R (m/+~/)x(2~/-n) , and 7" E ~R nx(2~i-n) such that

f(! = T UT Hp = - T UT H!

where f(l = TXf , for some similarity transformation T. The system matrices [A, B, C',/)] are then obtained by solving an over-determined linear system of equations.

2 C A N O N I C A L C O R R E L A T I O N A N A L Y S I S

Suppose ZII p = MTHp is an oversized (mi + n) • j state vector sequence corresponding to a non-minimum system. Then, we know that there is an n th order sub-system that has minimum dimension and at the same time is both observable and controllable. The idea then is to compress the state vector sequence so that the non-observable/non-controllable states are deleted. That is, find an n • (n -k mi) compression matrix 7", such that ffl = 7"MTHp" In essence, this data compression operation resembles a principal component extraction, an idea popularized in system theory by [1, 8].

To fix the ideas, we need the output and state equations in subspace form [3, 6], i.e.,

Yp = r Xp + HT Up (3)

= r x j + HT U! (4)

X I = A i xp + A Up (5)

where HT E ~ixmi is a lower triangular block Toeplitz matrix with the Markov param-

Canonical Correlation Analysis 435

eters, while r E ~tixn and A E ~,~xmi axe, respectively, the observability and reversed controllability matrices. From the subspace equations (3)-(5), one can show that both Hp and H I have dual representations. That is,

Up Omixn Imixmi Omixmi ] c a

J

[_~s]x~ lip r HT O a x m i ,4 0 Xp l p (6) ] -- Omixn Omi• Imixmi Up - C1 ]32 U I

YI r A ~ r A HT

Similarly, by reversing the direction of time, we get

lip rb HbT Oa,,,~i .Ab 0 XII p (7) f = O~i• O..• I,~i• Up = B~ B2 US

y] r Oiixmi HT UI

where the subscript/superscript b denotes backwards, in relation to a backwards model, and rb = r A -~ and H b = H T - FA- iA are, respectively, the backwards observability matrix and the upper triangular Toeplitz matrix of backwards Markov parameters. The aforementioned duality stems from the fact that by reversing the direction of time and exchanging the roles of Hp and HI, we get another representation dual to (7), i.e.,

YI r HT Otixmi A 0 Xll I (8)

yp rb o~i• H~r Up

Let us now formally define the (n + mi) • j "state plus input" sequences

X p l , = [ X, XI [ XI u~ ' x s ls = us ' x s l, = v~

Let us also define the following covariance matrices

j-~oo 3 UI ~pf

A : j--oolim 13 Xfl fUp [ % Iv; ] - Ay,

, .oo 7 H ,IHf -

ZI

Alp Ap

7Z] We should point out that rank{rip} = rank{n]} = rank{~]p} = mi 4- n, thus, the data covaxiance matrices are rank deficient [2, 5].

It is rather tedious but straight forward to show that n~ ~ n~ - ~s = ~ ( ~ ; i r ~ , s _ r~s)~ (9)

n~ ~ nz - n , = ~ (ay~AT~a. - A,)(~)~ (10) If we now pre-multiply (9) and (10) by their respective orthogonal complements, we get


where

~-(~) ~ ~ ~'•162 - [ - ~'~ ~ 1 ~ ' ~ ]

for nonsingular matrices (P, Pb) E ~ti• whose general form can be shown to be

P = r • Pb= ' rb•

where (.)• denotes orthogonal complement and (.)~ denotes pseudo inverse. One can further show that there exists matrices U and V such that Np and N! can be decomposed as

~7 = .~- = .~.

[v~ -rtA~lrt ] N ~ = A• = A•

where U and V are the canonical solutions we are looking for, i.e., they satisfy 2s = u r n , = vrHs (la)

It is well known [3, 13] that if dim{spanrow{Hp}Nspanrow{H ! }} = n, then the canonical angles between the subspaces spanned by the rows of Hp and H! satisfy : 81 = 82 . . . . . 8,, = 0, and 8n+1 >_ 8n+2 >_ "" >_ 8n+mi >_ O. This result states that the first n canonical correlations (si = cos{0i}) are equal to one. Therefore, we can write (11)-(12) as a pair of generalized eigenvalue-eigenvector problems of the form

where S = diag{si = 1}~=1 are the first n canonical correlations associated with the canonical eigenvectors U = [u~, u2 , . . . , un] and Y = [v~, v2, . . . , v,~].

2.1 PROPERTIES OF THE CANONICAL MATRICES [U, V!

Let us now form the (rai + n) • j pair of variables to which we will perform the canonical correlations. First, recall from (7)-(8) that

x/l / - At H/, xtl ~ = A~ Hp Let us also define the following transformed variables

Z/I. = IX/I =PbX! [ ' X / ] = PX! ~- l,, zs ls= ~) is

where ~ j = TXs for ~ny ~imnarity transformation m~trix T, ~', ~nd ~) ~re tr~n~orm~tion~ to Up and U!, respectively, and P and Pb have the general form

7 ) =

Then ;Ell ~, and ;Ell ! form a full rank pair of canonical variates defined as z~,l, = M r H , , Z~, b, = L r H , (is)


where M T = 79 b .A~ and L T = P .A~ are the canonical transformations which are to be determined from the data matrices Hp and H I.

One can further show the connection of the canonical variables to a pair of forward- backwards models, a concept well developed in stochastic realization theory [4]. From r

s

and ~4b T, one can show that At=[-FtHT rt ] I m i x m i Omixgi

l,.i x,.i O,,,.i x.a Im~ x,~i O,,-,ixti where

F b = F A - i H b = H T - F A - i A

= H T - Fb A

= H T + F A b Ab = - A - i A

Notice that H~ is upper triangular, corresponding to an anti-causal backwards system. It is then rather straight forward to show that the above relationships are associated with a pair of forward-backwards models of the form"

Forward Mode l xk+l = A xk + B uk

Yk = C x k + D u k

Backwards Mode l zk-1 = Ab zk + Bb uk

Yk -- Cb Zk + Db uk where, for the backwards model, Ab = A -1, Bb = - A -1 B, Cb = C A -1, Db = D - C A - 1 B ,

and Zk-1 -- Xk.

Furthermore, from (13) one can show that f(. f H T - uTT~,p - v T ' ~ . f p

Thus, we have the following system of equations

7-lip ~ y from which we get the optimality conditions

v~t~.T, ~ u,', - ~,] = o.• v~tn~ u~ n ~ - u~] = o,,• U T = vTT-I# Tt~

V T -- u T ' ] - ~ T ~ ]

This can easily be implemented using a Q R decomposition.

(17)

(18)

(20)

(21)


3 B A L A N C E D R E A L I Z A T I O N S

Here we will show that the orthogonal complement matrices Np and N! satisfy certain constraints that lead to the balanced conditions similarly derived in [7]. First, let us point out that Np and N! satisfy the following system of equations

Now. since the two systems of equations are equivalent, one needs to solve only one system. Let [ T, IT4 ]= [ NTI-NT ], then from (22) we have

T2r + T4rA ~ = 0~• (24) T1 + T2HT + T4EA = 0~i• (25)

T3 + T4HT = 0tixmi (26) Since the product r & is similarity invariant, one can specify a similarity transformation

directly from the equations. That is, from (25) we can specify T* = T4F as the pseudoinverse of the compression matrix. Substituting this result in the above system of equations leads to a dual relationship of the form T t = (T2T4T3 - I'1)A~. One can then find the compression matrix by specifying the following balancing constraints

(Tt)T(TT4 )TT2T* = Abe, (27) T(T2T4T3 - Tx)(T2T4T3 - T1)TT T = Abal (28)

where Abat is a diagonal matrix.

Let us finally remark that (24)-(26) can also be expressed as

] [ I- �9 r ] -- [0~ , • (29) where rl:2i and H)r :2i are extensions of r and HT to include 2i block rows. Since the

system matrices [A, B, C, D] are contained in the unknowns, one may find other interesting canonical forms, leading perhaps to alternative constrained system identification algorithms.

4 G E N E R A L I Z E D D E T E R M I N I S T I C I D E N T I F I C A T I O N P R O B L E M

The deterministic realization problem can be seen as a classical multivariate analysis problem, where one has a pair of data matrices and would like to transform them to a coordinate system where they have maximum correlation [3, 5, 10, 11]. This leads to a number of alternatives for modeling the deterministic realization problem. Due to lack of space we shall only present a summary of the results. Let us write the main decomposition as follows :

L r o l l = A j ( 3 0 )

M T O p M = Ap (31)

7-lln = Of LA~�89 (32)

where S " mi+,, = d*ag{si}i=l , Ap and A/ are diagonal matrices, and 0p and 0 f are to be

defined below. From the above solution we can specify the constraints according to the


following algorithms [12]"

Case 1" Canonical Correlation Analysis QPQ-SVD of {Hf, H f , Hp, H / }

_ .r~ < 1 0 < t~,tji:l __

Op = 7Zp

0~ = ~ f U = first n rows ofL T

V T = first n rows o f M T

Case 2: Principal Components of Instrumental Variables PQ-SVD of { HI , Hp, H T, } 0 < {si}m=il +n < no upper bound (variances) 0p = ~p

O] = I(mi+~)• U T = a T S M T

V T = aT1L T

where a l is the result of a second approximation, i.e,

Due to lack of space we will not go into any details as to the GSVD part of the algorithms. Instead, we will refer the interested reader to the references [3, 12] where an algorithmic approach has been elegantly outlined. Finally, the computation of L and M can also be done from the ordinary SVD by performing the following computations �9 First, compute the SVD's of Hp and H/

Hp = VpSpV T , H / - Vf S f Vf then compute the SVD of the product of the V factors

vfv = u s v Set m f -- S and Ap = S. Finally, compute L and M as

1 1

L T = A ~ u T s T I u ~ , M T = A g v T s ; 1 U [


We have studied the deterministic realization problem from a classical canonical correlation analysis point of view. We have shown that there exists an inherent duality between a forward-backwards pair of models. Furthermore, we have derived the canonical weights as a function of system matrices such as the observability and controllability matrices. This introduces some flexibility in designing constrained system identification algorithms that may require a given canonical form. For the balancing conditions it was shown how to derive the constraints explicitly. However, for other canonical forms, this remains an open problem. Finally, we derived some optimality conditions and showed how to correct for these when using Principal Components based algorithms.


References

[1] K.S. Arun, D.V. Bhaskar Rao, and S.Y. Kung A new predictive efficiency criterion for approximate stochastic realization. Proc. 22nd IEEE Conf. Decision Contr., San Antonio, TX, pp. 1353-1355, 1983.

[2] A. Bjork and G. H. Golub. Numerical methods for computing angles between linear subspaces. Math. Comp., Vol. 27, pp. 579-594, 1973.

[3] B. De Moor. Mathematical concepts and techniques for modeling of static and dynamic systems. Ph. D. dissertation, Katholieke Universiteit Leuven, Leuven, Belgium, 1988.

[4] U.B. Desai, D. Pal, and R. Kirkpatrick. A realization approach to stochastic model reduction. Int. Journal of Control 42, No. 4, pp. 821-838, 1985.

[5] D. G. Kabe. On some multivariate statistical methodology with applications to statistics, psychology, and mathematical programming. The Journal of the Industrial Math- ematics Society, Vol. 35, Part 1, pp. 1-18, 1985.

[6] M. Moonen, B. De Moor, L. Vandenberghe, J. Vandewalle. On- and off-line identification of linear state space models. Int. Journal of Control 49, No. 1, pp. 219-232, 1989.

[7] M. Moonen and J. A. Ramos. A subspace algorithm for balanced state-space system identification. IEEE Trans. Automat. Contr. AC-38, No. 11, pp. 1727-1729, 1993.

[8] J.B. Moore. Principal component analysis in linear systems: controllability, observability, and model reduction. IEEE Trans. Automat. Contr. AC-26, No. 1, pp. 17-32, 1981.

[9] C.T. Mullis and R.A. Roberts. Synthesis of minimum roundoff noise in fixed point digital filters. IEEE Trans. Circuits and Systems CAS-23, No. 9, pp. 551-562, 1976.

J. A. Ramos and E. I. Verriest. A unifying tool for comparing stochastic realization algorithms and model reduction techniques. Proc. of the 1984 Automatic Control Con- ference, San Diego, CA, pp. 150-155, June 1984.

P. Robert and Y. Escoufier. A unifying tool for linear multivariate statistical methods: the RV-coefficient. Applied Statistics, 25, NO. 3, pp. 257-265, 1976.

J. Suykens, B. De Moor, and J. Vandewalle GSVD-based stochastic realization. ESAT Laboratory, Katholieke Universiteit Leuven, Leuven, Belgium. Internal report No. ESAT-SISTA 1990-03, July 1990.

D. S. Watkins. Fundamentals of Matrix Computations. John Wiley & Sons, 1991.

[10]

[11]

[12]

[13]


441

A N U P D A T I N G A L G O R I T H M F O R O N - L I N E M I M O S Y S T E M I D E N T I F I C A T I O N

M. STEWART Coordinated Science Laboratory University of Illinois Urbana, IL USA stewart @monk. csl. uiuc. edu

P. VAN DOOREN CESAME Universitd Catholique de Louvain Louvain-la-Neuve Belgium vandooren @anma. ucl. ac. be

ABSTRACT. This paper describes the application of a generalized URV decomposition to an on-line system identification algorithm. The algorithm updates estimates of a state space model with O(n 2) complexity.

KEYWORDS. Identification, MIMO systems, updating, URV.


Identification of a state space model for a MIMO system from input/output data is a computationally intensive problem. A reliable algorithm was given in [1], but as presented it depends on the SVD to make crucial rank decisions and identify subspaces. Unfortunately, there have been no exact algorithms proposed for updating the SVD when input/output measurements are added which are faster than O(n3). An approximate approach to SVD updating which might be considered for use here was developed in [2]. However the problem is really more difficult than just updating one SVD. What is desired is the intersection of the range spaces of two matrices. In [1] this is computed using two SVD's, and they both must be updated simultaneously.

The URV decomposition, [4], is an easily updated decomposition which, in some applications, may be used to replace the SVD. The fact that an intersection of the range spaces

442 M. Stewart and P. Van Dooren

of two matrices is required suggests that a generalization of the URV decomposition along the lines of [3] might be helpful. Such a decomposition was introduced in [5] along with an O(n 2) updating algorithm. This paper will give a brief description of the decomposition and show how it can be used as part of an on-line identification algorithm.

Given a sequence of m x 1 input vectors, u(k), we assume that the sequence of l • 1 output vectors, y(k), are generated from the state space equations,

x(k + 1) = Akz(k) + Bku(k)

y(k) = Ckz(k) + Dku(k).

(1)

Assuming we have observations of the input and output vectors, the identification problem is to find an order, n, and time-varying matrices, {Ak, Bk, Ck, Dk}, which satisfy (1) for some n x 1 state sequence, x(k). Generally it is assumed that the state space model is slowly time-varying. We then wish to provide an algorithm which will track the model.

The algorithm uses the same basic approach developed in [1]. It can be summarized in two steps: find an estimate of the state sequence and then obtain the system matrices from the least squares problem

[ x ( k + i + j - 1 ) . . . x ( k + i + l ) ] y(k + i + j - 2) . . . y(k + i) Wj-l =

(2) Cj Dj u ( k + i + j - 2 ) . . . u (k+i ) Wj-1,

where Wj is a diagonal weighting matrix defined by

[1 o wj= o ,~wj_~

for lal < 1 and W1 = 1. The index k is the time at which observations begin and k + i + j - 1 is the time at which the latest observations have been made. Indices k and i are fixed, but j grows as observations are made. To keep the notation compact, the indexing of the system matrices will show only the dependence on j , though {Aj,Bj , Cj, Dj} will depend on observations up to u(k + i + j - 1) and y(k + i + j - 1)

An appropriate exponentially weighted state vector sequence can be determined from the intersection of the row space of two Toeplitz matrices�9 Define the (m + l)i x j block Toeplitz matrix

T(k) =

u(k + j - 1) u(k + j - 2) . . . u(k) u(k + j - 1) y(k + j - 2) . . . y(k)

�9 , ,

u ( k + j + i - 2 ) u ( k + j + i - 3 ) . . . u ( k + i - 1 ) y ( k + j + i - 2 ) y ( k + j + i - 3 ) . . . y ( k + i - 1 )

If T1 = T(k) and 7'2 = T(k + i) then in the time invariant case, the intersection of the row spaces of T1 and T2 generically has dimension n, the order of the model (1) generating y(k) from u(k). [1]

If the rows of some X form a basis for the intersection then the columns of X are a

On-line MIMO System Identification 443

sequence of state vectors for a time invariant model generating y(k) from u(k). A proof of this fact can be found in [1].

If we compute the intersection of the row spaces of T1Wj and T2Wj and let X denote the basis for this space, then we use X as the exponentially weighted state vector sequence,

= [ x ( k + i + j - 1 ) x ( k + i + j - 2 ) . . . x(k+i) ]Wj. (3) X

The decomposition of [5] can be used to track the intersection of the row spaces. The contribution of this paper is to show how the system matrices can be obtained efficiently at the same time.

2 T H E D E C O M P O S I T I O N

This section will deal with the T matrices in transposed form so that the problem becomes one of tracking column spaces as rows are added.

The decomposition has the form

o v2

Rn E12 S13 R14 E15 0 E22 R23 E24 E25 0 E32 0 F34 E35 0 E42 0 0 F45 0 F52 0 0 0 0 0 0 0 0

(4)

where R n , R23 and R14 are upper triangular and full rank. Rl l and R23 are square.

Each F block is an upper triangular matrix with norm less than the tolerance. Each E block is an arbitrary matr ix with norm less than the tolerance. The S block is an arbitrary matrix. If this is the case, then the decomposition gives estimates of the range spaces of WiT T and WiT T. In fact, it can be shown that if the E and F blocks are zero, then the first columns, U1, of U corresponding to the number of columns in R14 are a basis for the intersection of the range space of WiT T and WjT T. Details concerning decompositions of this type can be found in [3].

If we parti t ion V2 in a manner which matches the decomposition

[ ] and assume that the E and F blocks are zero then

Villi4 = W~TTV24

and the full rank property of R14 imply that WjT2TV24 is also a basis for the intersection. This fact makes it possible to avoid storing U and will avoid the problem of growing memory storage as rows are added to WiT1T and WiT T.

Details on updating the decomposition can be found in [5]. A brief summary of relevant features will be given here. We assume that the decomposition has already been computed and we are interested in having the decomposition for the same matrices, but with an added row. The process can be initialized by setting U and V to the identity and letting


the decomposition equal zero.

If two rows, a T and b T are added to WiT T and WiT T respectively and each row of the old matrix is weighted by 0 < a < 1 then we wish to restore the form of (4) to

[1 ~ Ir l 0 U T o~WjT T aWjT T 0 V2 " (5)

This looks much like (4), but with an additional row along the top. The problem is to update the orthogonal matrices U and V to restore the structure of the decomposition and to deal with possible rank changes in the R matrices. The key feature of the algorithm as presented in [5] which has a bearing on the system identification algorithm is that the structure of (5) can be restored by applying plane rotations from the left which operate on adjacent rows and plane rotations from the right which operate on adjacent columns.

The approach is similar to that of UP~V updating as given in [4]. The algorithm can be broken into two stages. The first updates the overall structure of the decomposition when new rows are added to WiT T and WiT T. After the update, the decomposition has the same general form, but the triangular R matrices are possibly larger and might no longer have full rank. The second stage looks for small singular values of the R blocks and recursively deflates these blocks using the scheme described in [4] until they have full rank.

3 U P D A T I N G T H E S Y S T E M M A T R I C E S

As mentioned earlier, the intersection of the range spaces of WiT T and WiT T is given by WjTTV24 which also gives an estimate of the exponentially weighted state vectors, (3).

Thus the least squares problem can be written as

where

U(j)= [ u ( k + i + j - 1 ) u ( k + i + j - 2 ) ... u(k+i) ]T

and

Y( j )= [ y ( k + i + j - 2 ) y ( k + i + j - 3 ) ... y ( k + i - 1 ) ]T.

We will give an updating scheme for the Q R decomposition associated with the least squares problem which can carried out in conjunction with the decomposition updating to provide a solution to the system identification problem. It would be nice if the updating could be performed in O(n2), and in a sense this is possible: Unfortunately there is a problem: The system matrices will be updated through several intermediate stages during the process of updating the decomposition of WiT T and WiT T. If at one of these stages R14 is ill conditioned, as would be expected prior to a deflation, then the least squares problem will also be ill conditioned. The connection between the conditioning of R14 and that of the least squares problem is obvious since R14 is, neglecting the small elements, the


n • n principle submatrix of the QR decomposition for (6). This temporary ill conditioning cart introduce large errors in the system matrices as updating is carried out.

Thus there are two possible approaches to updating the system matrices presented in this paper. The first is the numerically safe approach of updating the Q R decomposition for (6) and then do a back substitution to get {Aj, Bj, Cj, Dj}. This avoids large errors due to a temporarily ill conditioned R14, but it is, unfortunately, an O(n 3) process. The other possibility is art O(n 2) algorithm which updates {Aj, Bj, Cj, Dj} as the generalized URV decomposition is updated.

Both approaches require the Q R factorization of

I so that will be dealt with first. As the generalized URV decomposition is updated, the size of WjTTV24 can change due to changes in the size of R14 during the updating. This can be dealt with by computing the Q R decomposition of the expanded matrix,

P =

We then obtain the required R factor as the (n + m) • (n + m) principal submatrix of the expanded R.

Suppose a row is added to WiT T and WiT T. This corresponds to a row being added to P,

o QT = ~R , (v) o Q r

where

is a square orthogonal matrix and Q1 has the same number of columns as P. If P has full column rank, then Q1 is art orthogonal basis for the range space of P. Otherwise, range of P is contained in the range of Q1.

To deal with the right hand side, we define

and keep track of the matrix QTs. When a row is added we get

[0 o] ~QTS 1" (s/ Before any updating is done on the generalized URV decomposition, we can apply a

standard QR updating to (7). The rotations which accomplish this are applied to (8) at the same time. Since it is not necessary to store Q, the memory required by this approach does not grow with time.

Once the Q R decomposition has been updated, the generalized URV updating can be


performed. Each time a right rotation is performed and V2 is updated, a corresponding right rotation is performed on P and S. The rotation performed on P destroys the QR decomposition of P. Since all of the right rotations which are used to update the generalized UI~V decomposition operate on adjacent columns, there are clearly three ways in which the QR decomposition can be damaged. The simplest is when the update to V2 only affects one of the matrices V23, V24, or V25. In this case the rotation operates on two adjacent columns of P and hence merely creates a single nonzero element on the sub diagonal of the R factor of P. To zero this element requires just one left rotation which is applied to both P and S.

The other possibilities are when the update to V2 affects the last column of V24 and the first column of V2s or the first column of V24 and the last column of V23. Since they do not correspond to adjacent columns of P, they create more nonzeros than the first case. To restore the QR decomposition after one such right rotation is an O(n 2) process. Fortunately, the number of these rotations is bounded independently of n, so that the overall process is still O(n2). It can easily be shown that it is possible to deal with changes in the block sizes of P and S due to changes in the sizes of R14 and R23 in O(n 2) by using similar techniques.

Once the URV updating has been completed and the Q R decomposition of P has been maintained, we have t h e / / f a c t o r for the least squares problem in the form of the (n + m) x (n + m) principal submatrix of the R factor of P. Similarly if we take as the right hand side the (n + m) x (n + l) principal submatrix of QT1s , then we can do a triangular backsolve to find the system matrices.

There are three sorts of updates which must be performed on the least squares solution. The first is to deal with a new row which is added to the problem. If we look at the submatrix of P,

and the submatrix of S

([ S1

that define the least squares problem

AT cT P1 B~ D~ =$1,

then when a row is added, we would like to find the solution to the least squares problem

pT] A T. cT] ,s T ]

The normal equations are

AT cT

Using the Sherman-Morrison-Woodbury formula, the solution can be written as

[A T.~. C T ]aT . = (P1Tp1)_lpTs 1 + x ( pTx sT_ xTP1TS1) .


where x = (pTp1)-lp. The new least squares solution is just a rank one modification of the old solution. This modification can be computed in O(n 2) flops using the R factor for P1, the updating of which was described earlier.

Once the new row has been incorporated into the least squares solution, the Q R decomposition for P with the new row added can be computed as described earlier. When that has been done, the generalized URV updating can commence with the Q R decomposition being updated as V2 is changed. The final part of the identification problem is to update {Aj,Bj, Cj, Dj} as V2 is changed and the partitioning is changed.

Rotations which only affect V23 and V25 will not affect the least squares solution. The simplest case in which something actually has to be done is one in which the rotation affects only V24. Suppose some rotation V is applied to the state estimate portions of P1 and $1. Then the normal equations become

0 Im pTp1 V 0 V 0 pT.,c I V 0 o I m B = O I m O I t

so the new solution is

AT. c T. V O ] T [ 0 I t "

The same rotation which is applied to P1 and 5'1 can be applied to the right and left of the old solution to get the new system matrices.

Because the right rotations involved in updating the generalized URV decomposition always act on adjacent columns, we need only make special consideration of the case in which a rotation acts at the boundaries of one of the blocks of V:. It turns out that such rotations always occur when there is a change in the size of the R14 block. They can be dealt with by viewing the process as adding either the last column of V23 or the first column of V25 to V24 and then performing a rotation which acts purely within II24.

All that we need to deal with are rotations which act solely within one of the blocks V23, V24 or V25, which has already been covered, and the process of adding or deleting a column from the least squares problem. Each time R14 grows, we must bring a column from the W.iT2TV23 or WjTTV25 into the WjTTV24 block of P. Since the WjTTV24 and the U(j) blocks of P define, the least squares problem, this amounts to adding a column to the least squares problem. The same thing applies to S. Similarly, whenever a column is removed from R14, the least squares problem shrinks by a column.

Suppose we have the Q R decomposition of/)1 with the column p appended,

[ R l l r12] 0 0 0

and we have a solution to the least squares problem


Then the solution satisfies

0 r22 xT1 x22 -- qTS1 qTs "

/,From this we get four equations which can be used to update the solution

~11X12 "~- r12X22 --" Q1Ts

= qTs

and

T22X22 _. qT.

/,From the first of these equations, it is clear that if we wish to delete a column, we get a solution to the new least squares problem P1X = $1 of

x = + (9)

This can easily be computed in O(n2).

The reverse process, that of going from a solution, X, of the smaller problem, to the solution of the larger problem makes use of all four of the equations. First z22 can be computed from the last equation, x21 from the third, x12 from the second, and Xll using (9). The necessary products QITs, qTs1 and qTs will be available from the part of the identification algorithm which updates the Q R decomposition and the transformed right hand side. Again, the whole process can be carried out in O(n 2) flops.

Acknowledgments

This research was supported by the National Science Foundation under Grant CCI~ 9209349

References

[1] M. Moonen, B. De Moor, L. Vandenberghe and J. Vandewalle. On- and Off-line Iden- tification of Linear State-space Models. Int. J. Control 49, pp 219-232, 1989.

[2] M. Moonen. Jacobi-type Updating Algorithms for Signal Processing, Systems Identifi- cation and Control. PhD Thesis, Katholieke Universiteit Leuven, 1990.

[3] C. C. Paige. Some Aspects of Generalized QR Factorizations. In : M. G. Cox and S. Hammarling (Eds.), Reliable Numerical Computation, Oxford Univ. Press, pp. 73-91, 1990.

[4] G. W. Stewart. An Updating Algorithm for Subspace Tracking. IEEE Transactions on Signal Processing 40, pp. 1535-1541, 1992.

[5] M. Stewart and P. Van Dooren, A QURV Updating Algorithm. In : J. G. Lewis (Ed.), Proceedings of the Fifth SIAM Conference on Applied Linear Algebra, SIAM, pp. 269- 273, 1994.


449

S U B S P A C E T E C H N I Q U E S I N B L I N D M O B I L E R A D I O C H A N N E L I D E N T I F I C A T I O N A N D E Q U A L I Z A T I O N U S I N G F R A C T I O N A L S P A C I N G A N D / O R M U L T I P L E A N T E N N A S

D.T.M. SLOCK Mobile Communications Department Eurecom Institute 2229 route des Crates, BP 193 F-06904 Sophia Antipolis Cedex, France [email protected]

ABSTRACT. Equalization for digital communications constitutes a very particular blind deconvolution problem in that the received signal is cyclostationary. Oversampling (w.r.t. the symbol rate) of the cyclostationary received signal leads to a stationary vector-valued signal (polyphase representation). Oversampling also leads to a fractionally-spaced channel model and equalizer. In the polyphase representation, channel and equalizer can be considered as an analysis and synthesis filter bank. Zero-forcing (ZF) equalization corresponds to a perfect-reconstruction filter bank. We show that in the oversampling case FIR (Finite Impulse Response) ZF equalizers exist for a FIR channel. In the polyphase representation, the noise-free multichannel power spectral density matrix has rank one and the channel can be found as the (minimum-phase) spectral factor. The multichannel linear prediction of the noiseless received signal becomes singular eventually, reminiscent of the single-channel prediction of a sum of sinusoids. As a result, a ZF equalizer can be determined from the received signal second-order statistics by linear prediction in thenoise-free case, and by using a Pisarenko-style modification when there is additive noise. In the given data case, Music (subspace) or Maximum Likelihood techniques can be applied. We also present some Cramer-Rao bounds and compare them to the case of channel identification using a training sequence.

KEYWORDS. Subspace techniques, multichannel linear prediction, eigen decomposition, blind equalization, channel identification, cyclostationarity, music, maximum likelihood.

450 D.T.M. Slock

1 F R A C T I O N A L L Y - S P A C E D C H A N N E L S A N D E Q U A L I Z E R S , A N D FIL- T E R B A N K S

Consider linear digital modulation over a linear channel with additive Gaussian noise so that the received signal can be written as

y(t) = ~ a k h ( t - k T ) + v(t) (1) k

where the ak axe the transmitted symbols, T is the symbol period, h(t) is the (overall) ~h~nnel impulse re~pon~e. A ~ u m i n g the (ak} ~nd (~ ( t ) } to be (wide-~en~e) ~ t ~ t i o ~ r r , the process {y(t)} is (wide-sense) cyclostationaxy with period T. If {y(t)} is sampled with period T, the sampled process is (wide-sense) stationary and its second-order statistics contain no information about the phase of the channel. Tong, Xu and Kailath [6] have proposed to oversample the received signal with a period A = T/m, m > 1. This leads to m symbols-spaced channels. The results presented here generalize the results in [3] where an oversampling factor ra = 2 was considered. As an alternative to oversampling, multiple channels could also arise from the use of multiple antennas. Corresponding to each antenna signal, there is a channel impulse response. Each antenna signal could furthermore be oversampled. The total number of symbol rate channels is then the product of the number of antennas and the oversampling factor. In what follows, we use the terminology of the case of one antenna. The case of multiple synchronous transmitting sources is treated in [4].

We assume the channel to be FIR with duration of approximately NT. With an oversampling factor m, the sampling instants for the received signal in (1) are to+T(k + ~ ) for integer k and j = 1, 2 , . . . , m. We introduce the polyphase description of the received signal: yj(k) = y(to+T(k + L~_.!)) for j = 1 , . . . , m are the m phases of received signal, and similarly for the channel impulse response and the additive noise. The oversampled received signal can now be represented in vector form at the symbol rate as

N-1 y(k) = ~ h(i)ak_i + v(k) = HNAN(k) + v(k) ,

i-o

�9 , v ( k ) = �9 , h ( k ) = �9 (2)

y(k)= y ik) h ik)J [ . . ]" HN = [ h ( 0 ) . . . h ( N - 1 ) ] , A ~ ( k ) = ak ""ak_N§

where superscript H denotes Hermitian transpose. We formalize the finite duration N T assumption of the channel as follows

(AFIR) �9 h(0) 7~ 0, h ( N - 1 ) 7~ 0 and h(/) = 0 for i < 0 or i >__ N. (3)

The z-transform of the channel response at the sampling rate ~- is/-/(z) = ~ z-(J-1)/ / j (z~). j-1

Similarly, consider a fractionally-spaced (~-)equalizer of which the z-transform can also be decomposed into its polyphase components" F(z) = ~ = 1 z(J-1)Fj(z'~), see Fig. 1. A1-

Blind Mobile Radio Channel Identification 451

though this equalizer is slightly noncausal, this does not cause a problem because the discrete-time filter is not a sampled version of an underlying continuous-time function. In fact, a particular equalizer phase z(J-1)Fj(z m) follows in cascade the corresponding channel phase z-(J-1)Hj(z m) so that the cascade Fj(zm)Hj(z m) is causal. We assume the equalizer phases to be causal and FIR of length L: Fj(z) - x--,L-1 f rk~z-k, j = 1, . . . ,m. - - z - . , k = O J~, /

[qli!iii!i!iiii!i i!iiiiiiiiii!i]_ ................................................ Hi!ii!i!i!i!i! !iiiiiii!!!iiiiill iiii!iiiiiiiN!iii!i!i!i!!l -

Figure 1" Polyphase representation of the T/m fractionally-spaced channel and equalizer for m = 2.

2 F I R Z E R O - F O R C I N G (ZF) E Q U A L I Z A T I O N

We introduce f(k) . . . . [fl(k) fro(k)], FL = [f(0) .. �9 f(L-1)], H(z) = ~k=og-1 h(k)z-k and F ( z ) L-~ -k z - ~ = ~k=o f(k)z . The condition for the equalizer to be ZF is F(z)H(z) - where n = 0, 1 , . . . , N + L - 2 . The ZF condition can be written in the time-domain as

FL T L ( H N ) : [0 . . .0 1 0 . . .0] (4)

where the 1 is in the n+ 1 st position and TM (x) is a (block) Toeplitz matrix with M (block)

and Ix 0p• as first (block) row (p is the number of rows in x). (4)is a r o w s system of L+N-1 equations in Lm unknowns. To be able to equalize, we need to choose the equalizer length L such that the system of equations (4) is exactly or underdetermined. Hence

- - , ~ 1 ' (5)

We assume that HN has full rank if N >_ m. If not, it is still possible to go through the developments we consider below. But lots of singularities will appear and the non-singular part will behave in the same way as if we had a reduced number of channels, equal to the row rank of HN. Reduced rank in HN can be detected by inspecting the rank of Ey(k)yH(k). If a reduced rank in HN is detected, the best way to proceed (also when quantities are estimated from data) is to preprocess the data y(k) by transforming them into new data of dimension equal to the row rank of HN.

The matrix TL (HN) is a generalized Sylvester matrix. It can be shown that for L _ L_. it has full column rank if the FIR assumption (3) is satisfied, and if H(z) # 0, Vz or in other words if the Hi(z) have no zeros in common. This condition coincides with the identifiability condition of Tong et al. on H(z) mentioned earlier. Assuming TL (HN) to have full column rank, the nullspace of T H (HN) has dimension L(m-1 ) -N+I . If we take the entries of any vector in this nullspace as equalizer coefficients, then the equalizer output is zero, regardless of the transmitted symbols.

452 D.T.M. Slock

To find a ZF equalizer (corresponding to some delay n), it suffices to take an equalizer length equal to L__. We can arbitrarily fix L _ ( m - 1 ) - N + I equalizer coefficients (e.g. take L_ . (m-1) -N+I equalizer phases of length L_-i only). The remaining L__+N-1 coefficients can be found from (4) i f H(z) # 0, Vz. This shows that in the oversampled case, a FIR equalizer suffices for ZF equalization! With an oversampling factor m = N, the minimal required total number of equalizer coefficients N is found (L__. = 1).

3 C H A N N E L I D E N T I F I C A T I O N F R O M S E C O N D - O R D E R S T A T I S T I C S : F R E Q U E N C Y D O M A I N A P P R O A C H

Consider the noise-free case and let the transmitted symbols be uncorrelated with variance a 2. Then the power spectral density matrix of the stationary vector process y(k) is

Syy(z) = a 2 H ( z ) H H ( z - * ) . (6)

The following spectral factorization result has been brought to our attention by Loubaton [1]. Let g ( z ) be a m • 1 rational transfer function that is causal and stable. Then g ( z ) i s called minimum-phase if K(z) # 0, Iz] > 1. Syy(z) is a rational m • m spectral density matrix of rank 1. Then there exists a rational m • 1 transfer matrix K(z) that is causal, stable , minimum-phase, unique up to a unitary constant, of (minimal) McMillan degree deg(g) = �89 deg(Syy)such that

Syy(Z) = g ( z ) E l l ( z - * ) . (7)

In our case, Syy is polynomial (FIR channel) and H(z) i s minimum-phase since we assume H(z) ~ 0, Vz. Hence, the spectral factor K(z)identifies the channel

K ( z ) = as e jr H(z) (8)

up to a constant ac, eJr So the channel identification from second-order statistics is simply a multivariate MA spectral factorization problem.

4 GRAM-SCHMIDT ORTHOGONALIZATION, TRIANGULAR FACTOR- IZATION AND LINEAR PREDICTION

4.1 UDL FACTORIZATION OF THE INVERSE COVARIANCE MATRIX

Consider a vector of zero mean random variables Y = yH y H . . . y . We shall intro-

duce the notation Yl:M -- Y . Consider Gram-Schmidt orthogonalization of the components of Y. We can determine the linear least-squares (lls) estimate yi of yi given Yl:i-1 and the associated estimation error Y'i as

y~ = ~1~:,_~ = R~,~:,_~ R~:,_~:,_~ yl:~-~ , ~ = ~l~: ,_~ = y ~ - y~ (9)

where Rab = Eab H for two random column vectors a and b. The Gram-Schmidt orthogo-

.a z io. p ocess r of ge,er i.g r162 s r i.g with

y~ = yl. We can write the relation

L Y = Y (10)


where L is a unit-diagonal lower triangular matrix. The first i - 1 elements in row i of L R -1 ~From (10), we obtain axe -.Ryiyl:i_ 1 Y l : i - l Y l : i - 1 "

E ( L Y ) ( L y ) H = E ? ? H ~ L R y y L H = D = R ~ . (11)

D is indeed a diagonal matrix since the Y'i are decorrelated. Equation (11) can be rewritten as the UDL triangular factorization of Ry~

R ~ = L ' D - ~ L . (12)

If Y is filled up with consecutive samples of a random process, Y = [yH(k) y H ( k - 1 ) . . . y H ( k - M + l ) ] H, then the L become backward prediction errors of order i - 1 , the corresponding rows in L are backward prediction filters and the corresponding diagonal elements in D are backward prediction error variances. If the process is stationary, then R y y is Toeplitz and the backward prediction errors filters and variances (and hence the UDL factorization of R y e ) can be determined using a fast algorithm, the Levinson algorithm. If

Y is filled up in a different order, i.e. r = [ y g ( k ) y H ( k + l ) . . . y g ( k + M - 1 ) ] H, then the

backward prediction quantities become forward prediction quantities, which for the prediction error filters and variances are the same as the backward quantities if the process y(.) is scalar valued.

If the process y(.) is vector valued (say with m components), then there are two ways to proceed. We can do the Gram-Schmidt procedure to orthogonalize a vector component with respect to previous vector components. In this way, we obtain successively Yi = .~ilyl:~_ 1.

Applied to the multichannel time-series case, we obtain vector valued prediction errors and multichannel prediction quantities. The UDL factorization of R y y now becomes

R~ry = L 'HD'- IL ' (13)

in which L' and D' are block matrices with m x m blocks. L' is block lower triangular and its block rows contain the multichannel prediction error filter coefficients. The diagonal blocks in particular are Im. D' is block diagonal, the diagonal containing the m x m prediction

!

error variances Di. Alternatively, we can carry out the Gram-Schmidt factorization scalar component by scalar component. This is multichannel linear prediction with sequential processing of the channels. By doing so, we obtain the (genuine) UDL factorization

R;2y = L ~D -~L (14)

in which L is again lower triangular with unit diagonal and D is diagonal. When the vector process y(.) is stationary, R y y is block Toeplitz and appropriate versions of the multichannel Levinson algorithm can generate both triangular factorizations in a fast way. The relationship between the two triangular factorizations (13) and (14) is the following.

I I

Consider the UDL factorizations of the inverses of the blocks D i on the diagonal of D

D~-I H " = L i D i - l L i (15)

and let LII II II II .

= b t o ~ k d i a g ( L ~ , . . . , Z , M } , D = b t o ~ k d i a g ( D ~ , . . . , D M } (16)

454 D.T.M. Slock

Then we get from (13)-(16)

R ~ r y = L H D - 1 L = L 'HL"HD"- IL "L' =~ L = L"L' , D = D" (17)

by uniqueness of UDL triangular factorizations.

If the matrix R y y is singular, then there exist linear relationships between certain components of Y. As a result, certain components Yi will be perfectly predictible from the previous components and their resulting orthogonalized version Yi will be zero. The corresponding diagonal entry in D will hence be zero also. For the orthogonalization of the following components, we don't need this yi. As a result, the entries under the diagonal in the coresponding column of L can be taken to be zero. The (linearly independent) row vectors in L that correspond to zeros in D are vectors that span the null space of R y y . The number of non-zero elements in D equals the rank of R y y .

4.2 LDU FACTORIZATION OF A COVARIANCE MATRIX

Assume at first that R y y is nonsingular. Since the Y'i form just an orthogonal basis in the space spanned by the yi, Y can be perfectly estimated from Y. Expressing that the covariance matrix of the error in estimating Y from Y is zero leads to

0 = R y y - R y ~ l : l y ~ l : l . ~ y (is)

=~ R y y = ( R y ~ -1 R ~ ) R ~ (Rv~R?y) = V H D U

where D is the same diagonal matrix as in (14) and U = L -H is a unit-diagonal upper triangular matrix. (18) is the LDU triangular factorization of R y y . In the stationary multichannel time-series case, R y y is block Toeplitz and the rows of U and the diagonal elements of D can be computed in a fast way using a sequential processing version of the multichannel Schur algorithm.

When R y v is singular, then D will contain a number of zeros, equal to the dimension of the nullspace of R y y . Let J be a selection matrix (the rows of J are rows of the identity matrix) that selects the nonzero elements of D so that J D J H is a diagonal matrix that contains the consecutive non-zero diagonal elements of D. Then we can write

R y y = ( j u ) H ( j D - 1 j H ) ( j U ) (19)

which is a modified LDU triangular factorization of the singular R y y . (JU) H is a modified lower triangular matrix, its columns being a subset of the columns of the lower triangular matrix U g. A modified version of the Schur algorithm to compute the generalized LDU factorization of a singular block Toeplitz matrix R y y has been recently proposed in [7].

5 S I G N A L A N D N O I S E S U B S P A C E S

Consider now the measured data with additive independent white noise v(k) with zero 2 (in the complex case, real mean and assume Ev(k )vH(k) = a2Im with unknown variance a v

and imaginary parts are assumed to be uncorrelated, colored noise with known correlation structure but unknown variance could equally well be handled). A vector of L measured


data can be expressed as

YL(k) = TL (HN) AL+N-I(k) + VL(k) (20)

where V L ( k ) = [ y H ( k ) . . . y H ( k - L + I ) ] H and VL(k)is similarly defined. Therefore, the

structure of the covariance matrix of the received signal y(k) is 2 R y E Y L ( k ) y H ( k ) = TL (HN)R ~ 7;, H (HN) + a, ImL (21) "- L + N - 1 L

where R~ = EAL(k)AH(k). We assume R ~ to be nonsingular for any M. For L >_ L_., and assuming the FIR assumption (3) and the no zeros assumption, H(z) ~ 0, Yz, to hold,

2 then TL (HN) has full column rank and a v can be identified as the smallest eigenvalue of R y. Replacing R y by R y -a2ImL gives us the covariance matrix for noise-free data. Given

the structure of R y in (21), the column space of TL (HN) is caned the signal subspace and its orthogonal complement the noise subspace.

Consider the eigendecomposition of R y of which the real positive eigenvalues are ordered in descending order:

L + N - 1 m L

R [ = , v,. + v,. v, = A v [ + v . A v # ( 2 2 ) i=x i = L + N

2 where A2r = avI(m_l)L_N+l (see (21)). The sets of eigenvectors Vs and VX are orthogonal: 2 i = 1 . . . , L + N - 1 . We then have the following equivalent v H v H = 0, and Ai > a , ,

descriptions of the signal and noise subspaces

Range {Vs} = Range {TL (HN)} , VffTL (HN) = 0. (23)

ZF E Q U A L I Z E R A N D N O I S E S U B S P A C E D E T E R M I N A T I O N F R O M S E C O N D - O R D E R S T A T I S T I C S BY M U L T I C H A N N E L L I N E A R P R E - D I C T I O N

We consider now the noiseless covariance matrix or equivalently assume noisefree data: v(t) - O. We shall also assume the transmitted symbols to be uncorrelated, R ~ = a2aIM, though the noise subspace parameterization we shall obtain also holds when the transmitted symbols are correlated.

Consider now the Gram-Schmidt orthogonalization of the consecutive (scalar) elements in the vector YL(k). We start building the UDL factorization of (RY) -x and obtain the consecutive prediction error filters and variances. No singularities are encountered until we arrive at block row L in which we treat the elements of y(k-L_.+l). From the full column rank of TL_(HN), we infer that we will get m = mL__-(L+N-1) E {0, 1 , . . . , m - 2 } singularities. If m > 0, then the following scalar components of Y become zero after orthogonalization: ~ ' i (k-L+l) = 0, i = r e + l - m , . . . , m . So the corresponding elements in the diagonal factor D are also zero. We shall call the corresponding rows in the triangular factor L singular prediction filters.

For L = L__+I, T_L+I (HN) has m more rows than TL(HN) but only one more column. Hence the (column) rank increases by one only. As a result ~l(k-L_.) is not zero

456 D.T.M. Mock

in general while y i (k-L) = 0, i = 2 , . . . , m (we assume h i ( N - l ) ~ 0. The ordering of the channels can always be permuted to make this true since h ( N - 1 ) ~ 0). Fur- thermore, since TL(HN) has full column rank, the orthogonalization of y l ( k - L ) w.r.t. YL_(k) is the same as the orthogonalization of y l (k -L ) w.r.t. AL+N-I(k). Hence, since the ak are assumed to be uncorrelated, only the component of yl(k-L__) along ak-L-N+l remains" ~'l(k-L_.) = hl(N--1)ak-L-N+l. This means that the corresponding prediction filter is (proportional to) a ZF equalizer! Since the prediction error is white, a further increase in the length of the prediction span will not improve the prediction. Hence y"~(k-L) = h~(N-1)ak-L-y+~, L > L and the prediction filters in the corresponding rows of L will be appropriately shifted versions of the prediction filter in row m L + 1. Simi- larly for the prediction errors that are zero, a further increase of the length of the prediction span cannot possibly improve the prediction. Hence ~i(k-L) = O, i = 2 , . . . , m , L >_ L. The singular prediction filters further down in L are appropriately shifted versions of the first m - 1 singular prediction filters. Furthermore, the entries in these first m - 1 singular prediction filters that appear under the l 's ("diagonal" elements) are zero for reasons we explained before in the general orthogonalization context. So we get a (rank one) white prediction error with a finite prediction order. Hence the channel ouput process y(k) is autoregressive. Due to the structure of the remaining rows in L being shifted versions of the first ZF equalizer and the first m - 1 singular prediction filters, after a finite "transient", L becomes a banded lower triangular block Toeplitz matrix.

Consider now L > L and let us collect all consecutive singular prediction filters in the triangular factor L into a ( (m-1) (L - L_)+m) x (mL) matrix ~L. The row space of gL is the (transpose of) the noise subspace. Indeed, every singular prediction filter belongs to the noise subspace since ~L"ff L (HN) = 0 , all rows in gL are linearly independent since they are a subset of the rows of a unit-diagonal triangular matrix, and the number of rows in ~L equals the noise subspace dimension. ~L is a banded block Toeplitz matrix of which the first m - l - m rows have been omitted. ~L is in fact parameterized by the first m - 1 singular prediction filters. Let us collect the nontrivial entries in these m - 1 singular prediction filters into a column vector GN. So we can write ~L(GN). The length of GN can be calculated to be

m ( ( L _ _ - l ) m + m - m ) + ( m - l - m ) ( ( L _ . - 1 ) m + m - m + l ) = m N - I (24)

which equals the actual number of degrees of freedom in HN (the channel can only be determined up to a scalar factor, hence the -1) . So ~L(GN) represents a minimal linear parameterization of the noise subspace.

7 C H A N N E L I D E N T I F I C A T I O N BY C O V A R I A N C E M A T R I X T R I A N G U - L A R F A C T O R I Z A T I O N

Consider now the triangular factorization of R y. Since R y is singular, we shall end up with a factorization of the form (19). For L = L_, we have exactly m singularities. Going from L = L to L = L+I , the rang increases by only one and only one colum gets added to (JU) g. Since the corresponding nonzero orthogonalized variable is ~'l(k-L__) = hl(N--1)ak-L-g+l,


the corresponding column in the factor (JU) H of R y is

2[01• . .hH(0) 0 . . ] H (25) E Y~(k)~H(k-_L) = h H ( N - 1 ) a~ _ �9 �9

which hence contains the channel impulse response, apart from a multiplicative factor. For the remaining columns of (JU) H, we have ~ l ( k - L ) = hl (W-1)ak-L-N+X , L >_ L_, and hence the remaining columns of (JU) H are just shifted down versions of the column in (25). Hence, after a finite transient, (JU) H becomes a lower triangular block Toeplitz matrix. The elements of this block Toeplitz matrix are a certain multiple of the multichannel impulse response coefficients. Since the channel output is obviously a multichannel moving average process, R y and its triangular factor (JU) H are banded.

This is the time-domain equivalent of the frequency domain spectral factorization result. In the frequency domain, the channel is obtained as the minimum-phase spectral factor of the power spectral density matrix. In the time-domain, the channel is obtained by triangular factorization of the covariance matrix. Due to a combination of the FIR assumption and the singularity of the power spectral density matrix, this time-domain factorization reaches a steady-state after a finite number of recursions. Recall that in the single-channel case (MA process), the minimum-phase spectral factor can also be obtained from the triangular factorization of the covariance matrix. However, the factorization has to be pursued until infinity for convergence of the last line of the triangular factor to the minimum-phase spectral factor to occur.

8 C H A N N E L E S T I M A T I O N F R O M A N E S T I M A T E D C O V A R I A N C E SE- Q U E N C E B Y S U B S P A C E F I T T I N G

See [5] for a discussion of this approach.

9 C H A N N E L E S T I M A T I O N F R O M DATA U S I N G D E T E R M I N I S T I C M L

The transmitted symbols ak are considered deterministic, the stochastic part is considered to come only from the additive noise, which we shall assume Gaussian and white with

2 We assume the data YM(k) to be available. The zero mean and unknown variance a v. maximization of the likelihood function boils down to the following least-squares problem

rain [JYM(k) -- 7 M (HN) AM+N-I(k)[[~ �9 (26) HN,AM+N-1 (k)

The optimization problem in (26) is separable. Eliminating A u + N - l ( k ) in terms of HN, we get

IP II rain P• )YM(k) (27) HN ~'M (HN 2

subject to a nontriviality constraint on HN. In order to find an attractive iteratNe procedure for solving this optimization problem, we should work with a minimal parameterization of the noise subspace, which we have obtained before. Indeed,

P~M (HN) = PanM(aN)" (28)

458 D.T.M. Slock

The number of degrees of freedom in HN and GN is both raN-1 (the proper scaling factor cannot be determined). So HN can be uniquely determined from GN and vice versa. Hence, we can reformulate the optimization problem in (27) as

min GN 2

Due to the (almost) block Toeplitz character of GM, the product GMYM(k) represents a convolution. Due to the commutativity of convolution, we can write gM(GN)YM(k) = YN(YM(k))[1 GH] H for some properly structured YN(YM(k)). This leads us to rewrite (29) ~

[ ]H 1] (30) min 1 Y~v(YM(k)) (~HM(GN) ~M(GN)) -1YN(YM(k)) GN aN GN

This optimization problem can now easily be solved iteratively in such a way that in each iteration, a quadratic problem appears [2]. An initial estimate may be obtained from the subspace fitting approach discussed above. Such an initial estimate is consistent and hence one iteration of (30) will be sufficient to generate an estimate that is asymptotically equivalent to the global optmizer of (30). Cramer-Rao bounds have been obtained and analyzed in [5].

References

[1] Ph. Loubaton. "Egalisation autodidacte multi-capteurs et systhmes multivariables". GDR 134 (Signal Processing) working document, february 1994, France.

[2] L.L. Scharf. Statistical Signal Processing. Addison-Wesley, Reading, MA, 1991.

[3] D.T.M. Slock. "Blind Fractionally-Spaced Equalization, Perfect-Reconstruction Filter Banks and Multichannel Linear Prediction". In Proc. ICASSP 94 Conf., Adelaide, Australia, April 1994.

[4] D.T.M. Slock. "Blind Joint Equalization of Multiple Synchronous Mobile Users Using Oversampling and/or Multiple Antennas". In Proc. 28th Asilomar Conf. on Signals, Systems and Computers, Pacific Grove, CA, Oct. 31- Nov. 2 1994.

[5] D.T.M. Slock and C.B. Papadias. "Blind Fractionally-Spaced Equalization Based on Cyclostationarity". In Proc. Vehicular Technology Conf., Stockholm, Sweden, June 1994.

[6] L. Tong, G. Xu, and T. Kailath. "A New Approach to Blind Identification and Equal- ization of Multipath Channels". In Proc. of the 25th Asilomar Conference on Signals, Systems ~J Computers, pages 856-860, Pacific Grove, CA, Nov. 1991.

[7] K. Gallivan, S. Thirumalai, and P. Van Dooren. "A Block Toeplitz Look-Ahead Schur Algorithm". In Proc. 3rd International Workshop on SVD and Signal Processing, Leu- ven, Belgium, Aug. 22-25 1994.


459

R E D U C T I O N O F G E N E R A L B R O A D - B A N D N O I S E I N S P E E C H B Y T R U N C A T E D Q S V D : I M P L E M E N T A T I O N A S P E C T S

S.H. JENSEN ESAT--Department of Electrical Engineering Katholieke Universiteit Leuven Kardinaal Mercierlaan 9~ B-3001 Heverlee, Belgium Soren.Jensen @esat.kuleuven. ac. be

P.C. HANSEN UNI . C, Building 304 Technical University of Denmark DK-2800 Lyngby, Denmark Per. [email protected]

S.D. HANSEN, J.A. SORENSEN Electronics Institute, Building 3~9 Technical University of Denmark DK-2800 Lyngby, Denmark { sdh,jaas} @ei.dtu.dk

ABSTRACT. In many speech processing applications an appropriate filter is needed to remove the noise. The truncated SVD technique has a noise filtering effect and, provided that the noise is white, it can be applied directly in noise reduction algorithms. However, for non-white broad-band noise a pre-whitening operation is necessary. In this paper, we focus on implementation aspects of a newly proposed QSVD-based algorithm for reduction of general broad-band noise in speech. A distinctive advantage of the algorithm is that the pre- whitening operation is an integral part of the algorithm, and this is essential in connection with updating issues in real-time applications. We compare the existing implementation (based on the QSVD) with an implementation based on the ULLV decomposition that can be updated at a low cost.

KEYWORDS. Speech processing, speech enhancement, noise reduction, quotient singular value decomposition, ULLV decomposition.

460 S.H. Jensen et al.


At a noisy site, e.g., the cabin of a moving vehicle, speech communication is affected by the presence of acoustic noise�9 This effect is particularly serious when linear predictive coding (LPC) [7] is used for the digital representation of speech signals at low bit rates as, for instance, in digital mobile communication. Low-frequency acoustic noise severely affects the estimated LPC spectrum in both the low- and high-frequency regions�9 Consequently, the intelligibility of digitized speech using LPC often falls below the minimum acceptable level. In [4], we described an algorithm, based on the quotient singular value decomposition (QSVD) [2], for reduction of general broad-band noise in speech�9 Our algorithm, referred to as the truncated QSVD algorithm hereafter, first arranges a segment of speech-plus-noise samples and a segment of noise samples in two separate Hankel matrices, say H and N. Then it computes the QSVD of ( H , N ) and modifes H by filtering and truncation, and finally it restores the Hankel structure of the modified H. In this way, the pre-whitening operation becomes an integral part of the algorithm. The resulting modified data segment can be considered as enhanced speech.

In this paper, we focus on implementation aspects of the truncated QSVD algorithm. Section 2 summarizes the truncated QSVD algorithm. Section 3 addresses some updating issues related to the algorithm and suggest an implementation by means of a related decomposition, the rank-revealing ULLV decomposition [6]. Section 4 contains experiments that compare the truncated QSVD algorithm with an implementation based on the ULLV decomposition�9

2 T H E T R U N C A T E D Q S V D A L G O R I T H M

We consider a noisy signal vector of N samples:

= [~o, ~ , . . . ~ N - ~ ] r, (i) and we assume that the noise is additive,

x = ~ T n , (2)

where .~ contains the signal component and n represent the noise. From x we construct the following L • M Hankel matrix H, where M -t- L = N - 1 and L >_ M:

X0 Xl "'" XM-1/ H = xl X2 �9 �9 �9 XM

X L - I XL �9 �9 �9 X N - I

(3)

We can always write H as

H = I=I + N, (4)

where I=I and N represent, respectively, the Hankel matrices derived from ~ and n in (2). Moreover, we assume that I=I is rank deficient, rank(I=I) = K < M, and that H and N have full rank, rank(H) = rank(N) = M. This assumptions are, e.g., satisfied when the samples xi of ~ consist of a sum of K sinusoids, and the samples ni of n consist of white noise

Reduction of General Broad-band Noise in Speech 461

or non-white broad-band noise. A sinusoidal model has often been attr ibuted to speech signals, cf. [9].

Our interest in the ordinary SVD is to formulate a general noise reduction algorithm that can be applied directly to the data matrix H = I=I + N, and give reliabel estimates of I=I. Provided that the signal is uncorrelated with the noise in the sense that H is orthogonal to N' I:ITN - 0, and the noise is white in the sense that N has orthogonal columns and

2 every column of N has norm anoise: NTN = anoiseI, then the MV estimate of I=I can be found [1] from the SVD of H'

H = U d i a g ( a l , . . . , aM) V T, (5)

where U and V are matrices with orthonormal columns. Specificly, by setting the K • K filter matrix

F - diag 1 - an~ .. 1 - an~ (6) , . , ,

the MV-estimate of H is given by

( F ~ I 0 ) v T (7) I~'Iest "-- U 0 0 "

Unfortunately, IT"Iest is not Hankel. So, to obtain a signal vector ~ corresponding to the MV estimates we need to make a Hankel matrix approximation to I=I~t. A simple way to compute a Hankel matrix approximation is to arithmetically average every antidiagonal of I=Iest, and put each average-value as a common element in the corresponding diagonal of a new Hankel matrix of the same dimension [10].

If the noise is broad-band but not white, NTN ~ 2 anoiseI , then a pre-whitening matrix R -1 can always be applied to H. From N, which is constructed from x in "silent" periods in the speech, we can compute R via a QR decomposition,

N=QR. (8)

In the colored-noise case, we then consider the matrix

X = H R -I = ITIR -1 + NR-1; ( 9 )

the pre-whitening operation does not change the nature of the linear model while it diagonalizes the covariance matrix of the noise as shown by ( N R - 1 ) T ( N R -1) = QTQ = I. It follows that the MV estimate of I=IR -1 can be found by applying the same procedure as outlined above. The only modification is that the MV estimate of I=IR -1 should be de-whitened before restoring the Hankel structure.

The MV estimate is, of course, not the only possible estimate of I=I or I=IR -1. For example, by setting

F = d i a g 1 - an~ . . . 1 - an~ ( 1 0 ) , , ,

we obtain the well-known estimate used in [5] and by setting

F - - - I K , ( 1 1 )

462 S . H . J e n s e n e t al.

we obtain the classical LS estimate.

We are now in a position to formulate the general SVD-based noise reduction algorithm using conventional pre-whitening.

Algor i thm 1

Input : (H, N) and K.

Output: I:I.

1. Compute the QR decomposition of N:

N = Q R .

2. Perform a pre-whitening of H:

X = HR -1.

3. Compute the SVD of X

X = U diag(o'l,... , O'M) V T

4. Truncate X to rank K and filter diag(al , . . . ,OK) by F:

"Xest -- U ( F d i a g ( a l ' " " " ' 00) VT"

5. Perform a de-whitening of Xest:

(12)

(14)

(15)

xi = ~ _ a + l ~ z ( i - k + 2 ' k ) ' (18) k=a

with a = max(l, i - L + 2) and # = min(M, i + 1).

In principle, one should repeat Steps 2 - 6 until I:I converges to a Hankel matrix with exact rank K. In practise, we do not perform this iteration.

A major disadvantage of Algorithm 1 is that the explicit use of the matrix R -1 may result in loss of accuracy in the data. Moreover, it is complicated to update the matrix X = H R -1 when H and N are updated, e.g. in a recursive algorithm. The explicit use of R, and also the QR decomposition of N, can be avoided by working directly with H and N using the QSVD of the pair of matrices (H, N), which delivers the required factorization

where

;~0 Xl .. . ;~M-1/ i2 I = ;~1 X2 . . . ~M (17)

�9 : : �9

;~L-1 ;~L .. . ;~N-1

6. Compute I:I from Z by arithmetic averaging along its antidiagonals:

z = X~,R. (16)


without forming quotients and products. To see that this is true, write the QSVD of the matrix pair (H, N) as

H = U diag(~l, ~2, . . . , ~M) | -1, N - V diag(#l, # 2 , . . . , #M) | -1, (19)

where U and V are matrices with orthonormal columns, and | is nonsingular. In addition, we use that R. = QTN. By substituting this and (19) into X = H R -1 we obtain

X = H i t -1 = U diag(~x/#l , . . . , ~M/#M)(QTv)T; (20)

i.e., U, d iag(~l /# t , . . . ,~M/~M) and Q T v in the QSVD of (H ,N) are identical to the SVD of X = HR. -1, with ai = ~i/#i. Accordingly, Algorithm 1 can be reformulated by means of the QSVD as outlined below:

A lgo r i t hm 2 (T runca t ed QSVD)

Input: (H, N) and K.

Output: I:I.

1. Compute the QSVD of (H, N):

H = U diag(~l , . . . ,~M) | N = V d iag(# l , . . . ,#M) | (21)

2. Truncate H to rank K and filter diag(~l , . . . ,~g) by F:

I'[Iest---- U ( F d i a g ( ~ o ' ' " ~ K ) oO) 0 _ 1. (22)

3. Compute I:I from Hest by arithmetic averaging along its antidiagonals, cf. Step 6 in Algorithm 1.

Notice that the pre-whitening operation is now an integral part of the algorithm, not a separate step. We mention in passing that a similar use of the truncated QSVD is suggested in connection with regularization problems in [3].

3 I M P L E M E N T A T I O N A S P E C T S

In real-time signal processing, it is desirable to update matrix decompositions instead of recomputing them. In connection with the truncated QSVD algorithm, we see from (22) that we, in general, need to update U, diag(~l, ~2,. . . , ~M), d iag(#l ,#2 , . . . , #M), and | Notice that F in (6) and (10) requires the first K singular values of Hl:t -1, i.e., the first K quotient singular values of (H, N); and the quantity an| e 2 , which can be estimated as

2 = (M K) -1 ~MK+ 1 a 2. Algorithms for updating the QSVD is a topic of current ffnoise research. Anyway, the QSVD would be difficult to update, and may not be the best choice in a practical application.

A promising alternative is to use the rank-revealing ULLV decomposition that can be updated at a low cost. In this decomposition, the matrix pair (H, N) is written as

H = U L L1 W T, N -- 'Q L1 W T, (23)


where I~I, ~r, and W are matrices with orthonormal columns, while L and L1 are lower triangular matrices. In particular,

L= G E '

where LK is K x K with ~min(LK)~ aK, and IIGII~- IIEII~ ~ a~+ 1 +...+~; ~min(LK) denotes the smallest singular value of LK and IIGIIF (and I]EIIF)is the Frobenious norm of G (and E). By analogy with (20), we can then write

i.e., the rank K of H R -1 is displayed in the matrix L in that LK is well conditioned, and that [[(~'I]F and I]EI]F are small. In addition, 01 = [51,. . . , IlK] (and 02 = [UK+I , ' " , UM]) represents approximately the same space as U1 = [Ul," ", UK] (and U2 = [UK+l, ' ' ", UM]). The same is true for V and V. Hence, the ULLV decomposition in (23) yields essentially the same rank and subspace information as the QSVD does, and the approximate subspaces are typically very accurate.

The advantage of the ULLV decomposition is that the matrices H and N can be recursively updated in an efficient way, which will lead to an adaptive algorithm for reduction of broad-band noise in speech.

The implementation of the truncated QSVD algorithm by means of the ULLV decomposition is straightforward when the LS estimate is used. In this case, Step 2 in Algorithm 2 becomes:

0 w r. (26) ffILs=fd (LO K o ) L1

On the other hand, when other estimates are used, the implementation by means of the ULLV decomposition is not so simple. In this case a refinement step [8], which moves a triangular matrix toward diagonality, may be useful in order to obtain an estimate of the singular values of HR. -1 from LK so that the "filters" (6), (10), or (11) can be used. This is a topic of current research.

4 E X P E R I M E N T S

Algorithm 2 (Truncated QSVD) with Step 2 implemented respectively by the QSVD and the ULLV decomposition were programmed in MATLAB; the QSVD was computed using a stable QSVD algorithm implemented along the lines described in [11] and the ULLV decomposition was computed using an adaptive ULLV-decomposition algorithm implemented along the lines described in [6]. The output of the truncated QSVD algorithm implemented respectively by the QSVD and the ULLV decomposition was compared by the R.MS log spectral distortion, which is widely used in many speech processing systems.

Define f to be the normalized frequency. Let 17"~10(f)l be the 10'th order LPC model spectrum of the original speech segment ~ = [~'0, 21,...Z,g-1] T, and let ~10(f) be the 10'th order LPC model spectrum of the the enhanced speech segment ~ = [xo,~l,... 'I 'N-1] T.


(a) 20 �9 �9 �9

I \ -20 ' ' ' '

0 1000 2 0 0 0 3000 4 0 0 0 f r equency in Hz

(c) 20 �9 �9 �9

m 1o ._

"Q o r

t~ E -10

-20

(b) 20

123 "O .~ 10 (D

t - c~ 0 E

-10 0

20

m 10 ._c

-o 0 r

0 ~

-]0

lo'oo x;oo 3ooo f r e q u e n c y in Hz

(d)

4 0 0 0

-2o o looo 2ooo 3ooo 4ooo o l o~o 2o~o 3ooo 4ooo f r equency in Hz f r equency in Hz

Figure 1: LPC model spectrum of (a) segment containing noise-free voiced speech sounds. (b) segment containing noisy speech; SNR = 5 dB. (c) segment containing enhanced speech; QSVD. (d) segment containing enhanced speech; ULLV decomposition.

Then the RMS log spectral distortion, in dB, is defined as

[ /1 /2 ] 1/2 . (27) d2 = 20 tJ_l/2(log 17~1o(f)1 - log 17S[lo(f)l)2df[.

We used real speech signals sampled at 8 kHz, noise with spectral density of the form S(f) = (4 s- - cos(27rf)) -1, and (N,M,K) = (160,20,14). Our experiments show that for the LS estimate, the algorithm implemented by means of ULLV decomposition computes the enhanced speech segment such that d2 in most cases is less that 1 dB above d2 obtained with the algorithm implemented by means of QSVD. A difference of 1 dB is normally not audible.

In Figure 1 we show an example where the speech segment contains voiced speech sounds and the signal-to-noise ration (SNR) is 5 dB. We see that the LPC spectrum of the enhanced speech segment matches the LPC spectrum of the noise-free speech segment much more closely in the regions near the peaks (formants) than the noisy one does. We also see that the enhanced speech segment obtained with the ULLV decomposition (d2 = 3.04 dB) is close to the segment obtained with the QSVD (d2 = 2.67 dB). This shows that the ULLV decomposition is a promising method in noise reduction of speech signals.


Acknowledgements

Scren Holdt Jensen's research was supported by the European Community Itesearch Pro- gram HCM, contract no. EItBCHBI-CT92-0182. We thank Franklin Luk of Itensselaer Polytechnic Institute and Sanzheng Qiao of McMaster University for MATLAB routines for computing the ULLV decomposition.

References

[1] B. De Moor. The singular value decomposition and long and short spaces of noisy matrices. IEEE Trans. Signal Processing 41, pp 2826-2838, 1993.

[2] B. De Moor and H. Zha. A tree of generalizations of the ordinary singular value decomposition. Lin. Alg. and its Applic. 147, pp 469-500, 1991.

[3] P.C. Hansen. Regularization, GSVD and truncated GSVD. BIT 29, pp 491-504, 1989.

[4] S.H. Jensen, P.C. Hansen, S.D. Hansen, and J.A. Scrensen. Reduction of broad-band noise in speech by truncated QSVD. Report ESAT-SISTA/TR 1994-16, Dept. Electrical Engineering, Katholieke Universiteit Leuven, March 1994 (21 pages); revised version of Report UNIC-93-06, UNIoC, July 1993 (29 pages); IEEE Trans. Speech and Audio Processing.

[5] S.Y. Kung, K.S. Arun, and D.V.B. Rao. State-space and singular value decomposition- based approximation methods for the harmonic retrieval problem. J. Opt. Soc. Am. 73, pp 1799-1811, 1983.

[6] F.T. Luk and S. Qiao. A new matrix decomposition for signal processing. In" M.S. Moonen, G.H. Golub, and B.L.It. De Moor (Eds.), Linear algebra for large scale and real- time applications. Kluver Academic Publishers, Dordrecht, The Netherlands, pp 241- 247, 1993.

[7] J.D. Markel and A.H. Gray Jr. Linear prediction of speech. Springer-Verlag, New York, N.Y., 1976.

[8] It. Mathias and G.W. Stewart. A block QIt algorithm and the singular value decomposition, Lin. Alg. and its Appl. 182, pp 91-100, 1993.

[9] It.J. McAulay and T.F. Quatieri. Speech analysis/synthesis based on a sinusoidal representation. IEEE Trans. Acoust., Speech, Signal Processing 34, pp 744-754, 1986.

[10] S. Van Huffel. Enhanced resolution based on minimum variance estimation and exponential data modeling, Signal Processing 33, pp 333-355, 1993.

[11] C.F. Van Loan. Computing the CS and generalized singular value decomposition. Nu- met. Math. 46, pp 479-491, 1985.


467

S V D - B A S E D M O D E L L I N G O F M E D I C A L N M R S I G N A L S

It. DE BEEIt, D. VAN OItMONDT, F.T.A.W. WAJEIt, Delft Univ. of Technology, Applied Physics Laboratory, P.O. Box 5046, 2600 GA Delft, The Netherlands. ormo@si, tn. tudelft, nl.

S. CAVASSILA, D. GItAVERON-DEMILLY, Univ. Claude Bernard LyonI, Laboratoire RMN, 69622 Villeurbanne Cedex, France. graveron @muzelle. univ-lyon l . fr.

S. VAN HUFFEL, Katholieke Univ. Leuven, E S A T Laboratory, 3001 Heverlee, Belgium. sabine, va nh u f f el @esat. kule u ven. ac. be.

ABSTRACT. A Magnetic Resonance (MR) scanner enables one to noninvasively detect and quantify biochemical substances at selected positions in patients. The MR signal detected by the scanner comprises a number of damped sinusoids in the time domain. Each chemical substance contributes at least one sinusoid, at a specific frequency. Often, metabolite quantification is severely hampered by interference with strong MR signals of water and other substances (e.g. fat) resident in the human body. The damping function of the interfering sinusoids is nonexponential, which aggravates the situation. We investigate subtraction of interfering sinusoids by means of SVD-based state space modelling. Among other things, modelling of all details of the signal is attempted (zero-error modelling). Rank criteria for three alternative Hankel data matrices guaranteeing zero-error modelling, are provided. Measures to meet the criteria are proposed and tested on simulated signals. Although zero-error modelling can indeed be achieved, it turns out that such modelling does not guarantee exact separation of the unwanted sinusoids from the wanted sinusoids.

KEYWORDS. SVD, magnetic resonance spectroscopy, in vivo, state space, zero-error modelling

468 R. de Beer et al.


A Magnetic Resonance (MR) scanner enables one to noninvasively detect and quantify biochemical substances at selected positions in patients [1]. Often the signal processing attendant on quantification is hampered by interference from MR signals of water and other substances (e.g. fat) resident in the human body. Removal of such signals prior to the quantification step is desired. Fig. 1 shows a typical example, transformed from the measurement domain (= time domain) to the frequency domain by FFT for display reasons. See caption of Fig.1 for details.

| !

-- w a t e r

~ r a

! !

-0.05 0 0.05 0.1 0.15 0.2 0.25 c0/2~

Figure 1: State space processing of an in vivo MR time domain signal from the right parietal lobe of a human brain. The results are displayed in the frequency domain by applying FFT and plotting the real part. a) The spectral region of interest of the raw data minus the two first data points which are corrupted. A huge unwanted peak of nondescript (i.e., no model function available) shape originating from water dominates, b) Spectrum after subtraction of the water contribution as parametrized with seven sinusoids using the SVD-based state space algorithm of Kung et al. [2]. In addition, two relatively strongly damped sinusoids were subtracted at w/2r= 0.101 and 0.252, WNyquis t = ~. (Note that the mentioned omission of initial datapoints causes perturbation of the baseline that may be hard to distinguish from genuine MR features. This is a mere display problem.)

Apparently, Kung et al.'s SVD-based state space method [2] is capable of modelling nondescript (i.e., model function unknown) components in terms of exponentially damped sinusoids. Other examples of this feat were shown in the recent past [3]. However, several important aspects of modelling nondescript signals are yet to be resolved. First, criteria guaranteeing success of the modelling seem lacking. This is crucial in the context of automated processing of large numbers of signals. Success is defined here as the degree to

SVD-based Modelling o f Medical NMR Signals 469

which all details of the data can be accounted for. Ref.[4] advocates 'zero-error modelling', which aims to reduce the residue of model minus data to zero. Second, little is known about the extent to which successful modelling yields the true physical parameters. This is important for establishing whether removal of unwanted components from the signal by subtracting the related part of the model does affect the wanted components. In the present contribution we address both aspects�9

2 M E T H O D

2.1 PRELIMINARIES

For reasons of space, we assume that the reader is familiar with the essence of Kung et al.'s SVD-based state space method [2, 5, 6, 7]. We recall that the starting point of the method is rearrangement of the complex-valued data xn, n = 0 , 1 , . . . , N - 1, into an L x M Hankel data matrix X whose elements are Xl,m = Xl+m-2, l = 1, 2 , . . . , L, and m = 1, 2 , . . . , M, and L + M = N + 1, i.e.

XO ;r,1 X2 �9 �9 " X M - 1

X l X2 �9 �9 �9 X M

X = ~ . ( I ) �9 .

X L - 1 XL �9 �9 �9 X N - 1

Next, X is subjected to SVD, according to X = U A V H. If the signal xn comprises K exponentially damped sinusoids, Ck exp[(c~k + iwk)n], and no noise, the rank of X equals K (k = 1 , 2 , . . . , K , with K <_ L , M ; i = x/L-T). The frequencies Wk and damping factors C~k can easily be estimated from the K leading singular vectors of U or V [2], using linear least squares or total least squares [8]. The complex-valued amplitudes Ck can subsequently be estimated by linear least squares fitting of the sinusoids to the signal. Once the Wk, ak, and Ck have been estimated, a 'Vandermonde decomposition' of X has been accomplished [2], according to

[ Z 0 Z 0 . . . Z ~ . Z f - 1

Z 1 Z 1 ZlK C O Z 0 Z 1 . . z M _ I

C 1 O Z~. Z~. "'" . , (2) x _ - . . 0 . �9

\ 4 _ 1 4 _ ~ ~I_1/ ~K ~} ~} ""

de] with Zk = exp(o~k +iwk), k - l, 2 , . . . , K. For later use, we mention that the model function for each data point can be expressed as

K ~. = ~ ~z~ ~ (3)

k - 1


For the purpose of making a robust automated estimator based on the above, it is important to know whether the rank statement is reversible, i.e. does Rank(X) = K always imply that the signal comprises K sinusoids? The answer is negative�9 In view of this we propose a simple 'regularization' of the SVD-based state space method that should render all conceivable signals tractable, albeit at a cost.

2.2 RANK CRITERION FOR SUCCESSFUL MODELLING

In this subsection we indicate without our proof how one can establish whether state space modelling is capable of zero-error modelling the given data. To this end we write two additional data matrices that differ from the one defined in Eq.(1), through their numbers of rows and columns�9 (If L and M are specified, either X I or X" can be dropped�9

XO X l ;T2 " " ' ~ g M - 1 ;TO X l X 2 �9 �9 �9 ;TM

X l X 2 . . . X M X l X 2 . . . X M + 1

X' x2 X" x2 (4) �9 �9 . .

X L - 2 X L - 1 ' �9 �9 X N - 2 X L - 2 X L - 1 �9 �9 �9 X N - 1

Criterion for possibility of exact (i.e., zero-error) modelling: If R a n k ( X ) = Rank(X') = Rank(X") = J, with J _< m i n ( Z - 1, M), then it is possible to model data point xn exactly as

K xn = ~ ( ~ ) r (r ~k, (5)

k - 1

_ T denotes transposition, ~k is a #k X 1 amplitude vector, r is a where J d~.f ~ k # k >--- K , #k X #k Jordan matrix, and Uk a #k X 1 unit vector:

cO ) z k 1

k zk 0 o A2) i 1 ~k

~,= . , r = . . . . . . , ~ k =

c(~,) 0 z k 1 0 z k

(6)

It can be seen that Eq.(6) reduces to Eq.(3) when all #k equal 1. If a signal comprises a time polynomial of order P, then #k = P + 1. During execution of the algorithm, this becomes apparent through (P + 1)-fold degeneracy of an eigenvalue of a Jordan matrix�9 However, a small amount of added noise already lifts the degeneracy�9 One must then consider whether to still model with a polynomial, or with P exponentially damped sinusoids. The latter aspect needs further research. For MR signals the given criterion can be strictly satisfied only by choosing L - 1 = M = N / 2 , assuming N is even. This is because in the real-world MR signals are corrupted by noise, and moreover, their damping is only approx-

SVD-based Modelling of Medical NMR Signals 471

imately exponential. As a result, an MR data matrix most often has full rank, regardless of its size. The mentioned size choice yields R a n k ( X ) = Rank(X') - R a n k ( X " ) = N/2. See also [9], page 167, and [4]. The modelling being full, it may enable one to remove unwanted features from the signal, which is the ultimate goal of this research.

2.3 REGULARIZATION

Removal of unwanted features is particularly useful if it can be fully automated, obviating the need of human intervention in e.g. a hospital environment. This requires numerical stability of the computations. Under the circumstances, it may be hazardous to trust that the noise present in the signal always guarantees full rank. Therefore we propose to provide the guarantee by adding a deterministic perturbation to the signal. The perturbation should be simple, so as to allow (future) analytical calculation of its effect. In this work the perturbation is e(0, 0 , . . . , 0,1, 0, 0 , . . . , 0,1), where e is an adjustable constant. This amounts to adding an outlier e to data points no. N / 2 - 1 and N - 1, N being even and the numbering starting at zero. In the sequel, this measure is called regularization. The resulting regularized data matrix, Xr, is

X ~ = X + E

I i ~

1

0 i

0

(7)

The ratio of the largest and smallest singular values of Xr can be set by varying e. In this study the regularizing signal is simultaneously assigned the role of possible background signal which is to be subtracted after modelling has been achieved. Its N Fourier coefficients are alternatingly nonzero and zero, which results in considerable overlap with the spectrum of the wanted signal. Controlled overlap is desired in order to better study its effect on the state space estimates.

If all Jordan blocks have size #k = 1, the original state space algorithm of Kung et al. can be applied. If #k > 1, for one or more values of k, the model function must be adapted according to Eq.(6). The latter is required only in the second stage where the amplitudes are estimated. Once zero-error modelling has been achieved, the next task is to identify those sinusoids that represent the original wanted signal, and to evaluate the extent to which the unwanted components affect the estimates of the wanted parameters.

3 R E S U L T S A N D D I S C U S S I O N

The method described above was tried on a noiseless simulated signal comprising a single exponentially damped sinusoid perturbed by the 2 outliers e, shown in Fig.2. Two cases, differing in frequency of the sinusoid, were considered. Details about the chosen parameters


and the results of the zero-error modelling are listed in Tablel. For w/21r = 0.1230, the singular values of the 17x 16 regularized data matrix Xr are )q = 7.644, A2 = 1.366, )~3 through Als = 1.000, )q6 = 0.876, which indicates that the rank of Xr is well defined and indeed full. Nearly the same singular values are found for w/2~r = 0.15625. The small residues quoted

1

0 . 5 �9176 � 9

0 . �9 i . . " . i �9 �9 �9 �9 ~

- 0 . 5 o �9 ~

-~0 ~o ~o ~0 n

Figure 2: Simulated signal, comprising one noiseless, exponentially damped, sinusoid and two outliers of magnitude e, at n = 15 and 31, indicated by arrows. Table 1 lists the results of zero-error modelling for e = 1 and two different frequencies of the sinusoid.

in the footnote of Table 1 show that zero-error modelling can be achieved. For both cases, the original sinusoid can be located at k = 6. Since all parameters have been estimated, subtraction of unwanted components on the basis of zero-error modelling is feasible. At the same time, this particular example shows limitations of the procedure. The estimates for k = 6 can be seen to deviate somewhat from the true parameters, listed immediately underneath. The deviations depend on the frequency, which is related to the fact that the power spectrum of the regularizing signal peaks at w/2~r = m / 1 6 , m = - 8 , - 7 , . . . , 7. For m = 2, w/2~r = 0.12500, which is near 0.1230; for m = 2.5, w/21r = 0.15625. Apparently, the parameters are better when the frequency of the wanted sinusoid is close to that of an unwanted sinusoid. Note the 'reverse damping' and small initial amplitude for k = 5, w / 2 r = 0.15625. At present we can not yet offer quantitative explanations of the observed phenomena. An analytic state space solution for the simple signal of Fig.2 is sought. Fi- nally, we point out that in real-world cases, it may be difficult to distinguish between a wanted component and an unwanted component occupying the same frequency region. It is unfortunate that prior knowledge about the wanted part of the signal, such as relative frequencies and phases, cannot yet be exploited by SVD-based state space modelling.

4 S U M M A R Y A N D C O N C L U S I O N S

�9 Zero-error state space modelling is possible if the data matrix satisfies certain rank criteria. (Proof omitted for reasons of space.)

�9 A procedure for imposing the rank criteria (regularization) on the data matrix of an arbitrary signal, is devised. This regularization does not entail addition of random noise to the data.

�9 Our state space solution is not restricted to exponentially damped sinusoids.

SVD-based Modelling of Medical NMR Signals 473

�9 Zero-error state space modelling is capable of quantification of nondescript unwanted components in MR time domain signals. The spectrum of the unwanted components strongly overlaps with that of wanted components. The perturbation of parameter estimates of the wanted components is studied.

Table 1: Results of zero-error a SVD-based state space modelling for a single exponentially damped sinusoid augmented by two unit outliers as indicated in Fig.1. The true values of the sinusoid are listed below line k = 6. L = 17, M = 16, N = 32. ck = [ck[ exp(i~k). Two frequencies are considered: w/27r = 0.1230, 0.15625, with WNyquis t - - - - 71". • " - - 1.

k I~kl ~o~ ~k ~k/2~ 1 0.0608 155.6 0.0017 0.4378 2 0.0611 132.4 0.0014 0.3755 3 0.0615 109.0 0.0008 0.3131 4 0.0624 84.8 -0.0003 0.2509 5 0.0653 58.1 -0.0039 0.1890 6 1.0270 0.6 -0.0617 0.1243

true 1.0000 0.0 -0.0670 0.1230 7 0.0596 32.9 0.0014 0.0607 8 0.0594 4.6 0.0029 -0.0008 9 0.0596 -19.9 0.0029 -0.0630 10 0.0598 -43.5 0.0027 -0.1253 11 0.0600 -66.7 0.0026 -0.1877 12 0.0601 -89.7 0.0025 -0.2501 13 0.0603 -112.7 0.0024 -0.3125 14 0.0604 -135.6 0.0022 -0.3749 15 0.0605 -158.5 0.0021 -0.4373 16 0.0607 178.6 0.0019 -0.4997

;0) Ickl <p~, o<k ~k/2~ 0.0723 166.2 -0.0092 0.4359 0.0720 147.4 -0.0086 0.3727 0.0719 131.0 -0.0077 0.3091 0.0732 122.6 -0.0059 0.2440 0.0039 -29.5 0.1464 0.1637 1.1638 13.0 -0.0937 0.1502 1.0000 0.0 -0.0670 0.15625 0.0850 -5.1 -0.0163 0.0677 0.0783 -15.0 -0.0135 0.0028 0.0761 -31.8 -0.0123 -0.0608 0.0751 -50.8 -0.0117 -0.1239 0.0744 -70.8 -0.0112 -0.1869 0.0739 -91.3 -0.0108 -0.2498 0.0736 -112.0 -0.0105 -0.3126 0.0732 -132.7 -0.0102 -0.3754 0.0729 -153.4 -0.0099 -0.4383 0.0726 -173.8 -0.0096 0.4989

a) The relative rms residues of the modelling are 6.4 x 10 -15 and 3.4 • 10 -15, for w/2r = 0.1230, 0.15625 respectively, which is indicative of 'zero error'.

A c k n o w l e d g m e n t s

This work is supported by the Dutch Technology Foundation (STW) and the EU programme Human Capital and Mobility, Networks. SVH is sponsored by the Belgian N.F.W.O. and Programme on Interuniversity Poles of Attraction (IUAP- no. 50 and 17). The in vivo MR signal was provided by P. Gilligan in the context of EU Concerted Action Biomed 1, no. PL 920432. DvO acknowledges illuminating discussions of rank criteria for state space modelling with K.S. Arun.

R e f e r e n c e s

[1] F. Wehrli. The Origins and Future of Nuclear Magnetic Resonance Imaging. Physics


Today 45, pp 34-42, 1992.

[2] S.Y. Kung, K.S. Arun, D.V. Bhaslmr Rao. State Space and Singular Value Decompos- ition-Based Approximation Methods for the Harmonic ttetrieval Problem. J. Amer. Opt. Soc. 73, pp 1799-1811, 1983.

[3] W.W.F. Pijnappel, A. van den Boogaart, R. de Beer, D. van Ormondt. SVD-Based Quantification of Magnetic Resonance Signals. J. Magn. Reson. 97, pp 122-134, 1992.

[4] I. Dologlou and G. Carayannis. LPC/SVD Analysis of Signals with Zero-Modelling Error. Signal Processing 23, pp 293-299, 1991.

[5] D.V. Bhaskar Rao and K.S. Arun. Model-Based Processing of Signals : A State Space Approach. Proc. IEEE, pp 283-309, 1992.

[6] A.J. van der Veen, E.F. Deprettere, A.L. Swindlehurst. Subspace-Based Signal Analysis Using Singular Value Decomposition. Proc. IEEE, pp 1277-1308, 1993.

[7] W.W.F. Pijnappel, R. de Beer, D. van Ormondt. State Space Modelling of Medical NMR Signals. In : J.P. Veen and M.J. de Ket (Eds.), Proc. ProRISC, STW, Utrecht, pp 225-230, 1992.

[8] S. Van Huffel, H. Chen, C. Decanniere, P. Van Hecke. Algorithm for Time-Domain NMR. Data Fitting Based on Total Least Squares, J. Magn. Reson. A l l 0 , 1994 (in print).

[9] M. Fiedler. Special Matrices and their Applications in Numerical Mathematics. Mart- inus Nijhoff, Dordrecht, 1986.


475

I N V E R S I O N O F B R E M S S T R A H L U N G S P E C T R A E M I T T E D B Y S O L A R P L A S M A

M. PIANA Dipartimento di Fisica dell'Universith di Genova and Istituto Nazionale di Fisica Nucleate via Dodecaneso 33 1-161~6 Genova, Italy

ABSTRACT. Bremsstrahlung of energetic electrons with the ions of the plasma is considered the dominant hard X-ray emission mechanism in correspondence with a solar flare. If no assumption is made about the thermodynamic conditions in which the process happens, it is possible to link the electron distribution function, averaged on the ion density, to the photon spectrum, through a Volterra equation whose kernel is given by the bremsstrahlung cross section. In the case of discrete data, the elements of the Gram matrix have been analytically calculated and then the singular system of the Volterra operator has been computed. On the other hand, if thermal conditions are assumed, the photon spectrum coincides with the Laplace transform of the differential emission measure, whose profile gives information about the temperature structure of the plasma. The SVD of the Laplace operator is computed in different function spaces. The knowledge of the singular system of the Volterra and Laplace operators allows the application of Tikhonov algorithm to the inversion of a real photon spectrum; in this way, stable reconstructions of the averaged electron distribution function and of the emission measure are obtained.

KEYWORDS. Solar flares: hard X-rays - regularisation: Tikhonov.


Solar flares are transient phenomena typical of some active regions of the solar photosphere, characterised by the sudden release of an enormous quantity of energy (,,~ 103Serg). Typical flare manifestations are the heating of the solar atmosphere and the increasing of emission over the whole electromagnetic spectrum, caused by the interaction of accelerated particles

476 M. Piana

with the electromagnetic fields and the ions of the plasma. One of the main problems in solar flare physics [6] regards the description of the acceleration mechanism. To this aim, several theoretical models have been formulated; in particular, for electron energies in the range 1 0 - 100 keV, there is little doubt that acceleration can be described in terms of a stochastic mechanism in which the electron distribution function assumes a fundamental role. This is the reason why it becomes particularly significant to analyse the emission spectra observed at Earth in order to infer information about the distribution function of the charged particles (particularly electrons) which emit this radiation through the most frequent radiative processes, such as bremsstrahlung, nuclear reactions, synchrotron.

Collisional bremsstrahlung of electrons with the ions of the plasma is considered the main origin of hard X-rays radiation (photon energies higher than 10 keV) [4]. From a general point of view, it is possible to link the photon spectrum to the local electron distribution function by the equation:

g(e) = n(r) F(E,r)Q(e,E)dEdr (1)

where V is the volume of the emitting region, n(r) is the density of the ions in the plasma, E is the electron energy, e is the photon energy, F(E, r) is the local electron distribution function, Q(e, E) is the bremsstrahlung cross section, g(e) is the total rate of photon emission, in (photon/sec) per unit e (in this equation the relativistic and absorption effects have been neglected). Equation (1) can be transformed to a simpler form by introducing the averaged distribution function:

1 /vn(r)r(Z,r)dr (2) T(E) = ~V

where ~ is the volumetric mean proton density. If the Bethe-Heitler approximation for the cross section is assumed [10], one obtains:

fecr 1+ ~ / 1 - ~ dE g(e) -- _1 F (E) log (3)

where all the multiplicative constants have been put equal to unity. It must be observed that equation (3) describes any bremsstrahlung source model in terms of the source-averaged F ( E ) characterizing it.

A further interpretation of equation (1) can be developed by assuming particular hypotheses on the dynamic and thermodynamic characteristics of the region in which the process happens. For example, in the case of the thermal model [5], one assumes that the local electron distribution function F ( E , r ) is given by a Maxwellian, so that, after some physical approximations and suitable changes of variables [8], equation (1) becomes:

A [co ~(T) ezp(- e g(e) = "-[ -~ (kT)�89 -~-~)dT (4)

where ~(T) is the differential emission measure

n2(r) ~(T) = JS[T IVTI dsT (5)

Bremsstrahlung Spectra Emitted by Solar Plasma 477

and ST is a constant temperature surface. The importance of the differential emission measure ~(T) in solar flare physics resides in the fact that it provides information about the electron density and temperature distribution in the emitting region.

The two integral equations (3) and (4) can be analytically solved. As regards the Volterra equation (3), it can be easily transformed to an Abel equation [7] whose solution is:

T ( E ) = E�89 d f f ( d(eg(~)) 1 ) (6) 7r dE de v/e E

As regards the Fredholm equation (4), it can be reduced to the Laplace transform by the changes of variables y = ~-~ and:

f(y) = a ~(~)~ (7) ky~

so that

/5 1 f (y)exp(-ey)dy (8)

This equation can be immediately solved by inverting the Laplace transform, for instance by means of the Mellin transform.

It is interesting to note that these analytic solutions of the two integral equations are not physically meaningful. In fact, in order to obtain the electron distribution function and the differential emission measure from them, it would be necessary to know the photon spectrum on the whole energy interval, up to infite photon energy, and with a great accuracy (sufficient, for example in the Volterra case, to estimate its second derivative). On the contrary, the spectrum is recorded only in N finite bins so that the two integral equations must be considered in the discretised form:

gn = -- F(E) log n = I N (9) e. ~, 1 - . / 1 - .~- E " ' " V ~:.,

and

l f 0 ~ gn = - - f(y)exp(-e,~y)dy n = 1, . . . , N (10) en

where g~ = g(e,~). The iU-posedness of equations (9) and (10)is evident (for example, the solution is not unique). Nevertheless it is possible to provide stable (and so physically meaningful) estimates of the solutions by introducing a suitable regularisation method whose computation is made possible by the knowledge of the singular system of the integral operators.

2 SVD OF T H E I N T E G R A L O P E R A T O R S

2.1 VOLTERRA OPERATOR

Equations (9) and (10) can be easily transposed to a closer form if one introduces a linear operator L : X --* Y where X is a Hilbert space and Y is a euclidean space endowed with

478 M. Piana

the scalar product' N

(g ,h) = ~ gnhnwn (11) n--1

in which the weights wn depends on the kind of sampling. In the case of the Volterra equation, if X = L2(0, oo), the scalar product (F, r of the distribution function T with the function

0 E _< en (12) 1 1+Y/'f:~" E > e , ~ r = ~ l o g x-VT=~

can be introduced, so that the equation can be written in the form:

g = L F (13)

where L F is the vector whose components are given by:

(LF)n = (Y,r n = 1 , . . . , N (14) N As L is a finite rank operator, it is possible to introduce its singular system {an; v,.,, un}n=l

[2] so that:

I ,v , = an un Z* un = an vn (15)

From this definition it immediately follows that the singular values and the singular vectors axe respectively the square root of the eigenvalues and the eigenvectors of the operator LL* which can be decomposed in the form:

LL* = G T w (16)

with W the weight matrix defined by:

Wnm = 6nmWm n , m = 1 , . . . , N (17)

and G the Gram matrix whose (n, m) entry is given by:

Grim = (r162 n , m = 1 , . . . , N (lS)

The elements of the Gram matrix have been computed analytically [12]; the result is: f o r m > n:

4 Gmn = 3 log

for m = n: 1

G.~.~ - (em)s 8 log 2

F o r m < n:

1 + ~ 2

1 - ~ en(em) :z log e__.~_~ +

Em En(Em)2 =~

(20)


2.2 LAPLACE TRANSFORM

It is well-known that the inversion of the Laplace transform is a severely ill-posed problem [1]. It follows that, in order to obtain stable estimates of the solution, it may be necessary to include apriori information on the solution in the regularisation algorithm. An example of this fact is given by the reconstruction of functions which are assumed to be zero in the origin. In this case it is useful to choose, as the source space X, the Sobolev space Hi(O, a) endowed with the scalar product:

(/, g)x = /'(y)g'(y)dy (22)

and with the property:

f(0) = 0 (23)

The linear inverse problem with discrete data (10) can be written again in the operator form (13), with L f = (f, Cn)x, though, this time, the functions Cn are given by the solution of the boundary value problem:

r = o

r = 0

One obtains explicitly:

1 1 exp(-eny) - Y exp(-ena)

and so the (n, m) entry of the Gram matrix is given by: 1 " { 1 exp[-(en + era)a]+

+ + aexp[-(en + era)a] + en -I- em 1 1

+--exp[- (en + era)a]- - -exp( -ena) + f-m s

1 1 exp(-ema) "~ +--exp[-(en + era)a]- e--~ J (-n

(24)

= 1 , . . . , N (25)

3 NUMERICAL RESULTS

The knowledge of the singular system of the two integral operators allows to apply Tikhonov regularisation [13] to the reconstruction both of the averaged distribution function F (E) and of the differential emission measure ~(T). Tikhonov regularisation solution is defined as the function which minimises the functional:

~m[f] = I I L f - gll~" + AIIfll~ (27) We remark that, if X is the Sobolev space Hi(0, a), then the minimisation of (27) implies the minimisation of the functional:

/i �9 ~[f] = IILI- gll} + A If'(Y)12dY (28)

480 M. Piana

in the subset of the functions satisfying condition (23). In general, the regularised solution can be represented in terms of the singular system of the integral operator, explicitly [3]:

N Or n

f~ = ~ ~ + ~ (g, u~)~ (29) r~=l

The choice of the regularisation parameter )~ is a crucial problem in Tikhonov method. To this aim, there exist several criteria whose efficiency sensitively depends on the particular problem which is studied. In this paper two methods have been applied [9]: Morozov's Discrepancy Principle, according to which the best )~ is given by the solution of the equation:

IIL/~ - glIY - ~ (30)

(where ~ is the RMS error on the data) and the Generalised Cross Validation, which provides the value of A minimising the function:

V(A) = ( N - 1 T r [ I - A(A)])-2(N-11[[I- A(A)]gll 2) (31)

It is interesting to note that, owing to their iU-posedness, the solutions of the inverse problems (9) and (10) call be known ollly with infinite uncertainty. Nevertheless it is possible to estimate the propagation error from the data to the regularised solution by calculating the so called confidence limit which is obtained by performing several regularised reconstructions corresponding with different realisations of the data vector computed by modifying the real data with random components with zero mean and variance equal to unity. The result is a "confidence strip" whose upper and lower borders are the confidence limits of the reconstructed function.

As regards the estimate of the resolution, it is sufficient to observe that not all the singular functions significatively contribute to the regularised solution through equation (29). More precisely, this sum can be truncated at the value n = M so that the relative variation between the truncated solution:

M o" n f ~ = ~ ~ + ~(g, u~)~ (32)

r~=l

and (29) is less than the relative error on the data. As the singular function of order n has n - 1 zeroes in (0, oo), it follows that the regularised solution cannot contain details in the interval between two adjacent zeroes of the last significant singular function.

Besides several simulated cases, Tikhonov regularisation has been applied to a representative sample (figure 1) of the temporal series of spectra recorded by Germanium detectors during the solar flare of 27 June 1980 [11], in order to recover both the averaged electron distribution function and the differential emission measure. Figure 2a represents the "confidence strip" corresponding to equation (9); the value of the regularisation parameter is obtained by GCV and the error and resolution bars have been plotted on the strip at some discrete points, representing the geometric means between adjacent zeroes of the last significant singular function. In figure 2b, there is another reconstruction of F ( E ) , in which the value of A has been chosen by Discrepancy Principle. As one can see, GCV is characterised by undersmoothing properties while Discrepancy Principle provides over- smoothed reconstructions; this different performance between the two methods is even more


103

10 2

I01 3

>. 100 c_

4.*

L 10 "1 JO

3 o , 10_ 2

I I I

' ' ' ' ' ' ' ' I ' ' -

I o ,

,

! I

I I I

I I

! I

I I I

i i i i | | | I l u "31 .0 - , 50 100

e (keY]

I I i I

'IIll I ' I

Figure 1: Real data vector from the HIREX instrument for the June 27 1980 solar flare

104

10 3

3

~" 10 2 L

L

L d

100

- I I i i I i i i I I i

i , , , , i i l l i i 20 80 100 200

| I i i I i i i I i i

b

i i | , , , , I i l 20 80 100 200

E [keV] E [keV]

Figure 2: Regularised reconstruction of the electron averaged distribution function F(E): (a) A = 2.4x10 -6 (GCV). (b) A = 1.13x10 -5 (Discrepancy Principle).

482 M. Piana

evident in the case of Laplace inversion, where the reconstruction provided by GCV is still completely unstable; on the contrary, figure 3 shows the regularised differential emission measure obtained by Tikhonov technique, applied in the Sobolev space Hi(0, a), when the regularisation parameter is chosen by Discrepancy Principle.

10-1

10-2

10-3 P. w 4.,

10-4

o h .,~ 10"5

i-- I0 -{~

10-7

::' '1 . . . . . . . . I

f !11111 _ . . . . : ~ ' ? ~ ' ~ " I ' . % ~

_

, I 10 7

' ' ' ' ' ' " 1

( ~ \ \ ~ .

" l I I l I i l

. . . . . . . I . . . . . . . . I 10 8 10 9 T {K}

Figure 3: Regularised reconstruction of the differential emission measure ~(T); A = 6.7x10 -1~ (Discrepancy Principle) .

4 C O M M E N T S AND C O N C L U S I O N S

The study of ill-posed problems is usually developed by projecting the integral equation onto a finit dimensional space and then by treating the corresponding ill-conditioned linear system with opportune numerical methods. On the contrary, in this paper, we have considered discrete data but we have maintained the solution in infinite-dimensional spaces; this approach has been favoured by the possibility to analytically compute the Gram matrix both in the case of the Volterra operator in L2(0, oc) and in the case of the Laplace operator in Hi(0, a).

Tikhonov regularisation in L2(0, c~) seems to be efficient in order to recover the averaged electron distribution function F(E) from the real data of figure 1. In particular, from a phenomenological point of view, the two reconstructions obtained by GCV and Discrepancy Principle put in evidence a double power law, typical of flare manifestations, with a spectral slope at electron energy E _ 40 keV. Nevertheless, in the GCV regularised solution it is possible to note a further spectral slope at E ~ 80 keV.

The inversion of the Laplace transform is a more severely ill-conditioned problem and so, in order to exhibit a physically significant reconstruction of the differential emission measure, it has been necessary to adopt higher order smoothness assumptions. Then Tikhonov regularisation has been applied in a Sobolev space with the prescription that the solution is zero at the origin. The regularised ~(T) form comprises two components: the first one shows a distribution of relatively cool material with a peak at T _ 107 ~ the second


one is an "ultra hot" component peaking at 4.5x10 s ~ Such a ~(T) structure has numerous interpretations in the field of the theoretical modelisation of magnetic reconnection mechanisms typical of solar flares.

Acknowledgements

It is a pleasure to thank R.P. Lin and C. Johns for providing the experimental data. This work has been partly supported by Consorzio INFM.

References

[1]

[2]

M. Bertero, P. Brianzi and E.R. Pike. On the recovery and resolution of exponential relaxation rates from experimental data. III. The effect of sampling and truncation of data on the Laplace transform inversion. Proc. R. Soc. Lond. A 398, pp 23-44, 1985.

M. Bertero, C. De Mol and E.R. Pike. Linear inverse problems with discrete data. I: General formulation and singular system analysis. Inverse Problems 1, pp 301-330, 1985.

[3]

[4]

[5]

[~]

[7]

M. Bertero, C. De Mol and E.R. Pike. Linear inverse problems with discrete data: II. Stability and regularisation. Inverse Problems 4, pp 573-594, 1988.

J.C. Brown. The deduction of energy spectra of non-thermal electrons in flares from the observed dynamic spectra of hard X-ray bursts. Solar Phys. 18, pp 489-502, 1971.

J.C. Brown and A.G. Emslie. Analytic limits on the forms of spectra possible from optically thin coUisional bremsstrahhng source models. Astrophys. J. 331, pp 554- 564, 1988.

J.C. Brown and D.F. Smith. Solar flares. Rep. Prog. Phys. 43, pp 125-197, 1980.

R. Courant and D. Hilbert. Methods of mathematical physics. Interscience, New York, 1989.

[8] I.J.D. Craig and J.C. Brown. Inverse problems in astronomy. Adam Hilger, Bristol, 1986.

[9] A.R. Davies. Optimality in regularisation. In : M. Bertero and E.R. Pike (Ed.), Inverse problems in scattering and imaging, Adam Hilger, Bristol, pp 393-410, 1992.

[10] H.W. Koch and J.W. Motz. Bremsstrahlung cross section formulas and related data. Rev. Mod. Phys. 31, pp 920-955, 1959.

[11] R.P Lin and R.A. Schwartz. High spectral resolution measurements of a solar flare hard X-ray burst. Astrophys. J. 312, pp 462-474, 1987.

[12] M. Piana. Inversion of bremsstrahlung spectra emitted by solar plasma. Astron. As- trophys. 288, pp 949-959, 1994.

484 M. Piana

[13] A.N. Tikhonov. Solution of incorrectly formulated problems and the regularisation method. Soy. Math. Dokl. 4, pp 1035-1038, 1963.

485

A U T H O R S I N D E X

Ammann L.P., 333 Barlow J.L., 167 Beltrami E., 5 Berry M.W., 123 Bertero M., 341 Boley D., 3 Boley D., 295 Bunch J.R., 175 Cardoso J.-F., 375 Cavassila S., 467 Chen H., 391 de Beer R., 467 De Lathauwer L., 383 De Mol C., 341 De Moor B.L.R., 61 De Moor B.L.R., 383 DeGroat R.D., 227 DeGroat R.D., 235 Dehaene J., 259 Deprettere E.F., 267 Deprettere E.F., 277 Dewilde P., 209 Dologlou I., 323 DoMing E.M., 227 DoMing E.M., 235 Drma(: Z., 107 Drma~ Z., 115 Fierro R.D., 175 Fierro R.D., 183 Fu Z., 235 Gallivan K., 199 Gander W., 349 Gersemsky F., 287 Golub G.H, 139 Golub G.H., 349 GStze J., 251 Graveron-Demilly D., 467

Hanke M., 131 Hansen P.C., 131 Hansen P.C., 183 Hansen P.C., 459 Hansen S.D., 459 Helmke U., 33 Hosur S., 295 Hiiper K., 251 Jensen S.H., 459 Ks B., 207 Kirsteins I.P., 415 Kruth J.P., 367 Kullstam J.A., 423 Lemmerling P., 191 Linebarger D.A., 227 Linebarger D.A., 235 Lorenzelli F., 243 Luk F.T., 305 Ma W., 367 Moonen M., 259 Moonen M., 267 O'Leary D.P., 315 Otte D., 357 Pan C.-T., 157 Park H., 399 Paul S., 251 Piana M., 475 Qiao S., 149 Ramos J.A., 433 Rosen J.B., 399 Shah A.A., 407 Skowronski J., 323 Slock D.T.M., 449 Solna K., 139 Sorensen D.C., 21 Sorensen J.A., 459 Stewart M., 441

Strebel R., 349 Tang P.T.P., 157 Tewfik A.H., 295 Thao N.T., 79 Thirumalai S., 199 Tufts D.W., 407 Tufts D.W., 415 Vaccaro R.J., 407 van Dijk H.W., 277 Van Dooren P., 139 Van Dooren P., 199 Van Dooren P., 441 Van Huffel S., 191 Van Huffel S., 391 Van Huffel S., 399 Van Huffel S., 467 van Ormondt D., 467 Vandevoorde D., 305 Vandewalle J., 259 Vandewalle J., 383 Vandewalle J., 391 Vanhamme L., 191 Vanpoucke F., 267 Varadhan S., 123 Verriest E.I., 423 Verriest E.I., 433 Veselid K., 115 Vetterli M., 79 von Matt U., 99 Wayer F.T.A.W., 467 Yang B., 287 Yao K., 243 Ye H., 227 Yoon P.A., 167 Zha H., 167


Documents

[Marc Moonen] SVD and Signal Processing III Algor(BookFi.org)