1–1

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Dynamical Systems for Extreme Eigenspace Computations

Maziar Nikpour

UCL Belgium

1–2

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Co-workers

Iven M. Y. Mareels

Jonathan H. Manton

University of Melbourne, Australia.

Vadym Adamyan

Odessa State University, Ukraine.

Uwe Helmke

University of Wurzberg, Germany.

1–3

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Problem• For Hermitian matrices (A, B), with B > 0;

find the non-trivial solutions (, x) of

with the smallest or largest generalised eigenvalues .

n – size of matrices (A,B)k – no. of desired generalised eigenvalue/eigenvector pairs.

1–4

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Outline

• Introduction• Motivation• Brief history of literature• Penalty function approach• Gradient flow• Convergence• Discrete-time Algorithms• Applications• Conclusions

1–5

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Motivation

• Signal Processing

• Telecommunications

• Control

• Many others…

1–6

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Brief History of Problem

• Numerical Linear Algebra Literature– Methods for general A and B:

• QZ algorithm, Moler and Stewart 1973.

(what MATLAB does when you type ‘eig’)

– Methods for large and sparse A, B.• Trace minimisation method, Sameh & Wisiniewski, 1981.

• Engineering Literature• Methods largely for computing largest/smallest generalised evs

adaptively• Mathew and Reddy 1998 (inflation approach, special case of

approach in this work).

• Strobach, 2000 (tracking algorithms).

1–7

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Brief History of Problem

• Dynamical systems literature– Brockett flow

– Oja

• Above approaches cannot be adapted to the Generalised Eigenvalue problem without manipulating A and/or B.

• Recent paper by Manton et al. presents an approach that can…

1–8

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Penalty Function Approach• The minimisation of the following cost can lead to

algorithms for computing extreme generalised evs.

1–9

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Dynamical Systems for Numerical Computations

Gradient descent like flows on a cost function.

Discretisation of flows.

Efficient numerical algorithms.

1–10

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Examples

• Power flow:

• Oja subspace flow:

• Brockett flow:

1–11

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Contributions• Gradient flow on f(A, B)

• Discretisation of Gradient Flow– Steepest Descent

– Conjugate Gradient

• Stochastic minor/principal component tracking algorithms

• The case B = I, and Z real has already been treated.

(see Manton et al. 2003).

Extending the domain to the complex matrices complicates the analysis substantially…

Allowing B to be any p.d. matrix expands the range of applications…

1–12

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Gradient Flow

• Main Result: For almost all initial conditions, solutions of

converge to a single point in the stable invariant set of the flow.

1–13

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Gradient Flow

• The stable invariant set is:

1–14

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Critical Points of f(A, B)

• Hessian of f(A, B) is degenerate at critical points,

N.B. • Proposition:

1–15

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Stability analysis of critical points

• Linear stability analysis will not suffice.• Use center manifold theorem at each c.p.• Proposition:

Why?

Nullspace of hessian of cost func. = Tangent space of critical subman.

1–16

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Stability analysis of critical pointsReduction principle of dynamical systems

stable

unstable

center

1–17

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

• Main result follows….

• Proposition: level sets are compact => flow converges to one of the critical components.

• Center manifold thm. + reduction principle => converge to a single point on a critical component.

• Converges to stable invariant set for an open dense set of initial conditions.

Stability analysis of critical points

1–18

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Remarks• Conditions used in proof => f(A, B) is a Morse-Bott function

=> solutions converge to a single point instead of a set (see Helmke & Moore, 1994).

• Also f(A, B) is a real analytic function (Cn x k considered as a real vector space) => convergence to a single point (Lojasiewicz, 1984).

1–19

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Further Remarks• Generalised eigenvectors not unique but

convergence to particular g.evs can be achieved by the following flow in reduced dimensions:

where trunc{X} denotes X with imaginary components of diagonal set to 0.

Flow converges to an element of critical component with real diagonal elements.

1–20

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Systems of Flows

• Consider the system of cost functions:

1–21

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Systems of Flows

• System of partial gradient descent flows allows the possibility to add or take away components without affecting the computation of others

• Proposition: Z(t) converges to smallest generalised eigenvalues for a generic initial condition.

1–22

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Discrete-time algorithms

• Since flow evolves on a Euclidean space – discretisation is not complicated:

• Steepest descent:

• Conjugate gradient

1–23

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Discrete-time algorithms

• Can solve the Hermitian definite GEVP without any factorisation or manipulation of A or B.

• Only matrix – small matrix multiplications are required.

• Suitable for cases where A and B are large and sparse.

• Conjugate gradient algorithm – superlinear convergence but no increase in order of computational complexity.

• Complexity O(n2k).

• Exact line search can be performed.

1–24

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

1–25

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Discrete-time algorithms

• Tracking algorithm:

• O(nk2) complexity when Rnn = I.

- signal plus noise model

1–26

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

1–27

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Conclusion

• Proposing and deriving convergence theory of a gradient flow for solving GEVP.

• Modular system of flows.

• Discretisation: CG and SD algorithms.

• Application to Minor component tracking.

1–28

Wor

ksh

op o

n D

ynam

ical

Sys

tem

s an

d C

ompu

tatio

n

Questions