Lecture 17Introduction to Eigenvalue Problems

Shang-Hua Teng

Eigenvalue Problems• Eigenvalue problems occur in many areas of

science and engineering– E.g., Structure analysis

• It is important for analyzing numerical and linear algebra algorithms– Impact of roundoff errors and precision requirement

• It is widely used in information management and web-search

• It is the key ingredient for the analysis of Markov process, sampling algorithms, and various approximation algorithms in computer science

Eigenvalues and Eigenvectors

• Standard Eigenvalue Problem: Given an n by n matrix A, find a scalar and nonzero vector x such that

A x = x is eigenvalue, and x is corresponding

eigenvector

Spectrum of Matrices

• Spectrum(A) = (A) = set of all eigenvalues of A

• Spectral radius (A) = (A) = max {||: in (A)}

• Spectral analysis

• Spectral methods

Geometric Interpretation

• Matrix expands or shrinks any vector lying in direction of eigenvector by scalar factor

• Expansion of contraction factor given by corresponding eigenvalue

• Eigenvalues and eigenvectors decompose complicated behavior of general linear transformation into simpler actions

Examples: Eigenvalues and Eigenvectors

1

0,2 ,

0

1,1

20

01

2211 xx

A

Note: x1 and x2 are perpendicular to each other

Examples: Eigenvalues and Eigenvectors

1

1,2 ,

0

1,1

20

11

2211 xx

A

Note: x1 and x2 are not perpendicular to each other

Examples: Eigenvalues and Eigenvectors

1

1,4 ,

1

1,2

31

13

2211 xx

A

Note: x1 and x2 are perpendicular to each other

Examples: Eigenvalues and Eigenvectors

1

1,1 ,

1

1,2

5.15.0

5.05.1

2211 xx

A

Note: x1 and x2 are perpendicular to each other

Examples: Eigenvalues and Eigenvectors

1 :where

1, ,

1,

01

10

2211

- i

ixi

ixi

A

Note: x1 and x2 are not perpendicular to each other : eigenvalues or eigenvectors may not be real!!!

Simple Facts of Eigenvalue Problem

• If (,x) is a eigenvalue-eigenvector pair of A, then for any k, (k, x) is a eigenvalue-eigenvector pair of Ak.

• If (,x) is a eigenvalue-eigenvector pair of A, then for any c, (c, x) is a eigenvalue-eigenvector pair of cA.

Algebraic Interpretation:Equation for the Eigenvalues

A x = x

(A - I ) x = 0• The eigenvectors make up the

nullspace of (A – I ) if we know .

Eigenvalue First

• If (A - I ) x = 0 has a nonzero solution, then– A - I is not invertible

– The determinant of A - I must be zero.

Characteristic Equation for Eigenvalues

• The number l is an eigenvalue of A if and only if (A - I ) is singular:

det( A - I ) = 0

Characteristic Polynomial for Eigenvalues

• det (A - I ) = 0 is a polynomial in of degree at most n.

• The spectrum of A is the set of roots of this characteristic polynomial:

• Fundamental Theorem of Algebra implies that n by n matrix A always has n eigenvalues, but they need be neither distinct nor real

Examples: Characteristic Polynomial

So Spectrum(A) = {1,2}

2120

01det

10

01

20

01det)det(

20

01

IA

A

Examples: Characteristic Polynomial

So Spectrum(A) = {2, 4}

4286

13331

13det

10

01

31

13det)det(

31

13

2

IA

A

Examples: Characteristic Polynomial

So Spectrum(A) = {i, -i}

ii

IA

A

11

1det

10

01

01

10det)det(

01

10

2

A Possible Methods for Solving the Eigenvalue Problems

• Compute the characteristic polynomial of A in by expanding det(A – I) = 0

• Find the roots of the characteristic polynomial

• For each eigenvalue , solve (A – I) x=0 to find an eigenvector x.

Practical Difficulties

• Computing eigenvalues using characteristic polynimial is not recommended or used because– Roots of polynomial of degree > 4 cannot

always be computed in finite number of steps– A lot of work is needed in computing

coefficients of the characteristic polynomial– Computer has round-off errors

Examples: Characteristic Polynomial

So Spectrum(A) = {1+, 1-}

11

111

1det )det(

an smaller thslighly number a is where 1

1

2

machine

IA

A

But in machine, 2 < machine is equal to 0So, the algorithm will returnSpectrum(A) = {1,1}

Theory and Practice

• Characteristic polynomial is a powerful theoretical tool but usually is not useful computationally.

Special Matrix

• What is Spectrum( I )?– Multiplicity is the number of times root appears

when polynomial written as product of linear factors

– det(I – I ) = (1-)n

• What is Spectrum( upper or lower triangular matrix )?

Bad News

• Elimination does not preserve the ’s.

2 and 0 has 11

11

1 and 0 has 00

11

21

21

Diagonalizing A Matrix• Suppose the n by n matrix A has n linearly

independent eigenvectors x1, x2,…, xn.

• Eigenvector matrix S: x1, x2,…, xn are columns of S.

• Then

n

ASS

11

is the eigenvalue matrix

Matrix Power Ak

• S-1AS = implies A = S S-1

• implies A2 = S S-1 S S-1 = S S-1

• implies Ak = S kS-1

Random walks

How long does it take to get completely lost?

0

0

0

0

0

1

Random walks Transition Matrix

1

2

345

6

0

0

0

0

0

1

02

1

4

100

2

13

10

4

1000

3

1

2

10

2

1

3

10

004

10

3

10

004

1

2

10

2

13

1000

3

10

100

P

Matrix Powers

• If A is diagonalizable as A = S S-1 then for any vector u, we can compute Aku efficiently

– Solve S c = u– Aku = S kS-1 S c = S k c

• As if A is a diagonal matrix!!!!

Independent Eigenvectors from Different Eigenvalues

• Eigenvectors x1, x2,…, xk that correspond to distinct (all different) eigenvalues are linear independent.

• An n by n matrix that has n different eigenvalues (no repeated ’s) must be diagonalizable

Proof: Show that

implies all ci = 0

011 kk xcxc