Understanding the QR algorithm, Part X · 2. Fundamentals of Matrix Computations, Wiley, 1991 3....

Understanding the QR algorithm,Part X

David S. Watkinswatkins@math.wsu.edu

Department of Mathematics

Washington State University

Glasgow 2009 – p. 1

1. Understanding the QR algorithm, SIAM Rev., 1982

Glasgow 2009 – p. 2

1. Understanding the QR algorithm, SIAM Rev., 1982

2. Fundamentals of Matrix Computations, Wiley, 1991

Glasgow 2009 – p. 2

1. Understanding the QR algorithm, SIAM Rev., 1982

2. Fundamentals of Matrix Computations, Wiley, 1991

3. Some perspectives on the eigenvalue problem, 1993

Glasgow 2009 – p. 2

1. Understanding the QR algorithm, SIAM Rev., 1982

2. Fundamentals of Matrix Computations, Wiley, 1991

3. Some perspectives on the eigenvalue problem, 1993

4. QR-like algorithms—an overview of convergencetheory and practice, AMS proceedings, 1996

Glasgow 2009 – p. 2

1. Understanding the QR algorithm, SIAM Rev., 1982

2. Fundamentals of Matrix Computations, Wiley, 1991

3. Some perspectives on the eigenvalue problem, 1993

4. QR-like algorithms—an overview of convergencetheory and practice, AMS proceedings, 1996

5. QR-like algorithms for eigenvalue problems, JCAM,2000

Glasgow 2009 – p. 2

1. Understanding the QR algorithm, SIAM Rev., 1982

2. Fundamentals of Matrix Computations, Wiley, 1991

3. Some perspectives on the eigenvalue problem, 1993

4. QR-like algorithms—an overview of convergencetheory and practice, AMS proceedings, 1996

5. QR-like algorithms for eigenvalue problems, JCAM,2000

6. Fundamentals of Matrix Computations, Wiley, 2002

Glasgow 2009 – p. 2

1. Understanding the QR algorithm, SIAM Rev., 1982

2. Fundamentals of Matrix Computations, Wiley, 1991

3. Some perspectives on the eigenvalue problem, 1993

4. QR-like algorithms—an overview of convergencetheory and practice, AMS proceedings, 1996

5. QR-like algorithms for eigenvalue problems, JCAM,2000

6. Fundamentals of Matrix Computations, Wiley, 2002

7. The Matrix Eigenvalue Problem: GR and KrylovSubspace Methods, SIAM, 2007.

Glasgow 2009 – p. 2

1. Understanding the QR algorithm, SIAM Rev., 1982

2. Fundamentals of Matrix Computations, Wiley, 1991

3. Some perspectives on the eigenvalue problem, 1993

4. QR-like algorithms—an overview of convergencetheory and practice, AMS proceedings, 1996

5. QR-like algorithms for eigenvalue problems, JCAM,2000

6. Fundamentals of Matrix Computations, Wiley, 2002

7. The Matrix Eigenvalue Problem: GR and KrylovSubspace Methods, SIAM, 2007.

8. The QR algorithm revisited, SIAM Rev., 2008.

Glasgow 2009 – p. 2

Some names associated withthe QR algorithm

Glasgow 2009 – p. 3

Some names associated withthe QR algorithm (short list)

Glasgow 2009 – p. 3

Some names associated withthe QR algorithm (short list)

Rutishauser

Glasgow 2009 – p. 3

Some names associated withthe QR algorithm (short list)

Rutishauser

Kublanovskaya

Glasgow 2009 – p. 3

Some names associated withthe QR algorithm (short list)

Rutishauser

Kublanovskaya

Francis

Glasgow 2009 – p. 3

Some names associated withthe QR algorithm (short list)

Rutishauser

Kublanovskaya

Francis

Implicitly Shifted QR algorithm

Glasgow 2009 – p. 3

Some names associated withthe QR algorithm (short list)

Rutishauser

Kublanovskaya

Francis

Implicitly Shifted QR algorithmHow should we understand it?

Glasgow 2009 – p. 3

Some names associated withthe QR algorithm (short list)

Rutishauser

Kublanovskaya

Francis

Implicitly Shifted QR algorithmHow should we understand it? . . . view it?

Glasgow 2009 – p. 3

Some names associated withthe QR algorithm (short list)

Rutishauser

Kublanovskaya

Francis

Implicitly Shifted QR algorithmHow should we understand it? . . . view it?. . . teach it to our students?

Glasgow 2009 – p. 3

The Standard Approach . . .

Glasgow 2009 – p. 4

The Standard Approach . . .. . . dating from the work of Francis

Glasgow 2009 – p. 4

The Standard Approach . . .. . . dating from the work of Francis

Start with the basic algorithm . . .

Glasgow 2009 – p. 4

The Standard Approach . . .. . . dating from the work of Francis

Start with the basic algorithm . . .

A = QR

Glasgow 2009 – p. 4

The Standard Approach . . .. . . dating from the work of Francis

Start with the basic algorithm . . .

A = QR RQ = A

Glasgow 2009 – p. 4

The Standard Approach . . .. . . dating from the work of Francis

Start with the basic algorithm . . .

A = QR RQ = A repeat!

Glasgow 2009 – p. 4

The Standard Approach . . .. . . dating from the work of Francis

Start with the basic algorithm . . .

A = QR RQ = A repeat!

This is simple,

Glasgow 2009 – p. 4

The Standard Approach . . .. . . dating from the work of Francis

Start with the basic algorithm . . .

A = QR RQ = A repeat!

This is simple, appealing,

Glasgow 2009 – p. 4

The Standard Approach . . .. . . dating from the work of Francis

Start with the basic algorithm . . .

A = QR RQ = A repeat!

This is simple, appealing, does not require muchpreparation,

Glasgow 2009 – p. 4

The Standard Approach . . .. . . dating from the work of Francis

Start with the basic algorithm . . .

A = QR RQ = A repeat!

This is simple, appealing, does not require muchpreparation, but . . .

Glasgow 2009 – p. 4

The Standard Approach . . .. . . dating from the work of Francis

Start with the basic algorithm . . .

A = QR RQ = A repeat!

This is simple, appealing, does not require muchpreparation, but . . .

. . . it is far removed from versions of theQRalgorithm that are actually used.

Glasgow 2009 – p. 4

Refinements

Glasgow 2009 – p. 5

Refinementsshifts of origin

Glasgow 2009 – p. 5

Refinementsshifts of origin

reduction to Hessenberg form

Glasgow 2009 – p. 5

Refinementsshifts of origin

reduction to Hessenberg form

implicit shift technique (Francis)

Glasgow 2009 – p. 5

Refinementsshifts of origin

reduction to Hessenberg form

implicit shift technique (Francis)

double shiftQR

Glasgow 2009 – p. 5

Refinementsshifts of origin

reduction to Hessenberg form

implicit shift technique (Francis)

double shiftQR

multiple shiftQR

Glasgow 2009 – p. 5

Refinementsshifts of origin

reduction to Hessenberg form

implicit shift technique (Francis)

double shiftQR

multiple shiftQR

implicit-Q theorem

Glasgow 2009 – p. 5

Refinementsshifts of origin

reduction to Hessenberg form

implicit shift technique (Francis)

double shiftQR

multiple shiftQR

implicit-Q theoremvs.Krylov subspaces

Glasgow 2009 – p. 5

Refinementsshifts of origin

reduction to Hessenberg form

implicit shift technique (Francis)

double shiftQR

multiple shiftQR

implicit-Q theoremvs.Krylov subspaces

Introducing Krylov subspaces improvesunderstanding,

Glasgow 2009 – p. 5

Refinementsshifts of origin

reduction to Hessenberg form

implicit shift technique (Francis)

double shiftQR

multiple shiftQR

implicit-Q theoremvs.Krylov subspaces

Introducing Krylov subspaces improvesunderstanding, allows more general results,

Glasgow 2009 – p. 5

Refinementsshifts of origin

reduction to Hessenberg form

implicit shift technique (Francis)

double shiftQR

multiple shiftQR

implicit-Q theoremvs.Krylov subspaces

Introducing Krylov subspaces improvesunderstanding, allows more general results, andprepares students for Krylov subspace methods.

Glasgow 2009 – p. 5

The Implicitly Shifted QR Iteration

Glasgow 2009 – p. 6

The Implicitly Shifted QR Iterationmatrix is in upper Hessenberg form

Glasgow 2009 – p. 6

The Implicitly Shifted QR Iterationmatrix is in upper Hessenberg form

pick some shiftsρ1, . . . ,ρm

Glasgow 2009 – p. 6

The Implicitly Shifted QR Iterationmatrix is in upper Hessenberg form

pick some shiftsρ1, . . . ,ρm (m = 1, 2, 4, 6)

Glasgow 2009 – p. 6

The Implicitly Shifted QR Iterationmatrix is in upper Hessenberg form

pick some shiftsρ1, . . . ,ρm (m = 1, 2, 4, 6)

p(A) = (A − ρ1I) · · · (A − ρmI)

Glasgow 2009 – p. 6

The Implicitly Shifted QR Iterationmatrix is in upper Hessenberg form

pick some shiftsρ1, . . . ,ρm (m = 1, 2, 4, 6)

p(A) = (A − ρ1I) · · · (A − ρmI) expensive!

Glasgow 2009 – p. 6

The Implicitly Shifted QR Iterationmatrix is in upper Hessenberg form

pick some shiftsρ1, . . . ,ρm (m = 1, 2, 4, 6)

p(A) = (A − ρ1I) · · · (A − ρmI) expensive!

computep(A)e1

Glasgow 2009 – p. 6

The Implicitly Shifted QR Iterationmatrix is in upper Hessenberg form

pick some shiftsρ1, . . . ,ρm (m = 1, 2, 4, 6)

p(A) = (A − ρ1I) · · · (A − ρmI) expensive!

computep(A)e1 cheap!

Glasgow 2009 – p. 6

The Implicitly Shifted QR Iterationmatrix is in upper Hessenberg form

pick some shiftsρ1, . . . ,ρm (m = 1, 2, 4, 6)

p(A) = (A − ρ1I) · · · (A − ρmI) expensive!

computep(A)e1 cheap!

Build unitaryQ0 with q1 = αp(A)e1.

Glasgow 2009 – p. 6

The Implicitly Shifted QR Iterationmatrix is in upper Hessenberg form

pick some shiftsρ1, . . . ,ρm (m = 1, 2, 4, 6)

p(A) = (A − ρ1I) · · · (A − ρmI) expensive!

computep(A)e1 cheap!

Build unitaryQ0 with q1 = αp(A)e1.

Perform similarity transformA → Q∗0AQ0.

Glasgow 2009 – p. 6

The Implicitly Shifted QR Iterationmatrix is in upper Hessenberg form

pick some shiftsρ1, . . . ,ρm (m = 1, 2, 4, 6)

p(A) = (A − ρ1I) · · · (A − ρmI) expensive!

computep(A)e1 cheap!

Build unitaryQ0 with q1 = αp(A)e1.

Perform similarity transformA → Q∗0AQ0.

Hessenberg form is disturbed.

Glasgow 2009 – p. 6

An Upper Hessenberg Matrix@

@@

@@

@@

@@

@@

@@

Glasgow 2009 – p. 7

After the Transformation ( Q∗0AQ0)

@@

@@

@@

@@

@@

Glasgow 2009 – p. 8

After the Transformation ( Q∗0AQ0)

@@

@@

@@

@@

@@

Now return the matrix to Hessenberg form.

Glasgow 2009 – p. 8

Chasing the Bulge@

@@@

@@

@@

@@@

Glasgow 2009 – p. 9

Chasing the Bulge@

@@

@@

@@

@@

@

Glasgow 2009 – p. 10

Done@

@@

@@

@@

@@

@@

@@

Glasgow 2009 – p. 11

Done@

@@

@@

@@

@@

@@

@@

The implicitQR step is complete!

Glasgow 2009 – p. 11

Summary of Implicit QR Iteration

Glasgow 2009 – p. 12

Summary of Implicit QR IterationPick some shifts.

Glasgow 2009 – p. 12

Summary of Implicit QR IterationPick some shifts.

Computep(A)e1. (p determined by shifts)

Glasgow 2009 – p. 12

Summary of Implicit QR IterationPick some shifts.

Computep(A)e1. (p determined by shifts)

Build Q0 with first columnq1 = αp(A)e1.

Glasgow 2009 – p. 12

Summary of Implicit QR IterationPick some shifts.

Computep(A)e1. (p determined by shifts)

Build Q0 with first columnq1 = αp(A)e1.

Make a bulge. (A → Q∗0AQ0)

Glasgow 2009 – p. 12

Summary of Implicit QR IterationPick some shifts.

Computep(A)e1. (p determined by shifts)

Build Q0 with first columnq1 = αp(A)e1.

Make a bulge. (A → Q∗0AQ0)

Chase the bulge. (return to Hessenberg form)

Glasgow 2009 – p. 12

Summary of Implicit QR IterationPick some shifts.

Computep(A)e1. (p determined by shifts)

Build Q0 with first columnq1 = αp(A)e1.

Make a bulge. (A → Q∗0AQ0)

Chase the bulge. (return to Hessenberg form)

A = Q∗AQ

Glasgow 2009 – p. 12

Question

Glasgow 2009 – p. 13

QuestionThis differs a lot from the basicQR step.

Glasgow 2009 – p. 13

QuestionThis differs a lot from the basicQR step.

A = QR RQ = A

Glasgow 2009 – p. 13

QuestionThis differs a lot from the basicQR step.

A = QR RQ = A

Can we carve a reasonable pedagogical path thatleads directly to the implicitly-shiftedQR algorithm,

Glasgow 2009 – p. 13

QuestionThis differs a lot from the basicQR step.

A = QR RQ = A

Can we carve a reasonable pedagogical path thatleads directly to the implicitly-shiftedQR algorithm,bypassing the basicQR algorithm entirely?

Glasgow 2009 – p. 13

QuestionThis differs a lot from the basicQR step.

A = QR RQ = A

Can we carve a reasonable pedagogical path thatleads directly to the implicitly-shiftedQR algorithm,bypassing the basicQR algorithm entirely?

That’s what we are going to do today.

Glasgow 2009 – p. 13

Ingredients

Glasgow 2009 – p. 14

Ingredientssubspace iteration (power method)

Glasgow 2009 – p. 14

Ingredientssubspace iteration (power method)

Krylov subspaces

Glasgow 2009 – p. 14

Ingredientssubspace iteration (power method)

Krylov subspaces and subspace iteration

Glasgow 2009 – p. 14

Ingredientssubspace iteration (power method)

Krylov subspaces and subspace iteration

(unitary) similarity transformation(change of coordinate system)

Glasgow 2009 – p. 14

Ingredientssubspace iteration (power method)

Krylov subspaces and subspace iteration

(unitary) similarity transformation(change of coordinate system)

reduction to Hessenberg form

Glasgow 2009 – p. 14

Ingredientssubspace iteration (power method)

Krylov subspaces and subspace iteration

(unitary) similarity transformation(change of coordinate system)

reduction to Hessenberg form

Hessenberg form and Krylov subspaces(instead of implicit-Q theorem)

Glasgow 2009 – p. 14

Ingredientssubspace iteration (power method)

Krylov subspaces and subspace iteration

(unitary) similarity transformation(change of coordinate system)

reduction to Hessenberg form

Hessenberg form and Krylov subspaces(instead of implicit-Q theorem)

No Magic Shortcut!

Glasgow 2009 – p. 14

Power Method, Subspace Iteration

Glasgow 2009 – p. 15

Power Method, Subspace Iterationv, Av, A2v, A3v, . . .

Glasgow 2009 – p. 15

Power Method, Subspace Iterationv, Av, A2v, A3v, . . .

convergence rate|λ2/λ1 |

Glasgow 2009 – p. 15

Power Method, Subspace Iterationv, Av, A2v, A3v, . . .

convergence rate|λ2/λ1 |

S, AS, A2S, A3S, . . .

Glasgow 2009 – p. 15

Power Method, Subspace Iterationv, Av, A2v, A3v, . . .

convergence rate|λ2/λ1 |

S, AS, A2S, A3S, . . .

subspaces of dimensionj

Glasgow 2009 – p. 15

Power Method, Subspace Iterationv, Av, A2v, A3v, . . .

convergence rate|λ2/λ1 |

S, AS, A2S, A3S, . . .

subspaces of dimensionj (|λj+1/λj |)

Glasgow 2009 – p. 15

Power Method, Subspace Iterationv, Av, A2v, A3v, . . .

convergence rate|λ2/λ1 |

S, AS, A2S, A3S, . . .

subspaces of dimensionj (|λj+1/λj |)

Substitutep(A) for A

Glasgow 2009 – p. 15

Power Method, Subspace Iterationv, Av, A2v, A3v, . . .

convergence rate|λ2/λ1 |

S, AS, A2S, A3S, . . .

subspaces of dimensionj (|λj+1/λj |)

Substitutep(A) for A (shifts, multiple steps)

Glasgow 2009 – p. 15

Power Method, Subspace Iterationv, Av, A2v, A3v, . . .

convergence rate|λ2/λ1 |

S, AS, A2S, A3S, . . .

subspaces of dimensionj (|λj+1/λj |)

Substitutep(A) for A (shifts, multiple steps)

S, p(A)S, p(A)2S, p(A)3S, . . .

Glasgow 2009 – p. 15

Power Method, Subspace Iterationv, Av, A2v, A3v, . . .

convergence rate|λ2/λ1 |

S, AS, A2S, A3S, . . .

subspaces of dimensionj (|λj+1/λj |)

Substitutep(A) for A (shifts, multiple steps)

S, p(A)S, p(A)2S, p(A)3S, . . .

convergence rate|p(λj+1)/p(λj) |

Glasgow 2009 – p. 15

Krylov Subspaces . . .

Glasgow 2009 – p. 16

Krylov Subspaces . . .. . . and Subspace Iteration

Glasgow 2009 – p. 16

Krylov Subspaces . . .. . . and Subspace IterationDef: Kj(A, q) = span

{

q, Aq,A2q, . . . , Aj−1q}

Glasgow 2009 – p. 16

Krylov Subspaces . . .. . . and Subspace IterationDef: Kj(A, q) = span

{

q, Aq,A2q, . . . , Aj−1q}

j = 1, 2, 3, . . . (nested subspaces)

Glasgow 2009 – p. 16

Krylov Subspaces . . .. . . and Subspace IterationDef: Kj(A, q) = span

{

q, Aq,A2q, . . . , Aj−1q}

j = 1, 2, 3, . . . (nested subspaces)

Kj(A, q) are “determined byq”.

Glasgow 2009 – p. 16

Krylov Subspaces . . .. . . and Subspace IterationDef: Kj(A, q) = span

{

q, Aq,A2q, . . . , Aj−1q}

j = 1, 2, 3, . . . (nested subspaces)

Kj(A, q) are “determined byq”.

p(A)Kj(A, q) = Kj(A, p(A)q)

Glasgow 2009 – p. 16

Krylov Subspaces . . .. . . and Subspace IterationDef: Kj(A, q) = span

{

q, Aq,A2q, . . . , Aj−1q}

j = 1, 2, 3, . . . (nested subspaces)

Kj(A, q) are “determined byq”.

p(A)Kj(A, q) = Kj(A, p(A)q)

. . . becausep(A)A = Ap(A)

Glasgow 2009 – p. 16

Krylov Subspaces . . .. . . and Subspace IterationDef: Kj(A, q) = span

{

q, Aq,A2q, . . . , Aj−1q}

j = 1, 2, 3, . . . (nested subspaces)

Kj(A, q) are “determined byq”.

p(A)Kj(A, q) = Kj(A, p(A)q)

. . . becausep(A)A = Ap(A)

Conclusion: Power method induces nested subspaceiterations on Krylov subspaces.

Glasgow 2009 – p. 16

power method: p(A)kq

Glasgow 2009 – p. 17

power method: p(A)kq

nested subspace iterations:

p(A)kKj(A, q) = Kj(A, p(A)kq) j = 1, 2, 3, . . .

Glasgow 2009 – p. 17

power method: p(A)kq

nested subspace iterations:

p(A)kKj(A, q) = Kj(A, p(A)kq) j = 1, 2, 3, . . .

convergence rates:

|p(λj+1)/p(λj) |, j = 1, 2, 3, . . .

Glasgow 2009 – p. 17

(Unitary) Similarity Transforms

Glasgow 2009 – p. 18

(Unitary) Similarity TransformsA → Q∗AQ preserves eigenvalues

Glasgow 2009 – p. 18

(Unitary) Similarity TransformsA → Q∗AQ preserves eigenvalues

transforms eigenvectors in a simple way(w → Q∗w)

Glasgow 2009 – p. 18

(Unitary) Similarity TransformsA → Q∗AQ preserves eigenvalues

transforms eigenvectors in a simple way(w → Q∗w)

is a change of coordinate system (v → Q∗v)

Glasgow 2009 – p. 18

(Unitary) Similarity TransformsA → Q∗AQ preserves eigenvalues

transforms eigenvectors in a simple way(w → Q∗w)

is a change of coordinate system (v → Q∗v)

triangular form (eigenvalues)

Glasgow 2009 – p. 18

(Unitary) Similarity TransformsA → Q∗AQ preserves eigenvalues

transforms eigenvectors in a simple way(w → Q∗w)

is a change of coordinate system (v → Q∗v)

triangular form (eigenvalues)

relationship of invariant subspaces to triangular form

Glasgow 2009 – p. 18

Subspace Iterationwith change of coordinate system

Glasgow 2009 – p. 19

Subspace Iterationwith change of coordinate system

takeS = span{e1, . . . , ej}

Glasgow 2009 – p. 19

Subspace Iterationwith change of coordinate system

takeS = span{e1, . . . , ej}

p(A)S = span{p(A)e1, . . . , p(A)ej}

= span{q1, . . . , qj} (orthonormal)

Glasgow 2009 – p. 19

Subspace Iterationwith change of coordinate system

takeS = span{e1, . . . , ej}

p(A)S = span{p(A)e1, . . . , p(A)ej}

= span{q1, . . . , qj} (orthonormal)

build unitaryQ = [q1 · · · qj · · ·]

Glasgow 2009 – p. 19

Subspace Iterationwith change of coordinate system

takeS = span{e1, . . . , ej}

p(A)S = span{p(A)e1, . . . , p(A)ej}

= span{q1, . . . , qj} (orthonormal)

build unitaryQ = [q1 · · · qj · · ·]

change coordinate system:A = Q∗AQ

Glasgow 2009 – p. 19

Subspace Iterationwith change of coordinate system

takeS = span{e1, . . . , ej}

p(A)S = span{p(A)e1, . . . , p(A)ej}

= span{q1, . . . , qj} (orthonormal)

build unitaryQ = [q1 · · · qj · · ·]

change coordinate system:A = Q∗AQ

qk → Q∗qk = ek

Glasgow 2009 – p. 19

Subspace Iterationwith change of coordinate system

takeS = span{e1, . . . , ej}

p(A)S = span{p(A)e1, . . . , p(A)ej}

= span{q1, . . . , qj} (orthonormal)

build unitaryQ = [q1 · · · qj · · ·]

change coordinate system:A = Q∗AQ

qk → Q∗qk = ek

span{q1, . . . , qj} → span{e1, . . . , ej}

Glasgow 2009 – p. 19

Subspace Iterationwith change of coordinate system

takeS = span{e1, . . . , ej}

p(A)S = span{p(A)e1, . . . , p(A)ej}

= span{q1, . . . , qj} (orthonormal)

build unitaryQ = [q1 · · · qj · · ·]

change coordinate system:A = Q∗AQ

qk → Q∗qk = ek

span{q1, . . . , qj} → span{e1, . . . , ej}

ready for next iterationGlasgow 2009 – p. 19

This version of subspace iteration . . .

Glasgow 2009 – p. 20

This version of subspace iteration . . .

. . . holds the subspace fixed

Glasgow 2009 – p. 20

This version of subspace iteration . . .

. . . holds the subspace fixed

while the matrix changes.

Glasgow 2009 – p. 20

This version of subspace iteration . . .

. . . holds the subspace fixed

while the matrix changes.

. . . moving toward a matrix under which

span{e1, . . . , ej}

is invariant.

Glasgow 2009 – p. 20

This version of subspace iteration . . .

. . . holds the subspace fixed

while the matrix changes.

. . . moving toward a matrix under which

span{e1, . . . , ej}

is invariant.

A →

[

A11 A12

0 A22

]

(A11 is j × j.)

Glasgow 2009 – p. 20

Reduction to Hessenberg form

Glasgow 2009 – p. 21

Reduction to Hessenberg formQ → Q∗AQ = H (a similarity transformation)

Glasgow 2009 – p. 21

Reduction to Hessenberg formQ → Q∗AQ = H (a similarity transformation)

can always be done (direct method,O(n3) flops)

Glasgow 2009 – p. 21

Reduction to Hessenberg formQ → Q∗AQ = H (a similarity transformation)

can always be done (direct method,O(n3) flops)

brings us closer to triangular form

Glasgow 2009 – p. 21

Reduction to Hessenberg formQ → Q∗AQ = H (a similarity transformation)

can always be done (direct method,O(n3) flops)

brings us closer to triangular form

makes computations cheaper

Glasgow 2009 – p. 21

Reduction to Hessenberg formQ → Q∗AQ = H (a similarity transformation)

can always be done (direct method,O(n3) flops)

brings us closer to triangular form

makes computations cheaper

First columnq1 can be chosen “arbitrarily”.

Glasgow 2009 – p. 21

Reduction to Hessenberg formQ → Q∗AQ = H (a similarity transformation)

can always be done (direct method,O(n3) flops)

brings us closer to triangular form

makes computations cheaper

First columnq1 can be chosen “arbitrarily”.

Example: q1 = αp(A)e1

Glasgow 2009 – p. 21

Krylov Subspaces . . .

Glasgow 2009 – p. 22

Krylov Subspaces . . .. . . and Hessenberg matrices . . .

Glasgow 2009 – p. 22

Krylov Subspaces . . .. . . and Hessenberg matrices . . .

. . . go hand in hand.

Glasgow 2009 – p. 22

Krylov Subspaces . . .. . . and Hessenberg matrices . . .

. . . go hand in hand.

A properly upper Hessenberg=⇒

Kj(A, e1) = span{e1, . . . , ej}.

Glasgow 2009 – p. 22

Krylov Subspaces . . .. . . and Hessenberg matrices . . .

. . . go hand in hand.

A properly upper Hessenberg=⇒

Kj(A, e1) = span{e1, . . . , ej}.

More generally . . .

Glasgow 2009 – p. 22

Krylov-Hessenberg Relationship

Glasgow 2009 – p. 23

Krylov-Hessenberg RelationshipIf H = Q∗AQ, andH is properly upper Hessenberg,then forj = 1, 2, 3, . . . ,

span{q1, . . . , qj} = Kj(A, q1).

Glasgow 2009 – p. 23

Krylov-Hessenberg RelationshipIf H = Q∗AQ, andH is properly upper Hessenberg,then forj = 1, 2, 3, . . . ,

span{q1, . . . , qj} = Kj(A, q1).

Proof (sketch):

Glasgow 2009 – p. 23

Krylov-Hessenberg RelationshipIf H = Q∗AQ, andH is properly upper Hessenberg,then forj = 1, 2, 3, . . . ,

span{q1, . . . , qj} = Kj(A, q1).

Proof (sketch): Induction onj.

Glasgow 2009 – p. 23

Krylov-Hessenberg RelationshipIf H = Q∗AQ, andH is properly upper Hessenberg,then forj = 1, 2, 3, . . . ,

span{q1, . . . , qj} = Kj(A, q1).

Proof (sketch): Induction onj. AQ = QH

Glasgow 2009 – p. 23

Krylov-Hessenberg RelationshipIf H = Q∗AQ, andH is properly upper Hessenberg,then forj = 1, 2, 3, . . . ,

span{q1, . . . , qj} = Kj(A, q1).

Proof (sketch): Induction onj. AQ = QH

Aqj =n

∑

i=1

qihij =

j∑

i=1

qihij + qj+1hj+1,j

Glasgow 2009 – p. 23

Aqj =

j∑

i=1

qihij + qj+1hj+1,j

Glasgow 2009 – p. 24

Aqj =

j∑

i=1

qihij + qj+1hj+1,j

qj+1hj+1,j = Aqj −

j∑

i=1

qihij

Glasgow 2009 – p. 24

Aqj =

j∑

i=1

qihij + qj+1hj+1,j

qj+1hj+1,j = Aqj −

j∑

i=1

qihij

Proof by induction follows immediately.

Glasgow 2009 – p. 24

Aqj =

j∑

i=1

qihij + qj+1hj+1,j

qj+1hj+1,j = Aqj −

j∑

i=1

qihij

Proof by induction follows immediately.

This also gives the student a preview of the Arnoldiprocess,

Glasgow 2009 – p. 24

Aqj =

j∑

i=1

qihij + qj+1hj+1,j

qj+1hj+1,j = Aqj −

j∑

i=1

qihij

Proof by induction follows immediately.

This also gives the student a preview of the Arnoldiprocess,the most important Krylov subspacemethod.

Glasgow 2009 – p. 24

and now,

Glasgow 2009 – p. 25

and now, the Implicit QR Iteration

Glasgow 2009 – p. 25

and now, the Implicit QR IterationWork with Hessenberg form to get . . .

Glasgow 2009 – p. 25

and now, the Implicit QR IterationWork with Hessenberg form to get . . .

. . . efficiency.

Glasgow 2009 – p. 25

and now, the Implicit QR IterationWork with Hessenberg form to get . . .

. . . efficiency.

. . . automatic nested subspace iterations.

Glasgow 2009 – p. 25

and now, the Implicit QR IterationWork with Hessenberg form to get . . .

. . . efficiency.

. . . automatic nested subspace iterations.

Get some shiftsρ1, . . . ,ρm to definep.

Glasgow 2009 – p. 25

and now, the Implicit QR IterationWork with Hessenberg form to get . . .

. . . efficiency.

. . . automatic nested subspace iterations.

Get some shiftsρ1, . . . ,ρm to definep.

Computep(A)e1. (power method)

Glasgow 2009 – p. 25

and now, the Implicit QR IterationWork with Hessenberg form to get . . .

. . . efficiency.

. . . automatic nested subspace iterations.

Get some shiftsρ1, . . . ,ρm to definep.

Computep(A)e1. (power method)

TransformA to upper Hessenberg form:

A = Q∗AQ

by a matrixQ that hasq1 = αp(A)e1.

Glasgow 2009 – p. 25

A = Q∗AQ where q1 = αp(A)e1.

Glasgow 2009 – p. 26

A = Q∗AQ where q1 = αp(A)e1.

q1 → Q∗q1 = e1

Glasgow 2009 – p. 26

A = Q∗AQ where q1 = αp(A)e1.

q1 → Q∗q1 = e1

power method with a change of coordinate system.Moreover . . .

Glasgow 2009 – p. 26

A = Q∗AQ where q1 = αp(A)e1.

q1 → Q∗q1 = e1

power method with a change of coordinate system.Moreover . . .

p(A)Kj(A, e1) = Kj(A, p(A)e1)

Glasgow 2009 – p. 26

A = Q∗AQ where q1 = αp(A)e1.

q1 → Q∗q1 = e1

power method with a change of coordinate system.Moreover . . .

p(A)Kj(A, e1) = Kj(A, p(A)e1)

i.e.p(A)span{e1, . . . , ej} = span{q1, . . . , qj}

Glasgow 2009 – p. 26

A = Q∗AQ where q1 = αp(A)e1.

q1 → Q∗q1 = e1

power method with a change of coordinate system.Moreover . . .

p(A)Kj(A, e1) = Kj(A, p(A)e1)

i.e.p(A)span{e1, . . . , ej} = span{q1, . . . , qj}

subspace iteration with a change of coordinatesystem

Glasgow 2009 – p. 26

A = Q∗AQ where q1 = αp(A)e1.

q1 → Q∗q1 = e1

power method with a change of coordinate system.Moreover . . .

p(A)Kj(A, e1) = Kj(A, p(A)e1)

i.e.p(A)span{e1, . . . , ej} = span{q1, . . . , qj}

subspace iteration with a change of coordinatesystem

j = 1, 2, 3, . . . ,n − 1

Glasgow 2009 – p. 26

A = Q∗AQ where q1 = αp(A)e1.

q1 → Q∗q1 = e1

power method with a change of coordinate system.Moreover . . .

p(A)Kj(A, e1) = Kj(A, p(A)e1)

i.e.p(A)span{e1, . . . , ej} = span{q1, . . . , qj}

subspace iteration with a change of coordinatesystem

j = 1, 2, 3, . . . ,n − 1

|p(λj+1)/p(λj) | j = 1, 2, 3, . . . ,n − 1

Glasgow 2009 – p. 26

Details

Glasgow 2009 – p. 27

Detailschoice of shifts

Glasgow 2009 – p. 27

Detailschoice of shifts

We change the shifts at each step.

Glasgow 2009 – p. 27

Detailschoice of shifts

We change the shifts at each step.

⇒ quadratic or cubic convergence

Glasgow 2009 – p. 27

Detailschoice of shifts

We change the shifts at each step.

⇒ quadratic or cubic convergence

Other Questions

Glasgow 2009 – p. 27

Detailschoice of shifts

We change the shifts at each step.

⇒ quadratic or cubic convergence

Other Questions. . . how to get BLAS 3 speed?

. . . how to parallelize?

Glasgow 2009 – p. 27

In Conclusion

Glasgow 2009 – p. 28

In ConclusionA careful study of

Glasgow 2009 – p. 28

In ConclusionA careful study of the power method and its extensions,

Glasgow 2009 – p. 28

In ConclusionA careful study of the power method and its extensions,similarity transformations,

Glasgow 2009 – p. 28

In ConclusionA careful study of the power method and its extensions,similarity transformations,Hessenberg form,

Glasgow 2009 – p. 28

In ConclusionA careful study of the power method and its extensions,similarity transformations,Hessenberg form,andKrylov subspaces

Glasgow 2009 – p. 28

In ConclusionA careful study of the power method and its extensions,similarity transformations,Hessenberg form,andKrylov subspacesleads directly to the implicitly shiftedQR algorithm.

Glasgow 2009 – p. 28

In ConclusionA careful study of the power method and its extensions,similarity transformations,Hessenberg form,andKrylov subspacesleads directly to the implicitly shiftedQR algorithm.

The basic, explicitQR algorithm is skipped.

Glasgow 2009 – p. 28

In ConclusionA careful study of the power method and its extensions,similarity transformations,Hessenberg form,andKrylov subspacesleads directly to the implicitly shiftedQR algorithm.

The basic, explicitQR algorithm is skipped.

The implicit-Q theorem is avoided.

Glasgow 2009 – p. 28

In ConclusionA careful study of the power method and its extensions,similarity transformations,Hessenberg form,andKrylov subspacesleads directly to the implicitly shiftedQR algorithm.

The basic, explicitQR algorithm is skipped.

The implicit-Q theorem is avoided.

Krylov subspaces are emphasized.

Glasgow 2009 – p. 28

In ConclusionA careful study of the power method and its extensions,similarity transformations,Hessenberg form,andKrylov subspacesleads directly to the implicitly shiftedQR algorithm.

The basic, explicitQR algorithm is skipped.

The implicit-Q theorem is avoided.

Krylov subspaces are emphasized.

Krylov subspace methods are foreshadowed.

Glasgow 2009 – p. 28

One Last Question

Glasgow 2009 – p. 29

One Last QuestionIn the implicitly shiftedQR algorithm

Glasgow 2009 – p. 29

One Last QuestionIn the implicitly shiftedQR algorithmtheQR decomposition is nowhere to be seen.

Glasgow 2009 – p. 29

One Last QuestionIn the implicitly shiftedQR algorithmtheQR decomposition is nowhere to be seen.

Should the implicitly-shiftedQR algorithm be givena different name?

Glasgow 2009 – p. 29

One Last QuestionIn the implicitly shiftedQR algorithmtheQR decomposition is nowhere to be seen.

Should the implicitly-shiftedQR algorithm be givena different name? Some possibilities: . . .

Glasgow 2009 – p. 29

One Last QuestionIn the implicitly shiftedQR algorithmtheQR decomposition is nowhere to be seen.

Should the implicitly-shiftedQR algorithm be givena different name? Some possibilities: . . .

. . . unitary bulge-chasing algorithm

Glasgow 2009 – p. 29

One Last QuestionIn the implicitly shiftedQR algorithmtheQR decomposition is nowhere to be seen.

Should the implicitly-shiftedQR algorithm be givena different name? Some possibilities: . . .

. . . unitary bulge-chasing algorithm

. . . Hessenberg-Krylov nonstationary progressivenested subspace iteration

Glasgow 2009 – p. 29

One Last QuestionIn the implicitly shiftedQR algorithmtheQR decomposition is nowhere to be seen.

Should the implicitly-shiftedQR algorithm be givena different name? Some possibilities: . . .

. . . unitary bulge-chasing algorithm

. . . Hessenberg-Krylov nonstationary progressivenested subspace iteration

. . . Francis’s algorithm

Glasgow 2009 – p. 29

One Last QuestionIn the implicitly shiftedQR algorithmtheQR decomposition is nowhere to be seen.

Should the implicitly-shiftedQR algorithm be givena different name? Some possibilities: . . .

. . . unitary bulge-chasing algorithm

. . . Hessenberg-Krylov nonstationary progressivenested subspace iteration

. . . Francis’s algorithm

Thank you for your attention.

Glasgow 2009 – p. 29