Research Matters Nick Higham February 25, 2009 Nick Higham ...higham/talks/an16_cvl.pdf · Black box: no need to choose/tune parameters. Nick Higham Charlie Van Loan 21 / 22 . Charlie

Research Matters

February 25, 2009

Nick HighamDirector of Research

School of Mathematics

1 / 6

Charlie Van Loanand the Matrix Exponential

Nick HighamSchool of Mathematics

The University of Manchester

http://www.maths.manchester.ac.uk/~higham@nhigham, nickhigham.wordpress.com

Numerical Linear and MultilinearAlgebra: Celebrating Charlie Van Loan

SIAM Annual Meeting, BostonJuly 13, 2016

http://www.manchester.ac.uk

http://www.maths.manchester.ac.uk/our-research/research-groups/numerical-analysis-and-scientific-computing/numerical-analysis/

http://www.maths.manchester.ac.uk/~higham/

http://www.maths.manchester.ac.uk

http://www.man.ac.uk

http://www.maths.manchester.ac.uk/~higham/

https://twitter.com/nhigham

http://nickhigham.wordpress.com

Charlie and MATLAB

Pioneer of the use of MATLAB in teaching.Popularized the colon notation in numerical linear algebra.

Nick Higham Charlie Van Loan 3 / 22

Laguerre (1867):

Peano (1888):

eA in Applied Mathematics

Frazer, Duncan & Collar, Aerodynamics Division ofNPL: aircraft flutter, matrix structural analysis.

Elementary Matrices & Some Applications toDynamics and Differential Equations, 1938.Emphasizes importance of eA.

Arthur Roderick Collar, FRS(1908–1986): “First book to treatmatrices as a branch of appliedmathematics”.


University of Manchester Period, 1974–1975

Science Research Council Research Fellow

Charles F. Van Loan. A study of the matrixexponential. Numerical Analysis Report No. 10,University of Manchester, Manchester, UK,August 1975.

“I became interested in the matrix exponentialduring the preparation of a talk I gave on thesubject in 1974 here at Manchester. Since thenI have been motivated by the work of B.N.Parlett [20] and by C. B. Moler with his ‘n badways to compute the matrix exponential(n ≥ 9)’ ”.


Contents

Defines f (A) by Cauchy integral.

Gives explicit, divided difference-based formula forf (T ), T triangular.

Bounds and perturbation bounds for etA.

Backward error result for Padé approximants.

Extensive use of Schur decomposition.“one of the most basic tenets of numericalalgebra, namely, anything that the Jordandecomposition can do, the Schurdecomposition can do better!”


A Bibliography of the Matrix Exponential“Dear Exponential Freak,I collect facts, algorithms, and referencesabout eAt . If you have one that is missing fromthese references, please write to me.” (Undated.)


Sensitivity of the Exponential

SINUM, 1977.

Jordan and Schur-based bounds.Bounded the condition number. Showed conditionnumber is minimal for normal A.

Contemporary work by Kågström.

Open Question

When is the condition number of eA large?


Integrals

exp([

A11 A12

0 A22

])=

eA11

∫ 1

0eA11(1−s)A12eA22s ds

0 eA22

.CVL, IEEE Trans. Automat. Control (1978):use in reverse direction to evaluate integrals involving eA

via exponentials of bock triangular matrices.

4th most cited paper!


Nineteen Dubious Ways (1)

Moler and Van Loan, SIREV, 1978. A ∈ Cn×n:

Power series Limit Scaling and squaring

I + A +A2

2!+

A3

3!+ · · · lim

s→∞(I + A/s)s (eA/2s

)2s

Cauchy integral Jordan form Interpolation

12πi

∫Γ

ez(zI − A)−1 dz Zdiag(eJk )Z−1n∑

i=1

f [λ1, . . . , λi ]

i−1∏j=1

(A − λj I)

Differential system Schur form Padé approximation

Y ′(t) = AY (t), Y (0) = I Qdiag(eT )Q∗ pkm(A)qkm(A)−1



Scaling and squaring: eA =(eA/s

)s.Hump phenomenon: ‖eA‖ � ‖eA/s‖s, instability.Backward error analysis for Padé approximants.S&S: Padé more efficient than Taylor. How to choosePadé degree to achieve given accuracy at min cost.

Pros and cons of various other methods explained;emphasis on numerical stability.Republished in SIREV 2003 with “Method 20: Krylovspace methods” and new sections.Taken together, CVL’s most cited-paper: ≥ 2800citations.



Scaling and squaring: eA =(eA/s

)s.Hump phenomenon: ‖eA‖ � ‖eA/s‖s, instability.Backward error analysis for Padé approximants.S&S: Padé more efficient than Taylor. How to choosePadé degree to achieve given accuracy at min cost.Pros and cons of various other methods explained;emphasis on numerical stability.Republished in SIREV 2003 with “Method 20: Krylovspace methods” and new sections.Taken together, CVL’s most cited-paper: ≥ 2800citations.


History of the S&S Method (1)

J. D. Lawson (1967): exponential integrators for ODE.Suggests using

(e2−sA

)2s

= eA with diagonal Padéapproximants (uncond. stable).

Ward (1977): m = 8, ‖2−sA‖1 ≤ 1.

MATLAB expm up to 2005: m = 6, ‖2−sA‖∞ ≤ 1/2.

In 2004 I started the eA chapter of my book Functions ofMatrices: Theory and Computation . . .



H (2005): choose (s,m) at run-time based on sharper,“non-explicit” error bounds.

m 3 5 7 9 13θm 0.015 0.25 0.95 2.1 5.4

Algorithm

for m = [3 5 7 9 13]if ‖A‖1 ≤ θm, X = rm(A), quit, end

end

A← A/2s with s ≥ 0 minimal s.t. ‖A/2s‖1 ≤ θ13 = 5.4X = r13

2s by repeated squaring.

Faster, and in practice more accurate, than previous alg.



H (2005): choose (s,m) at run-time based on sharper,“non-explicit” error bounds.

m 3 5 7 9 13θm 0.015 0.25 0.95 2.1 5.4

Algorithm

for m = [3 5 7 9 13]if ‖A‖1 ≤ θm, X = rm(A), quit, end

end

A← A/2s with s ≥ 0 minimal s.t. ‖A/2s‖1 ≤ θ13 = 5.4X = r13

2s by repeated squaring.

Faster, and in practice more accurate, than previous alg.



Overscaling (Kenney & Laub, 1998; Dieci & Papini, 2000) :large ‖A‖ causes larger than necessary s to be chosen,with harmful effect on accuracy.Al-Mohy & H (2009) “avoid” via sharper bounds.

Lemma

If k = pm1 + qm2 with p,q ∈ N and m1,m2 ∈ N ∪ {0},

‖Ak‖1/k ≤ max(‖Ap‖1/p, ‖Aq‖1/q).

Take {p,q} = {r , r + 1} for k ≥ r(r − 1).

Improved expm in MATLAB 2015b.


Software Scene

Al-Mohy & H (2009) scaling & squaring algorithm for eA is in

Julia,MATLAB,NAG Library,R,SciPy.

H & Deadman, A Catalogue of Software for MatrixFunctions (2016).


http://eprints.ma.man.ac.uk/2450

http://eprints.ma.man.ac.uk/2450

https://nickhigham.wordpress.com/2016/03/10/updated-catalogue-of-software-for-matrix-functions/

Action of the Exponential

Compute eAb without first computing eA.

What was originally wanted by Lawson (1967), butproblems small there!Krylov methods commonly used.


The Sixth Dubious WayMoler & Van Loan (1978, 2003)


Al-Mohy & Higham (2011)—expmv

Exploit, for integer s,

eAB = (es−1A)sB = es−1Aes−1A · · · es−1A︸︷︷︸s times

B.

Choose s so Tm(s−1A) =∑m

j=0(s−1A)j

j!≈ es−1A. Then

Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B

yields Bs ≈ eAB.

Choose s and m using b’err & ‖Ak‖1/k approach.


Al-Mohy & Higham (2011)—expmv

Exploit, for integer s,

eAB = (es−1A)sB = es−1Aes−1A · · · es−1A︸︷︷︸s times

B.

Choose s so Tm(s−1A) =∑m

j=0(s−1A)j

j!≈ es−1A. Then

Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B

yields Bs ≈ eAB.

Choose s and m using b’err & ‖Ak‖1/k approach.


Advantages of expmv

Versus one-step ODE integrators:

Fully exploits the linearity of the ODE.Variable order, up to m = 55.Backward error based; ODE integrator controls local(forward) errors.Overscaling avoided.

Versus Krylov methods:

Very competitive in cost and storage.Cost dominated by matrix–vector multiplications.Black box: no need to choose/tune parameters.


Charlie in Three Bullets

Aficionado of the Matrix Exponential.

Master of Matrix Computations.

Baseball nut andfamily man."Reproduced with thepermission of theCommissioner of MajorLeague Baseball".c©1988



Aficionado of the Matrix Exponential.Master of Matrix Computations.




Aficionado of the Matrix Exponential.Master of Matrix Computations.



References I

A. H. Al-Mohy and N. J. Higham.Improved inverse scaling and squaring algorithms forthe matrix logarithm.SIAM J. Sci. Comput., 34(4):C153–C169, 2012.

M. Aprahamian and N. J. Higham.The matrix unwinding function, with an application tocomputing the matrix exponential.SIAM J. Matrix Anal. Appl., 35(1):88–109, 2014.

L. Dieci and A. Papini.Padé approximation for the exponential of a blocktriangular matrix.Linear Algebra Appl., 308:183–202, 2000.


References II

R. A. Frazer, W. J. Duncan, and A. R. Collar.Elementary Matrices and Some Applications toDynamics and Differential Equations.Cambridge University Press, Cambridge, UK, 1938.xviii+416 pp.1963 printing.

N. J. Higham.Functions of Matrices: Theory and Computation.Society for Industrial and Applied Mathematics,Philadelphia, PA, USA, 2008.ISBN 978-0-898716-46-7.xx+425 pp.


References III

N. J. Higham and A. H. Al-Mohy.Computing matrix functions.Acta Numerica, 19:159–208, 2010.

N. J. Higham and E. Deadman.A catalogue of software for matrix functions. Version2.0.MIMS EPrint 2016.3, Manchester Institute forMathematical Sciences, The University of Manchester,UK, Jan. 2016.22 pp.Updated March 2016.


References IV

N. J. Higham and F. Tisseur.A block algorithm for matrix 1-norm estimation, with anapplication to 1-norm pseudospectra.SIAM J. Matrix Anal. Appl., 21(4):1185–1201, 2000.

R. A. Horn and C. R. Johnson.Topics in Matrix Analysis.Cambridge University Press, Cambridge, UK, 1991.ISBN 0-521-30587-X.viii+607 pp.

C. S. Kenney and A. J. Laub.A Schur–Fréchet algorithm for computing the logarithmand exponential of a matrix.SIAM J. Matrix Anal. Appl., 19(3):640–663, 1998.


References V

C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix.SIAM Rev., 20(4):801–836, 1978.

C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix, twenty-five years later.SIAM Rev., 45(1):3–49, 2003.

Multiprecision Computing Toolbox.Advanpix, Tokyo.http://www.advanpix.com.


http://www.advanpix.com

Documents

Research Matters Nick Higham February 25, 2009 Nick Higham ...higham/talks/an16_cvl.pdf · Black box: no need to choose/tune parameters. Nick Higham Charlie Van Loan 21 / 22 . Charlie