Nick Higham Research Matters School of Mathematics The ...higham/talks/talk11...Research Matters February 25, 2009 Nick Higham Director of Research School of Mathematics 1 / 6 The

Research Matters

February 25, 2009

Nick HighamDirector of Research

School of Mathematics

1 / 6

The f (A)b Problem

Nick HighamSchool of Mathematics

The University of Manchester

[email protected]://www.ma.man.ac.uk/~higham/

ICIAM 2011, Vancouver, July 2011.

http://www.ma.man.ac.uk/~higham/http://www.ma.man.ac.ukhttp://www.man.ac.ukmailto:[email protected]://www.ma.man.ac.uk/~higham/

The f (A)b Problem

Givenmatrix function f : Cn×n → Cn×n,A ∈ Cn×n, b ∈ Cn,

compute f (A)b without first computing f (A).

Most important casesf (x) = x−1

f (x) = ex .f (x) = log(x).f (x) = x1/2.f (x) = sign(x).

MIMS Nick Higham The f (A)b Problem 2 / 29

http://www.mims.manchester.ac.uk/

Application: Second Order ODE

d2ydt2

+ Ay = 0, y(0) = y0, y ′(0) = y ′0

has solution

y(t) = cos(√

At)y0 +(√

A)−1 sin(√At)y ′0.

But [y ′

y

]= exp

([0 −tA

t In 0

])[y ′0y0

].



A1/2b: Application in StatisticsChen, Anitescu & Saad (2011): sample x ∼ N(µ,C),C ∈ Rm×m, m ∈ [1012,1015].Let x ∼ N(0, I).

If C = LLT is Cholesky factorization,y = µ+ Lx ∼ N(µ,C).

Cholesky factorization may not be computable orstorable.

y = µ+ C1/2x ∼ N(µ,C).



A1/2 b via Contour Integration

f (A)b =1

2πi

∫Γ

f (z)(zI − A)−1b dz.

A 5× 5 Pascal matrix: λ(A) ∈ [0.01,92.3], f (z) = z1/2.Use repeated trapezium rule to integrate around circlecentre (λmin + λmax)/2, radius λmax/2.Need 32,000 (262,000) points for 2 (13) decimal digits.Hale, H & Trefethen (2008): conformally map

Ω = C \{(−∞,0] ∪ [m,M ]

}.

to an annulus: [m,M ]→ inner circle, (−∞,0]→ outercircle. Now 5 (35) points for 2 (13) decimal digits.Further refinements: 20 points yield 14 digits.



Aα b via Binomial Expansion

Write A = s(I − C).If λi > 0, s = (λmin + λmax)/2 yieldsρmin = (λmax − λmin)/(λmax + λmin).For any A, s = trace(A∗A)/trace(A∗) minimizes ‖C‖F .

(I − C)α =∞∑

j=0

(α

j

)(−C)j , ρ(C) < 1.

So

Aαb = sα∞∑

j=0

(α

j

)(−C)jb.

For M-matrices, required splitting with C ≥ 0 always exists.



Aα b via ODE IVP

dydt

= α(A− I)[t(A− I) + I

]−1y , y(0) = bhas unique solution

y(t) =[t(A− I) + I

]αbso

y(1) = Aαb.

Used by Allen, Baglama & Boyd (2000) for α = 1/2, spd A.



Exponential Integrators

u ′(t) = Au(t) + g(t ,u(t)), u(0) = u0, t ≥ 0,

Solution can be written

u(t) = etAu0 +∞∑

k=1

ϕk(tA)tk uk ,

where uk = g(k−1)(t ,u(t)) |t=0 and ϕ`(z) =∑∞

k=0 zk/(k + `)!.

u(t) ≈ û(t) = etAu0 +p∑

k=1

ϕk(tA)tk uk .

Exponential time differencing (ETD) Euler (p = 1):

yn+1 = ehAyn + hϕ1(hA)g(tn, yn).



Saad’s Trick (1992)

ϕ1(z) =ez − 1

z.

exp([

A b0 0

])=

[eA ϕ1(A)b0 1

]



Theorem (Al-Mohy & H, 2011)

Let A ∈ Cn×n, U = [u1,u2, . . . ,up] ∈ Cn×p, τ ∈ C, and define

B =[

A U0 J

]∈ C(n+p)×(n+p), J =

[0 Ip−10 0

]∈ Cp×p.

Then for X = eτB we have

X (1 : n,n + j) =∑j

k=1 τk ϕk(τA)uj−k+1, j = 1 : p.

Completely removes the need toevaluate ϕk functions!

Implementation of Exponential Integrators

We compute

û(t) = etAu0 +p∑

k=1

ϕk(tA)tk uk

as, with U = [up, . . . ,u1],

û(t) =[

In 0]

exp(

t[

A ηU0 J

])[u0

η−1ep

].

Choose η so that η‖U‖ ≈ 1.



Assumptions

Choice of algorithm depends on assumptions about A.

Examples:

Can Schur-factorize A.“Backslash matrix”: can solve (zI −A)x = b by (sparse)direct methods.A symmetric, can estimate [λmin, λmax], onlymatrix–vector products available.A is general, only matrix–vector products available.



Formulae for eA, A ∈ Cn×n

Taylor series Limit Scaling and squaring

I + A +A2

2!+

A3

3!+ · · · lim

s→∞(I + A/s)s (eA/2

s)2

s

Cauchy integral Jordan form Interpolation

12πi

∫Γ

ez(zI − A)−1 dz Zdiag(eJk )Z−1n∑

i=1

f [λ1, . . . , λi ]i−1∏j=1

(A − λj I)

Differential system Schur form Padé approximation

Y ′(t) = AY (t), Y (0) = I QeT Q∗ pkm(A)qkm(A)−1

Krylov methods: Arnoldi fact. AQk = Qk Hk + hk+1,k qk+1eTk withHessenberg H: eAb ≈ Qk eHk Q∗k b.



The Sixth Dubious WayMoler & Van Loan (1978, 2003)



Computing eAB

A︸︷︷︸n×n

, B︸︷︷︸n×n0

, n0 � n. Exploit, for integer s,

eAB = (es−1A)sB = es

−1Aes−1A · · · es−1A︸︷︷︸s times

B.

Choose s so Tm(s−1A) =∑m

j=0(s−1A)j

j!≈ es−1A. Then

Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B

yields Bs ≈ eAB.

How to choose s and m?



Backward Error Analysis

Lemma (Al-Mohy & H, 2009)

Tm(s−1A)sB = eA+∆AB, where ∆A = shm+1(s−1A) andhm+1(x) = log(e−x Tm(x)) =

∑∞k=m+1 ck x

k . Moreover,

‖∆A‖ ≤ s∞∑

k=m+1

|ck | αp(s−1A)k

if m + 1 ≥ p(p − 1), where

αp(A) = max(dp,dp+1), dp = ‖Ap‖1/p.



Why Use dp = ‖Ap‖1/p?ρ(A) ≤ dp ≤ ‖A‖.

‖Ak‖ ≤ ‖Apk‖1/p ≤ dkp .

With A =[

0.9 5000 −0.5

]:

k 2 5 10‖Ak‖1 2.0e2 2.2e2 1.2e2‖A‖k1 2.5e5 3.1e13 9.8e26

dk2 =(‖A2‖1/21

)k 2.0e2 5.7e5 3.2e11dk3 =

(‖A3‖1/31

)k 4.5e1 1.3e4 1.9e8Cheaply estimate ‖Ak‖1, for a few k (H & Tisseur,2000).



Why Use dp = ‖Ap‖1/p? — cont.

dp = ‖Ap‖1/p provide information about thenonnormality of A.Their use helps avoid overscaling.What other uses do they have?



Size of Taylor Series Argument

Constant

θm :=max

{θ :

∞∑k=m+1

|ck |θk−1 ≤ �}.

Computed via symbolic, high precision:

m 10 20 30 40 55single 1.0e0 3.6e0 6.3e0 9.1e0 1.3e1

double 1.4e-1 1.4e0 3.5e0 6.0e0 9.9e0



Choice of s and m

αp(A) :=max(dp,dp+1

)‖∆A‖‖A‖ ≤ � if m + 1 ≥ p(p − 1) and s

−1αp(A) ≤ θm.

Computational cost for Bs ≈ eAB is

Cm(A) = m max(dαp(A)/θme,1

)matrix products.

Cost decreases with m.Restrict 2 ≤ p ≤ pmax, p(p − 1)− 1 ≤ m ≤ mmax.Minimize cost over p, m.



Preprocessing

Expand the Taylor series about µ ∈ C:

eµ∞∑

k=0

(A− µI)k/k !

Choose µ so ‖A− µI‖ ≤ ‖A‖.Alg is based on 1-norm, but minimizing ‖A− µI‖F doesbetter empirically at minimizing dp(A− µI).Recover eA from

eµ[Tm(s−1(A− µI))

]s,

[eµ/sTm(s−1(A− µI))

]s.

First expression prone to overflow, so prefer second.Balancing is an option.



Termination Criterion

In evaluating

Tm(s−1A)Bi =m∑

j=0

(s−1A)j

j!Bi

we accept Tk(A)Bi for the first k such that

‖Ak−1Bi‖(k − 1)! +

‖AkBi‖k !

≤ �‖Tk(A)Bi‖.



Algorithm for F = etAB

1 µ = trace(A)/n2 A = A− µI3 [m∗, s] = parameters(tA) % Includes norm estimation.4 F = B, η = etµ/s

5 for i = 1: s6 c1 = ‖B‖∞7 for j = 1:m∗8 B = t AB/(sj), c2 = ‖B‖∞9 F = F + B

10 if c1 + c2 ≤ tol‖F‖∞, quit, end11 c1 = c212 end13 F = ηF , B = F14 end



George Forsythe “Pitfalls” (1970)



Conditioning of eAB

Relative forward error due to roundoff bounded by

ue‖A‖2‖B‖2/‖eAB‖F .

A normal implies κexp(A) = ‖A‖2. Then instability ife‖A‖2 � ‖eA‖2.A Hermitian implies spectrum of A− n−1trace(A)Ihas λmax = −λmin ⇒ (normwise) stability!



Comparison with the Sixth Dubious Way

Advantages of our method over the one-step ODEintegrator:

Fully exploits the linearity of the ODE.Backward error based; ODE integrator controls local(forward) errors.Overscaling avoided.



Experiment 1

Trefethen, Weideman & Schmelzer (2006):A ∈ R9801×9801, 2D Laplacian,-2500*gallery(’poisson’,99).

Compute eαAb, tol = ud .

α = 0.02 α = 1speed mv diff speed mv diff

Alg AH 1 1010 1 47702expv 2.8 403 7.7e-15 1.3 8835 4.2e-15phipm 1.1 172 3.1e-15 0.2 2504 4.0e-15rational 3.8 7 3.3e-14 0.1 7 1.2e-12



Experiment 2A = -gallery(’triw’,20,4.1), bi = cos i , tol = ud .

0 20 40 60 80 100

10−10

10−5

100

105

t

‖etAb‖2

Alg AH

phipm

rational

expv



Conclusions

f (A)b problem of growing interest.Many possible approaches (including Krylov—notdiscussed here).Method design/choice must take into account

assumptions about A,desired accuracy.

New Taylor series-based alg for eAB very competitivewith existing methods. Well suited for “black box” code.

Al-Mohy & H. Computing the action of the matrixexponential, with an application to exponentialintegrators. SISC, 2011.



References I

A. H. Al-Mohy and N. J. Higham.Computing the action of the matrix exponential, with anapplication to exponential integrators.SIAM J. Sci. Comput., 33(2):488–511, 2011.

E. J. Allen, J. Baglama, and S. K. Boyd.Numerical approximation of the product of the squareroot of a matrix with a vector.Linear Algebra Appl., 310:167–181, 2000.

J. Chen, M. Anitescu, and Y. Saad.Computing f (A)b via least squares polynomialapproximations.SIAM J. Sci. Comput., 33(1):195–222, 2011.



References II

P. I. Davies and N. J. Higham.Computing f (A)b for matrix functions f .In A. Boriçi, A. Frommer, B. Joó, A. Kennedy, andB. Pendleton, editors, QCD and Numerical Analysis III,volume 47 of Lecture Notes in Computational Scienceand Engineering, pages 15–24. Springer-Verlag, Berlin,2005.

G. E. Forsythe.Pitfalls in computation, or why a math book isn’tenough.Amer. Math. Monthly, 77(9):931–956, 1970.



References III

N. Hale, N. J. Higham, and L. N. Trefethen.Computing Aα, log(A) and related matrix functions bycontour integrals.SIAM J. Numer. Anal., 46(5):2505–2523, 2008.

N. J. Higham.Functions of Matrices: Theory and Computation.Society for Industrial and Applied Mathematics,Philadelphia, PA, USA, 2008.ISBN 978-0-898716-46-7.xx+425 pp.

N. J. Higham and A. H. Al-Mohy.Computing matrix functions.Acta Numerica, 19:159–208, 2010.



References IV

N. J. Higham and F. Tisseur.A block algorithm for matrix 1-norm estimation, with anapplication to 1-norm pseudospectra.SIAM J. Matrix Anal. Appl., 21(4):1185–1201, 2000.

C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix.SIAM Rev., 20(4):801–836, 1978.

C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix, twenty-five years later.SIAM Rev., 45(1):3–49, 2003.



References V

Y. Saad.Analysis of some Krylov subspace approximations tothe matrix exponential operator.SIAM J. Numer. Anal., 29(1):209–228, Feb. 1992.

L. N. Trefethen, J. A. C. Weideman, and T. Schmelzer.Talbot quadratures and rational approximations.BIT, 46(3):653–670, 2006.


Documents

Nick Higham Research Matters School of Mathematics The ...higham/talks/talk11...Research Matters February 25, 2009 Nick Higham Director of Research School of Mathematics 1 / 6 The