MA 580; Numerical Analysis I

Part IXa: Residual Correction Methods

MA 580; Numerical Analysis I

C. T. KelleyNC State University

tim [email protected]

Version of November 28, 2016

NCSU, Fall 2016Part VIe: Residual Correction

c©C. T. Kelley, I. C. F. Ipsen, 2016 Part IXa: Residual Correction Methods MA 580, Fall 2016 1 / 52


Residual Correction

Suppose you have a linear solver: S where

S(b) ≈ A−1b.

Residual correction methods try to overcome problems with thesolver by using it iteratively.



Algorithmic Sketch

x = x0r = b− Axwhile ‖r‖ > τ does = S(r)x← x + esr = b− Ax

end while

When S(b) = A−1b you converge in one iteration because

es = A−1r = A−1(b− Ax) = x∗ − x = e.



Classic Iterative Improvement

Suppose S is a single precision solve?

S(b) = U−1s L−1s b

where Ls and Us are the single precision LU factors of A.

AS=single(A);

[LS, US]=lu(AS);

will do the job.



The rules and the facts

Rule: computed residuals in full precision

Fact: relative error from solve will be single precision at best

Fact: cost of LU should be reduced by 1/8

Fact + Rule: you have to pay attention to thelanguage-dependent conversions from single to double andback.



Precision Management

To compute S(b) you need get prepared

Convert A to single to get As

Compute the single precision LU factorization LsUs = A

You use the preparation by

Compute xs = U−1s L−1s b in single.

Convert xs to double to get x.

Compute r = b− Ax in double

Compute es = U−1s L−1s r in singleYou will probably not have to convert r to single to do this

Convert es to double to get ed

Compute x← x + ed in double.



Some Matlab

Preparation:

AS=single(A);

[LS,US]=lu(AS);

Computing the correction:

xs=US\(LS\ b); xd=double(xs);

rd=b-A*xd;

es=US\(LS\rd); ed=double(es);

xd=xd+ed;



The iteration

AS=single(A);

[LS,US]=lu(AS);

xs=US\(LS\b);

xd=double(xs); rd=b-A*xd;

while norm(rd,inf) > 1.d-13

es=US\(LS\rd);

ed=double(es);

xd=xd+ed;

rd=b-A*xd;

end



Some Results: Method of Synthetic Solutions

Begin with

n=100;

A=rand(n,n); xe=rand(n,1); b=A*xe;

κ2(A) ≈ 1900.relative error relative residual

1.77e-05 1.52e-071.61e-11 2.25e-132.02e-14 1.02e-16



Analysis via Back-of-the-Envelope:I

Suppose δs is the relative error the single precision computations.

So x = U−1s L−1s b = x∗ − e,

e = O(δs),

and ed = U−1s L−1s r = e + ‖e‖O(δs) = e + O(δ2s ).

Which tells me that the correction is

x = xd + ed = x + e + O(δ2s ) = x∗ + O(δ2s )

So if δs = 10−k , you get k figures with every iteration.



Analysis via Back-of-the-Envelope:II

What’s δs?

εs ≈ 10−8, single precision roundoff?

κ(A)εs ≈ 1900× 10−8 ≈ 10−5?

Looks like δs = κ(A)εs .



A different perspective

Is this a stationary iterative method? Is the fixed point map

K(x) = x + S(x) ≡ x + I ds U−1s L−1s I sd (b− Ax)

So K is linear if the maps

I ds conversion from single to doubleNo problem: puts the same number is a bigger bucket

I sd conversion from double to singleThis rounding, just as linear as floating point addition

Bottom line: it’s close enough to linear.



What’s the iteration matrix?

The map is

K(x) = x + I ds U−1s L−1s I sd (b− Ax)

= (I− I ds U−1s L−1s I sdA)x + I ds U

−1s L−1s I sdb ≡Msx + xs

So it’s preconditioned Richardson iteration and the convergencerate is

ρ(Ms) = ρ(I− I ds U

−1s L−1s I sdA

).

How might we estimate that?



Estimating ‖Ms‖

How about evaluating ‖Msx− x‖ for a random x?We already did that and got

relative error relative residual

1.77e-05 1.52e-071.61e-11 2.25e-132.02e-14 1.02e-16

So things look consistent, and I ds U−1s L−1s I sd is a good approximate

inverse of A.Warning! Matlab does not support sparse single precision arrays.



There’s more!

Suppose you have two solvers S1 and S2 which are good atdifferent things.Suppose

x+ = S1(xc ,b) is a few iterations of an iterative solver

for Ax = b,with xc as the incoming iterate, andx+ as the new iterate.

S2(b) is a solver (iterative or direct) whichreturns a “converged” result for Ae = r.

Given the transfer maps, these solvers could have differentphysics, precisions, grids, . . .



Two-solver correction

You can let one correct the other via . . .

x = x0while Not happy dox1/2 = S1(x,b)r = b− Ax1/2e = S2(r)x = x1/2 + e

end while



Examples

Iterative refinement: S1 = I; S2 = U−1s L−1s .

MCSA: S1 is one step of Richardson: S2 is Monte Carlo solve

Two-grid: S1 is a couple steps of Jacobi, Gauss-Seidel, . . .S2 is a solve on a coarser grid.

Multigrid: Like two-grid, but S2 is a recursive call to MG.



Two grid for Laplacian in 1-D

Recall our sad story with Jacobi for this problem.

We had a simple example.

Jacobi did ok for the first few iterations

and then went very, very slowly.

It seemed to kill the high frequency error fast,

and get stuck on the low frequency stuff.

Let’s look again.



Jacobi Example

Let’s solve−u′′ = 0, u(0) = u(1) = 0.

with h = 1/101 and N = 100. The solution is u = 0. We will useas an initial iterate

u0 = x(1− x) +1

10sin(49πx)

We will take 100 Jacobi iterations.



Matrix Representation

Au = f

where A is tridiagonal and symmetric

Ah =1

h2

2 −1 0 . . . 0, 0−1 2 −1 , 0 . . . 0

0 −1 2 −1, . . . 0...

. . .. . .

. . .. . .

...0 . . . , , 0, −1 2 −10 . . . , . . . , , 0 −1 2



Eigenvalues and Eigenvectors

Theorem: A is symmetric positive definite. The eigenvalues are

λn = h−22 (1− cos(πnh)) .

The eigenvectors un = (ξn1 , . . . , ξnN)T are given by

ξni =√

2/h sin(niπh)



Initial Error as Function of x

0 0.2 0.4 0.6 0.8 1−0.1

−0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

x

erro

r

Initial Error as Function of $x$



Final Error as Function of x

0 0.2 0.4 0.6 0.8 10

0.05

0.1

0.15

0.2

0.25

x

erro

r

Final Error as Function of $x$



Final Error as Function of x

0 20 40 60 80 1000.24

0.245

0.25

0.255

Iterations

l∞ e

rror

Convergence History



How’s this for a great idea?

Start at mesh size h.

Do a few Jacobi’s (one?)We’ll call Jacobi (or whatever) the smoother.

Transfer residual to mesh 2h.

Do an exact solve of D2h2 e2h = r2h

Send e2h to h mesh and correct x← x + eh



And the details?

Exactly how many Jacobi’s do I need here?

Is Jacobi the best tool for the job?Gauss-Seidel? SOR? Something else?

Can I do the obvious thing for intergrid transfer?

Is this really likely to work?!?



Two-grid algorithm: notation

Ωh: vectors corresponding to the h-mesh Rnd where there aren points in each direction andd = 1, 2, 3.

Ah: discrete differential operator on Ωh.

I 2hh : fine-to-coarse intergrid transfer.

I h2h: coarse-to-fine intergrid transfer.

TG (vh, fh): a single two-grid iteration for Ahvh = fh.

Sν(vh, fh): ν iterations of the smoother.



Two-Grid Method

vh ← TG (vh, fh)

vh ← Sν1(vh, fh)rh = fh − Ahvh

r2h = I 2hh rh

Solve(?) A2he2h = r2h

eh = I h2he2h

vh = vh + eh

vh ← Sν2(vh, fh)



Testing . . .

Let’s make some questionable decisions.

One Jacobi presmooth (ν1 = 1, ν2 = 0)

I h2h: piecewise linear interpolation.

I 2hh : restriction by injection.



Jacobi in Matlab

function us=jacobi(u,b)

n=length(u); h=1/(n+1); h2=h*h; us=zeros(n

,1);

us(2:n-1)=h2*b(2:n-1) + u(3:n) + us(1:n-2);

us(1) = h2*b(1) + u(2); us(n) = h2*b(n) + u(

n-1);

us=us*.5;



Restriction by Injection

function uc=inject(uf)

% INJECT

% fine to coarse intergrid transfer by

injection

% function uc=inject(uf)

%

nf=length(uf);

uc=uf (2:2:nf -1);



Piecewise Linear Interpolation

function uf = ctof(uc)

% CTOF

% function uf = ctof(uc)

% Coarse to fine intergrid transfer

%

nc=length(uc); nf=2*(nc+1) - 1; uf=zeros(nf ,1);

% Use zero boundary conditions

uf(1) =.5*uc(1); uf(nf)=.5*uc(nc);

% Vectorize the rest.

uf(3:2:nf -1) =.5*(uc(1:nc -1)+uc(2:nc));

uf(2:2: nf)=uc;



Restriction by Injection, two-grid, Jacobi

That would be great, but it seems to get stuck.

iterations0 10 20 30 40 50

|| r

||/||

r 0 ||

10-3

10-2

10-1

100

81 = 1

81 = 4

81=8



Gauss-Seidel is better. Why?

iterations0 5 10 15 20 25 30 35 40 45

|| r

||/||

r 0 ||

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

81 = 1

81 = 4

81=8



What happened?

We had this idea that Jacobi did great on high-frequency andpoorly on low, so

I assumed that Gauss-Seidel did the same.

I was half right.

It’s time to do an experiment.

Take one Jacobi/Gauss-Seidel iteration forAu = 0 with u0 = sin(iπx) for 1 ≤ i ≤ nand see what the reduction rate is.



Results for Gauss-Seidel

wave number0 20 40 60 80 100 120 140

redu

ctio

n

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1



Results for Jacobi

wave number0 20 40 60 80 100 120 140

redu

ctio

n

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1



I’d like to save Jacobi because

it parallelizes and vectorizes well.One fix: damped Jacobi

unew = (1− ω)uold + ωujacobi

For our problem it’s

unewi = (1− ω)uoldi + ω(h2bi + uoldi−1 + uoldi+1)/2.

So what’s ω?

ω = 1 is Jacobi.

ω = 0 is do nothing.

The winner is ω = 2/3 because . . .



Analysis of Damped Jacobi

Damped Jacobi is

xn+1 = (1− ω)xn + ωD−1(L + U)xn + D−1b

with iteration matrix MhDJ = (1− ω)I + ωMJAC , where

MhJAC = −D−1(L + U).



Eigen-analysis

We showed that the eigenvalues of MJAC were

µn = 1− (h2/2)λn

with the same eigenvectors as Ah. Similarly

µn = (1− ω) + ω(1− (h2/2)λn) = 1− (h2/2)ωλn

= 1− ω(1− cos(πnh))



Optimal ω

Now suppose n ≥ N/2 (high frequency), then

µn = 1− ω(1− cos(πnh)) = 1− ω + ω cos(πnh)

because cos(πnh) ≤ cos(π/2) = 0, so

1− 2ω ≤ µn ≤ 1− ω.

Minimize |µn| to see that the optimal value of ω is 2/3. So

|µn| ≤ 1/3 for N/2 ≤ n ≤ N − 1



Results for Damped Jacobi

wave number0 20 40 60 80 100 120 140

redu

ctio

n

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1



Comparison

wave number0 20 40 60 80 100 120 140

redu

ctio

n

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Damped JacobiGauss-SeidelJacobi



Damped Jacobi’s the winner. Let’s solve something.

iterations0 5 10 15 20 25 30 35 40

|| r

||/||

r 0 ||

10-15

10-10

10-5

100

81 = 1

81 = 4

81=8



Can we do better?

One problem is that restriction by injection ignores half of the finegrid points.A better idea is restriction by full weighting

u2hi = (uh2i−1 + 2uh2i + uh2i+1)/4

function vcoarse=ftoc(vfine)

% FTOC fine to coarse intergrid transfer by full

weighting

nf=length(vfine);

nc =.5*( nf+1) -1;

vtmp=zeros(nf+2,1);

vcoarse =.5* vfine (2:2:nf -1);

vcoarse=vcoarse +.25*( vfine (1:2:nf -2)+vfine (3:2:nf));



Restriction by full weighting, two-grid, Damped Jacobi

iterations0 5 10 15 20 25

|| r

||/||

r 0 ||

10-14

10-12

10-10

10-8

10-6

10-4

10-2

100

81 = 1

81 = 4

81=8



Something more to like. Matrix representations

I h2h =1

2

1 0 . . . 0 0 02 0 . . . 0 0 01 1 . . . 0 0 00 2 . . . 0 0 00 1 1 . . . 0 00 0 2 . . . 0 00 0 1 1 . . . 0...

......

......

...0 0 0 . . . 1 10 0 0 . . . 0 20 0 0 . . . 0 1

, I 2hh =1

2(I h2h)T (full weighting)



Two-Grid is a stationary iterative method

Go through the loops and

MTG = Mν2S (I− I h2h(A2h)−1I 2hh Ah)Mν1

S

where MS is the iteration matrix for the smoother.One standard configuration is

Damped Jacobi smoother

ν1 = ν2 = 1



Estimate the spectral radius

Look at ratio of relative residuals.

The ratios should converge to the spectral radius

unless something strange happens.

Do this for various values of h to see if the spectral radiusdepends on h or not.



Feel-good Prediction

Each application of damped Jacobi reduces high-frequencyerror by 1/3

So, if the coarse grid correction eliminates the low-frequencyerrors completely . . .

thenρ(MTG ) = 1/9.

Really?



We have a winner! Ratio = 1/9

iteration0 2 4 6 8 10 12 14

resi

dual

rat

io

0

0.1

0.2

0.3

0.4

0.5

0.6h=1/16h=1/32h=1/64h=1/128



Where are we?

Good:

TG is a stationary iterative method.Solver work moved to coarse grid.Convergence rate independent of h.

Bad:

Coarse mesh solver is still a matrix factorization.

Fix: replace the coarse mesh solver with another two-griditeration and get . . .


Documents

MA 580; Numerical Analysis I