Asymptotic error expansion

Example 1: Numerical differentiation

– Truncation error via Taylor expansion

)(:2

)()()(': hI

h

hxfhxfxfI

2 3 4(2) (3) (4)

2 4(3) (5)

2 4 62 4 6

2 4 62 4 6

( ) ( ) '( ) ( ) ( ) ( )2 3! 4!

( ) ( )( ) : '( )

2

( ) ( )3! 5!

( )

h h hf x h f x h f x f x f x f x

f x h f x hE h f x

h

h hf x f x

A h A h A h

I h I A h A h A h

Asymptotic error expansion

Example 2: Numerical integration via midpoint rule

– Truncation error via Taylor expansion

66

44

22)( hAhAhAIhI

)(:)()()()()(: 2/12/122/112/1 hIxfxfxfxfhdxxfI N

b

a

Asymptotic error expansion

In general, we assume

In addition, we assume the asymptotic error expansion

– Convergence – Order of convergence: p1

Estimate order of convergence numerically– By log-log plot– By quotation

0 when0)( satisfying )( hhIIhII

321321 0 with)( 321 ppphAhAhAIhI ppp

Richardson extrapolation

Suppose we have the asymptotic error expansion

With two different meshes: h:=h1 < h2 321321 0 with)( 321 ppphAhAhAIhI ppp

1:

)()()(I

)(

)(

1

2131211

1

213

1

212

1

211

2322212

1312111

332211

332211

321

321

h

hcchAchAchAI

h

hhA

h

hhA

h

hhA

hAhAhAIhI

hAhAhAIhI

pppppp

pppppp

ppp

ppp

Richardson extrapolation

Eliminating the leading order error term

Equivalently, we have

Better approximation, with order of accuracy: p2

3

1

312

1

21

1

1

131221

1)1(

111

)()(:)( p

p

ppp

p

pp

p

p

hAc

cchA

c

ccI

c

hIhIchI

32

1

1

1)1(

31)1(

221

1)1(

1

)()(:)( pp

p

p

hAhAIc

hIhIchI

Richardson extrapolation

Specifically, if we choose

Similarly, we have

22 12 chh

32

1

1

321

)1(3

)1(2

)1(

321321

12

)2()(2:)(

0 with)(

ppp

p

ppp

hAhAIhIhI

hI

ppphAhAhAIhI

3

2

2

32

1

1

321

)2(3

)1()1()2(

)1(3

)1(2

)1(

321321

12

)2()(2:)(

12

)2()(2:)(

0 with)(

pp

p

ppp

p

ppp

hAIhIhI

hI

hAhAIhIhI

hI

ppphAhAhAIhI

Romberg algorithm

Choose a sequence of mesh

Romberg algorithm based on Richardson extrapolation

2/,,2/,2/,2/, 12312010 nn hhhhhhhhh

)()()()(rate econvergenc

)()()()(

)()()()(

)()()(

)()(

)(

4321

)3()2()1(

3)3(

3)2(

3)1(

33

2)2(

2)1(

22

1)1(

11

00

ppppnnnnn

hOhOhOhO

hIhIhIhIh

hIhIhIhIh

hIhIhIh

hIhIh

hIh

An example

Composite trapezoidal rule

– Asymptotic error expansion

– Richardson extrapolation

)(:)(2

1)()()()(

2

1)( 1210 hIxfxfxfxfxfhdxxfI NN

b

a

)( 63

42

21 hAhAhAIhI

6)2(34

)1()1(4)2(

6)1(3

4)1(22

2)1(

63

42

21

12

)2()(2:)(

12

)2()(2:)(

)(

hAIhIhI

hI

hAhAIhIhI

hI

hAhAhAIhI

An example

Romberg algorithm

Exponential convergence rate

)()()()(rate econvergenc

)()()()(

)()()()(

)()()(

)()(

)(

8642

)3()2()1(

3)3(

3)2(

3)1(

33

2)2(

2)1(

22

1)1(

11

00

hOhOhOhO

hIhIhIhIh

hIhIhIhIh

hIhIhIh

hIhIh

hIh

nnnnn

IhIhIhIhI nn )(,),(),(),( )(

2)2(

1)1(

0

Numerical result

Compute:Result by Romberg algorithm

)(2)sin(0

hIdxxI

00000000.200000000.200000000.200000000.200000103.299839336.132/

99999999.100000001.299999975.100001659.299357034.116/

00000555.299998313.100026917.297423160.18/

99857073.100455976.289611890.14/

09439511.257079633.12/

0

5

4

3

2

1

0

h

h

h

h

h

h

Order of convergence

00000000.000000000.000000000.000000000.000000103.000160664.032/

00000000.000000001.000000025.000001659.000642966.016/

00000555.000001687.000026917.002576840.08/

00142927.000455976.010388110.04/

09439511.042920367.02/

2

5

4

3

2

1

0

h

h

h

h

h

h

)( 2hO )( 4hO )( 8hO)( 6hO

Exercises

Suppose the asymptotic error expansion

– Design the Richardson extrapolation with

– Design the Richardson extrapolation with

– Design the Romberg algorithm with

4/,,4/,4/,4/, 12312010 nn hhhhhhhhh

3/,,3/,3/,3/, 12312010 nn hhhhhhhhh

1 with/,,/,/,/, 12312010 cchhchhchhchhh nn

321321 0 with)( 321 ppphAhAhAIhI ppp

Stability and conditioning

Example 1. Linear system

– Numerical solution with 3 digits

– Perturbation errors: small & stable

3,123/1

96

yxyx

yx

006.3~,999.0~2~333.0~

9~~6

yxyx

yx

006.0~,001.0~ yyxx

Stability and conditioning

Example 2. Linear system

– Numerical solution with 3 digits

– Perturbation errors: extremely large & unstable!!!!!

3,123/1

001.6001.3

yxyx

yx

1~,333.2~2~333.0~001.6~~001.3

yxyx

yx

4~,333.1~ yyxx

Stability and conditioning

In many cases, inaccuracies in computed results are much larger than the round-off errors and/or truncation errors introduced in the computationThe reason may be that the errors were ``amplified’’ by the algorithm.We say that a problem is stable if the solution depends continuously on the input parameters– If ``small’’ changes are made to the input parameters, then the resulting

changes in the solution will also be ``small– Mathematically,

bCxx ~

Stability and conditioning

If a problem is not stable, then it is said to be unstable

Condition number of a problem: – A measure of the sensitivity of its solution to small

perturbation of the input parameters– The ratio of the relative change in the solution to the relative

change in the input parameters– It is significant in many problems because the round-off errors

in the input to a problem may lead to large changes in the solution

Stability and conditioning

– Small: well-conditioned problem– Large: ill-conditioned problem– It is a property of the problem to be solved itself, but not of

the numerical algorithm employed to solve it!!!– It depends on the number of significant digits used in the

computation• Single precision • Double precision ---- adapted in most current scientific computations • 4 times precision• Infinite precision

Perturbation analysis

Solving the linear system

Vector and matrix norm

Perturbation in the right hand side

invertible is A suppose 1 bAxbxA

nnnnij

x

nTnn

aAx

xAAA

xxxxxxxxx

n

)(,sup:

),,,(,:

2

2

02

2122

2212

)(&

11

1

b

bAA

xA

bAA

x

bA

x

xbxA

bbxxAbxAxxxbbb

Perturbation analysis

– The largest ratio of the relative change

Condition number

– Two examples

AAb

b

x

x 1 /

1/ cond(A):k(A) minmax1 AA

33351)(cond,0849.38)(cond

3/11

1001.3

3/11

16

BA

BA

Some comments on condition number

– k(A) is great or equal than 1!! It depends on the norm.– A is well-conditioned if k(A)=O(1)– A is ill-conditioned if k(A)>>1– The linear system A x= b is well-conditioned (ill-conditioned) if

A is well-conditioned (ill-conditioned)– For well-conditioned linear systems, the relative change in the

solution is small if the relative change in the right-hand side is small

– For ill-conditioned linear systems, the relative change is the sloution can be very large even the right hand side is small!!

A simple example

Solve

– Condition number: ill-conditioned !!!!

– Perturb the right hand side

– Perturbation in the error

TxbxA

AbA

)11(

1000999

999998

1997

1999

998999

9991000 1

6211 10996.31999)(1999

AAAkAA

99.19

97.19

99.18

97.20

01.1997

99.1998

01.0

01.0xxxbbb

99.191999

01.01999)(99.19 2

b

bAk

x

x

Another example

Consider

– Plot the condition number of A for different degree– Solve

3/7

3/7

3/5

3/7

3/1

3/1

1

3/1

1000

0100

0210

0021

xAbxA

yxbyA

error plot the andy numericall

Condition number of A

Error of the solution

Some observations

A is not singular since det(A)=1 !! Condition number increase exponentially When n=73, the condition number exceeds the double precision!!We solve the linear system by back substitution, thus no truncation. So the errors are fully due to the round-off errors!!The error is directly proportional to the condition numberRound-off errors are important for large sparse matrix!

Perturbation on b

Result

Proof

AAAkb

bk(A)

x

x

bbxxAbxAxxxbbb

1)( with

)(&

11

1

b

bAA

xA

bAA

x

bA

x

x

Perturbation on A

Result

Proof: see details in class– Lemma 1.

– Lemma 2

)(1

)(

)(

))((&

1

A

AAk

A

AAk

x

xxAxAAx

bxxAAbxA

xxxAAA

invertible is them,1&matrix squarea is If XIX X

invertible is them,)(

1&matrix invertible squarea is If AA

AkA

A A

Perturbation on A & b

Result

Proof: see details in class

)(1

)(

)]([

))((&

&

1

A

A

b

b

A

AAk

Ak

x

x

xAxAbAx

bbxxAAbxA

xxxbbbAAA

Efficient computation: Fast algorithms

Example 1: compute power – Algorithm 1:

• Computational cost: 254 times multiplication

255x

0output

end;

00

2541 do

;0

s

*xss

,i

xs

xxxx 255

Efficient computation: Fast algorithms

– Algorithm 2:

• Computational cost: 14 times multiplication

0output

end;

100

;1*11

71 do

;1;0

s

*sss

sss

,i

xsxs

128643216842255 xxxxxxxxx

Efficient computation

Example 2: Evaluate polynomial

– Direction sum: cost is n(n+1)/2– Fast algorithm: O(n)!!!

012

21

1)( axaxaxaxaxP nn

nnn

)))((()( 1210 xaaxaxaxaxP nnn

sum

asumxsum

,ni

asum

i

n

output

end;

1,01 do

Review of numerical integration and differentiation

Numerical integration– Basic quadratures

• Midpoint rule• Trapezoidal rule• Simpson’s rule

– Composite techniques– Romberg algorithm – Richardson extrapolation– Gaussian quadratures --- high order

Numerical differentiation– Finite difference & stencils

Review of function approximation & interpolation

Function interpolation– Lagrange polynomial interpolation– Hermite polynomial interpolation– Cubic spine interpolation– Piecewise polynomial interpolation

Function approximation– Orthogonal functions approximation– Least square approximation