Computing Errors Radu Tr^ mbit˘a˘stradu/nlalgslides/lec4b_condstab.pdf · Radu Tr^ mbit˘a˘s ("Babe˘s-Bolyai" University) Conditioning and Stability March 18, 2009 8 / 24. Stability

Conditioning and StabilityComputing Errors

Radu Trımbitas

”Babes-Bolyai” University

March 18, 2009

Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 1 / 24

Conditioning

Absolute Condition Number of a differentiable problem f at x :

κ = supδx

‖δf ‖‖δx‖ = ‖J(x)‖

where the Jacobian J(x) = [Jij ] = [δfi /δxj ], and the matrix norm isinduced by the norms on δf and δx

Relative Condition Number

κ = supδx

‖δf ‖‖f (x)‖‖δx‖‖x‖

=‖J(x)‖

‖f (x)‖ / ‖x‖


Conditioning - Examples

Example: The function f (x) = αx

Absolute condition number κ = ‖J‖ = α

Relative condition number κ = ‖J‖‖f (x)‖/‖x‖ = α

αx/x = 1

Example: The function f (x) =√

x

Absolute condition number κ = ‖J‖ = 12√

x

Relative condition number κ = ‖J‖‖f (x)‖/‖x‖ = 1/(2

√x)√

x/x= 1

2

Example: The function f (x) = x1 − x2 (with ∞-norms)

Absolute condition number κ = ‖J‖ =∥∥∥(−1, 1)T

∥∥∥ = 2

Relative condition number κ = ‖J‖‖f (x)‖/‖x‖ = 2

|x1−x2|max{|x1|,|x2|}Ill-conditioned when x1 ≈ x2 (cancellation)


Condition of Matrix-Vector Product

Consider f (x) = Ax , with A ∈ Cm×n

κ =‖J‖

‖f (x)‖ / ‖x‖ = ‖A‖ ‖x‖‖Ax‖

For A square and nonsingular, use ‖x‖ / ‖Ax‖ ≤∥∥A−1

∥∥:

κ ≤ ‖A‖∥∥A−1

∥∥(equality achieved for the last right singular vector x = vm)

Also the condition number for f (b) = A−1b (solution of linearsystem)

Condition number of matrix A:

κ(A) = ‖A‖∥∥A−1

∥∥ for 2-norm κ(A) =σ1

σm


Condition of System of Equations

For fixed b, consider f (A) = A−1b

Perturb A by δA and find perturbation δx :

(A + δA)(x + δx) = b

Use Ax = b and assume (δA)(δx) ≈ 0:

(δA)x + A(δx) = 0 =⇒ δx = −A−1(δA)x

Condition number of problem f

κ =‖δx‖‖x‖

/‖δA‖‖A‖ ≤

∥∥A−1∥∥ ‖δA‖ ‖x‖‖x‖

/‖δA‖‖A‖ =

∥∥A−1∥∥ ‖A‖ = κ(A)


Accuracy

Consider an algorithm f for a problem f

A computation f (x) has absolute error∥∥∥f (x)− f (x)

∥∥∥ and relativeerror ∥∥∥f (x)− f (x)

∥∥∥‖f (x)‖

The algorithm is accurate if (for all x)∥∥∥f (x)− f (x)∥∥∥

‖f (x)‖ = O(εmachine)

where O(εmachine) is “on the order of εmachine” (more next slide)

Constant in O(εmachine) is likely to be large in many problems, sincebecause of rounding we are not even using the correct x


More on O(εmachine)

The notation ϕ(t) = O(ψ(t)) means there is a constant C such that,for t close to a limit (often 0 or ∞), |ϕ(t)| ≤ Cψ(t)Example: sin2 t = O(t2) as t → 0 means | sin2 t| ≤ Ct2 for some C

If ϕ depends on additional variables, the notation

ϕ(s, t) = O(ψ(t)) uniformly in s

means there is a constant C such that |ϕ(s, t)| ≤ Cψ(t) for any s

Example: (sin2 t)(sin2 s) = O(t2) uniformly as t → 0, but not ifsin2 s is replaced by s2

In bounds such as ‖x − x‖ ≤ Cκ(A)εmachine ‖x‖, C does not dependon A or b, but it might depend on the dimension m


Stability

An algorithm f for a problem f is stable if (for all x)∥∥∥f (x)− f (x)∥∥∥


for some x with‖x − x‖‖x‖ = O(εmachine)

“Nearly the right answer to nearly the right question”

An algorithm f for a problem f is backward stable if (for all x)

f (x) = f (x) for some x with‖x − x‖‖x‖ = O(εmachine)

“Exactly the right answer to nearly the right question”


Stability









Stability









Stability









Stability of Floating Point Arithmetic

The two floating point axioms imply backward stability for theoperations ~

(1) For all x ∈ R, there exists ε with |ε| ≤ εmachine such thatfl(x) = x(1 + ε)

(2) For all floating point x , y , there exists ε with |ε| ≤ εmachine such thatx ~ y = (x ∗ y)(1 + ε)

Example: Subtraction f (x1, x2) = x1 − x2 with floating pointalgorithm

f (x1, x2) = fl(x1) fl(x2)

(1) implies

fl(x1) = x1(1 + ε1), fl(x2) = x2(1 + ε2)

for some |ε1|, |ε2| ≤ εmachine



(example continued)

(2) implies

fl(x1) fl(x2) = (fl(x1)− fl(x2))(1 + ε3)

for some |ε3| ≤ εmachine

Combine:

fl(x1) fl(x2) = (x1(1 + ε1)− x2(1 + ε2))(1 + ε3)= x1(1 + ε1)(1 + ε3)− x2(1 + ε2)(1 + ε3)= x1(1 + ε4)− x2(1 + ε5)

for some |ε4|, |ε4| ≤ 2εmachine + O(ε2machine)

Therefore, fl(x1)− fl(x2) = x1 − x2



Example: Inner product f (x , y) = x∗y computed with ⊗ and ⊕ isbackward stable (more later)

Example: Outer product f (x , y) = xy ∗ computed with ⊗ is notbackward stable (unlikely that f is rank-1)

Example: f (x) = x + 1 computed by f (x) = fl(x)⊕ 1 is notbackward stable (consider x ≈ 0)

Example: f (x , y) = x + y computed by f (x , y) = fl(x)⊕ fl(y) isbackward stable




















Theorem (Accuracy of a Backward Stable Algorithm)

If a backward stable algorithm is used to solve a problem f with conditionnumber κ, the relative errors satisfy∥∥∥f (x)− f (x)

∥∥∥‖f (x)‖ = O(κ(x)εmachine)

Proof.

Backward stability means f (x) = f (x), for some x such that‖x−x‖‖x‖ = O(εmachine). The definition of condition number gives∥∥∥f (x)− f (x)

∥∥∥‖f (x)‖ = (κ(x) + o(1))

‖x − x‖‖x‖

where o(1)→ 0 as εmachine → 0. Combining these gives desiredresult.


Backward Stability of Householder QR

For a QR factorization A = QR computed by Householdertriangularization, the factors Q and R satisfy

QR = A + δA, ‖A‖ = O(εmachine)

Exactly the right QR factorization of a slightly perturbed A

As usual, R is the R computed by the algorithm using floating points

However, Q is a product of exactly unitary reflectors:

Q = Q1Q2 . . . Qn

where Qk is implicitly given by the computed vk (since Q is generallynot formed explicitly)


Backward Stability of Solving Ax = b with QR

Algorithm: Solving Ax = b by QR Factorization

1 Factor QR = A using Householder, represent Q byreflectors

2 y = Q∗b implicitly using reflectors

3 x = R−1y by back substitution

Step 1 is backward stable (from previous slide)

Step 2 can be shown to be backward stable:(Q + δQ

)y = b, ‖δQ‖ = O(εmachine)

Step 3 is backward stable (will be shown later):(R + δR

)x = y ,

‖δR‖∥∥∥R∥∥∥ = O(εmachine)



Put the three steps together to show backward stability of thealgorithm:

(A + ∆A) x = b,‖∆A‖‖A‖ = O(εmachine)

Proof. Steps 2 and 3 give(Q + δQ

) (R + δR

)x =

[QR + (δQ)R + Q(δR) + (δQ)(δR)

]x

Step 1 (backward stability of Householder) gives

b =[A + δA + Q(δR) + (δQ)(δR)

]x =

= (A + ∆A) x



δA is small compared to A, therefore∥∥∥R∥∥∥‖A‖ ≤

∥∥∥Q∗∥∥∥ ‖A + δA‖‖A‖ = O(1)

Now show that each term in ∆A is small:∥∥∥(δQ) R∥∥∥

‖A‖ ≤ ‖δQ‖

∥∥∥R∥∥∥‖A‖ = O(εmachine)∥∥∥Q(δR)

∥∥∥‖A‖ ≤

∥∥∥Q∥∥∥ ‖δR‖∥∥∥R∥∥∥∥∥∥R∥∥∥‖A‖ = O(εmachine)

‖(δQ)(δR)‖‖A‖ ≤ ‖δQ‖ ‖δR‖‖A‖ = O(ε2

machine)



Add the terms to show that A is small:

‖∆A‖‖A‖ ≤

‖δA‖‖A‖ +

∥∥∥(δQ)R∥∥∥

‖A‖ +

∥∥∥Q(δR)∥∥∥

‖A‖ +‖(δQ)(δR)‖‖A‖

= O(εmachine)

Since the algorithm is backward stable, it is also accurate:

‖x − x‖‖x‖ = O (κ(A)εmachine)


Backward Stability of Back Substitution

Solve Rx = b using back substitution: r11 . . . r1m

. . ....

rmm

x1

...xm

=

b1...

bm

xm = bm/rmm

xm−1 = (bm−1 − xmrm−1,m)/rm−1,m−1

xm−2 = (bm−2 − xm−1rm−2,m−1 − xmrm−2,m)/rm−2,m−2

...

xj =

(bj −

m

∑k=j+1

xk rjk

)/rjj


Backward Stability of Back Substitution

Back substitution is backward stable:

(R + δR) x = b,‖δR‖‖R‖ = O(εmachine)

Furthermore, each component of δR satisfies

|δrij ||rij |

≤ mεmachine + O(ε2machine)

Show in full detail for m = 1, 2, 3 as well as general m


Proof for Back Substitution (m = 1)

For m = 1, the algorithm is simply one floating point division. Usethe floating points axiom to get

x1 = b1 � r11 =b1

r11(1 + ε1) =

b1

r11 (1 + ε′1)

where |ε1| ≤ εmachine and |ε′1| ≤ εmachine + O(ε2machine)

Therefore, we solved a perturbed problem exactly:

(r11 + δr11)x1 = b1 with|δr11||r11|

≤ εmachine + O(ε2machine)



For m = 2, we first solve for x2 as before. Next we compute x1:

x1 = (b1 (x2 ⊗ r12))� r11 =(b1 − x2r12 (1 + ε2)) (1 + ε3)

r11(1 + ε4)

=b1 − x2r12 (1 + ε2)r11 (1 + ε′3) (1 + ε′4)

=b1 − x2r12 (1 + ε2)

r11 (1 + 2ε5)

where

|ε2|, |ε3|, |ε4| ≤ εmachine and |ε′3|, |ε′4|, |ε′5| ≤ εmachine + O(ε2machine)

Again this is an exact solution to (R + δR)x = b with[ |δr11||r11|

|δr12||r12||δr22||r22|

]=[

2|ε5| |ε2||ε1|

]≤[

2 11

]εmachine +O(ε2

machine)



For m = 3, compute x3 and x2 as before. Then compute

x1 = [(b1 (x2 ⊗ r12)) (x3 ⊗ r13)]� r11

=[(b1 − x2r12 (1 + ε4)) (1 + ε6)− x3r13 (1 + ε5)] (1 + ε7)

r11 (1 + ε′8)

=b1 − x2r12 (1 + ε4)− x3r13 (1 + ε5) (1 + ε′6)

r11 (1 + ε′6) (1 + ε′7) (1 + ε′8)

That is, (R + δR)x = b with|δr11||r11|

|δr12||r12|

|δr13||r13|

|δr22||r22|

|δr23||r23||δr33||r33|

≤ 3 1 2

2 11

εmachine + O(ε2machine)


Proof for Back Substitution (general m)

Similar analysis for general m gives the pattern (shown for m = 5):

|δR ||R | ≤ W εmachine + O(ε2

machine)

where W =0 1 1 1 1

0 1 1 10 1 1

0 10

︸︷︷︸

⊗

+

1

11

11

︸︷︷︸

�

+

4 0 1 2 3

3 0 1 22 0 1

1 00

︸︷︷︸

=

5 1 2 3 4

4 1 2 33 1 2

2 11


Figure: Alston S. Householder(1904-1993), Americanmathematician. Importantcontributions to mathematicalbiology and mainly to numericallinear algebra. His well knownbook ”The Theory of Matrices inNumerical Analysis” has a greatimpact on development ofnumerical analysis and computerscience.

Figure: James Hardy Wilkinson(1919-1986), Englishmathematician. Contribution tonumerical analysis, numericallinear algebra and computerscience. He received many awardsfor his outstanding work. He waselected a Fellow of the RoyalSociety in 1969.


Documents

Computing Errors Radu Tr^ mbit˘a˘stradu/nlalgslides/lec4b_condstab.pdf · Radu Tr^ mbit˘a˘s ("Babe˘s-Bolyai" University) Conditioning and Stability March 18, 2009 8 / 24. Stability