View
6
Download
0
Category
Preview:
Citation preview
Conditioning and StabilityComputing Errors
Radu Trımbitas
”Babes-Bolyai” University
March 18, 2009
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 1 / 24
Conditioning
Absolute Condition Number of a differentiable problem f at x :
κ = supδx
‖δf ‖‖δx‖ = ‖J(x)‖
where the Jacobian J(x) = [Jij ] = [δfi /δxj ], and the matrix norm isinduced by the norms on δf and δx
Relative Condition Number
κ = supδx
‖δf ‖‖f (x)‖‖δx‖‖x‖
=‖J(x)‖
‖f (x)‖ / ‖x‖
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 2 / 24
Conditioning - Examples
Example: The function f (x) = αx
Absolute condition number κ = ‖J‖ = α
Relative condition number κ = ‖J‖‖f (x)‖/‖x‖ = α
αx/x = 1
Example: The function f (x) =√
x
Absolute condition number κ = ‖J‖ = 12√
x
Relative condition number κ = ‖J‖‖f (x)‖/‖x‖ = 1/(2
√x)√
x/x= 1
2
Example: The function f (x) = x1 − x2 (with ∞-norms)
Absolute condition number κ = ‖J‖ =∥∥∥(−1, 1)T
∥∥∥ = 2
Relative condition number κ = ‖J‖‖f (x)‖/‖x‖ = 2
|x1−x2|max{|x1|,|x2|}Ill-conditioned when x1 ≈ x2 (cancellation)
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 3 / 24
Condition of Matrix-Vector Product
Consider f (x) = Ax , with A ∈ Cm×n
κ =‖J‖
‖f (x)‖ / ‖x‖ = ‖A‖ ‖x‖‖Ax‖
For A square and nonsingular, use ‖x‖ / ‖Ax‖ ≤∥∥A−1
∥∥:
κ ≤ ‖A‖∥∥A−1
∥∥(equality achieved for the last right singular vector x = vm)
Also the condition number for f (b) = A−1b (solution of linearsystem)
Condition number of matrix A:
κ(A) = ‖A‖∥∥A−1
∥∥ for 2-norm κ(A) =σ1
σm
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 4 / 24
Condition of System of Equations
For fixed b, consider f (A) = A−1b
Perturb A by δA and find perturbation δx :
(A + δA)(x + δx) = b
Use Ax = b and assume (δA)(δx) ≈ 0:
(δA)x + A(δx) = 0 =⇒ δx = −A−1(δA)x
Condition number of problem f
κ =‖δx‖‖x‖
/‖δA‖‖A‖ ≤
∥∥A−1∥∥ ‖δA‖ ‖x‖‖x‖
/‖δA‖‖A‖ =
∥∥A−1∥∥ ‖A‖ = κ(A)
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 5 / 24
Accuracy
Consider an algorithm f for a problem f
A computation f (x) has absolute error∥∥∥f (x)− f (x)
∥∥∥ and relativeerror ∥∥∥f (x)− f (x)
∥∥∥‖f (x)‖
The algorithm is accurate if (for all x)∥∥∥f (x)− f (x)∥∥∥
‖f (x)‖ = O(εmachine)
where O(εmachine) is “on the order of εmachine” (more next slide)
Constant in O(εmachine) is likely to be large in many problems, sincebecause of rounding we are not even using the correct x
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 6 / 24
More on O(εmachine)
The notation ϕ(t) = O(ψ(t)) means there is a constant C such that,for t close to a limit (often 0 or ∞), |ϕ(t)| ≤ Cψ(t)Example: sin2 t = O(t2) as t → 0 means | sin2 t| ≤ Ct2 for some C
If ϕ depends on additional variables, the notation
ϕ(s, t) = O(ψ(t)) uniformly in s
means there is a constant C such that |ϕ(s, t)| ≤ Cψ(t) for any s
Example: (sin2 t)(sin2 s) = O(t2) uniformly as t → 0, but not ifsin2 s is replaced by s2
In bounds such as ‖x − x‖ ≤ Cκ(A)εmachine ‖x‖, C does not dependon A or b, but it might depend on the dimension m
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 7 / 24
Stability
An algorithm f for a problem f is stable if (for all x)∥∥∥f (x)− f (x)∥∥∥
‖f (x)‖ = O(εmachine)
for some x with‖x − x‖‖x‖ = O(εmachine)
“Nearly the right answer to nearly the right question”
An algorithm f for a problem f is backward stable if (for all x)
f (x) = f (x) for some x with‖x − x‖‖x‖ = O(εmachine)
“Exactly the right answer to nearly the right question”
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 8 / 24
Stability
An algorithm f for a problem f is stable if (for all x)∥∥∥f (x)− f (x)∥∥∥
‖f (x)‖ = O(εmachine)
for some x with‖x − x‖‖x‖ = O(εmachine)
“Nearly the right answer to nearly the right question”
An algorithm f for a problem f is backward stable if (for all x)
f (x) = f (x) for some x with‖x − x‖‖x‖ = O(εmachine)
“Exactly the right answer to nearly the right question”
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 8 / 24
Stability
An algorithm f for a problem f is stable if (for all x)∥∥∥f (x)− f (x)∥∥∥
‖f (x)‖ = O(εmachine)
for some x with‖x − x‖‖x‖ = O(εmachine)
“Nearly the right answer to nearly the right question”
An algorithm f for a problem f is backward stable if (for all x)
f (x) = f (x) for some x with‖x − x‖‖x‖ = O(εmachine)
“Exactly the right answer to nearly the right question”
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 8 / 24
Stability
An algorithm f for a problem f is stable if (for all x)∥∥∥f (x)− f (x)∥∥∥
‖f (x)‖ = O(εmachine)
for some x with‖x − x‖‖x‖ = O(εmachine)
“Nearly the right answer to nearly the right question”
An algorithm f for a problem f is backward stable if (for all x)
f (x) = f (x) for some x with‖x − x‖‖x‖ = O(εmachine)
“Exactly the right answer to nearly the right question”
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 8 / 24
Stability of Floating Point Arithmetic
The two floating point axioms imply backward stability for theoperations ~
(1) For all x ∈ R, there exists ε with |ε| ≤ εmachine such thatfl(x) = x(1 + ε)
(2) For all floating point x , y , there exists ε with |ε| ≤ εmachine such thatx ~ y = (x ∗ y)(1 + ε)
Example: Subtraction f (x1, x2) = x1 − x2 with floating pointalgorithm
f (x1, x2) = fl(x1) fl(x2)
(1) implies
fl(x1) = x1(1 + ε1), fl(x2) = x2(1 + ε2)
for some |ε1|, |ε2| ≤ εmachine
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 9 / 24
Stability of Floating Point Arithmetic
(example continued)
(2) implies
fl(x1) fl(x2) = (fl(x1)− fl(x2))(1 + ε3)
for some |ε3| ≤ εmachine
Combine:
fl(x1) fl(x2) = (x1(1 + ε1)− x2(1 + ε2))(1 + ε3)= x1(1 + ε1)(1 + ε3)− x2(1 + ε2)(1 + ε3)= x1(1 + ε4)− x2(1 + ε5)
for some |ε4|, |ε4| ≤ 2εmachine + O(ε2machine)
Therefore, fl(x1)− fl(x2) = x1 − x2
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 10 / 24
Stability of Floating Point Arithmetic
Example: Inner product f (x , y) = x∗y computed with ⊗ and ⊕ isbackward stable (more later)
Example: Outer product f (x , y) = xy ∗ computed with ⊗ is notbackward stable (unlikely that f is rank-1)
Example: f (x) = x + 1 computed by f (x) = fl(x)⊕ 1 is notbackward stable (consider x ≈ 0)
Example: f (x , y) = x + y computed by f (x , y) = fl(x)⊕ fl(y) isbackward stable
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 11 / 24
Stability of Floating Point Arithmetic
Example: Inner product f (x , y) = x∗y computed with ⊗ and ⊕ isbackward stable (more later)
Example: Outer product f (x , y) = xy ∗ computed with ⊗ is notbackward stable (unlikely that f is rank-1)
Example: f (x) = x + 1 computed by f (x) = fl(x)⊕ 1 is notbackward stable (consider x ≈ 0)
Example: f (x , y) = x + y computed by f (x , y) = fl(x)⊕ fl(y) isbackward stable
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 11 / 24
Stability of Floating Point Arithmetic
Example: Inner product f (x , y) = x∗y computed with ⊗ and ⊕ isbackward stable (more later)
Example: Outer product f (x , y) = xy ∗ computed with ⊗ is notbackward stable (unlikely that f is rank-1)
Example: f (x) = x + 1 computed by f (x) = fl(x)⊕ 1 is notbackward stable (consider x ≈ 0)
Example: f (x , y) = x + y computed by f (x , y) = fl(x)⊕ fl(y) isbackward stable
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 11 / 24
Stability of Floating Point Arithmetic
Example: Inner product f (x , y) = x∗y computed with ⊗ and ⊕ isbackward stable (more later)
Example: Outer product f (x , y) = xy ∗ computed with ⊗ is notbackward stable (unlikely that f is rank-1)
Example: f (x) = x + 1 computed by f (x) = fl(x)⊕ 1 is notbackward stable (consider x ≈ 0)
Example: f (x , y) = x + y computed by f (x , y) = fl(x)⊕ fl(y) isbackward stable
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 11 / 24
Theorem (Accuracy of a Backward Stable Algorithm)
If a backward stable algorithm is used to solve a problem f with conditionnumber κ, the relative errors satisfy∥∥∥f (x)− f (x)
∥∥∥‖f (x)‖ = O(κ(x)εmachine)
Proof.
Backward stability means f (x) = f (x), for some x such that‖x−x‖‖x‖ = O(εmachine). The definition of condition number gives∥∥∥f (x)− f (x)
∥∥∥‖f (x)‖ = (κ(x) + o(1))
‖x − x‖‖x‖
where o(1)→ 0 as εmachine → 0. Combining these gives desiredresult.
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 12 / 24
Backward Stability of Householder QR
For a QR factorization A = QR computed by Householdertriangularization, the factors Q and R satisfy
QR = A + δA, ‖A‖ = O(εmachine)
Exactly the right QR factorization of a slightly perturbed A
As usual, R is the R computed by the algorithm using floating points
However, Q is a product of exactly unitary reflectors:
Q = Q1Q2 . . . Qn
where Qk is implicitly given by the computed vk (since Q is generallynot formed explicitly)
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 13 / 24
Backward Stability of Solving Ax = b with QR
Algorithm: Solving Ax = b by QR Factorization
1 Factor QR = A using Householder, represent Q byreflectors
2 y = Q∗b implicitly using reflectors
3 x = R−1y by back substitution
Step 1 is backward stable (from previous slide)
Step 2 can be shown to be backward stable:(Q + δQ
)y = b, ‖δQ‖ = O(εmachine)
Step 3 is backward stable (will be shown later):(R + δR
)x = y ,
‖δR‖∥∥∥R∥∥∥ = O(εmachine)
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 14 / 24
Backward Stability of Solving Ax = b with QR
Put the three steps together to show backward stability of thealgorithm:
(A + ∆A) x = b,‖∆A‖‖A‖ = O(εmachine)
Proof. Steps 2 and 3 give(Q + δQ
) (R + δR
)x =
[QR + (δQ)R + Q(δR) + (δQ)(δR)
]x
Step 1 (backward stability of Householder) gives
b =[A + δA + Q(δR) + (δQ)(δR)
]x =
= (A + ∆A) x
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 15 / 24
Backward Stability of Solving Ax = b with QR
δA is small compared to A, therefore∥∥∥R∥∥∥‖A‖ ≤
∥∥∥Q∗∥∥∥ ‖A + δA‖‖A‖ = O(1)
Now show that each term in ∆A is small:∥∥∥(δQ) R∥∥∥
‖A‖ ≤ ‖δQ‖
∥∥∥R∥∥∥‖A‖ = O(εmachine)∥∥∥Q(δR)
∥∥∥‖A‖ ≤
∥∥∥Q∥∥∥ ‖δR‖∥∥∥R∥∥∥∥∥∥R∥∥∥‖A‖ = O(εmachine)
‖(δQ)(δR)‖‖A‖ ≤ ‖δQ‖ ‖δR‖‖A‖ = O(ε2
machine)
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 16 / 24
Backward Stability of Solving Ax = b with QR
Add the terms to show that A is small:
‖∆A‖‖A‖ ≤
‖δA‖‖A‖ +
∥∥∥(δQ)R∥∥∥
‖A‖ +
∥∥∥Q(δR)∥∥∥
‖A‖ +‖(δQ)(δR)‖‖A‖
= O(εmachine)
Since the algorithm is backward stable, it is also accurate:
‖x − x‖‖x‖ = O (κ(A)εmachine)
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 17 / 24
Backward Stability of Back Substitution
Solve Rx = b using back substitution: r11 . . . r1m
. . ....
rmm
x1
...xm
=
b1...
bm
xm = bm/rmm
xm−1 = (bm−1 − xmrm−1,m)/rm−1,m−1
xm−2 = (bm−2 − xm−1rm−2,m−1 − xmrm−2,m)/rm−2,m−2
...
xj =
(bj −
m
∑k=j+1
xk rjk
)/rjj
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 18 / 24
Backward Stability of Back Substitution
Back substitution is backward stable:
(R + δR) x = b,‖δR‖‖R‖ = O(εmachine)
Furthermore, each component of δR satisfies
|δrij ||rij |
≤ mεmachine + O(ε2machine)
Show in full detail for m = 1, 2, 3 as well as general m
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 19 / 24
Proof for Back Substitution (m = 1)
For m = 1, the algorithm is simply one floating point division. Usethe floating points axiom to get
x1 = b1 � r11 =b1
r11(1 + ε1) =
b1
r11 (1 + ε′1)
where |ε1| ≤ εmachine and |ε′1| ≤ εmachine + O(ε2machine)
Therefore, we solved a perturbed problem exactly:
(r11 + δr11)x1 = b1 with|δr11||r11|
≤ εmachine + O(ε2machine)
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 20 / 24
Proof for Back Substitution (m = 2)
For m = 2, we first solve for x2 as before. Next we compute x1:
x1 = (b1 (x2 ⊗ r12))� r11 =(b1 − x2r12 (1 + ε2)) (1 + ε3)
r11(1 + ε4)
=b1 − x2r12 (1 + ε2)r11 (1 + ε′3) (1 + ε′4)
=b1 − x2r12 (1 + ε2)
r11 (1 + 2ε5)
where
|ε2|, |ε3|, |ε4| ≤ εmachine and |ε′3|, |ε′4|, |ε′5| ≤ εmachine + O(ε2machine)
Again this is an exact solution to (R + δR)x = b with[ |δr11||r11|
|δr12||r12||δr22||r22|
]=[
2|ε5| |ε2||ε1|
]≤[
2 11
]εmachine +O(ε2
machine)
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 21 / 24
Proof for Back Substitution (m = 3)
For m = 3, compute x3 and x2 as before. Then compute
x1 = [(b1 (x2 ⊗ r12)) (x3 ⊗ r13)]� r11
=[(b1 − x2r12 (1 + ε4)) (1 + ε6)− x3r13 (1 + ε5)] (1 + ε7)
r11 (1 + ε′8)
=b1 − x2r12 (1 + ε4)− x3r13 (1 + ε5) (1 + ε′6)
r11 (1 + ε′6) (1 + ε′7) (1 + ε′8)
That is, (R + δR)x = b with|δr11||r11|
|δr12||r12|
|δr13||r13|
|δr22||r22|
|δr23||r23||δr33||r33|
≤ 3 1 2
2 11
εmachine + O(ε2machine)
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 22 / 24
Proof for Back Substitution (general m)
Similar analysis for general m gives the pattern (shown for m = 5):
|δR ||R | ≤ W εmachine + O(ε2
machine)
where W =0 1 1 1 1
0 1 1 10 1 1
0 10
︸ ︷︷ ︸
⊗
+
1
11
11
︸ ︷︷ ︸
�
+
4 0 1 2 3
3 0 1 22 0 1
1 00
︸ ︷︷ ︸
=
5 1 2 3 4
4 1 2 33 1 2
2 11
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 23 / 24
Figure: Alston S. Householder(1904-1993), Americanmathematician. Importantcontributions to mathematicalbiology and mainly to numericallinear algebra. His well knownbook ”The Theory of Matrices inNumerical Analysis” has a greatimpact on development ofnumerical analysis and computerscience.
Figure: James Hardy Wilkinson(1919-1986), Englishmathematician. Contribution tonumerical analysis, numericallinear algebra and computerscience. He received many awardsfor his outstanding work. He waselected a Fellow of the RoyalSociety in 1969.
Radu Trımbitas (”Babes-Bolyai” University) Conditioning and Stability March 18, 2009 24 / 24
Recommended