Embed Size (px)
Citation preview
– Matrix Review– Solving Small Number of
Equations– Gauss Elimination– Gauss-Jordan Elimination– Non-linear Systems
Part 4a: NUMERICAL LINEAR ALGEBRA
Matrix Notation:
nxmnmn
m
aa
a
aaa
A
....
....
..
..
1
21
11211 row
column
mx1 a column vector1xn a row vectorm=n symmetric matrix
Addition/subtraction of two matrices:
BAC ijijij bac (add/subtract corresponding terms)
both A and B must have same sizes.
Multiplication of matrices:
BAC
n
kkjikij bac
1
nxlmxlnxm CBA > First matrix must have the same number of columns as the number of rows in the second matrix.
Division of matrices:
11 BBBBIinverse of B: exist only if matrix A is square and non-singular.
1/ BABAC İf B-1 exist, division is same as multiplicaton
EX: Calculate [X][Y] such that:
40
68
13
X
27
95Y
333231
232221
312111
aaa
aaa
aaa
A
Transpose:
333231
232221
312111
aaa
aaa
aaa
A
332331
322221
312111
aaa
aaa
aaa
A T
Trace:
333231
232221
312111
aaa
aaa
aaa
A 332211 aaaAtr
Augmentation:
3
2
1
333231
232221
312111
b
b
b
aaa
aaa
aaa
C
Determinant for 2x2 matrix:
2221
1211
aa
aaA 21122211
2221
1211 aaaaaa
aaA
Determinant for 3x3 matrix:
333231
232221
131211
aaa
aaa
aaa
A
3231
222113
3331
232112
3332
232211
333231
232221
131211
aa
aaa
aa
aaa
aa
aaa
aaa
aaa
aaa
A
Consider a system of n linear equations with n unknowns:
...
11212111 ... bxaxaxa nn
22222121 ... bxaxaxa nn
nnnnnn bxaxaxa ...2211
:'
:'
:'
sb
sx
sa coefficientsunknownsconstants
For small n’s (say, n <= 3) this can be done by hand, but for large n’s we need computer power ( numerical techniques).
In engineering, multi-component systems require solution of a set of mathematical equations that need to be solved simultaneously.
need to define pressure at every point (unknowns), on the surface, and solve the underlying physical equation simultenously.
Linear Algebraic Equations in Matrix form:
...
11212111 ... bxaxaxa nn
22222121 ... bxaxaxa nn
nnnnnn bxaxaxa ...2211
nnn
n
aa
a
aaa
A
....
....
..
..
1
21
11211
nb
b
b
b..2
1
nx
x
x
x..2
1
bAx
Symbolic form of the linear system
Matrix form of the linear system
Gauss Elimination
...
11212111 ... bxaxaxa nn
22222121 ... bxaxaxa nn
nnnnnn bxaxaxa ...2211
bAx
We want to solve
One way to solve it
bAAxA 11 bAx 1 if the inverse of A exist!
Even if an inverse of A exist, this method is computationally not efficient.
We have more efficient methods to solve the linear system.These solutions do not require operations involving calculating
the inverse of A.
Solving small number of linear equations:
1212111 bxaxa
2222121 bxaxa
1-Graphical Method:
12
11
12
112 a
bx
a
ax
22
11
22
212 a
bx
a
ax
EX: Use graphical method to solve:
1823 21 xx22 21 xx
2x
1x
9
0
3
4
1
intersection of two lines gives the solution
92
312 xx1
2
112 xx
For three equations (unknowns: x1 , x2 , x3 ), each equation represents a plane in a 3-D space. Solution is where three planes intersect.
For n>3, graphical method fails. It is useful for visualizing the behavior of linear systems.
2x
1x
No solution
2x
1x
Infinite solutions
1x
Ill-conditioned system
-0.5x 1+x 2
=1
-0.5x 1+x 2
=0.5
-0.5x 1+1x 2
=1
-x 1+2x 2
=2-0.46x 1
+x 2=1.1
-0.5x 1+x 2
=1
2x
2-Cramer’s Rule:
2x
1x
2x
1x 1x
-0.5x 1+x 2
=1
-0.5x 1+x 2
=0.5
-0.5x 1+1x 2
=1
-x 1+2x 2
=2-0.46x 1
+x 2=1.1
-0.5x 1+x 2
=1
1823 21 xx
22 21 xx
21
23A 82)1(23
21
23
A
For the previous example, calculate determinant of coefficients.
For special cases:
015.0
15.0
A 021
15.0
A 04.015.0
146.0
A
Singular systems have zero determinants; ill-conditioned systems have near-zero determinants.
In Cramer’s rule, we replace the column of the coefficients of the unknown by the column of the constants, and divide by the determinant, i.e. (for n=3),
A
aab
aab
aab
x 33323
23222
13121
1 A
aba
aba
aba
x 33331
23221
13111
2 A
baa
baa
baa
x 33231
22221
11211
3
EX: Use Cramer’s rule to solve:
01.052.03.0 321 xxx
67.09.15.0 321 xxx
44.05.03.01.0 321 xxx
For n>3, Cramer’s rule also becomes impractical and time-consuming for calculation of the determinants.
3-Elimination of Unknowns:
1212111 bxaxa
2222121 bxaxa
1212122111121 baxaaxaa
2112221112111 baxaaxaa
subtract first eqn. from the second one to eliminate one of the unknowns to get:
1212112122122211 babaxaaxaa
Then, solve for the second unknown
21122211
1212112 aaaa
babax
For the first unknown , use either of the original eqn:
21122211
2121221 aaaa
babax
EX: Use the elimination of unknowns to solve: 1823 21 xx 22 21 xx
Naive Gauss Elimination:
...
11212111 ... bxaxaxa nn
22222121 ... bxaxaxa nn
nnnnnn bxaxaxa ...2211
We apply the same method of elimination of unknowns to a system of n equations.> Eliminate unknowns until reaching a single unknown.> Back-substitute into the original equation to find other unknowns.
To eliminate (x1 ), multiply the first equation by a21/a11 :
111
211
11
21212
11
21121 ... b
a
axa
a
axa
a
axa nn
Elimination:
Divison by a11 is also called “normalization”
subtract this equation from the second one:
111
1221
11
122212
11
2122 ... b
a
abxa
a
aaxa
a
aa nnn
x1 eliminated
or'2
'22
'22 ... bxaxa nn
Same procedure is applied for the third equation, i.e., multiply the first equation by a31/a11 and subtract from the third equation. This will eliminate (x1) from the third equation.
...
11313212111 ... bxaxaxaxa nn '2
'23
'232
'22 ... bxaxaxa nn
'3
'33
'332
'32 ... bxaxaxa nn
'2
'3
'32
'2 ... bxaxaxa nnnnn
x1 eliminated from all equations except the first one.
pivotelement
pivotequation
Elimination of the second unknown (x2):
x2 eliminated from all equations except the first and second one.
...
11313212111 ... bxaxaxaxa nn '2
'23
'232
'22 ... bxaxaxa nn
''3
''33
''33 ... bxaxa nn
''2
''3
''3 ... bxaxa nnnn
The process can be repeated for all other unknowns to get an upper-triangular system:
...
11313212111 ... bxaxaxaxa nn '2
'23
'232
'22 ... bxaxaxa nn
''3
''33
''33 ... bxaxa nn
)1()1( nnn
nnn bxa
indicates the number of operations performed until the upper triangular system forms.
Back-substitution:
Solve for (xn) simply by:
)1(
)1(
n
nn
n
nna
bx
Value of xn can be back-substituted into the upper equation in upper triangular system to solve for (xn-1). The procedure is repeated for all remaining unknowns.
)1(
1
)1()1(
i
n
ijj
iij
ii
i
iia
xab
x1,...,2,1 nnifor
EX: Use Gauss elimination to solve (carrying 6 S.D.’s):
85.72.01.03 321 xxx
3.193.071.0 321 xxx
4.71102.03.0 321 xxx
Operation Counting:The time of execution in the elimination/back-substitution
processes depends on the total number of addition/subtraction and multiplication/division operations (combined called as floating point operations- FLOPs).
Total number of FLOPs for the solution of a system size n using Gauss elimination can be counted from the algorithm.
n Elimination Back-substitution Total FLOPS
10 375 55 430
100 338250 5050 343300
1000 3.34E+08 500500 3.34E+08
FLOPs for Gauss Elimination n3/3 + O(n2)
Most of the FLOPS are due to the elimination stage.
Pitfalls of the elimination:The following topics concerns all elimination techniques as well as
Gauss elimination.
Divison by Zero: In naive Gauss elimination, if one of the coefficients is zero, a
division-by-zero occurs during normalization. If the coefficient is nearly zero problems still arise (due to round-
off errors).Pivoting (discussed later) will provide a partial remedy.
Round-off errors:Due to limited amount of S.F. in computer numbers, a round-off
error in the results will always occur.They can be important when large number of operations (>100)
involved.Using more significant figures (double precision) lead to lower
round-off errors.
EX: a) Solve
b) Now solve
c) Compute the determimnants
Ill-conditioned systems:Small changes in the coefficients results in large changes in the
solution (ill-conditioned system). In a different approach a wide range of solutions approximately
satisfy the equations. In these systems, small changes in the coefficients due to round-
off errors lead to large errors in solution. In these systems, numerical approximations lead to larger errors.
102 21 xx
4.1021.1 21 xx
102 21 xx
4.10205.1 21 xx
Near-zero determinants of the system indicate ill-conditoning.
A small change in the coefficient results in very different solutions.
It is difficult to detemine how close to zero a determinant indicates ill-conditioning. This is due to the fact that magnitude of the determinant can be changed by scaling the equation even though the solution does not change.
EX: Compare determinants for
a)
b)
c)
1823 21 xx
22 21 xx
102 21 xx
4.1021.1 21 xx
1002010 21 xx
1042011 21 xx
x10
scale equations such that the maximum coefficient for each row is 1.
EX: Scale the systems of equations in the previous example and recompute the determinants.
Determinant of a triangular matrix can simply be computed as the product of its diagonal elements.
Determinant (after scaling)
System condition
How can we calculate determinant for large systems?Gauss elimination has the extra bonus of calculating the
determinant:
...
11313212111 ... bxaxaxaxa nn '2
'23
'232
'22 ... bxaxaxa nn
''3
''33
''33 ... bxaxa nn
)1(2
)1( nn
nnn bxa
pnnnaaaaD )1(... )1(''
33'2211
p= number of times pivoting applied (discussed later)
Singular systems:This is the worse case of ill-conditioning where two or more
equations are identical. In this case, system loose a degree of freedom which makes
solution impossible a singular system. In large systems, it may not be easy to see that the system is
singular. We can detect singularity by calculating the determinant.
In Gauss elimination, this means a “zero” is encountered in diagonal elements.
If a zero is encountered during elimination terminate the calculation.
If D=0 Singular system
So far we mentioned the possible problems in naive Gauss elimination; here we discuss some solutions of these problems.
Improvements on the elimination:
Pivoting: If pivot=0, a divison-by-zero occurs during normalization. As remedy (partial pivoting),
Find the element below the pivot column whose absolute value is the largest. Switch the pivot row with row of the the largest element.
If both rows and columns are searched for the largest element complete pivoting (not a common practice)
Pivoting is in general adventegous to reduce round-off errors during the elimination even if the pivot element is not zero.
By these adventages, pivoting is routinely applied in Gauss elimination.
EX: Use Gauss elimination to solve the system
a) Naive elimination (multiply the first eqn by 1/0.0003 and subtract to yield)
Back-substition:
b) Now apply pivoting
Elimination
back-substition:
0001.20000.30003.0 21 xx0000.10000.10000.1 21 xx
66669999 2 x
0001.20000.30003.0 21 xx..6666.02 x
0003.0
)3/2(30001.21
x
S.F. x1 x2
3 0.667 -3.33
4 0.6667 0.0000
5 0.66667 0.30000
6 0.666667 0.330000
7 0.6666667 0.3330000
0
0001.20000.30003.0 21 xx
0000.10000.10000.1 21 xx
Exact solutionx1=0.3333.. ; x2=0.6666..
0000.10000.10000.1 21 xx
9998.19997.2 2 x
..6666.02 x
..3333.01 x
Note that the system is not ill-conditioned.
EX: Use Gauss elimination to solve the system (using 3 significant figures):
a) Elimination with pivoting (without scaling):
Scaling: In engineering applications, equations with widely differents
units may have to be solved simultenously, e.g.,
This may result in large variations in the coefficients and constants.
This results in large round-off errors during elimination.
000,100000,1002 21 xx221 xx
note the scale problem!
000,100000,1002 21 xx
000,50000,50 2 x00.12 x
00.01 x (100% error)elimination b-substitution
Exact solution:x1=1.00002 x2=0.99998
..
..
)(
)(
....
....
..1
..1
kilovoltsx
millivoltsx
nm
m
b) Repeat with scaling (i.e. divide each raw by the largest coefficient):
c) Now, apply scaling for just pivoting (keep original eqn’s):
100002.0 21 xx
221 xx 100002.0 21 xx221 xx
scaling pivoting
221 xx
00.12 x12 x11 xelimination b-substitution
pivoting221 xx
000,100000,1002 21 xx elimination221 xx
000,100000,100 2 x
b-substitution 12 x11 x
We used scaling just to determine whether pivoting was necessary.
Eqn’s did not require scaling to arrive the correct result.
Since scaling introduces extra round-off errors, we apply scaling only as a criterion for pivoting.
If determinant is not needed (which is the case most of the time), the strategy is scale for just pivoting, but use the original coefficients for the elimination and back-substitution.
Scaling result in the correct result (for 3 S.F.)
Gauss-Jordan Elimination: In this method, unknowns are eliminated from all the raws; not just
from the subsequent ones. So, instead of an upper triangular matrix, one gets a diagonal matrix.
In addition, all raws are normalized by dividining them to the pivot element. So, the final matrix is an identity matrix.
...
11313212111 ... bxaxaxaxa nn '2
'23
'232
'22 ... bxaxaxa nn
''3
''33
''33 ... bxaxa nn
)1()1( nnn
nnn bxa
...
)(1nb)(
2nb)(
3nb
)(nnb
1x
2x
3x
nx
...
Gauss Elimination Gauss-Jordan Elimination
It is not necessary to apply back-substitution! shows total number of operations applied
EX: Gauss-Jordan technique to solve the system of equations (6 S.D.):
First, form an augmented system:
Normalize the first raw:
85.72.01.03 321 xxx
3.193.071.0 321 xxx
4.71102.03.0 321 xxx
4.71102.03.0
3.193.071.0
85.72.01.03
4.71102.03.0
3.193.071.0
61667.2066667.00333333.01
Eliminate x1 term from the second and third raws:
Normalize the second raw
Eliminate x2 term from the first and third raws:
6150.700200.10190000.00
5617.19293333.000333.70
61667.2066667.00333333.01
6150.700200.10190000.00
79320.20418848.010
61667.2066667.00333333.01
0843.7001200.1000
79320.20418848.010
52356.20680629.001
Normalize the third raw:
Finally, eliminate x3 term from the first and second raws:
00003.7100
79320.20418848.010
52356.20680629.001
00003.7100
50001.2010
00000.3001 00000.31 x50001.22 x
00003.73 x
The same pivoting strategy as in Gauss elimination can be applied.Number of processes (FLOPs) is slightly larger than Gauss
elimination.
FLOPS for Gauss-Jordan n3/2 + O(n2) about 50% more operations in G-J
Working with Complex Variables: In some cases, we may have to face with complex variables in the
system of equations.
where
If the language you are using supports complex variables (such as Fortran, Matlab), then you don’t need to do anything.
Alternatively, the complex system can be rewritten by substituting the real and imaginary parts, and equating real and imaginary parts separately. Thus,
WZC
BiAC YiXZ ViUW
UYBXA
VYAXB
V
U
Y
X
AB
BAinstead of an nxn complex system, we have a 2nx2n real system
Nonlinear System of Equations:
0),...,,( 211 nxxxf
Consider a system of n non-linear equations with n unknowns:
In the previous chapter, we developed a solution method for a system of n=2 (multi-equation Newton-Raphson method).
In order to evaluate the problem as a linear system, we can expand the equations using first-order Taylor series expansion, i.e., for k-th row
0),...,,( 212 nxxxf
...0),...,,( 21 nn xxxf
n
ikinin
ikii
ikiiikik x
fxx
x
fxx
x
fxxff
,
,1,2
,,21,2
1
,,11,1,1, )(...)()(
set to zero unknowns
n
ikin
iki
ikiik
n
ikin
iki
iki x
fx
x
fx
x
fxf
x
fx
x
fx
x
fx
,
,2
,,2
1
,,1,
,1,
2
,1,2
1
,1,1 ...)(...)()(
re-arranging the terms
Define matrices
n
ininin
n
iii
n
iii
x
f
x
f
x
f
x
f
x
f
x
fx
f
x
f
x
f
Z
,
2
,
1
,
,2
2
,2
1
,2
,1
2
,1
1
,1
...
............
...
...
in
i
i
i
x
x
x
X
,
,2
,1
...
1,
1,2
1,1
1 ...
in
i
i
i
x
x
x
X
in
i
i
i
f
f
f
F
,
,2
,1
...
Then,
iii XZFXZ 1
This equation is in the form of Ax=b, and can be solved using Gauss elimination
Note that the solution is reached iteratively.
