Embed Size (px)
Citation preview
Linear SystemsLinear Systems
Numerical MethodsNumerical Methods
Linear equationsLinear equations• NN unknowns, unknowns, MM equations equations
wherewhere
coefficient coefficient matrixmatrix
3
Determinants and Cramer’s RuleDeterminants and Cramer’s Rule
[A] : [A] : coefficient matrixcoefficient matrix bxA
3
2
1
3
2
1
333231
232221
131211
b
b
b
x
x
x
aaa
aaa
aaa
333231
232221
131211
aaa
aaa
aaa
A
D
aab
aab
aab
x 33323
23222
13121
1 D
aba
aba
aba
x 33331
23221
13111
2 D
baa
baa
baa
x 33231
22221
11211
3
D : Determinant of A matrix
Solving methodsSolving methods
• Direct methodsDirect methods– Gauss eliminationGauss elimination
– Gauss-Jordan eliminationGauss-Jordan elimination
– LU decompositionLU decomposition
– Singular value decompositionSingular value decomposition
– ……
• Iterative methodsIterative methods– Jacobi iterationJacobi iteration
– Gauss-Seidel iterationGauss-Seidel iteration
– ……
Linear SystemsLinear Systems
• Solve Ax=b, where A is an Solve Ax=b, where A is an nnnn matrix and matrix andb is an b is an nn1 column vector1 column vector
• Can also talk about non-square systems Can also talk about non-square systems wherewhereA is A is mmnn, b is , b is mm1, and x is 1, and x is nn11– OverdeterminedOverdetermined if if mm>>nn::
“more equations than unknowns”“more equations than unknowns”
– UnderdeterminedUnderdetermined if if nn>>mm::“more unknowns than equations”“more unknowns than equations”Can look for best solution using least squaresCan look for best solution using least squares
6
Gauss EliminationGauss Elimination
• Solve Solve AxAx = = bb
• Consists of two phases:Consists of two phases:–Forward eliminationForward elimination–Back substitutionBack substitution
• Forward EliminationForward Eliminationreduces reduces AxAx = = bb to an upper to an upper triangular system triangular system TxTx = = b’b’
• Back substitutionBack substitution can then can then solve solve TxTx = = b’b’ for for xx
''3
''33
'2
'23
'22
1131211
3333231
2232221
1131211
00
0
ba
baa
baaa
baaa
baaa
baaa
ForwardElimination
BackSubstitution
11
21231311
'22
3'23
'2
2''33
''3
3
a
xaxabx
a
xabx
a
bx
Gauss EliminationGauss Elimination
• Fundamental operations:Fundamental operations:1.1. Replace one equation with linear combinationReplace one equation with linear combination
of other equationsof other equations
2.2. Interchange two equationsInterchange two equations
3.3. Re-label two variablesRe-label two variables
• Combine to reduce to trivial systemCombine to reduce to trivial system
• Simplest variant only uses #1 operations,Simplest variant only uses #1 operations,but get better stability by addingbut get better stability by adding#2 (partial pivoting) or #2 and #3 (full #2 (partial pivoting) or #2 and #3 (full pivoting)pivoting)
8
Gaussian EliminationGaussian EliminationForward Elimination
x1 - x2 + x3 = 6 3x1 + 4x2 + 2x3 = 9 2x1 + x2 + x3 = 7
x1 - x2 + x3 = 6 0 +7x2 - x3 = -9 0 + 3x2 - x3 = -5
x1 - x2 + x3 = 6 0 7x2 - x3 = -9 0 0 -(4/7)x3=-(8/7)
-(3/1)
Solve using BACK SUBSTITUTION: x3 = 2 x2=-1 x1 =3
-(2/1) -(3/7)
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0 0
9
Back SubstitutionBack Substitution
1x0 +1x1 –1x2 +4x3 8=
– 2x1 –3x2 +1x3 5=
2x2 – 3x3 0=
2x3 4=x3 = 2
10
1x0 +1x1 –1x2 0=
– 2x1 –3x2 3=
2x2 6=
Back SubstitutionBack Substitution
x2 = 3
11
1x0 +1x1 3=
– 2x1 12=
Back SubstitutionBack Substitution
x1 = –6
1x0 9=
Back SubstitutionBack Substitution
x0 = 9
13
for i n down to 1 do
/* calculate xi */
x [ i ] b [ i ] / a [ i, i ]
/* substitute in the equations above */for j 1 to i-1 do
b [ j ] b [ j ] x [ i ] × a [ j, i ]endfor
endfor
Back Substitution(* Pseudocode *)
Time Complexity? O(n2)
14
Forward EliminationForward Elimination
4x0 +6x1 +2x2 – 2x3 = 8
2x0 +5x2 – 2x3 = 4
–4x0 – 3x1 – 5x2 +4x3 = 1
8x0 +18x1 – 2x2 +3x3 = 40
-(2/4)
MULTIPLIERS
-(-4/4)
-(8/4)
15
4x0 +6x1 +2x2 – 2x3 = 8
+4x2 – 1x3 = 0
+3x1 – 3x2 +2x3 = 9
+6x1 – 6x2 +7x3 = 24
– 3x1
-(3/-3)
MULTIPLIERS
Forward Elimination
-(6/-3)
16
4x0 +6x1 +2x2 – 2x3 = 8
+4x2 – 1x3 = 0
1x2 +1x3 = 9
2x2 +5x3 = 24
– 3x1
??
MULTIPLIER
Forward Elimination
17
4x0 +6x1 +2x2 – 2x3 = 8
+4x2 – 1x3 = 0
1x2 +1x3 = 9
3x3 = 6
– 3x1
Forward EliminationForward Elimination
18
Gauss-Jordan EliminationGauss-Jordan Elimination
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0 0
0
a11
x22
0
x33
0
0
x44
0
0
0
x55
0
0
0
0
x66
x77
x88
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
b11
b22
b33
b44
b55
b66
b77
b66
b660 0 0 x99
19
Gauss-Jordan Elimination: Gauss-Jordan Elimination: ExampleExample
10
1
8
|
|
|
473
321
211
:Matrix Augmented
10
1
8
473
321
211
3
2
1
x
x
x
1 1 2 | 8
0 1 5 | 9
0 4 2| 14
R2 R2 - (-1)R1
R3 R3 - ( 3)R1
Scaling R2:
R2 R2/(-1)
R1 R1 - (1)R2
R3 R3-(4)R2
1 1 2 | 8
0 1 5| 9
0 4 2| 14
22
9
17
|
|
|
1800
510
701
Scaling R3:
R3 R3/(18)
9/11
9
17
|
|
|
100
510
701
222.1
888.2
444.8
|
|
|
100
010
001
R1 R1 - (7)R3
R2 R2-(-5)R3
RESULT:
x1=8.45, x2=-2.89, x3=1.23
Time Complexity? O(n3)
Gauss-Jordan EliminationGauss-Jordan Elimination
• Solve:Solve:
• Only care about numbers – form Only care about numbers – form “tableau” or “augmented matrix”:“tableau” or “augmented matrix”:
1354
732
21
21
xx
xx
1354
732
21
21
xx
xx
13
7
54
32
13
7
54
32
Gauss-Jordan EliminationGauss-Jordan Elimination
• Given:Given:
• Goal: reduce this to trivial systemGoal: reduce this to trivial system
and read off answer from right columnand read off answer from right column
13
7
54
32
13
7
54
32
?
?
10
01
?
?
10
01
Gauss-Jordan EliminationGauss-Jordan Elimination
• Basic operation 1: replace any row byBasic operation 1: replace any row bylinear combination with any other rowlinear combination with any other row
• Here, replace row1 with Here, replace row1 with 11//22 * row1 + 0 * * row1 + 0 *
row2row2
13
7
54
32
13
7
54
32
1354
1 27
23
1354
1 27
23
Gauss-Jordan EliminationGauss-Jordan Elimination
• Replace row2 with row2 – 4 * row1Replace row2 with row2 – 4 * row1
• Negate row2Negate row2
1354
1 27
23
1354
1 27
23
110
1 27
23
110
1 27
23
110
1 27
23
110
1 27
23
Gauss-Jordan EliminationGauss-Jordan Elimination
• Replace row1 with row1 – Replace row1 with row1 – 33//22 * row2 * row2
• Read off solution: xRead off solution: x1 1 = 2, x= 2, x2 2 = 1= 1
110
1 27
23
110
1 27
23
1
2
10
01
1
2
10
01
Augmented MatricesAugmented Matrices
• Matrices are Matrices are rectangular arrays of numbersrectangular arrays of numbers that can aid us by that can aid us by eliminating the need to eliminating the need to write the variableswrite the variables at each step of the at each step of the reduction.reduction.
• For example, the For example, the systemsystem
may be represented by the may be represented by the augmentedaugmented matrixmatrix
CoefficiCoefficient ent
MatrixMatrix
2 4 6 22
3 8 5 27
2 2
x y z
x y z
x y z
2 4 6 22
3 8 5 27
2 2
x y z
x y z
x y z
2 4 6 22
3 8 5 27
1 1 2 2
2 4 6 22
3 8 5 27
1 1 2 2
Matrices and Gauss-JordanMatrices and Gauss-Jordan
• Every stepEvery step in the in the Gauss-Jordan elimination Gauss-Jordan elimination methodmethod can be expressed with can be expressed with matricesmatrices, rather , rather than systems of equations, thus simplifying the than systems of equations, thus simplifying the whole process:whole process:
• Steps expressed as Steps expressed as systems of equationssystems of equations::
• Steps expressed as Steps expressed as augmented matricesaugmented matrices::
2 4 6 22
3 8 5 27
2 2
x y z
x y z
x y z
2 4 6 22
3 8 5 27
2 2
x y z
x y z
x y z
2 4 6 22
3 8 5 27
1 1 2 2
2 4 6 22
3 8 5 27
1 1 2 2
Toggle slides Toggle slides back and back and forth to forth to compare compare
before and before and changeschanges
Matrices and Gauss-JordanMatrices and Gauss-Jordan
• Every stepEvery step in the in the Gauss-Jordan elimination Gauss-Jordan elimination methodmethod can be expressed with can be expressed with matricesmatrices, , rather than systems of equations, thus rather than systems of equations, thus simplifying the whole process:simplifying the whole process:
• Steps expressed as Steps expressed as systems of equationssystems of equations::
• Steps expressed as Steps expressed as augmented matricesaugmented matrices::
1 2 3 11
0 2 4 6
1 1 2 2
1 2 3 11
0 2 4 6
1 1 2 2
2 3 11
2 4 6
2 2
x y z
y z
x y z
2 3 11
2 4 6
2 2
x y z
y z
x y z
Toggle slides Toggle slides back and back and forth to forth to compare compare
before and before and changeschanges
Matrices and Gauss-JordanMatrices and Gauss-Jordan• Every stepEvery step in the in the Gauss-Jordan elimination Gauss-Jordan elimination
methodmethod can be expressed with can be expressed with matricesmatrices, , rather than systems of equations, thus rather than systems of equations, thus simplifying the whole process:simplifying the whole process:
• Steps expressed as Steps expressed as systems of equationssystems of equations::
• Steps expressed as Steps expressed as augmented matricesaugmented matrices::
1 2 3 11
0 2 4 6
0 3 5 13
1 2 3 11
0 2 4 6
0 3 5 13
2 3 11
2 4 6
3 5 13
x y z
y z
y z
2 3 11
2 4 6
3 5 13
x y z
y z
y z
Toggle slides Toggle slides back and back and forth to forth to compare compare
before and before and changeschanges
Matrices and Gauss-JordanMatrices and Gauss-Jordan• Every stepEvery step in the in the Gauss-Jordan elimination Gauss-Jordan elimination
methodmethod can be expressed with can be expressed with matricesmatrices, , rather than systems of equations, thus rather than systems of equations, thus simplifying the whole process:simplifying the whole process:
• Steps expressed as Steps expressed as systems of equationssystems of equations::
• Steps expressed as Steps expressed as augmented matricesaugmented matrices::
1 2 3 11
0 1 2 3
0 3 5 13
1 2 3 11
0 1 2 3
0 3 5 13
2 3 11
2 3
3 5 13
x y z
y z
y z
2 3 11
2 3
3 5 13
x y z
y z
y z
Toggle slides Toggle slides back and back and forth to forth to compare compare
before and before and changeschanges
Matrices and Gauss-JordanMatrices and Gauss-Jordan• Every stepEvery step in the in the Gauss-Jordan elimination Gauss-Jordan elimination
methodmethod can be expressed with can be expressed with matricesmatrices, , rather than systems of equations, thus rather than systems of equations, thus simplifying the whole process:simplifying the whole process:
• Steps expressed as Steps expressed as systems of equationssystems of equations::
• Steps expressed as Steps expressed as augmented matricesaugmented matrices::
1 0 7 17
0 1 2 3
0 3 5 13
1 0 7 17
0 1 2 3
0 3 5 13
7 17
2 3
3 5 13
x z
y z
y z
7 17
2 3
3 5 13
x z
y z
y z
Toggle slides Toggle slides
back and back and forth to forth to compare compare
before and before and changeschanges
Matrices and Gauss-JordanMatrices and Gauss-Jordan• Every stepEvery step in the in the Gauss-Jordan elimination Gauss-Jordan elimination
methodmethod can be expressed with can be expressed with matricesmatrices, , rather than systems of equations, thus rather than systems of equations, thus simplifying the whole process:simplifying the whole process:
• Steps expressed as Steps expressed as systems of equationssystems of equations::
• Steps expressed as Steps expressed as augmented matricesaugmented matrices::
1 0 7 17
0 1 2 3
0 0 11 22
1 0 7 17
0 1 2 3
0 0 11 22
7 17
2 3
11 22
x z
y z
z
7 17
2 3
11 22
x z
y z
z
Toggle slides Toggle slides
back and back and forth to forth to compare compare
before and before and changeschanges
Matrices and Gauss-JordanMatrices and Gauss-Jordan• Every stepEvery step in the in the Gauss-Jordan elimination Gauss-Jordan elimination
methodmethod can be expressed with can be expressed with matricesmatrices, , rather than systems of equations, thus rather than systems of equations, thus simplifying the whole process:simplifying the whole process:
• Steps expressed as Steps expressed as systems of equationssystems of equations::
• Steps expressed as Steps expressed as augmented matricesaugmented matrices::
1 0 7 17
0 1 2 3
0 0 1 2
1 0 7 17
0 1 2 3
0 0 1 2
7 17
2 3
2
x z
y z
z
7 17
2 3
2
x z
y z
z
Toggle slides Toggle slides
back and back and forth to forth to compare compare
before and before and changeschanges
Matrices and Gauss-JordanMatrices and Gauss-Jordan• Every stepEvery step in the in the Gauss-Jordan elimination Gauss-Jordan elimination
methodmethod can be expressed with can be expressed with matricesmatrices, , rather than systems of equations, thus rather than systems of equations, thus simplifying the whole process:simplifying the whole process:
• Steps expressed as Steps expressed as systems of equationssystems of equations::
• Steps expressed as Steps expressed as augmented matricesaugmented matrices::
1 0 0 3
0 1 2 3
0 0 1 2
1 0 0 3
0 1 2 3
0 0 1 2
3
2 3
2
x
y z
z
3
2 3
2
x
y z
z
Toggle slides Toggle slides
back and back and forth to forth to compare compare
before and before and changeschanges
Matrices and Gauss-JordanMatrices and Gauss-Jordan• Every stepEvery step in the in the Gauss-Jordan elimination Gauss-Jordan elimination
methodmethod can be expressed with can be expressed with matricesmatrices, , rather than systems of equations, thus rather than systems of equations, thus simplifying the whole process:simplifying the whole process:
• Steps expressed as Steps expressed as systems of equationssystems of equations::
• Steps expressed as Steps expressed as augmented matricesaugmented matrices::
1 0 0 3
0 1 0 1
0 0 1 2
1 0 0 3
0 1 0 1
0 0 1 2
3
1
2
x
y
z
3
1
2
x
y
z
Row Row Reduced Reduced
Form of the Form of the MatrixMatrix
Toggle slides Toggle slides back and back and forth to forth to compare compare
before and before and changeschanges
