Upload
princess-aduana
View
19
Download
1
Embed Size (px)
DESCRIPTION
Quantitative Techniques
Citation preview
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
2 Systems of linear equations and their solution
For the purpose of computer simulation, physical structures(e.g., bridges, electric circuits) are often represented as dis-crete models. These models are usually linear systems Ax =
f , where the matrix A represents the structure, f is the vec-tor representing external forces, and x is the vector of un-known quantities of interest (e.g., displacements of points onthe bridge, currents in the electric circuit).
2.1 Definition of linear systems of equations
A system of equations of the form
a11x1 + a12x2 + . . . + a1nxn = b1
a21x1 + a22x2 + . . . + a2nxn = b2... ... ...
am1x1 + am2x2 + . . . + amnxn = bm
is called a linear system of m equations with n unknowns(x1, x2, . . . , xn).
Points to note:
• linear systems often occur in engineering applications;
• m does not need to be equal to n;
• generally, for n variables, n equations will give a uniquesolution; not always, though (as we shall see in the exam-ples below);
• the coefficients aij (i, j = 1, 2, . . . , n) do not need to beintegers, nor even “nice” real numbers.
2.1
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
2.2 Matrix solution of an n×n linear system of equations
Suppose that we have a system of n equations in n unknownsx1, x2, . . . , xn.
a11x1 + a12x2 + . . . + a1nxn = b1
a21x1 + a22x2 + . . . + a2nxn = b2... ... ...
an1x1 + an2x2 + . . . + annxn = bn
This can be represented in matrix form as
A x = b
where
A =
a11 a12 . . . a1n
a21 a22 . . . a2n... ... . . . ...
an1 an2 . . . ann
x =
x1
x2...
xn
b =
b1
b2...bn
If A is a non-singular matrix so that the inverse matrix A−1
exists, then there is a unique solution given by
x = A−1 b
2.2
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
Example
3x1 + 2x2 = 1
7x1 + 5x2 = 2
}
This can be represented in matrix form as(
3 2
7 5
)(x1
x2
)=
(1
2
)(1)
Multiply both sides on the left by the inverse matrix A−1
(x1
x2
)=
(5 −2
−7 3
)(1
2
)
which implies(
x1
x2
)=
(5× 1− 2× 2
−7× 1 + 3× 2
)=
(1
−1
)
Check by substitution into original equation (1)(
3 2
7 5
)(1
−1
)=
(3× 1− 2× 1
7× 1− 5× 1
)=
(1
2
)X
BUT, calculation of the inverse matrix is very expensive.
Solving linear systems of equations is at the heart of manynumerical algorithms in engineering.
Therefore, it is important to have efficient methods.
2.3
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
2.3 Solving linear systems of equations: The basic idea
Consider the case n = 2, e.g.
2x1 + 3x2 = 2 (2)4x1 + x2 = 1 (3)
Let us subtract two times equation (2) from equation (3)
2x1 + 3x2 = 2 (4)0x1 − 5x2 = −3 (5)
We have eliminated x1 from equation (5) and can solve it
−5x2 = −3 ⇒ x2 =3
5
We can now insert x2 into the first equation to obtain
2x1 = 2− 3× 3
5=
1
5⇒ x1 =
1
10
2.4
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
This process can be written in the matrix form A x = b as
(2 3
4 1
)(x1
x2
)=
(2
1
). (6)
Subtracting two times row 1 from row 2 implies
R2− 2R1
(2 3
0 −5
)(x1
x2
)=
(2
−3
)(7)
and back substituting for the components of x
x2 =3
5
x1 =1
2
(2− 3× 3
5
)=
1
10
gives the solution
x =
1
10
3
5
.
Summary We started with the linear system of equations (6)and transformed it by row manipulations into the equivalentsystem (7), from which we could easily obtain the solution x.
2.5
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
2.4 Back substitution in upper triangular systems
Solution of the system A x = b is easy if it has the form
a11x1 + a12x2 + a13x3 = b1
a22x2 + a23x3 = b2
a33x3 = b3
(8)
Provided ajj 6= 0 for j = 1, 2, 3 we simply start at the last rowand progressively back substitute to work out the values of x
x3 =1
a33b3
x2 =1
a22(b2 − a23x3)
x1 =1
a11(b1 − a13x3 − a12x2)
2.6
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
For any linear system of equations in which the the coefficientmatrix A is upper triangular with zero elements below themain diagonal
a11 a12 a13 . . . a1n
0 a22 a23 . . . a2n
0 0 a33 . . . a3n... ... . . . . . . . . .
0 0 . . . 0 ann
x1
x2
x3...
xn
=
b1
b2
b3...bn
the solution can be found by back substitution
xn =1
annbn,
xn−1 =1
an−1,n−1(bn−1 − an−1,nxn) ,
... = ...
x1 =1
a11
b1 −
n∑j=2
a1jxj
,
provided ajj 6= 0 for all j = 1, . . . , n.
In order to transform a given linear system of equations intoan equivalent one, where the matrix A is upper triangular weuse an algorithm called Gaussian Elimination.
Carl Friedrich Gauss (1777 – 1855) – a German mathemati-cian and scientist.
2.7
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
2.5 Gaussian Elimination
We explain this for a 3 × 3 example. The general algorithmwill be clear from this.
We start with the system of equations
1 2 3
2 2 2
1 8 1
x1
x2
x3
=
6
6
10
It is useful to rewrite it in the augmented form
1 2 3 6
2 2 2 6
1 8 1 10
Now introduce zeros in the first column below the main diag-onal by first subtracting 2 times the first row from the secondrow and then subtracting the first row from the third row
R2− 2R1
R3−R1
1 2 3 6
0 −2 −4 −6
0 6 −2 4
We are not yet finished. In order to achieve upper triangularform we need to create an additional zero in the last row
R3 + 3R2
1 2 3 6
0 −2 −4 −6
0 0 −14 −14
The linear system of equations is now in upper triangular formand we can easily recover the solution x =
(1 1 1
)T.
2.8
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
We now show the process of Gaussian elimination schemat-ically for a 4× 4 example
x x x x x
x x x x x
x x x x x
x x x x x
First reduce the first column
x x x x x
0 x x x x
x x x x x
x x x x x
−→
x x x x x
0 x x x x
0 x x x x
x x x x x
−→
x x x x x
0 x x x x
0 x x x x
0 x x x x
then the second column
−→
x x x x x
0 x x x x
0 0 x x x
0 x x x x
−→
x x x x x
0 x x x x
0 0 x x x
0 0 x x x
−→
and finally the third column
x x x x x
0 x x x x
0 0 x x x
0 0 0 x x
to generate an upper triangular augmented matrix.
Back substitute to determine the solution x.
2.9
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
2.6 Geometric Interpretation: Existence of solutions
For the case of a 2× 2 system of equations
a11x1 + a12x2 = b1
a21x1 + a22x2 = b2
we may interpret (x1, x2) as coordinates in the x1x2-plane.
The equations are straight lines.
The point (x1, x2) is a solution ⇐⇒ the point lies on bothlines. There are three cases:-
(i) No solution if the lines are parallel
(ii) Precisely one solution if they intersect
(iii) Infinitely many solutions if they coincide
Example (i) Two parallel lines
x + y = 1
x + y = 0⇒ detA = 0,
(1 1 1
1 1 0
)
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
1.5
2
x
y
R2-R1(1 1 1
0 0 −1
)
Contradiction
0×x+0×y = 0 = −1
NO SOLUTIONS
2.10
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
Example (ii) Lines intersect
x + y = 1
x− y = 0⇒ detA = −2,
(1 1 1
1 −1 0
)
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
1.5
2
x
y
R2-R1(1 1 1
0 −2 −1
)
⇒ y =1
2x =
1
2UNIQUESOLUTION
Example (iii) Lines coincide
x + y = 1
2x + 2y = 2⇒ detA = 0,
(1 1 1
2 2 2
)
−1 −0.5 0 0.5 10
0.5
1
1.5
2
x
y
R2-2R1(1 1 1
0 0 0
)
No Contradiction
0× x + 0× y = 0X
INFINITELYMANYSOLUTIONS
If detA = 0 there are either no solutions or infinitely many
2.11
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
2.7 Pivoting
Gaussian elimination is not possible if we encounter a diago-nal element that is zero in the current column.
Example
A x =
0 1 2
2 3 1
4 2 1
x =
4
9
9
The solution exists, x =(1 2 1
)T.
But we cannot start Gaussian Elimination because a11 = 0.
This can be solved by a process called pivoting.
Since the order does not matter, we can just exchange the firstequation with the third equation to obtain
4 2 1
2 3 1
0 1 2
x =
9
9
4
.
Now we can start Gaussian Elimination.
In practice one always looks for the element with the largestmagnitude in the current column and exchanges the row con-taining this element with the top row of the current sub-block.This should be done each time elimination takes place (i.e.,for each column) until we arrive at the upper triangular formof the matrix. This process is called Partial Pivoting.
2 −1 2 1
−6 0 −2 0
8 −1 5 4
NB We don’t need to pivota22 because we are NOTeliminating elements in the2nd column.
2.12
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
Pivoting becomes even more important when Gaussian Elimi-nation is implemented on a computer. This is because comput-ers operate with approximations of real numbers up to someprecision. Unavoidably this leads to round-off errors, whichmay propagate and have dramatic effect on the final result (ap-proximate solution to the linear system). Let us demonstratethis on a simple example.
Example. Consider a linear system:
0.0001x + y = 1 [1]
−x + 2y = 1 [2]
It has the exact solution x = 1000010002, y = 10001
10002 (check this!).
Therefore, x ≈ 1, y ≈ 1 is a good approximation.
Now, let us try solving this system using Gaussian Elimina-tion, rounding all results to 4 significant digits.
Multiply row [1] by −10.0001 and subtract the result from row [2]:
0.0001x + y = 1 [1]
(2− −10.0001 · 1)y = 1− −1
0.0001 · 1 [2′]
Recall that we round all results to 4 significant digits. E.g.,one can use MATLAB to compute
>> digits(4)>> vpa(2-(-1)*1/0.0001)ans =10000.0>> vpa(1-(-1)*1/0.0001)ans =10000.0
2.13
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
Thus, we find0.0001x + y = 1 [1]
10000.0y = 10000.0 [2′]Hence,
0.0001x + y = 1 [1]
y = 1.0 [2′]Then we back-substitute to find x:
0.0001x + 1.0 = 1 [1]
y = 1.0 [2′]such that our approximate solution is
x = 0.0
y = 1.0
As you can see, we have a real problem: the propagation ofround-off errors results in a drastically wrong “solution”.
Now, let us swap the equations around in the above linear sys-tem:
−x + 2y = 1 [1∗]0.0001x + y = 1 [2∗]
As before, the exact solution is x = 1000010002, y = 10001
10002,and x ≈ 1, y ≈ 1 is a good approximation.
Again, let us solve this system using Gaussian Elimination,rounding all results to 4 significant digits.
One has−x + 2y = 1 [1∗]
(1− 0.0001−1 · 2)y = 1− 0.0001
−1 · 1 [2′∗]This gives
−x + 2y = 1 [1∗]1.0y = 1.0 [2′∗]
2.14
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
Hence−x + 2y = 1 [1∗]
y = 1.0 [2′∗]Then we back-substitute to find x:
−x + 2 · 1.0 = 1 [1∗]y = 1.0 [2′∗]
such that our approximate solution is
x = 1.0
y = 1.0
Clearly, this is a much better approximation than what we hadbefore.
Points to note:
• If we do not use exact arithmetic, then Gaussian Elimi-nation for the same linear system may give different an-swers depending on the order in which equations of thesystem are written. One of these answers may be obvi-ously wrong! This is due to propagation of round-off er-rors.
• The process of Partial Pivoting switches the equationsaround (as explained above). In many cases this keepsround-off errors under control and produces adequate ap-proximations.
• Note that in Partial Pivoting we look for the elementwith the largest magnitude (i.e., with the largest absolutevalue). In other words, given a choice between 0.3 and−0.5, we must select −0.5.
2.15
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
Example. Solve the following linear system using GaussianElimination with partial pivoting
x − 3y = 1
−2x + 2y − 4z = 2
x + 3y − 2z = 3
Let us write the system in the augmented matrix form:
R1
R2
R3
1 −3 0 1
–2 2 −4 2
1 3 −2 3
The element with the largest magnitude in the 1st column is−2, and it is in the 2nd row (R2). Therefore, we do pivoting,i.e., we interchange rows (R2) and (R1), and then use Gaus-sian elimination to introduce zeros in the 1st column:
R2
R1
R3
−2 2 −4 2
1 −3 0 1
1 3 −2 3
∼
R2
R1+12R2
R3+12R2
−2 2 −4 2
0 –2 −2 2
0 4 −4 4
.
Now, in this new matrix, we look at the elements in the 2ndcolumn of the 2nd and the 3rd rows (marked in boldface), andfind the one with the largest magnitude: it is 4 and it is in the3rd row. Thus, we need to pivot again. We interchange the2nd and the 3rd rows, and then apply Gaussian elimination:
R1′
R2′
R3′
−2 2 −4 2
0 4 −4 4
0 −2 −2 2
∼
R3′+12R2′
−2 2 −4 2
0 4 −4 4
0 0 −4 4
.
The system is now in the upper triangular form, and we canuse back-substitution to find the solution: x=1, y=0, z=− 1.
2.16
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
? <2.8 LU Factorization
The nonsingular matrix A has an LU-factorization if it canbe expressed as the product of a lower-triangular matrix L andan upper triangular matrix U, e.g.
A =
a11 a12 a13
a21 a22 a23
a31 a32 a33
=
l11 0 0
l21 l22 0
l31 l32 l33
u11 u12 u13
0 u22 u23
0 0 u33
= LU
It turns out that this factorization (when it exists) is not unique.
If L has ones on it’s diagonal, then it is called a Doolittlefactorization. If U has ones on its diagonal, then it is calleda Crout factorization. When U = LT it is called a Choleskydecomposition.
2.8.1 Using LU factorization to solve a system
Assuming A x = b can be factorized into
LU x = b
Then let y = U x and use forward substitution to solve
L y = b
for y. Then use backward substitution to solve
U x = y
for x. This has the advantage that once A = LU is knownwe only have to do a forward and back substitution again fora different righthand side b.
2.17
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
2.8.2 Doolittle’s Method
A =
a11 a12 a13
a21 a22 a23
a31 a32 a33
=
1 0 0
l21 1 0
l31 l32 1
u11 u12 u13
0 u22 u23
0 0 u33
= LU
Multiplying out
A =
u11 u12 u13
l21u11 l21u12 + u22 l21u13 + u23
l31u11 l31u12 + l32u22 l31u13 + l32u23 + u33
Comparing coefficients implies that
u11 = a11, u12 = a12, u13 = a13,
l21 = a21u11
, u22 = a22 − l21u12, u23 = a23 − l21u13,
l31 = a31u11
, l32 = a32−l31u12u22
, u33 = a33 − l31u13 − l32u23.
Sometimes it is necessary to use pivoting.
Example Use LU factorization to solve
−1 1 −1
−2 4 −2
1 1 4
x1
x2
x3
=
−2
−6
9
Doolittle’s method implies
−1 1 −1
−2 4 −2
1 1 4
=
u11 u12 u13
l21u11 l21u12 + u22 l21u13 + u23
l31u11 l31u12 + l32u22 l31u13 + l32u23 + u33
2.18
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
Comparing coefficients we find
u11 = −1, u12 = 1, u13 = −1
l21 = −2−1 = 2, u22 = 4− 2× 1 = 2, u23 = −2− 2× (−1) = 0
l31 = 1−1 = −1, l32 = 1−(−1)×1
2 = 1 u33 = 4− 1− 0 = 3
Now let y = U x ⇒ LU x = b becomes L y = b, which issolved by forward substitution (i.e. start at top of matrix):
L y =
1 0 0
2 1 0
−1 1 1
y1
y2
y3
=
−2
−6
9
y1 = −2
2y1 + y2 = −6
−y1 + y2 + y3 = 9
⇒y1 = −2
y2 = −6− 2× (−2) = −2
y3 = 9 + (−2)− (−2) = 9
Then solve U x = y for x by back substitution:−1 1 −1
0 2 0
0 0 3
x1
x2
x3
=
−2
−2
9
3y3 = 9
2x2 = −2
−x1 + x2 − x3 = −2
⇒x3 = 3
x2 = −1
x1 = −2
Note if we now wanted the result for another b we would onlyneed to do the forward and back substitution.
> ?
2.19
2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution
2.9 Programs and libraries
There are several packages that implement Gaussian Elimina-tion and LU decomposition.
For example
• MATLAB
• Maple
• Mathematica
And there is a standard programming library called LAPACKthat can easily be integrated into self-written programs.
2.20