2E1 Linear Algebra Notes 2

2E1: Linear Algebra | Lecture Notes §2 Systems of linear equations and their solution

2 Systems of linear equations and their solution

For the purpose of computer simulation, physical structures(e.g., bridges, electric circuits) are often represented as dis-crete models. These models are usually linear systems Ax =

f , where the matrix A represents the structure, f is the vec-tor representing external forces, and x is the vector of un-known quantities of interest (e.g., displacements of points onthe bridge, currents in the electric circuit).

2.1 Definition of linear systems of equations

A system of equations of the form

a11x1 + a12x2 + . . . + a1nxn = b1

a21x1 + a22x2 + . . . + a2nxn = b2... ... ...

am1x1 + am2x2 + . . . + amnxn = bm

is called a linear system of m equations with n unknowns(x1, x2, . . . , xn).

Points to note:

• linear systems often occur in engineering applications;

• m does not need to be equal to n;

• generally, for n variables, n equations will give a uniquesolution; not always, though (as we shall see in the exam-ples below);

• the coefficients aij (i, j = 1, 2, . . . , n) do not need to beintegers, nor even “nice” real numbers.

2.1


2.2 Matrix solution of an n×n linear system of equations

Suppose that we have a system of n equations in n unknownsx1, x2, . . . , xn.

a11x1 + a12x2 + . . . + a1nxn = b1

a21x1 + a22x2 + . . . + a2nxn = b2... ... ...

an1x1 + an2x2 + . . . + annxn = bn

This can be represented in matrix form as

A x = b

where

A =

a11 a12 . . . a1n

a21 a22 . . . a2n... ... . . . ...

an1 an2 . . . ann

x =

x1

x2...

xn

b =

b1

b2...bn

If A is a non-singular matrix so that the inverse matrix A−1

exists, then there is a unique solution given by

x = A−1 b

2.2


Example

3x1 + 2x2 = 1

7x1 + 5x2 = 2

}

This can be represented in matrix form as(

3 2

7 5

)(x1

x2

)=

(1

2

)(1)

Multiply both sides on the left by the inverse matrix A−1

(x1

x2

)=

(5 −2

−7 3

)(1

2

)

which implies(

x1

x2

)=

(5× 1− 2× 2

−7× 1 + 3× 2

)=

(1

−1

)

Check by substitution into original equation (1)(

3 2

7 5

)(1

−1

)=

(3× 1− 2× 1

7× 1− 5× 1

)=

(1

2

)X

BUT, calculation of the inverse matrix is very expensive.

Solving linear systems of equations is at the heart of manynumerical algorithms in engineering.

Therefore, it is important to have efficient methods.

2.3


2.3 Solving linear systems of equations: The basic idea

Consider the case n = 2, e.g.

2x1 + 3x2 = 2 (2)4x1 + x2 = 1 (3)

Let us subtract two times equation (2) from equation (3)

2x1 + 3x2 = 2 (4)0x1 − 5x2 = −3 (5)

We have eliminated x1 from equation (5) and can solve it

−5x2 = −3 ⇒ x2 =3

5

We can now insert x2 into the first equation to obtain

2x1 = 2− 3× 3

5=

1

5⇒ x1 =

1

10

2.4


This process can be written in the matrix form A x = b as

(2 3

4 1

)(x1

x2

)=

(2

1

). (6)

Subtracting two times row 1 from row 2 implies

R2− 2R1

(2 3

0 −5

)(x1

x2

)=

(2

−3

)(7)

and back substituting for the components of x

x2 =3

5

x1 =1

2

(2− 3× 3

5

)=

1

10

gives the solution

x =

1

10

3

5

.

Summary We started with the linear system of equations (6)and transformed it by row manipulations into the equivalentsystem (7), from which we could easily obtain the solution x.

2.5


2.4 Back substitution in upper triangular systems

Solution of the system A x = b is easy if it has the form

a11x1 + a12x2 + a13x3 = b1

a22x2 + a23x3 = b2

a33x3 = b3

(8)

Provided ajj 6= 0 for j = 1, 2, 3 we simply start at the last rowand progressively back substitute to work out the values of x

x3 =1

a33b3

x2 =1

a22(b2 − a23x3)

x1 =1

a11(b1 − a13x3 − a12x2)

2.6


For any linear system of equations in which the the coefficientmatrix A is upper triangular with zero elements below themain diagonal

a11 a12 a13 . . . a1n

0 a22 a23 . . . a2n

0 0 a33 . . . a3n... ... . . . . . . . . .

0 0 . . . 0 ann

x1

x2

x3...

xn

=

b1

b2

b3...bn

the solution can be found by back substitution

xn =1

annbn,

xn−1 =1

an−1,n−1(bn−1 − an−1,nxn) ,

... = ...

x1 =1

a11

b1 −

n∑j=2

a1jxj

,

provided ajj 6= 0 for all j = 1, . . . , n.

In order to transform a given linear system of equations intoan equivalent one, where the matrix A is upper triangular weuse an algorithm called Gaussian Elimination.

Carl Friedrich Gauss (1777 – 1855) – a German mathemati-cian and scientist.

2.7


2.5 Gaussian Elimination

We explain this for a 3 × 3 example. The general algorithmwill be clear from this.

We start with the system of equations

1 2 3

2 2 2

1 8 1

x1

x2

x3

=

6

6

10

It is useful to rewrite it in the augmented form

1 2 3 6

2 2 2 6

1 8 1 10

Now introduce zeros in the first column below the main diag-onal by first subtracting 2 times the first row from the secondrow and then subtracting the first row from the third row

R2− 2R1

R3−R1

1 2 3 6

0 −2 −4 −6

0 6 −2 4

We are not yet finished. In order to achieve upper triangularform we need to create an additional zero in the last row

R3 + 3R2

1 2 3 6

0 −2 −4 −6

0 0 −14 −14

The linear system of equations is now in upper triangular formand we can easily recover the solution x =

(1 1 1

)T.

2.8


We now show the process of Gaussian elimination schemat-ically for a 4× 4 example

x x x x x

x x x x x

x x x x x

x x x x x

First reduce the first column

x x x x x

0 x x x x

x x x x x

x x x x x

−→

x x x x x

0 x x x x

0 x x x x

x x x x x

−→

x x x x x

0 x x x x

0 x x x x

0 x x x x

then the second column

−→

x x x x x

0 x x x x

0 0 x x x

0 x x x x

−→

x x x x x

0 x x x x

0 0 x x x

0 0 x x x

−→

and finally the third column

x x x x x

0 x x x x

0 0 x x x

0 0 0 x x

to generate an upper triangular augmented matrix.

Back substitute to determine the solution x.

2.9


2.6 Geometric Interpretation: Existence of solutions

For the case of a 2× 2 system of equations

a11x1 + a12x2 = b1

a21x1 + a22x2 = b2

we may interpret (x1, x2) as coordinates in the x1x2-plane.

The equations are straight lines.

The point (x1, x2) is a solution ⇐⇒ the point lies on bothlines. There are three cases:-

(i) No solution if the lines are parallel

(ii) Precisely one solution if they intersect

(iii) Infinitely many solutions if they coincide

Example (i) Two parallel lines

x + y = 1

x + y = 0⇒ detA = 0,

(1 1 1

1 1 0

)

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

1.5

2

x

y

R2-R1(1 1 1

0 0 −1

)

Contradiction

0×x+0×y = 0 = −1

NO SOLUTIONS

2.10


Example (ii) Lines intersect

x + y = 1

x− y = 0⇒ detA = −2,

(1 1 1

1 −1 0

)

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

1.5

2

x

y

R2-R1(1 1 1

0 −2 −1

)

⇒ y =1

2x =

1

2UNIQUESOLUTION

Example (iii) Lines coincide

x + y = 1

2x + 2y = 2⇒ detA = 0,

(1 1 1

2 2 2

)

−1 −0.5 0 0.5 10

0.5

1

1.5

2

x

y

R2-2R1(1 1 1

0 0 0

)

No Contradiction

0× x + 0× y = 0X

INFINITELYMANYSOLUTIONS

If detA = 0 there are either no solutions or infinitely many

2.11


2.7 Pivoting

Gaussian elimination is not possible if we encounter a diago-nal element that is zero in the current column.

Example

A x =

0 1 2

2 3 1

4 2 1

x =

4

9

9

The solution exists, x =(1 2 1

)T.

But we cannot start Gaussian Elimination because a11 = 0.

This can be solved by a process called pivoting.

Since the order does not matter, we can just exchange the firstequation with the third equation to obtain

4 2 1

2 3 1

0 1 2

x =

9

9

4

.

Now we can start Gaussian Elimination.

In practice one always looks for the element with the largestmagnitude in the current column and exchanges the row con-taining this element with the top row of the current sub-block.This should be done each time elimination takes place (i.e.,for each column) until we arrive at the upper triangular formof the matrix. This process is called Partial Pivoting.

2 −1 2 1

−6 0 −2 0

8 −1 5 4

NB We don’t need to pivota22 because we are NOTeliminating elements in the2nd column.

2.12


Pivoting becomes even more important when Gaussian Elimi-nation is implemented on a computer. This is because comput-ers operate with approximations of real numbers up to someprecision. Unavoidably this leads to round-off errors, whichmay propagate and have dramatic effect on the final result (ap-proximate solution to the linear system). Let us demonstratethis on a simple example.

Example. Consider a linear system:

0.0001x + y = 1 [1]

−x + 2y = 1 [2]

It has the exact solution x = 1000010002, y = 10001

10002 (check this!).

Therefore, x ≈ 1, y ≈ 1 is a good approximation.

Now, let us try solving this system using Gaussian Elimina-tion, rounding all results to 4 significant digits.

Multiply row [1] by −10.0001 and subtract the result from row [2]:

0.0001x + y = 1 [1]

(2− −10.0001 · 1)y = 1− −1

0.0001 · 1 [2′]

Recall that we round all results to 4 significant digits. E.g.,one can use MATLAB to compute

>> digits(4)>> vpa(2-(-1)*1/0.0001)ans =10000.0>> vpa(1-(-1)*1/0.0001)ans =10000.0

2.13


Thus, we find0.0001x + y = 1 [1]

10000.0y = 10000.0 [2′]Hence,

0.0001x + y = 1 [1]

y = 1.0 [2′]Then we back-substitute to find x:

0.0001x + 1.0 = 1 [1]

y = 1.0 [2′]such that our approximate solution is

x = 0.0

y = 1.0

As you can see, we have a real problem: the propagation ofround-off errors results in a drastically wrong “solution”.

Now, let us swap the equations around in the above linear sys-tem:

−x + 2y = 1 [1∗]0.0001x + y = 1 [2∗]

As before, the exact solution is x = 1000010002, y = 10001

10002,and x ≈ 1, y ≈ 1 is a good approximation.

Again, let us solve this system using Gaussian Elimination,rounding all results to 4 significant digits.

One has−x + 2y = 1 [1∗]

(1− 0.0001−1 · 2)y = 1− 0.0001

−1 · 1 [2′∗]This gives

−x + 2y = 1 [1∗]1.0y = 1.0 [2′∗]

2.14


Hence−x + 2y = 1 [1∗]

y = 1.0 [2′∗]Then we back-substitute to find x:

−x + 2 · 1.0 = 1 [1∗]y = 1.0 [2′∗]

such that our approximate solution is

x = 1.0

y = 1.0

Clearly, this is a much better approximation than what we hadbefore.

Points to note:

• If we do not use exact arithmetic, then Gaussian Elimi-nation for the same linear system may give different an-swers depending on the order in which equations of thesystem are written. One of these answers may be obvi-ously wrong! This is due to propagation of round-off er-rors.

• The process of Partial Pivoting switches the equationsaround (as explained above). In many cases this keepsround-off errors under control and produces adequate ap-proximations.

• Note that in Partial Pivoting we look for the elementwith the largest magnitude (i.e., with the largest absolutevalue). In other words, given a choice between 0.3 and−0.5, we must select −0.5.

2.15


Example. Solve the following linear system using GaussianElimination with partial pivoting

x − 3y = 1

−2x + 2y − 4z = 2

x + 3y − 2z = 3

Let us write the system in the augmented matrix form:

R1

R2

R3

1 −3 0 1

–2 2 −4 2

1 3 −2 3

The element with the largest magnitude in the 1st column is−2, and it is in the 2nd row (R2). Therefore, we do pivoting,i.e., we interchange rows (R2) and (R1), and then use Gaus-sian elimination to introduce zeros in the 1st column:

R2

R1

R3

−2 2 −4 2

1 −3 0 1

1 3 −2 3

∼

R2

R1+12R2

R3+12R2

−2 2 −4 2

0 –2 −2 2

0 4 −4 4

.

Now, in this new matrix, we look at the elements in the 2ndcolumn of the 2nd and the 3rd rows (marked in boldface), andfind the one with the largest magnitude: it is 4 and it is in the3rd row. Thus, we need to pivot again. We interchange the2nd and the 3rd rows, and then apply Gaussian elimination:

R1′

R2′

R3′

−2 2 −4 2

0 4 −4 4

0 −2 −2 2

∼

R3′+12R2′

−2 2 −4 2

0 4 −4 4

0 0 −4 4

.

The system is now in the upper triangular form, and we canuse back-substitution to find the solution: x=1, y=0, z=− 1.

2.16


? <2.8 LU Factorization

The nonsingular matrix A has an LU-factorization if it canbe expressed as the product of a lower-triangular matrix L andan upper triangular matrix U, e.g.

A =

a11 a12 a13

a21 a22 a23

a31 a32 a33

=

l11 0 0

l21 l22 0

l31 l32 l33

u11 u12 u13

0 u22 u23

0 0 u33

= LU

It turns out that this factorization (when it exists) is not unique.

If L has ones on it’s diagonal, then it is called a Doolittlefactorization. If U has ones on its diagonal, then it is calleda Crout factorization. When U = LT it is called a Choleskydecomposition.

2.8.1 Using LU factorization to solve a system

Assuming A x = b can be factorized into

LU x = b

Then let y = U x and use forward substitution to solve

L y = b

for y. Then use backward substitution to solve

U x = y

for x. This has the advantage that once A = LU is knownwe only have to do a forward and back substitution again fora different righthand side b.

2.17


2.8.2 Doolittle’s Method

A =

a11 a12 a13

a21 a22 a23

a31 a32 a33

=

1 0 0

l21 1 0

l31 l32 1

u11 u12 u13

0 u22 u23

0 0 u33

= LU

Multiplying out

A =

u11 u12 u13

l21u11 l21u12 + u22 l21u13 + u23

l31u11 l31u12 + l32u22 l31u13 + l32u23 + u33

Comparing coefficients implies that

u11 = a11, u12 = a12, u13 = a13,

l21 = a21u11

, u22 = a22 − l21u12, u23 = a23 − l21u13,

l31 = a31u11

, l32 = a32−l31u12u22

, u33 = a33 − l31u13 − l32u23.

Sometimes it is necessary to use pivoting.

Example Use LU factorization to solve

−1 1 −1

−2 4 −2

1 1 4

x1

x2

x3

=

−2

−6

9

Doolittle’s method implies

−1 1 −1

−2 4 −2

1 1 4

=

u11 u12 u13

l21u11 l21u12 + u22 l21u13 + u23

l31u11 l31u12 + l32u22 l31u13 + l32u23 + u33

2.18


Comparing coefficients we find

u11 = −1, u12 = 1, u13 = −1

l21 = −2−1 = 2, u22 = 4− 2× 1 = 2, u23 = −2− 2× (−1) = 0

l31 = 1−1 = −1, l32 = 1−(−1)×1

2 = 1 u33 = 4− 1− 0 = 3

Now let y = U x ⇒ LU x = b becomes L y = b, which issolved by forward substitution (i.e. start at top of matrix):

L y =

1 0 0

2 1 0

−1 1 1

y1

y2

y3

=

−2

−6

9

y1 = −2

2y1 + y2 = −6

−y1 + y2 + y3 = 9

⇒y1 = −2

y2 = −6− 2× (−2) = −2

y3 = 9 + (−2)− (−2) = 9

Then solve U x = y for x by back substitution:−1 1 −1

0 2 0

0 0 3

x1

x2

x3

=

−2

−2

9

3y3 = 9

2x2 = −2

−x1 + x2 − x3 = −2

⇒x3 = 3

x2 = −1

x1 = −2

Note if we now wanted the result for another b we would onlyneed to do the forward and back substitution.

> ?

2.19


2.9 Programs and libraries

There are several packages that implement Gaussian Elimina-tion and LU decomposition.

For example

• MATLAB

• Maple

• Mathematica

And there is a standard programming library called LAPACKthat can easily be integrated into self-written programs.

2.20

Documents

2E1 Linear Algebra Notes 2