Numerical Analysis with Python

Linear algebraic Equations

LECTURE 05

• Linear algebraic equations

• Matrix Notations

• Determinant

• Cramer's rules

• Gauss Elimination

• Gauss-Jordan Elimination

Linear Algebraic Equations

• Linear algebraic equation is an algebraic equation of the first degreein all unknowns that simultaneously satisfy set of equations

• linear algebraic equations that are of the general form

• where the a’s are constant coefficients, the b’s are constants, and n isthe number of equations.

𝑎11𝑥1 + 𝑎12𝑥2 + 𝑎13𝑥3 +⋯+ 𝑎1𝑛𝑥𝑛 = 𝑏1𝑎21𝑥1 + 𝑎22𝑥2 + 𝑎23𝑥3 +⋯+ 𝑎2𝑛𝑥𝑛 = 𝑏2𝑎31𝑥1 + 𝑎32𝑥2 + 𝑎33𝑥3 +⋯+ 𝑎3𝑛𝑥𝑛 = 𝑏3

.

.

.

𝑎𝑛1𝑥1 + 𝑎𝑛2𝑥2 + 𝑎𝑛3𝑥3 +⋯+ 𝑎𝑛𝑛𝑥𝑛 = 𝑏𝑛

𝐸𝑞 5.1

Linear Algebraic Equations

• For small numbers of equations (n<=3), linear equations can besolved readily by simple techniques (substitution & elimination).

• However, for four or more equations, solutions become arduous andcomputers must be utilized.

• Larger Linear algebraic equations are represented in matrix form.Example Eq (5.1) can be represented as

𝑎11 𝑎12 𝑎13 … 𝑎1𝑛𝑎21 𝑎22 𝑎23 … 𝑎2𝑛𝑎31 𝑎32 𝑎33 … 𝑎3𝑛. . . … .. . . … .

𝑎𝑛1 𝑎𝑛2 𝑎3𝑛 … 𝑎𝑛𝑛

𝑥1𝑥2𝑥3..𝑥𝑛

=

𝑏1𝑏2𝑏3..𝑏𝑛

𝐴 𝑋 = 𝐵

𝐸𝑞 5.2

Matrix Notation

• Matrix dimension is represented as (𝑚 × 𝑛) where 𝒎 is thenumber of rows and 𝒏 is the number of columns.

• Column Vectors: is a matrix where number of columns 𝑛 =1

𝐴 =

𝑎11 𝑎12 𝑎13 𝑎14𝑎21 𝑎22 𝑎23 𝑎24𝑎31 𝑎32 𝑎33 𝑎34𝑎41 𝑎42 𝑎43 𝑎44

Number of rows 𝑚 =4

Number of columns 𝑛 =4

𝑋 =

𝑥1𝑥2𝑥3𝑥4

Matrix Notation

• Square matrix: a matrices where 𝑚 = 𝑛 is called a squaredmatrix.

• symmetric matrix: is one where 𝑎𝑖𝑗 = 𝑎𝑗𝑖 for all i’s and j’s. For

example,

• Upper triangular: matrix is one where all the elements belowthe main diagonal are zero

𝐴 =

𝑎11 𝑎12 𝑎13 𝑎14𝑎21 𝑎22 𝑎23 𝑎24𝑎31 𝑎32 𝑎33 𝑎34𝑎41 𝑎42 𝑎43 𝑎44

A =5 1 21 3 72 7 8

Matrix Notation

• Diagonal Matrix: is a square matrix where all elements off themain diagonal are equal to zero

• Identity: diagonal matrix where all elements on the main diagonalare equal to 1,

𝐴 =

𝑎11 0 0 00 𝑎22 0 00 0 𝑎33 00 0 0 𝑎44

𝐴 =

1 0 0 00 1 0 00 0 1 00 0 0 1

Matrix Notation

• Upper triangular: matrix is one where all the elements belowthe main diagonal are zero

• Lower triangular matrix : matrix is one where all the elementsabove the main diagonal are zero

𝐴 =

𝑎11 𝑎12 𝑎13 𝑎140 𝑎22 𝑎23 𝑎240 0 𝑎33 𝑎340 0 0 𝑎44

𝐴 =

𝑎11 0 0 0𝑎21 𝑎22 0 0𝑎31 𝑎32 𝑎33 0𝑎41 𝑎42 𝑎43 𝑎44

Matrix Notation

• Matrix Transpose: transpose of a matrix involves transforming itsrows into columns and its columns into rows.

• Augmented matrix: matrix augmentation is achieved by joiningtwo matrix together

𝐴 =1 2 34 5 67 8 9

𝑡𝑟𝑎𝑛𝑠𝑝𝑜𝑠𝑒 𝑜𝑓 𝐴, 𝐴𝑇 =1 4 72 5 83 6 9

𝑎11 𝑎12 𝑎13𝑎21 𝑎22 𝑎23𝑎31 𝑎32 𝑎33

=

𝑏1𝑏2𝑏3

⇒

𝑎11 𝑎12 𝑎13 . 𝑏1𝑎21 𝑎22 𝑎23 . 𝑏2𝑎31 𝑎32 𝑎33 . 𝑏3

Determinant and Cramer's Rule

• Determinant: the determinant has relevance to the evaluation of theill-conditioning of a matrix. Determinant D of a matrix is a singlenumber.

• Determinant for 3 × 3 matrix

𝐵 =

𝑎11 𝑎12 𝑎13𝑎21 𝑎22 𝑎23𝑎31 𝑎32 𝑎33

𝐷 𝐵 =

𝑎11 𝑎12 𝑎13𝑎21 𝑎22 𝑎23𝑎31 𝑎32 𝑎33

D 𝐵 = 𝑎11𝑎22 𝑎23𝑎32 𝑎33

− 𝑎12𝑎21 𝑎23𝑎31 𝑎33

+ 𝑎13𝑎21 𝑎22𝑎31 𝑎32

𝐴 =𝑎11 𝑎12𝑎21 𝑎22

, Determinant of A D A =𝑎11 𝑎12𝑎21 𝑎22

D A = 𝑎11 × 𝑎22 + 𝑎12 × 𝑎21

Determinant and Cramer's Rule

• Systems that have determinant D=0 are called singular system andare unstable.

• Systems that have determinant D close to zero are ill-conditionedsystems

•𝑎22 𝑎23𝑎32 𝑎33

,𝑎21 𝑎23𝑎31 𝑎33

and𝑎21 𝑎22𝑎31 𝑎32

are called theminors

• Determinant evaluation by expansion of minors was impractical forlarge sets of equations

D = 𝑎11𝑎22 𝑎23𝑎32 𝑎33

− 𝑎12𝑎21 𝑎23𝑎31 𝑎33

+ 𝑎13𝑎21 𝑎22𝑎31 𝑎32

Determinant and Cramer's Rule

Problem statement 1:

Find the determinant of the matrix A.

Solution:

𝐴 =0.3 0.52 10.5 1 1.90.1 0.3 0.5

𝐷 = 0.31 1.90.3 0.5

− 0.520.5 1.90.1 0.5

+ 10.5 10.1 0.3

= 0.3 1 ∗ 0.5 − (0.3 ∗ 1.9) − 0.52 0.5 ∗ 0.5 − (0.1 ∗ 1.9)

+ 1 0.5 ∗ 0.3 − (0.1 ∗ 1)

= 0.3 −0.07 − 0.52 0.06 + 1 0.05

= −0.0022

Determinant and Cramer's Rule

• Cramer's rule: This rule states that each unknown in a system oflinear algebraic equations may be expressed as a fraction of twodeterminants with denominator D and with the numerator obtainedfrom D by replacing the column of coefficients of the unknown

• Example for system of linear algebraic equations represented below

• Cramer's solution to unknow 𝑥1, 𝑥2 and 𝑥3 are as follows

𝑥1 =

𝑏1 𝑎12 𝑎13𝑏2 𝑎22 𝑎23𝑏3 𝑎32 𝑎33

𝐷, 𝑥2=

𝑎11 𝑏1 𝑎13𝑎21 𝑏2 𝑎23𝑎31 𝑏3 𝑎33

𝐷, 𝑥3 =

𝑎11 𝑎12 𝑏1𝑎21 𝑎22 𝑏2𝑎31 𝑎32 𝑏3

𝐷

𝑎11 𝑎12 𝑎13𝑎21 𝑎22 𝑎23𝑎31 𝑎32 𝑎33

𝑥1𝑥2𝑥3

=

𝑏1𝑏2𝑏3

Determinant and Cramer's Rule

Problem statement 2:

Use Cramer’s rule to solve the following system of linear equations

Solution:

Matrix form;

Determinant 𝐷 = 0.31 1.90.3 0.5

− 0.520.5 1.90.1 0.5

+ 10.5 10.1 0.3

0.3𝑥1 + 0.52𝑥2 + 𝑥3 = −0.010.5𝑥1 + 𝑥2 + 1.9𝑥3 = 0.67

0.1𝑥1 + 0.3𝑥2 + 0.5𝑥3 = −0.44

0.3 0.52 10.5 1 1.90.1 0.3 0.5

𝑥1𝑥2𝑥3

=−0.010.67−0.44

𝐷 =-0.0022

Determinant and Cramer's Rule

𝑥1 =

−0.01 0.52 10.67 1 1.9−0.44 0.3 0.5

−0.0022=0.03278

−0.0022= −14.9

𝑥2 =

0.3 −0.01 10.5 0.67 1.90.1 −0.44 0.5

−0.0022=

0.0649

−0.0022= −29.5

𝑥3 =

0.3 0.52 −0.010.5 1 0.670.1 0.3 −0.44

−0.0022=−0.04356

−0.0022= 19.8

Naïve Gauss Elimination

• Gauss elimination is the most basic for solving systems of linearequations using forward elimination and then backward substitution

Forward Elimination of Unknowns. The first phase is designed toreduce the set of equations to an upper triangular system.

• To eliminate 𝑎21 multiply through row 1 by (𝑎21

𝑎11) and then subtract

the result from row 2

𝑎11 𝑎12 𝑎13𝑎21 𝑎22 𝑎23𝑎31 𝑎32 𝑎33

=

𝑏1𝑏2𝑏3

⟹

𝑎11 𝑎12 𝑎130 𝑎22 𝑎230 0 𝑎33

=

𝑏1𝑏2𝑏3

Naïve Gauss Elimination

• multiply through row 1 by (𝑎31

𝑎11) and then subtract the results from

row 3

• To eliminate 𝑎32 multiply through row 2 by (𝑎32

𝑎22) and then subtract

the results from row 3

𝑎11 𝑎12 𝑎13𝑎21 𝑎22 𝑎23𝑎31 𝑎32 𝑎33

=

𝑏1𝑏2𝑏3

⟹

𝑎11 𝑎12 𝑎130 𝑎22 𝑎230 0 𝑎33

=

𝑏1𝑏2𝑏3

Naïve Gauss Elimination

Problem statement 3:

Use Gauss Elimination to solve the following system of linear

equations

Solution: express the systems in matrix form

• To eliminate 0.1 in row 2 column 1, multiply row 2 by (0.1/3) then

subtract the result from 2

3𝑥1 − 0.1𝑥2 − 0.2𝑥3 = 7.850.1𝑥1 + 7𝑥2 − 0.3𝑥3 = −19.30.3𝑥1 − 0.2𝑥2 + 10𝑥3 = 71.4

3𝑥1 −0.1𝑥2 −0.2𝑥30.1𝑥1 7𝑥2 −0.3𝑥30.3𝑥1 −0.2𝑥2 10𝑥3

=7.85−19.371.4

Naïve Gauss Elimination

• To eliminate 0.1 in row 2, multiply row 2 by (0.1/3) then subtract

the result from 2

• To eliminate 0.3 in row 3 column 1, multiply row 3 by (0.3/3) then

subtract the result from row 2

3𝑥1 −0.1𝑥2 −0.2𝑥30 7.00333𝑥2 −0.293333𝑥3

0.3𝑥1 −0.2𝑥2 10𝑥3

=7.85

−19.561771.4

3𝑥1 −0.1𝑥2 −0.2𝑥30 7.00333𝑥2 −0.293333𝑥30 −0.190𝑥2 10.0200𝑥3

=7.85

−19.561770.6150

Naïve Gauss Elimination

• To eliminate the element -0.190 in row 3 column 2, multiply row 2

by (-0.190/7.00333) then subtract the result from row 3

• Back substitution: We can now solve these equations by back

substitution.

3𝑥1 −0.1𝑥2 −0.2𝑥30 7.00333𝑥2 −0.293333𝑥30 0 10.0120𝑥3

=7.85

−19.561770.0843

10.0120𝑥3 = 70.0843; 𝑥3 =70.0843

10.0120= 7.00

7.00333𝑥2 − 0.293333𝑥3 = −19.561

𝑥2 =−19.561 + 0.293333𝑥3

7.00333= −2.5

Naïve Gauss Elimination

3𝑥1 − 0.1𝑥2 − 0.2𝑥3 = 7.85

𝑥1 =7.85 + 0.1𝑥2+0.2𝑥3

3

𝑥1 =7.85 + 0.1∗(−2.5)+0.2 ∗ (7)

3= 3

Naïve Gauss Elimination

Problems associated with naïve Gauss elimination

• During both forward elimination and backward substitution, its

possible to encounter division by zero (this is why it is called naïve)

• Round-off errors

• Not suitable for ill-conditioned system whereby a small change in

coefficient (i.e. due to round-off error) causes large changes in the

solution

Naïve Gauss Elimination

Scaling and pivoting

• Pivoting: To write a better computer program for naïve Gauss

Elimination algorithm’ pivoting can be used to reorder the columns

of the matrix so that the column with biggest coefficient is placed

first.

• Scaling: It involves dividing each row element by the biggest

coefficient element of that row there by making the largest element

to be one. This operation helps improve some issues with ill-

conditioned systems

Gauss-Jordan Method

• The major difference is that when an unknown is eliminated in the

Gauss-Jordan method, it is eliminated from all other equations

rather than just the subsequent ones.

• In addition, all rows are normalized by dividing them by their pivot

elements

• Thus, the elimination step results in an identity matrix rather than a

triangular matrix.

• back substitution is not required in Gauss-Jordan method to obtain

the solution.

Gauss-Jordan Method

Problem statement 4:

Use Gauss-Jordan to solve the following system of linear equations

Solution: express the systems as an augmented matrix

3𝑥1 − 0.1𝑥2 − 0.2𝑥3 = 7.850.1𝑥1 + 7𝑥2 − 0.3𝑥3 = −19.30.3𝑥1 − 0.2𝑥2 + 10𝑥3 = 71.4

3 −0.1 −0.2 7.850.1 7 −0.3 −19.30.3 −0.2 10 71.4

Gauss-Jordan Method

• Then normalize the first row by dividing it by the pivot element, 3,

to yield.

• To eliminate 0.1 in row 2 column 1, multiply row 1 by (0.1/1) then

subtract the results from row 2

• Normalize the 2 row by dividing it by the pivot element, 7.00333,

to yield.

1 −0.0333333 −0.066667 2.616670.1 7 −0.3 −19.30.3 −0.2 10 71.4

1 −0.0333333 −0.066667 2.616670 7.00333 −0.293333 −19.56170.3 −0.2 10 71.4

1 −0.0333333 −0.066667 2.616670 1 −0.0418848 −2.793200.3 −0.2 10 71.4

Gauss-Jordan Method

• To eliminate 0.3 in row 3 column 1 multiply row 1 by (0.3/1) then

subtract the results from row 3.

• Eliminate elements in row 1 column 2 and row 3 column 2 using the

same elimination method to get

• Normalize row 3 by dividing it by 10.0120

1 −0.0333333 −0.066667 2.616670 1 −0.0418848 −2.793200 −0.190000 10.0200 70.6150

1 0 −0.0680629 2.523560 1 −0.0418848 −2.793200 0 10.01200 70.0843

1 0 −0.0680629 2.523560 1 −0.0418848 −2.793200 0 1 7.0000

Gauss-Jordan Method

• Eliminate elements in row 1 column 3 and row 2 column 3 using the

same elimination method to get

• Hence,

1 0 0 3.00 1 0 −2.50 0 1 7.0

𝑥1 = 3.0𝑥2 = −2.5𝑥3 = 7.0

Gauss-Jordan Method

• Exercises