Linear Systems of Equations Ax = b Marco Lattuada Swiss Federal Institute of Technology - ETH...

Linear Systems of EquationsLinear Systems of EquationsAx = bAx = b

Linear Systems of EquationsLinear Systems of EquationsAx = bAx = b

Marco Lattuada

Swiss Federal Institute of Technology - ETHInstitut für Chemie und BioingenieurwissenschaftenETH Hönggerberg/ HCI F135 – Zürich (Switzerland)

E-mail: lattuada@chem.ethz.chhttp://www.morbidelli-group.ethz.ch/education/index

Marco Lattuada

Swiss Federal Institute of Technology - ETHInstitut für Chemie und BioingenieurwissenschaftenETH Hönggerberg/ HCI F135 – Zürich (Switzerland)

E-mail: lattuada@chem.ethz.chhttp://www.morbidelli-group.ethz.ch/education/index

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 2

Definition of the ProblemDefinition of the Problem

We want to solve:

where x is the vector of the unknowns, while A and b are given.

Hypotheses:

1. The number of equations is equal to the number of unknowns

(that is, A is a square matrix)

1. The coefficients of A, b and x are real

2. The solution of the system exists and it is unique

We want to solve:

where x is the vector of the unknowns, while A and b are given.

Hypotheses:

1. The number of equations is equal to the number of unknowns

(that is, A is a square matrix)

1. The coefficients of A, b and x are real

2. The solution of the system exists and it is unique

( , ) ( ,1) ( ,1)n n n n A x b

• A-1 exists• A is not singular• A's columns are linearly independent• A's lines are linearly independent• det(A) is non-zero• rank(A) is equal to n• Ax = 0 only if x is a null vector

Analytical ApproachAnalytical Approach

Cramer’s rule (1750):

The solution of a system of equations:

Is given by:

where Ai is defined as follows:

Cramer’s rule (1750):

The solution of a system of equations:

Is given by:

where Ai is defined as follows:

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 3

det

deti

ix A

A

x b A

1,1 1,2 1, 1 1 1, 1 1,

2,1 2,2 2, 1 2 2, 1 2,

,1 ,2 , 1 , 1 ,

i i n

i i ni

n n n i n n i n n

a a a b a a

a a a b a a

a a a b a a

A

b replaces the ith column

Calculation of the determinantCalculation of the determinant

How to compute the determinant of a square matrix?

Laplace formula (1772):

where Ci,j is the cofactor of element ai,j. The cofactor Ci,j is the determinant of the submatrix obtained by removing the ith row and the jth column of the matrix, multiplied by (-1)i+j:

How to compute the determinant of a square matrix?

Laplace formula (1772):

where Ci,j is the cofactor of element ai,j. The cofactor Ci,j is the determinant of the submatrix obtained by removing the ith row and the jth column of the matrix, multiplied by (-1)i+j:

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 4

1, 1,1

det( )N

i ji

a C

A

, ,1 deti j

i j i jC M

1,1 1,2 1, 1 1, 1 1,

1,1 1,2 1, 1 1, 1 1,

,1,1 1,2 1, 1 1, 1 1,

,1

j j n

i i i j i j i n

i ji i i j i j i n

n

a a a a a

a a a a a

a a a a a

a

M

,2 , 1 , 1 , n n j n j n na a a a

No ith row

No jth column

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 5

Numerical approach: Gauss Elimination MethodNumerical approach: Gauss Elimination Method

Let us consider the system:

Let us consider the following operations:

1. I multiply one line by a constant

2. I substitute one line with a linear combination of the others

3. I operate a permutation of the lines

The result does not change

Let us consider the system:

Let us consider the following operations:

1. I multiply one line by a constant

2. I substitute one line with a linear combination of the others

3. I operate a permutation of the lines

The result does not change

1

2

3

3 2 1 1

1 2 2 5

1 1 1 2

x

x

x

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 6

Gauss Elimination MethodGauss Elimination Method

Numerical ExampleNumerical Example

• Multiply by -3Multiply by -3• Sum it to 1st lineSum it to 1st line• Multiply by -3Multiply by -3

• Sum it to 1st lineSum it to 1st line• Multiply by -4Multiply by -4• Sum it to 2nd lineSum it to 2nd line

Triangular SystemTriangular System

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 7

Gauss Elimination MethodGauss Elimination Method

General CaseGeneral Case

11 12 13

21 22 23

31 32 33

a a a

A a a a

a a a

I want to replace I want to replace aa2121 with a zero with a zero

I define the multiplier I define the multiplier ll2121::( ) ( )21 21 21 11( ) ( )21

21 22 22 21 1211 ( ) ( )

23 23 21 13

0new old

new old

new old

a a l aa

l a a l aa

a a l a

11 12 13 11 12 13( ) ( )

21 21 22 23 22 23

31 32 33 31 32 33

1 0 0

1 0 0

0 0 1

new new

a a a a a a

l a a a a a

a a a a a a

Note that:Note that:

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 8

Gauss Elimination MethodGauss Elimination Method

Gauss Transformation MatrixGauss Transformation Matrix

(0) (0) (0) (0) (0) (0)11 12 13 11 12 13

(0) (0) (1)(0) (0) (0) (0) (1) (1)21 21 22 23 22 23(0) (0) (0) (0) (1) (1)31 31 32 33 32 33

1 0 0

1 0 0

0 1 0

a a a a a a

l a a a a a

l a a a a a

M A A

where:where:( )

( )( )

kijk

ij kjj

al

a

(0) (0) (0) (0) (0) (0)11 12 13 11 12 13

(1) (1) (2)(1) (1) (1) (1)22 23 22 23

(1) (1) (1) (2)21 32 33 33

1 0 0

0 1 0 0 0

0 1 0 0 0

a a a a a a

a a a a

l a a a

M A A

Solution:Solution:( 1) ( 2) (1) (0) (0)

( 1) ( 1)

( 1) ( 2) (1) (0) (0)

n n

n n

n n

A M M M AA x b

b M M M b

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 9

Gauss Elimination MethodGauss Elimination Method

Numerical ExampleNumerical Example

n(n-1) operations (flops)n(n-1) operations (flops)(n-1)(n-2) operations (flops)(n-1)(n-2) operations (flops)

Total number of Total number of operations requiredoperations required

31 12

2 2( 1)

3

n n

j j

nj j j

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 10

Gauss Transformation MethodGauss Transformation Method

Let us change our point of view!

can be used to transform A

Gauss Elimination Method1. Changes the matrix A2. Needs the coefficient vector b3. Must re-run the method if b is changed

( 2) (1) (0)nM M M M

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 11

Gauss Transformation MethodGauss Transformation Method

Properties

• The final matrix A is a right triangular matrix

• The matrix M is a left triangular matrix

• The inverse of M is also a left triangular matrix

• The matrix L = M-1 has the simple form:

Properties

• The final matrix A is a right triangular matrix

• The matrix M is a left triangular matrix

• The inverse of M is also a left triangular matrix

• The matrix L = M-1 has the simple form:

1 (0)21(0) (1)31 31

1 0 0

1 0

1

l

l l

L M

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 12

Consider the following expression:

Let us multiply by L = M-1 both sides:

Consider the following expression:

Let us multiply by L = M-1 both sides:

LR (LU) FactorizationLR (LU) Factorization

( 2) (0)n A MA

Right triangular

(0)R MA

1 (0) (0) LR M MA A A LR

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 13

Starting matrix A is transformed (factorized) as:

Let us solve a linear system with a generic vector b:

Starting matrix A is transformed (factorized) as:

Let us solve a linear system with a generic vector b:

LR FactorizationLR Factorization

A LR

Ly bAx LRx b

Rx y

1. For every vector b, two simple triangular systems must be solved without factorizing again

2. The matrices LR can be stored using the elements of A3. If A is modified, it is often possible to modify L and R

accordingly without factorizing

Marco Lattuada – Statistical and Numerical Methods for Chemical EngineersSolution of Linear Systems of Equations – Page # 14

Starting matrix:

Consider the following system:

Consider the following similar system:

Starting matrix:

Consider the following system:

Consider the following similar system:

Problems of Gaussian Elimination and LRProblems of Gaussian Elimination and LR

(0) (0) (0)11 12 13

(0) (0) (0) (0)21 22 23(0) (0) (0)31 32 33

a a a

a a a

a a a

A Pivot value must be ≠ 0

2

1 2

1

0.4 0.3 0.1

x

x x

a11 = 0 I switch the lines x1 = 1 and x2 = 1

201 2

1 2

2 10 1

0.4 0.3 0.1

x x

x x

Manual LR FactorizationManual LR Factorizationwithout pivotingwithout pivoting