CHEE6330 LectureNotes Matrix Algebra(16)

7/23/2019 CHEE6330 LectureNotes Matrix Algebra(16)

http://slidepdf.com/reader/full/chee6330-lecturenotes-matrix-algebra16 1/113

CHEE 6330

Foundation of Mathematical Methods

in Chemical Engineering

Matrix Algebra

Lecture Notes

© Michael Nikolaou

Chemical & Biomolecular Engineering Department



CHEE 6330 Lecture Notes Matrix Algebra Michael Nikolaou

- 2 -

How To Use These Notes:

•

As the cover page suggests, this text is a set of Lecture Notes, NOT a textbook!

• A number of sources, in addition to the textbook, as well as the author’s personal experience

have served as basis.

• While certain topics covered in detail in the required textbook of the course are presented

rather telegraphically, others are elaborated on, particularly when they refer to material not

covered in the textbook.

•

In many places throughout the notes some space has been intentionally left blank, for the

student to understand a certain topic by being forced to fill in the missing material. That is

frequently done during lecture time. That is why lectures are important!

•

In many other places assignments are given for HomeWork Not To Hand In (HWNTHI). Please

make sure that you complete all of that!

•

The examples have been carefully selected to correspond to a variety of problems of interest to

the evolving nature of chemical engineering. While the emphasis in these examples is on

mathematical methods, the physical picture in the background is also important and should be

understood.

• There are three basic software tools used throughout: Matlab, Mathematica, and Excel. Each

does certain tasks particularly well, while being adequate for others. While the particular

software or programming language a student learns is not important, it is important to be

familiar with at least one basic computational tool, along with the mathematical and

programming principles of computation.

•

The code included with some examples is intentionally kept simple, to illustrate concepts.

Professional code is a lot more complicated, although the numerical recipe involved is usually

not very different.




- 3 -

Table of Contents

1. IN LIEU OF PREFACE ................................................................................................................................................... 4

2. SYSTEMS OF LINEAR ALGEBRAIC EQUATIONS ............................................................................................................ 5

2.1 WHY? ............................................................................................................................................................................. 5 2.2 DEFINITION ....................................................................................................................................................................... 5 2.3 SPECIAL MATRICES .............................................................................................................................................................. 5 2.4 MATRIX OPERATIONS .......................................................................................................................................................... 6 2.5 SYSTEMS OF LINEAR ALGEBRAIC EQUATIONS IN MATRIX FORM ..................................................................................................... 7 2.6 GEOMETRIC INTERPRETATIONS .............................................................................................................................................. 9 2.7 SCALAR, VECTOR, AND TENSOR OPERATIONS: A MATRIX APPROACH TO TRANSPORT PHENOMENA NOTATION ..................................... 11 2.8 SOLUTION METHODS FOR SYSTEMS OF LINEAR ALGEBRAIC EQUATIONS ........................................................................................ 23

2.8.1 Naïve Gauss elimination .................................................................................................................................... 24 2.8.2 The two phases of Gauss elimination ................................................................................................................ 25

2.8.3 Gauss elimination algorithm for Ax b ....................................................................................................... 27 2.8.4 Gauss elimination software ............................................................................................................................... 28 2.8.5 Time requirements for Gauss elimination.......................................................................................................... 30 2.8.6 Gauss elimination for equations with unknowns ..................................................................................... 32

2.8.7 LU decomposition .............................................................................................................................................. 34 2.8.8 Computation of the inverse of a matrix ............................................................................................................. 42 2.8.9 Computation of the determinant of a matrix .................................................................................................... 46 2.8.10 Numerical precision and pivoting ................................................................................................................. 48 2.8.11 Numerical precision and matrix conditioning ............................................................................................... 52 2.8.12 Matrix condition number .............................................................................................................................. 56

2.9 MATRIX RANK ................................................................................................................................................................. 66 2.9.1 Linear independence .......................................................................................................................................... 66 2.9.2 Basic facts about matrix rank ............................................................................................................................ 68 2.9.3 Matrix rank and solution of systems of linear equations .................................................................................. 69

3. EIGENVALUE/EIGENVECTOR PROBLEMS .................................................................................................................. 74

3.1 REVIEW

.......................................................................................................................................................................... 74 3.1.1 Solution of polynomial equations ...................................................................................................................... 80 3.1.2 Software for polynomial equations ................................................................................................................... 80 3.1.3 Software for eigenvalues ................................................................................................................................... 80

3.2 WHY EIGENVALUES AND EIGENVECTORS? .............................................................................................................................. 81 3.3 Q UADRATIC FORMS .......................................................................................................................................................... 91

3.3.1 Visualize quadratic forms .................................................................................................................................. 94 3.3.2 Optimization with quadratic forms and linear inequalities (quadratic programming) ..................................... 97 3.3.3 Quadratic programming in general ................................................................................................................. 101

4. SINGULAR-VALUE DECOMPOSITION (SVD) ............................................................................................................. 104

4.1 WHAT IS SINGULAR-VALUE DECOMPOSITION (SVD)? ............................................................................................................ 104 4.2 WHY SVD? .................................................................................................................................................................. 105

4.3 SOFTWARE FOR SVD ...................................................................................................................................................... 105 Notation:

Uppercase, boldface: Matrices. e.g. M Lowercase, boldface: vectors. e.g. v Lowercase, italics: scalars. e.g. f

m n




- 4 -

1. IN LIEU OF PREFACE

In November 1964 Richard P. Feynman (1918-1988), one of the most brilliant theoretical physicists,

Nobel laureate, and distinguished educator, was invited to deliver the Messenger Lectures at Cornell

University. This is an excerpt from his lectures taken from "The Character of Physical Law" by Richard

P. Feynman, MIT Press, 19671:

To summarize, I would use the words of Jeans, who said that "the Great Architect

seems to be a mathematician". To those who do not know mathematics it is difficult to get

across a real feeling as to the beauty, the deepest beauty, of nature. C.P. Snow talked about

two cultures. I really think that those two cultures separate people who have and people who

have not had this experience of understanding mathematics well enough to appreciate nature

once.

It is too bad that it has to be mathematics, and that mathematics is hard for some

people. It is reputed - I do not know if it is true - that when one of the kings was trying to

learn geometry from Euclid he complained that it was difficult. And Euclid said, "There is no

royal road to geometry". And there is no royal road. Physicists cannot make a conversion to

any other language. If you want to learn about nature, to appreciate nature, it is necessary to

understand the language that she speaks in. She offers her information only in one form; we

are not so unhumble as to demand that she change before we pay any attention.

All the intellectual arguments that you can make will not communicate to deaf ears

what the experience of music really is. In the same way all the intellectual arguments in the

world will not convey an understanding of nature to those of "the other culture". Philosophers

may try to teach you by telling you qualitatively about nature. I am trying to describe her.

But it is not getting across because it is impossible. Perhaps it is because their horizons are

limited in the way that some people are able to imagine that the center of the universe is man.

Also of interest on the subject:

Eugene Wigner, "The Unreasonable Effectiveness of Mathematics in the Natural Sciences,"

Communications in Pure and Applied Mathematics, vol. 13, No. I (February 1960).2

1 http://www.physicsteachers.com/pdf/The_Character_of_Physical_Law.pdf 2 http://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html

http://www.physicsteachers.com/pdf/The_Character_of_Physical_Law.pdf



http://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html








- 5 -

2. SYSTEMS OF LINEAR ALGEBRAIC EQUATIONS

2.1 Why?

a) Inherent in ChE problems (EXAMPLE 2, below)

b)

Many mathematical problems (analytical or numerical) reduced to linear algebraic equations

c)

Visit www.netlib.org.

2.2 Definition

11 12 13 1

21

1 2

rowˆ ˆ

|

|

column

n

ij

m m mn

a a a a

a

a A A m i

a a a

n

j

A

dim( ) m n A , m n A

2.3 Special matrices

− Symmetric

− Diagonal

−

Identity

− Upper triangular

− Lower triangular

−

Banded

−

Row matrix

− Column matrix

HWNTHI: Explore special matrix construction commands in Matlab, by typing

> hel p t oepl i t z> hel p hankel

> hel p eye> hel p zeros> hel p ones> hel p di ag

HWNTHI: Explore special matrix construction commands in Mathematica, such as the following.

Toepl i t zMat r i x, Hankel Mat r i x, I dent i t yMat r i x, Di agonal Mat r i x, Band.

http://www.netlib.org/







- 6 -

2.4 Matrix operations

1

:

, ( ) ( )

:

:

ij ij ij

ij ij

n

ij ik kj k

c a b

g b ga

c a b

C A B

A B B A A B C A B C

B A

C AB

[ ] [ ] [ ]m l m n n l

C A B

:T

ij ji a a A

( )

( )

T T T

T T T

A B A B

AB B A

( ) ( )AB C A BC ( Associative)

( ) , ( ) A B C AB AC A B C AC BC (Distributive, both left and right)

But

AB BA (Non-commutative)

1 1 1: A AA A A I 1 1 1( ) AB B A

1

trace( ) ˆn

ii i

a

A

B

A=

j

i

j

C

i




- 7 -

2.5 Systems of linear algebraic equations in matrix form

EXAMPLE 1 – VECTORS AND VECTOR OPERATIONS

2 3

4 5 6 15

7 8 15

w y

w y z

y z

1 2 0 3

4 5 6 15

0 7 8 15

w

y

z

xA b

1 Ax b x A b

or

1[ ] { } { } { } [ ] { }A x b x A b

11 1 12 2 1 1

21 1 22 2 2 2

1 1 2 2

.....

.....

.....

n n

n n

n n nn n n

a x a x a x b

a x a x a x b

a x a x a x b




- 8 -

EXAMPLE 2 – MASS BALANCE VIA SYSTEM OF LINEAR ALGEBRAIC EQUATIONS

Fi gur e 1 – React i on- separat i on pr ocess uni t wi t h r ecycl i ng

Process data

100% conversion of A in R1

B4/B5 = 0.01; (moles)

D4/D5 = 100.

Material balances

Mixer:

A1 – A2 = 0 (1)

B1 + B5 – B2 = 0 (2)

D5 – D2 = 0 (3)

Reactor:

B3 – B2 + A2 = 0 (4)

D3 – D2 – A2 = 0 (5)

Separator:

D4 – 100D5 = 0 (6)

B4 – 0.01B5 = 0 (7)

B3 – B4 – B5 = 0 (8)

D3 – D4 – D5 = 0 (9)

11 Unknowns: A1, B1, A2, B2, D2, B3, D3, D4, B4, B5, D5

9 Equations

To be able to solve, must specify feed (A1, B1) or product (B4, D4)

HWNTHI: Write and solve the above equations in vector/matrix form Ax b .

Mixer M1A1B1

Reactor R1A B D

A2B2

D2

B3

D3

S e p a r a t o r C 1

B4D4

B5D5




- 9 -

2.6 Geometric interpretations

11 1 1

1 1

1

1

n

n n

m mn n

n

a a x

x x

a a x

a a

a a

(10)

EXAMPLE 3 – MATRIX-VECTOR MULTIPLICATION

1 2 3 1 2 3

1 2 0 1 2 0 1 2 0

4 5 6 4 5 6 4 5 6

0 7 8 0 7 8 0 7 8

w

y w y z w y z

z

xA a a a a a a

EXAMPLE 4 – MATRIX-VECTOR MULTIPLICATION: SPECIAL CASE

1 1

1 1 1 1 1 1

1 1

1 0 1 1

0 0 0 0 0 0

0 1 1 1n n n n n n

n n

n n

x x

x x x x x x x x

x x

e e e e

x I x e e e e

1{ , ..., }

n e e called standard unit vectors in

n .




- 10 -

EXAMPLE 5 – MATRIX-VECTOR MULTIPLICATION IN 3D EUCLIDEAN SPACE: IMPORTANT SPECIAL CASE

1 0 0 1 0 0

0 1 0 0 1 0

0 0 1 0 0 1

x x

y y x y z x y z

z z

i j k

v I v i j k

(11)

Fi gur e 2 – Gr aphi cal r epr esent ati on of uni t vect ors i n 3D Eucl i dean space.

{ , , }i j k (or1 2 3

{ , , }e e e or1 2 3

{ , , } or…) called standard unit vectors in3 (3D Euclidean space).

Incidentally: What is a space?

Definition 1 – Vector space

(Vector or linear) space is a set, V , associated with

(a) a field F (usually real or complex numbers);

(b) two operations:

addition of elements ,v w in V : v w , following the properties of addition of numbers, and

scalar multiplication of v in the set V by f in the field F ;

(c) two special but very simple properties: Every addition (sum v w of any two elements ,v w in V )

and every scalar multiplication (product fv of any v in V and any f in F ) results in an element in .V

EXAMPLE 6 – SETS THAT ARE OR ARE NOT SPACES

• Is 3 space over the field of reals ?

o Yes. Addition and scalar multiplication results in element of 3 .

• Is 3

1 2 3 1{[ ] | 1 1}T

V v v v v space over the field of reals ?

o No. Not every addition and scalar multiplication results in element of V .

1

1

0ˆ ˆ

0

i

2

01ˆ ˆ

0

j

1

1

3

0

0ˆ ˆ

1

k

x

y

z

1

0




- 11 -

2.7 Scalar, vector, and tensor operations: A matrix approach to transport phenomena notation

- What are scalars, vectors, and tensors?

o For the purposes of transport phenomena

Fi gur e 3 – Scal ar , ( col umn) vect or , and ( t wo- ) t ensor .

-

What are they good for?

o

Make life (thinking, modeling, computation, calculation) a lot easier

EXAMPLE 7 – THE INNER (SCALAR, DOT) PRODUCT

vw not well defined (except as dyadic product – see EXAMPLE 9 – The Dyadic Product, below).

Inner (or scalar or dot) product of real vectors ,v w :

ˆ ˆT T

i i i

v w v w w v v w (12)

Cauchy-Schwarz inequality3

2 2

1 1

v w

v w for any two vectors ,v w

2 2cos v w v w (13)

Geometric interpretation in Euclidean space (Figure 4): is the angle between ,v w .

Fi gur e 4 – Gr aphi cal r epr esent ati on of i nner product of t wo vect ors i n Eucl i dean space.

Is v w w v ?

Is ( ) ( ) z v w zw v ? (Hint: ( ) ( )T z v w z v w

3 Simple proof of Cauchy-Schwarz inequality: 20 ( ) ( ) ( ) (2 )c c c c v w v w v v v w w w for all c except

for c v w 0 Discriminant 0 2( ) ( )( ) 0 v w v v w w , with equal sign for c v w ( ,v w aligned).

cos v

w

v




- 12 -

EXAMPLE 8 – INNER (SCALAR, DOT) PRODUCT OF TWO VECTORS

1

2 1 2 3

3

T

v v ,

4

5 4 5 6

6

T

w w

4

1 2 3 5 32ˆ

6

T

i i i

v w

v w v w inner (or scalar or dot) product for real vectors ,v w .

EXAMPLE 9 – INNER PRODUCT OF UNIT VECTORS

1 1 1 1 1ˆ T e e e e (

1 1 1 1 1ˆ T )

1 2 1 2 0ˆ

T

e e e e ( 1 2 1 2 0ˆ

T

)General formula in terms of Kronecker delta,

ij :

1,ˆ

0,T

i j i j ij

i j

i j

(HWNTHI: Confirm.)

EXAMPLE 10 – THE DYADIC PRODUCT

1 4 5 6

2 4 5 6 8 10 12ˆ

3 12 15 18

T

vw vw often called dyadic product for real vectors ,v w .

Is vw wv ?

What is ( )T wv in terms of ,w v ?

Even though the dyadic product uses the same notation as the ordinary product between two vectors,

it is entirely different. Misinterpretation of one product for the other is a frequent source of errors.

It is a safe practice to always interpret the dyadic product vw as T vw and proceed with subsequent

calculations using standard vector/matrix algebra.




- 13 -

EXAMPLE 11 – THE CROSS PRODUCT: A SPECIAL PRODUCT IN 3D EUCLIDEAN SPACE

2 3 3 2

1 2 3 2 3 3 2 3 1 1 3 1 2 2 1 3 1 1 3

1 2 3 1 2 2 1

( ) ( ) ( )ˆ

v w v w

v v v v w v w v w v w v w v w v w v w

w w w v w v w

i j k

v w i j k

Is v w w v ?

v w is orthogonal to both v and w , because

( ) _______________________ 0 v w v

( ) _______________________ 0 v w w

(HWNTHI: Verify.)

Geometric interpretation in 3D Euclidean space.

Fi gur e 5 – Gr aphi cal r epr esent ati on of cr oss product i n 3D Eucl i dean space.

EXAMPLE 12 –CROSS PRODUCT OF UNIT VECTORS

1 2 3

1 2 1 2 3 3

0

1 0 0 (0 0) (0 0) (1 0) 0ˆ

0 1 0 1

1 2 3

1 1 1 2 3

0

1 0 0 (0 0) (0 0) (0 0) 0ˆ1 0 0 0

General formula in terms of permutation epsilon:3

1i j ijk k k

where (the Levi-Civita is)

1, 123, 231, 312

1, 321, 132, 213

0, any two indices equalijk

ijk

ijk

2 3 3 2

3 1 1 3 2 2

1 2 2 1

sin

v w v w

v w v w

v w v w

vw v w

x y

z

1

2

3

v

v

v

1

2

3

w

w

w




- 14 -

EXAMPLE 13 – A SPECIAL VECTOR IN 3D EUCLIDEAN SPACE: THE DEL OPERATOR AND ASSOCIATED CONSTRUCTS

1 0 0 1 0 0

0 1 0 0 1 0ˆ

0 0 1 0 0 1

x x

y y

z z

x y z x y z

i j k

I i j k

(14)

Note:x y z

i j k unless partial derivatives are understood not to act on , ,i j k .

Misinterpretation leads to errors.

Gradient of scalar f :

1 0 0 1 0 0

0 1 0 0 1 0ˆ

0 0 1 0 0 1

f f

x x

f f

y y

f f

z z

f f f f f f f

x y z x y z

f

i j k

I i j k

(15)

Is f f ?

Divergence of vector v :

1

1 2 32

3

ˆ T

x y z

T

v v v v

v x y z

v

v v

v

(16)

Is v v ?

Gradient of vector v :

1

2

3

x

y

z

v

v

v

v not well defined (unless understood as the dyadic product ).

1 2 3

1 2 3

1 2 3

1 2 3

v v v

x x x x v v v T

y y y y

v v v

z z z z

v v v

v (17)




- 15 -

Curl or rot of vector w in 3D Euclidean space ( 3 ):

3 2

1 3

2 1

1 2 3

3 2 1 3 2 1

ˆ

( ) ( ) ( )

x y z

y z z x x y w w

y z

w w

z x

w w

x y

w w w

w w w w w w

i j k

w

i j k (18)

Note: The determinant expression for w contains some abuse of notation, in that the

determinant can only be expanded as shown in the above equation. Alternative expansions, e.g. along

the second or third row or along columns of the matrix

1 2 3

x y z

w w w

i j k

would lead to erroneous results.

EXAMPLE 14 – DOT PRODUCT OF VECTOR AND MATRIX (RECALL EQN. (12), DOT PRODUCT BETWEEN VECTORS)

( )T T T v A A v v A (19)

1 2 3

1 2 3

1 2 3

1 1 1

1 2 3 1 2 3 1 2 3

1 2 3 1

( ) [ ] [ ] [ ]

T T v v v

x x x x v v v T T T T

y y y y

v v v z z z z

v v v

x y z

v v v v v v v v v

v v v v

v v v v

2 2 2 3 3 3

1 1 1

2 2 2

3 3 3

2 3 1 2 3

1 2 3

1 2 3

1 2 3

T v v v v v v

x y z x y z

v v v

x y z

v v v

x y z

v v v

x y z

v v v v v

v v v

v v v

v v v

or

1 1 1 2 2 2 3 3 3

1 2 3 1 2 3 1 2 3 1 2 3

1 2 3 1 2 3 1 2 3

( ) [ ] [ ] [ ][ ]

...

T

x T T T T T

y x y z

z

T v v v v v v v v v

x y z x y z x y z

v v v v v v v v v v v v

v v v v v v v v v

v v v v

or




- 16 -

1 2 3 1 1 1 1 1 1

1 2 3 2 2 2

1 2 3 3 3 3

1 2 31 1

2 2

3 3

( )

T v v v v v v v v v

x x x x y z x y

v v v v v v T T T

y y y x y z

v v v v v v

z z z x y z

v v v v v

v v

v v

v v v v 2 2 2

3 3 3

1 2 3

1 2 3

z

v v v

x y z

v v v

x y z

v v v

v v v

vv not well defined (unless vv is interpreted as dyadic product ).

1 1

1

2 1 2 3

3

1 1 1 2 1 3

2 1 2 2 2 3

3 1 3 2 3 3

( ) ( ( )) [ ] [ ]

[ ]

T

T T T T

x y z

T

x y z

v v

x

v

v v v v

v

v v v v v v

v v v v v v

v v v v v v

vv vv

2 1 3 1 1 2 2 2 3 2 1 3 2 3 3 3

1 1 2 1 3 1

1 2 2 2 3 2

1 3 2 3 3 3

T v v v v v v v v v v v v v v v v

y z x y z x y z

v v v v v v

x y z v v v v v v

x y z v v v v v v

x y z

Is ( ) ( )T T T T vv v v ?




- 17 -

EXAMPLE 15 – A SPECIAL VECTOR: THE LAPLACE OPERATOR

2 ˆ (20)

Is 2 ?

Laplacian of scalar f :

2 2 22

2 2 2[ ] [ ]ˆ

f

x x f

x y z y x y z y

f

z z

f f f f f f

x y z

or

2 2 2 2 2 22

2 2 2 2 2 2[ ]ˆ

x

x y z y

z

f f f f f f f

x y z x y z

Laplacian of vector v :2 v v not well defined (unless the concept of dyadic product is used).

1 2 3

1 2 3

1 2 3

1 2 3

1 2 3

[ ]

[ ] [ ]

[ ]

x x

T

y y

z z

T

x

x y z y

z

v v v

x x x

v v v

x y z y y y

v v v

z z z

v v v

v v v

v

2 2 2 2 2 2 2 2 21 1 1 2 2 2 3 3 3

2 2 2 2 2 2 2 2 2

2 2 21 1 1

2 2 2

2 2 22 2 2

2 2 2

2 2 23 3 3

2 2 2

T

T v v v v v v v v v

x y z x y z x y z

v v v

x y z

v v v

x y z

v v v

x y z

or




- 18 -

2 2 2

2 2 2

2 2 21 1 1

2 2 2

2 2 22 2

2 2

1 2 3

1 2 3

1 2 3

[ ]

[ ] [ ]

( )[ ]

x x

T

y y

z z

T

x

x y z y

z

T

x y z

v v v

x y z

v v v

x y

v v v

v v v

v v v

v

2

2

2 2 23 3 3

2 2 2

z

v v v

x y z

Note: Cannot write1 2 3

1 2 3

1 2 3

1 2 3[ ]

T v v v

x x x x x x v v v T

y y y y y y

v v v

z z z z z z

v v v

v

(Why?)




- 19 -

EXAMPLE 16 – VECTOR/MATRIX CALCULATIONS WITH THE STRESS TENSOR

Think of the stress tensor as a matrix

ˆxx xy xz

yx yy yz

zx zy zz

(21)

or

11 12 13

21 22 23

31 32 33

ˆ

(22)

EXAMPLE 17 – MATRIX REPRESENTATION IN TERMS OF UNIT VECTORS

11 12 13

21 22 23

31 32 33

11 12 33

11 12

1

1 1

ˆ

1 0 0 0 1 0 0 0 0

0 0 0 0 0 0 ... 0 0 0

0 0 0 0 0 0 0 0 1

1 1

0 1 0 0 0 0 1 0

0 0T

33

2 3

33 3

1 1

0

... 0 0 0 1

1T T

T

ij i j i j

(23)

(Note: T

i j is standard vector/matrix notation of dyadic product

i j .)




- 20 -

EXAMPLE 18 – THE DOUBLE-DOT PRODUCT BETWEEN TENSORS (MATRICES)

: îj j i

i j

a b A B (24)

It can be shown that

: tr( ) tr( ) :̂ A B AB BA B A (25)

EXAMPLE 19 – DOUBLE-DOT PRODUCT FOR STRESS TENSOR

v not well defined (unless v is dyadic product)

1

2

3

1 2 3

1 1 1

1 2 3

2 2 2

1 2 3

3 3 3

11 12 13

21 22 23 1 2 3

31 32 33

11 12 13

21 22 23

31 32 33

:ˆ

:

x

T

x

x

v v v

x x x

v v v

x x x v v v

x x x

v v v

v

1 2 3

1 1 1

1 2 3

2 2 2

1 2 3

3 3 3

11 12 13

21 22 23

31 32 33

tr

i

j

v v v

x x x

v v v

x x x

v v v

x x x

v

ij x i j




- 21 -

EXAMPLE 20 – DOUBLE-DOT PRODUCT FOR UNIT VECTORS IN 3D EUCLIDEAN SPACE

BSL eqns. (A.3-1) – (A.3-6)4

3

13

1

( : ) ( )( )

] ( )

] ( )] ( )

] ]

] [ ]

i j k l j k i l jk il

i j k i j k i jk

i j k i j k ij k

i j k l i j k l jk i l

i j k i j k jkl i l l

i j k i j k ijl l k l

(26)

Translation into standard vector/matrix notation

3 3

1 13

1

: tr(( )) tr( ( ) ))

( )

( )

( )

( )

T T T T T T

i j k l i j k l i j k l jk il

T T

i j k i j k i jk

T T T T

i j k i j k ij k

T T T T T

i j k l i j k l jk i l

T T T

i j k i jkl l jkl i l l l

T

i j k ijl l l

3

1

T T

k ijl l k l

(27)

4 Note typo in eq. (A.3-4): extra .




- 22 -

EXAMPLE 21 – THE EQUATION OF MOTION

BSL eqn. (3.2-7), p. 80

[ ]t

v g (28)

Eqn. (28) in standard vector/matrix notation

( )

[ ]

x

y

z

yx xx zx

xy yy zy

xz

T T

t

T v

t xx xy xz x v

yx yy yz y t x y z

v

zx zy zz z t

x y z

x y z

g

g

g

v g

yz zz

x

y

z x y z

g

g

g

(29)

BSL eqn. (3.2-9), p. 80

[ ] [ ]t

p

v vv g (30)

Eqn. (30) in standard vector/matrix notation

( ( )) ( )

y x y x x x z x xx

y x y y y z y

y z z x z z z

T T T

t

v v v v v v v pt x y z x x v v v v v v v p

t x y z y

v v pv v v v v

z t x y z

p

v vv g

x zx

xy yy zy

yz xz zz

y z x

y x y z

z x y z

g

g

g

(31)

EXAMPLE 22 – THE EQUATION OF MECHANICAL ENERGY

BSL eqn. (3.3-2), p. 812 21 1

2 2ˆ ˆ( ) ( ( ) ) ( ) ( ) ( [ ]) ( : )

t v v p p

v v v v v (32)

Eqn. (32) in standard vector/matrix notation2 21 1

2 2ˆ ˆ( ) (( ) ) ( ) ( ) : T

t v v p p

v v v v v (33)




- 23 -

2.8 Solution methods for systems of linear algebraic equations

Two categories of methods

Direct : Gauss elimination

For dense matrices

Iterative: Gauss-Seidel (successive over-relaxation)

For large sparse matrices ( 510n )

(Mostly resulting from other problems,e.g. ODE, PDE, optimization)

Will not be covered in this course




- 24 -

2.8.1 Naïve Gauss elimination

EXAMPLE 23– NAÏVE GAUSS ELIMINATION

3 5

1 2 3 4 9

5 7 9 21

x y z

S x y z

x y z

2 2 6 10

2 12 3 4 9

x y z y z

x y z

5 5 15 252 6 4

5 7 9 21

x y z y z

x y z

Thus:

3 5

2 2 1

2 6 4

x y z

S y z

y z

2 4 2

2 22 6 4

y z z

y z

Therefore:

3 5 3.1

3 2 1 3.2

2 2 3.3

x y z S

S y z S

z S

3.2 3.1

3.3 1 2 1 1 1 3 5 1

S S

S z y y x x

2

5

2




- 25 -

2.8.2 The two phases of Gauss elimination

11 12 13 1

21 22 23 2

31 32 33 3

11 12 13 1

22 23 2

33 3

Forwardelimination

0

0 0

a a a b

a a a b

a a a b

a a a b

a a b

a b

33

33

2 23 3

222

1 12 2 13 31

11

Back-substitution

bx

a

b a x x

a b a x a x

x a




- 26 -

EXAMPLE 24 – EXAMPLE 22 IN MATRIX NOTATION

1 1 3 5

2 3 4 9 2

5 7 9 21 5

1 1 3 5

0 1 2 1

0 2 6 4 2

1 1 3 5

0 1 2 1

0 0 2 2




- 27 -

2.8.3 Gauss elimination algorithm for Ax b

1st phase: Forward elimination for [ | ]A b

For 1,..., 1

1,...,

, 1,..., , 1,...,

, 1,...,

ik

i kk

ij ij i kj

i i i k

k n

a m i k n

a a a m a i k n j k n

b b m b i k n

Note: If 0kk

a , switch rows first, to make 0kk

a .

2nd phase: Back–substitution

1 , 1,...,1

n

n

nn n

k kj j j k

k

kk

bx

a

b a x

x k n a




- 28 -

2.8.4 Gauss elimination software

• Matlab>> A = [ 1 2; 3 4] ;>> b = [ 5; 6] ;>> x = A\ b

•

Mathematica A = {{1, 2}, {3, 4}};b = {{5}, {6}};x = Li near Sol ve[ A, b]

• Fortran, C

The web site www.netlib.org is a great repository of FORTRAN, C and C routines for numerical computation,

available free of charge. There is a taxonomy of the numerical computations performed by corresponding

routines at http://www.netlib.org/bib/gams.html. On that page you may click on D2a1 to go to a collection

of routines used for the Solution of systems of linear equations Ax b corresponding to General matrices

A. Scroll to the bottom of the page and go to Results Page 3. On that page click on toms/576, which takes you

to the desired final page, namely http://netlib3.cs.utk.edu/toms/576. This page contains

a. 10 main programs that can be used to test the subsequent subroutines;

b. The subroutines

- MODGE: Solves Ax b for general matrices A, using Gaussian elimination.

- OUTPT1: Shows the solution

solx of Ax b .

- RESID1: Shows the residualssol

Ax b .

- REFINE: Performs iterative refinement of the solutionsol

x of Ax b .

- OUTPT2: Shows the iteratively refined solutionref

x of Ax b .

- RESID2: Shows the residuals

ref Ax b .

c. Data for tests 1 and 4.

You may cut and paste the entire text into a text file. The Gaussian elimination solver is contained between the

lines

SUBROUTI NE MODGE( N, NDI M, A, B, EPS, TOLER, P, Q, Z, MOD 10

and

99995 FORMAT ( E18. 7)END

To test the above solver you may use the main program in Test 2, which does not require and external

data. After you have compiled and run this test program, you are ready to solve your linear system of

equations.




http://www.netlib.org/bib/gams.html



http://netlib3.cs.utk.edu/cgi-bin/search.pl?query=gams/D2a1%2A



http://netlib3.cs.utk.edu/cgi-bin/search.pl?query=gams/D2%2A






http://netlib3.cs.utk.edu/toms/576
















- 29 -

• Excel: Use Solver

EXAMPLE 25 – SOLUTION OF LINEAR EQUATIONS AS MINIMIZATION OF A Q UADRATIC FUNCTION

EXAMPLE 22 in Excel.

B C D E F G H

2 A x b Ax Ax-b

3 1 1 3 1.00034436147488 5 =MMULT(B3:D3,E$3:E$5) =G3-F3

4 2 3 4 0.999839433693534 9 =MMULT(B4:D4,E$3:E$5) =G4-F4

5 5 7 9 0.999935570349651 21 =MMULT(B5:D5,E$3:E$5) =G5-F5

6 SSE = =SUMSQ(H3:H5)

B C D E F G H

2 A x b Ax Ax-b

3 1 1 3 1.0003 5 1 1

4 2 3 4 0.9998 9 2 3

5 5 7 9 0.9999 21 5 7

6




- 30 -

2.8.5 Time requirements for Gauss elimination

# of the required divisions and multiplications:

1st phase (Forward elimination)

For 1, ..., 1k n we need

n k divisions fori

m

2( )n k multiplications forij

a

n k multiplications fori

b

Given that1

1

( 1)

2

n

k

n n k

and

12

1

( 1) (2 1)

6

n

k

n n n k

we get that total # for phase 1 is

2( 1) ( 1)

3 2

n n n n

2nd phase (Back-substitution)

For eachk

x we needmultiplications

1 division

n k

, i.e.( 1)

2

n n total.

Thus, grand total is

32

3 3

n n n (34)

If n is large, the total number of operations is

3( )O n (35)

For 100n in eqn. (34) we get 343,300 multiplications.

By comparison, Cramer’s method (HWNTHI: What is Cramer's method?) requires

( 1)! 2

n

n n n

e

(36)

(HWNTHI: Search Stirling formula.)

For 100n in eqn. (36) we get

100

15810025.1 10

2.718282

multiplications

If a fast PC can do 109 floating-point operations per second (FLOPS) how long would it take to solve a

linear system of 100 equations using each approach?




- 31 -

HWNTHI: Solve

1

2

3

4

1 2 3 4 10

5 6 7 8 26

9 10 11 12 42

x

x

x

x

1

2

3

4

1 2 3 4 10

5 6 7 8 26

9 10 11 12 41

x

x

x

x




- 32 -

2.8.6 Gauss elimination for equations with unknowns

11 1 1 1

21 1 2 2

1 1

...

...

.

.

.

.

...

n n

n n

m mn n m

a x a x b

a x a x b

a x a x b

(37)

Apply Gauss elimination to bring [ | ]A b to row echelon (upper triangular) form and determine

whether eqn. (37) has

a)

No solution

b)

Infinite solutions

c) Unique solution

Visually:

m n




- 33 -

Using equations:

-

Initially we have three cases:

Case 1:1

0, 1, ..., j

a j n and1

0b . Then1

0 0b , contradiction no solution, STOP.

Case 2:1

0, 1, ..., j

a j n and1

0b . Then1

0 0b . Interchange this equation with the last one

and examine again the 3 cases.

Case 3:1

0 j

a for a certain j . Then interchange columns so that1 j

a comes in the first column.

- Do forward elimination for the column of1 j

a .

- Examine the above 3 cases for the ( 1) ( 1)m n subsystem.

By repeating the previous procedure we result in one of the following:

a) encounter an unsolvable equation (nonzero = 0)

b) transform eqn. (37) to

1 2

2

(1) (1) (1) (1) (1)

11 12 1 1 1

(2) (2) (2) (2)21 2 2 2

( ) ( ) ( )

... ...

... ...

...

...

0 0

...

0 0

r n

r n

r n

j j r j n j

j r j n j

r r r

rr j rn j r

a x a x a x a x b

a x a x a x br

a x a x b

m r

r n

(38)

Because r n , eqn. (38) is indeterminate.

c) Eqn. (38) has r n . Then, there exists a unique solution.

EXAMPLE 26 – HWNTHI

Ax b Ax c

1 2 3 0

4 5 6 3

7 8 9 0

A b

(No solution)

0 2 1

3 1 2

6 0 1

d

c x

(Infinite solutions)

-

Verify with Mathematica:LinearSolve[{{1,2,3}, {4,5,6}, {7,8,9}}, {0,3,0}] yields LinearSolve[{{1,2,3}, {4,5,6}, {7,8,9}}, {0,3,0}]

Solve[{{1,2,3}, {4,5,6}, {7,8,9}}. {x1, x2, x3} == {0,3,0},{x3, x2, x1}] yields {}

LinearSolve[{{1,2,3}, {4,5,6}, {7,8,9}}, {0,3,6}] yields {2, −1,0}

Solve[{{1,2,3}, {4,5,6}, {7,8,9}}. {x1, x2, x3} == {0,3,6},{x3, x2, x1}] yields

{{x2 → −1 − 2x3, x1 → 2 + x3}}




- 34 -

2.8.7 LU decomposition

• What :

Theorem 1 – LU decomposition of matrix A

A LU (39)

• Why : Solve series of systems of equations with same matrix A , particularly when A is sparse (e.g.

tridiagonal).

( )

fixed

i A x b , 1,2, 3,...i

•

How to use LU:

easy to solve for (Why?) Then...

easy to solve for (Why?)

Ld b dAx b L Ux b

Ux d xd

• How to find matrices ,L U : Gauss elimination!

* * *

* * * *

* * * *

* * *

0 * *

0 * * *

* * *

0 * *

0 0 *

U

1 0 0

* 1 0

* * 1

L

A

L

U




- 35 -




- 36 -

EXAMPLE 27 – LU DECOMPOSITION

1 1 3

2 3 4 2

5 7 9 5

1 1 3

0 1 2

0 2 6 2

1 1 3

0 1 2

0 0 2

1 1 3 1 0 0

0 1 2 , 2 1 0

0 0 2 5 2 1

U L

HWNTHI: Check that LU A . Solve Ax b by solving Ld b , then Ux d .




- 37 -

- More formally:

Definition 2 – Elementary row operations (ERO)

row row

row row ( 0)

row row row

i j

i i

j j i

c c

c

Definition 3 – Elementary column operations (ECO)

column column

column column ( 0)

column column column

i j

i i

j j i

c c

c

Definition 4 – Elementary matrix

A matrix resulting from performing an ERO (ECO) on the identity matrix.

EXAMPLE 28 – ELEMENTARY MATRICES

1 2

0 1 0

row row : 1 0 0

0 0 1

2 2

1 0 0

row 2 row : 0 2 0

0 0 1

2 2 1

1 0 0

row row 2 row : 2 1 0

0 0 1

Similar matrices for columns.




- 38 -

Theorem 2 – Inverse of elementary matrix

The inverse of every elementary matrix is also an elementary matrix.

EXAMPLE 29 – ELEMENTARY MATRIX INVERSE

1

1 0 0 1 0 02 1 0 2 1 0

0 0 1 0 0 1

1

0 1 0 0 1 0

1 0 0 1 0 0

0 0 1 0 0 1

1

1 0 0 1 0 0

0 1 / 2 0 0 2 00 0 1 0 0 1




- 39 -

Theorem 3 – Elementary row (column) operations through elementary matrices

The result of each ERO (ECO) on a matrix A can be represented as the product EA ( AE ), where E is

an elementary matrix.

EXAMPLE 30 – ELEMENTARY MATRICES

1 2

1

0 1 0 1 1 3 2 3 4

row row : 1 0 0 2 3 4 1 1 3

0 0 1 5 7 9 5 7 9

A A

2 2

1

1 0 0 1 1 3 1 1 3

row 2 row : 0 2 0 2 3 4 4 6 8

0 0 1 5 7 9 5 7 9

A A

2 2 1

1

1 0 0 1 1 3 1 1 3

row row 2 row : 2 1 0 2 3 4 4 5 10

0 0 1 5 7 9 5 7 9

A A

Similar idea for ECO (multiplication from the right).




- 40 -

EXAMPLE 31 – LU DECOMPOSITION USING ELEMENTARY MATRICES

For A as in EXAMPLE 26:

1

1 1

1 0 0 1 1 3 1 1 3 1 1 3 1 0 0 1 1 3

2 1 0 2 3 4 0 1 2 2 3 4 2 1 0 0 1 2

5 0 1 5 7 9 0 2 6 5 7 9 5 0 1 0 2 6

1 0 0

0 1 0

0 2 1

E A E

12 2

1 1 3 1 1 3 1 1 3 1 0 0 1 1 3

0 1 2 0 1 2 0 1 2 0 1 0 0 1 2

0 2 6 0 0 2 0 2 6 0 2 1 0 0 2

E E

1 1

1 2

1 1 3 1 0 0 1 0 0 1 1 3 1 0 0 1 1 3

2 3 4 2 1 0 0 1 0 0 1 2 2 1 0 0 1 2

5 7 9 5 0 1 0 2 1 0 0 2 5 2 1 0 0 2

A L UE E




- 41 -

Note: If pivoting (row and/or column exchange) is used to reduce A to upper triangular form, eqn.

(39) becomes

PAQ LU (40)

where ,P Q are corresponding row and column permutation (elementary) matrices, respectively.

(Why?)

EXAMPLE 32 – LU DECOMPOSITION WITH PIVOTING USING ELEMENTARY MATRICES

For

0 1 3

2 3 4

5 7 9

A :

1 1

0 1 0 0 1 3 2 3 4 0 1 3 0 1 0 2 3 4

1 0 0 2 3 4 0 1 3 2 3 4 1 0 0 0 1 30 0 1 5 7 9 5 7 9 5 7 9 0 0 1 5 7 9

P A A A P A

1

1 1 1 1

1 0 0 2 3 4 2 3 4 2 3 4 1 0 0 2 3 4

0 1 0 0 1 3 0 1 3 0 1 3 0 1 0 0 1 3

5 / 2 0 1 5 7 9 0 1 / 2 1 5 7 9 5 / 2 0 1 0 1 / 2 1

E A A E

12 2

1 0 0 2 3 4 2 3 4 2 3 4 1 0 0 2 3 4

0 1 0 0 1 3 0 1 3 0 1 3 0 1 0 0 1 3

0 1 / 2 1 0 1 / 2 1 0 0 1 / 2 0 1 / 2 1 0 1 / 2 1 0 0 1 / 2

E E

1 1

1 2

0 1 3 0 1 0 1 0 0 1 0 0 2 3 4

2 3 4 1 0 0 0 1 0 0 1 0 0 1 35 7 9 0 0 1 5 / 2 0 1 0 1 / 2 1 0 0 1 / 2

0 1 0

1 0 0

0 0 1

A P E E

1 0 0 2 3 4

0 1 0 0 1 3

5 / 2 1 / 2 1 0 0 1 / 2

P L U




- 42 -

2.8.8 Computation of the inverse of a matrix

EXAMPLE 33 – MATRIX INVERSE

1 2 3

1

3 1 1 1 0 0

1 2 1 0 1 01 1 1 0 0 1

x x x

A IX A

1

2

3

3 1 1 1

1 2 1 0

1 1 1 0

3 1 1 0

1 2 1 1

1 1 1 0

3 1 1 0

1 2 1 0

1 1 1 1

x

x

x




- 43 -

Use Gauss elimination three times OR once (simultaneously) for

F.E.

13

13

F.E.

25

B.S.

11 12

21

31

1

3 1 1 1 0 0

1 2 1 0 1 0

1 1 1 0 0 1

3 1 1 1 0 0

5 4 10 1 03 3 3

2 4 10 0 13 3 3

3 1 1 1 0 0

5 4 10 1 03 3 3

4 1 20 0 15 5 5

1 / 4

0 ,

1 / 4

x x

x x

x

x

13

22 23

32 33

2 3

1 / 2 3 / 4

1 , 1

1 / 2 5 / 4

x

x

x x

x x

Thus

1

1 1 34 2 40 1 1

1 1 54 2 4

A




- 44 -

General algorithm for AX I .

1 2

1

01 1

11 1

0

0

12 2

02

2 20

0

0

1

1 0

.... 0 0

0 1

[ ]{ } { }

[ ]{ } { }

[ ]{ } { }

n

n n

n

n n

A x e

A x e

A x e

A x x x

Ax eA x

Ax eA x

Ax eA x

Solve by Gauss elimination (LU decomposition)

1 2 1 2[ | ],[ | ],...,[ | ] [ | ... ]

n n A e A e A e A e e e

Even though in exact arithmetic Ax b and 1x A b are equivalent, x is hardly ever computed as

1x A b . Rather, x is computed as the solution of Ax b by whatever method is most

appropriate for a given A . (Gaussian elimination, iterative methods, optimization-based methods…)

Hence the Matlab command x = A\ b.




- 45 -

EXAMPLE 34 – MATRIX INVERSE VS. LU DECOMPOSITION

1. 0. 0. 0. 0. 4. 1. 0. 0. 0.

0.25 1. 0. 0. 0. 0. 3.75 1. 0. 0.

0. 0.2667 1. 0. 0. 0.

0. 0. 0.2679 1. 0.0. 0. 0. 0.2679 1

4 1

1 4 1

1 4 1

1 4 11 4 .

A L

0. 3.733 1. 0.

0. 0. 0. 3.732 1.0. 0. 0. 0. 3.732

U

0.2679 0.07179 0.01923 0.005128 0.001282

0.07179 0.2871 0.07692 0.02051 0.005128

0.01923 0.07692 0.2884 0.07692 0.01923

0.005128 0.02051 0.07692 0.2871 0.07179

0.001282 0.005128 0.01923 0.07179 0.2679

1

A

• Find x that sastisfies Ax b by solving LUx b or as1x A b ?




- 46 -

2.8.9 Computation of the determinant of a matrix

• Why?

− Value of determinant rarely needed

−

Useful for theoretical considerations and derivation of general results

HWNTHI: Review properties of determinants in the textbook: det( ) 1I , det( ) det( )T A A ,

1det( ) 1/ det( ) A A , det( ) det( )det( )AB A B , det( ) det( )n c c A A

HWNTHI: Review matrix inversion theorem:1 adj( )

det( )

AA

A.

•

Computation of det( )A

forward elimination forward elimination...

pivoting pivoting A U

Then

1

det( ) ( 1) det( ) ( 1)n

r r

ii i

u

A U

where r is # of row or column interchanges.

EXAMPLE 35 – DETERMINANT OF A MATRIX VIA GAUSS ELIMINATION

13

13

25

3 1 1

1 2 1

1 1 1

3 1 1

5 403 3

2 403 3

3 1 1

5 403 3

40 05

U

Thus




- 47 -

0 5 4det ( 1) (3) 4

3 5

A




- 48 -

2.8.10 Numerical precision and pivoting

Pivoting = Interchanging of columns or rows during Gauss elimination.

• Why pivoting?

EXAMPLE 36 – SENSITIVITY OF SINGLE LINEAR EQUATION TO PARAMETER ERRORS

2 10 5x x , 2 10.1 5.05x x

% error = ______________

EXAMPLE 37 – SENSITIVITY OF MULTIPLE LINEAR EQUATIONS TO PARAMETER ERRORS

1 20.0003 1.566 1.569x x (41)

1 20.3454 2.436 1.018x x (42)

Fi gur e 6 – Gr aphi cal r epr esent ati on of eqns. ( 41) and ( 42) .

- Infinite precision 1 2

10, 1x x

- Matlab by default reports numbers with four significant digits.

Using 4-significant-digit arithmetic

2

0.34541151

0.0003000m (43)

(41)2 1 2

0.3454 1802 1806m x x (44)

(42) – (44)2 2

1804 1805 1.001x x (45)




- 49 -

(41) and (45)1 1 1

1.569 1.5680.0003 (1.566)(1.001) 1.569 3.333

0.0003000x x x

% error = ______________

In matrix form:

0.0003000 1.566 1.569 0.0003000 1.566 1.569

0.3454 2.436 1.018 0 1804 1805

0.0003000 0 0.001000 1.000 0 3.333

0 1 1.001 0 1.000 1.001

(46)

•

What went wrong?

• How could it be fixed?

-

Source of inaccuracy in EXAMPLE 36:

11 21| | | |a a

- A brute-force fix: Increase precision of arithmetic to 8 digits.

2

0.345400001151.3333

0.00030000000II m (47)

(41)2 1 2

0.34540000 1803.0879 1806.5419II m x x (48)

(42)–(48)2 2

1805.5239 1805.5239 1.0000000x x (49)

(41) and (49) 10.00030000000 1.5660000 1.0000000 1.5690000x

1 1

1.5690000 1.566000010.000000

0.00030000000x x

% error = ______________

In matrix form:

0.00030000000 1.5660000 1.5690000 0.00030000000 1.5660000 1.5690000

0.34540000 2.4360000 1.0180000 0 1805.5239 1805.5239

0.00030000000 0 0.0030000000 1.0000000 0 1

0 1.0000000 1.0000000

0.000000

0 1.0000000 1.0000000

(50)




- 50 -

- A smart fix: Interchange rows (pivoting): Solve eqns.(42), (41) rather than (41), (42), using same

precision.

4-significant-digit arithmetic

'

2

0.00030000.0008686

0.3454m (51)

(41) – (42) '

2 2 2 11.568 1.568 1.000 10.01m x x x (52)

Solution is accurate!

0.3454 2.436 1.018 0.3454 2.436 1.018

0.0003000 1.566 1.569 0 1.568 1.568

0.3454 0 3.454 1.000 0 10.00

0 1.000 1.000 0 1.000 1.000

(53)

HWNTHI: Spot the differences among the above three different solutions of eqns. (41) and (42).

EXAMPLE 38 – ROW PIVOTING

ForwardElimination

12

ForwardElimination

23

Back-subst

0 1 2 1

1 1 1 0

2 1 0 5

2 1 0 5

1 1 1 0

0 0 1 2 1

2 1 0 5

30 3 / 2 1 5 / 2 (No pivoting since 1)2

0 1 2 1

2 1 0 5

0 3 / 2 1 5 / 2

0 0 4 / 3 8 / 3

itution

3 2 12, 3, 1x x x




- 51 -

General computation rule:

- Compute all intermediate results with highest precision

- Report final results with sensible precision

HWNTHI: Many universities report course grades as A, B, C,… and GPA as, say, 3.763. Anything wrongwith that?




- 52 -

2.8.11 Numerical precision and matrix conditioning

EXAMPLE 39 – SENSITIVITY OF VARIOUS LINEAR SYSTEMS OF EQUATIONS

2

!

2

121 =+− x x

2 x

1 x

12

121 =+− x x

2

20

4

4

6

6

8

8

2221

=+− x x

182321

=+ x x

( )3,4

2 x

1 x

22 21 =+− x x

2 x

1 x

12

121 =+− x x

12

121 =+− x x

2 x

1 x

1.15

3.221 =+− x x

• How to assess quantitatively?

• Given that det( ) 0 A Ax b does not have unique solution, does det( ) 0A imply the

solution of Ax b is very sensitive to small errors in ,A b ?




- 53 -

Definition 5 – Vector norm in n

The norm of the vector n x is a real-valued function : n , such that

a) 0 for all n x x

b)

0 x x 0 (What is the difference between 0 and 0 ?)

c) , for any , n x x x

d)

, for any ,n n x y x y x y (Triangle inequality)

- Why use vector norms?

Theorem 4– Vector norms in n

General formula for p-norm in n :

1/

1

pn p

i pi

x

x (54)

where 1p

- Various possibilities

1-norm:1

1

n

i i

x

x (55)

- Matlab: norm( x, 1)

Euclidean norm or inner-product norm or 2-norm

1

22

21

n T

i i

x

x x x (56)

- Matlab (default vector norm): norm( x, 2)

Infinity-norm max

1,... i x

i n

x (57)

- Matlab: nor m( x, I nf )

EXAMPLE 40 – VECTOR NORMS

1 2

37, 5, 4

4

v v v v




- 54 -

Definition 6 – Matrix norm

The norm of the matrix n n A is a real-valued function : n n , such that

a) 0 for all n n A A

b)

0 A A O (What is the difference between 0 and O ?)

c) , for any , n n A A A

d) , for any ,n n n n A B A B A B (Triangle inequality)

e)

, for any ,n n n n AB A B A B

- Why use matrix norms?

Theorem 5 – Induced matrix norms in n n

General formula for induced p -norm in n n :

max,

p

ip n

p

AxA

xx x 0 for 1p (58)

Above formula not immediately useful. However, the following explicit formulas can be shown:

Theorem 6 – Formulas for induced matrix norms in n n

1-norm (maximum column sum):1

1

max

1

n

ij i i

a

j n

A (59)

- Matlab: norm( A, 1)

Euclidean or 2-norm1/2

max2( )T

i

A A A (60)

- Matlab (default matrix norm): norm( A, 2) , norm( A)

Infinity-norm (maximum row sum)1

max1

n

ij i j

a i n

A (61)

- Matlab: nor m( A, I nf )




- 55 -

Theorem 7 – Frobenius matrix norm in n n (NOT an induced norm5)

1/2

2

1 1

n n

ij F i j

a

A

- Matlab: nor m( A, ' f r o' )

EXAMPLE 41 – MATRIX NORMS

2 2 4

0 5 3

2 1 2

A

1

2

_____________________ 9

_____________________ 5.881_____________________ 8

_____________________ 67

i

i

i

F

A

AA

A

5 Chellaboina, V. and Haddad, W.M., 1995. Is the Frobenius matrix norm induced? Automatic Control, IEEE Transactions on,

40(12): 2137-2139.




- 56 -

2.8.12 Matrix condition number

Definition 7 – Condition number

1cond( ) î i

A A A (62)

- It can be shown (how?) that

cond( ) 1A (63)

Proof: Eqn. (62) and property e) in Definition 6 – Matrix norm …

• cond( )A usually estimated without computing 1A .

- What is it good for?

EXAMPLE 42 – CONDITION NUMBER OF MATRIX FROM LINEAR SYSTEM OF EQUATIONS

1 41 11 100 11

cond( ) 109 100 9 1

A A A

Condition number for above matrix is "large".

EXAMPLE 43 – CONDITION NUMBER OF MATRIX IN EXAMPLE 36

1 4.498 2.89180.0003 1.560.63778 0.0005

6 cond( ) 160.3454 2.436 539

A A A

Condition number for above matrix is "large".

- How large is "large"?




- 57 -

Theorem 8 – Sensitivity of linear system of equations to matrix errors

Let the following two linear systems of equations have unique solutions:

( )( )

Ax b

A A x x b

Then

cond( )

x AA

x x A (64)

If, in addition, A is small enough that1 1 A A , then

cond( )

1 cond( )

AA

x A

x AAA

(65)

Theorem 9 – Sensitivity of linear system of equations to right-hand side vector errors

Let the following two linear systems of equations have unique solutions:

( )

Ax b

A x x b b

Then

cond( )

x b

Ax b

(66)

Large problems that can be solved (only) by computers are generally even more sensitive than small

ones.




- 58 -

EXAMPLE 44 – SENSITIVITY OF SOLUTION OF SYSTEM OF LINEAR ALGEBRAIC EQNS

System of equations

1 2 1

1 2 2

11

9 100

x x b

x x b

Case I:

1 2 1 21, 9.1 0.1, 0.1b b x x

Case II:

1 2 1 21, 9 1, 0b b x x

Percent change of solution from Case I to Case II:_____________________________________

Confirm: From EXAMPLE 41

410

x A

x x A

410

x b

x b

- What does that mean?




- 59 -

EXAMPLE 45 – THE HILBERT MATRIX

Appears in approximation of functions by polynomials.

1 1 12 3

1 1 1 12 3 4 1

1 1 1 11 2 2 1

1

ˆ

n

n

n

n n n n

H

(

1

1ij h i j )

18

2020 cond( ) 10n H

Matlab code (M-file Hi l ber t Mat r i x. m):

% Const r uct 20x20 Hi l ber t mat r i x%f or i =1: 20f or j =1: 20

Hi l bert Mat r i x20( i , j ) =1/ ( i +j - 1) ;

end end% Const r uct ant i ci pat ed sol ut i on of syst em of equat i ons% Hi l ber t Mat r i x20 * vect or = bvect or = ones( 20, 1)% Const r uct bb = Hi l ber t Mat r i x20 * vect or ;% Sol ve Hi l ber t Mat r i x20 * x = bx = Hi l bert Mat r i x20\ b

Matlab results

vect or =1111111111111

1111111

x = 1. 00001. 00001. 00010. 99671. 05050. 58493. 0370

- 5. 269413. 1243

- 12. 97809. 1255

- 0. 42992. 2268

- 1. 40794. 1174

- 11. 760327. 0189

- 23. 882612. 6007- 1. 1547




- 60 -

Mathematica code:

n=20;

H=HilbertMatrix[n]; MatrixForm[H]

⎣

11

2

1

3

1

4

1

5

1

6

1

7

1

8

1

9

1

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

201

2

1

3

1

4

1

5

1

6

1

7

1

8

1

9

1

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

211

3

1

4

1

5

1

6

1

7

1

8

1

9

1

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

21

1

221

4

1

5

1

6

1

7

1

8

1

9

1

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

21

1

22

1

231

5

1

6

1

7

1

8

1

9

1

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

21

1

22

1

23

1

241

6

1

7

1

8

1

9

1

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

21

1

22

1

23

1

24

1

25

17

18

19

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

1

8

1

9

1

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

21

1

22

1

23

1

24

1

25

1

26

1

271

9

1

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

21

1

22

1

23

1

24

1

25

1

26

1

27

1

281

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

21

1

22

1

23

1

24

1

25

1

26

1

27

1

28

1

291

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

21

1

22

1

23

1

24

1

25

1

26

1

27

1

28

1

29

1

30

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

1

13

1

14

1

15

1

16

1

17

1

18

1

19

1

20

1

21

1

22

1

23

1

24

1

25

1

26

1

27

1

28

1

29

1

30

1

31

1

321

14

1

15

1

16

1

17

1

18

1

19

1

20

1

21

1

22

1

23

1

24

1

25

1

26

1

27

1

28

1

29

1

30

1

31

1

32

1

331

15

1

16

1

17

1

18

1

19

1

20

1

21

1

22

1

23

1

24

1

25

1

26

1

27

1

28

1

29

1

30

1

31

1

32

1

33

1

341

16

1

17

1

18

1

19

1

20

1

21

1

22

1

23

1

24

1

25

1

26

1

27

1

28

1

29

1

30

1

31

1

32

1

33

1

34

1

351

17

1

18

1

19

1

20

1

21

1

22

1

23

1

24

1

25

1

26

1

27

1

28

1

29

1

30

1

31

1

32

1

33

1

34

1

35

1

361

18

1

19

1

20

1

21

1

22

1

23

1

24

1

25

1

26

1

27

1

28

1

29

1

30

1

31

1

32

1

33

1

34

1

35

1

36

1

371

19

1

20

1

21

1

22

1

23

1

24

1

25

1

26

1

27

1

28

1

29

1

30

1

31

1

32

1

33

1

34

1

35

1

36

1

37

1

381

20

1

21

1

22

1

23

1

24

1

25

1

26

1

27

1

28

1

29

1

30

1

31

1

32

1

33

1

34

1

35

1

36

1

37

1

38

1

39⎦




- 61 -

xOnes=Table[{1},{i,1,n}]; MatrixForm[xOnes]

⎣

⎦




- 62 -

b=H.xOnes; MatrixForm[b]

⎡ 55835135

1551950413684885

517316811333445

5173168678544345

356948592604180055

35694859213676707007

892371480012532641007

8923714800104294993063

8031343320097124150813

803134332002638126077577

23290895628002482853440057

232908956280072733748345767

72201776446800137946478311659

144403552893600131214377943659

14440355289360017878143110237

20629078984800119645914042379

144403552893600114631901789129

1444035528936004071493833381773

5342931457063200

391526776738577353429314570632003771059091081773

5342931457063200

⎤




- 63 -

x=LinearSolve[H,b]; MatrixForm[x]

⎣

1

1

111

11

11

111

11

1

111

1

1⎦




- 64 -

However:

n=20;

H=HilbertMatrix[n]//N; MatrixForm[H]

. . …

. . …. . …

. . …. . …

. . …. . …

. . …… … …

xOnes=Table[{1},{i,1,n}]; MatrixForm[xOnes]




- 65 -

b=H.xOnes; MatrixForm[b]

⎡ .

. . . .

. . . . . .

. . . .

.

. . . .

⎤

x=LinearSolve[H,b]; MatrixForm[x]

0.9999987102450404

1.0001984795168208

0.9925277635763163

1.11906409458715260.019414296716930644

5.4869056040266075−9.96710090253764310.156508406219984

20.683788328136238−55.15764016238065

36.6485198453427732.297361263125126−32.37348992021546

−20.942601206391025

3.286112567151407

56.4580572645654

−19.249110098461458−46.6230260964306845.64478439335456

−10.48027353766982

Li near Sol ve: : l uc: Resul t f or _ Li near Sol ve _ of badl y condi t i oned mat r i x

_ {{1. , 0. 5, 0. 333333, 0. 25, 0. 2, 0. 166667, 0. 142857, 0. 125, 0. 111111, 0. 1, 0. 0909091, 0. 0833333, 0. 0769231, 0. 0714286, 0. 0666667, 0. 0625, 0. 0588235, 0. 05555

56, 0. 0526316, 0. 05}, � 18� , {� 1� }} _ may cont ai n si gni f i cant numer i cal

err ors. �

http://reference.wolfram.com/mathematica/ref/message/General/luc.html






- 66 -

2.9 Matrix rank

• Why?

- Reaction network problems

- Dimensional analysis problems

- Approximation problems (cf. singular value decomposition)

2.9.1 Linear independence

Definition 8 – Linear independence of vectors in n

The vectors1,...,

k v v in n are linearly independent iff

1 1 1... ... 0

k k k c c c c v v 0 (67)

EXAMPLE 46 – LINEARLY INDEPENDENT VECTORS

1 2 3

1

1 1 2 2 3 3 1 2 3 2

3

1 2 3

4 , 5 , 6

7 8 10

1 2 3 1 2 3 0

4 5 6 4 5 6 0

7 8 10 7 8 10 0

c

c c c c c c c

c

v v v

v v v

1

2

3

3 2 1

1 2 3

1 2 3 0

... 0 3 6 0

0 0 1 0

c 0

, , are linearly independent

c

c

c

c c

v v v

EXAMPLE 47 – LINEARLY DEPENDENT VECTORS

1

1 2 3 1 1 2 2 3 3 1 2 3 2

3

1 2 3 1 2 3 1 2 3 0

4 , 5 , 6 4 5 6 4 5 6 0

7 8 9 7 8 9 7 8 9 0

c

c c c c c c c

c

v v v v v v

1

2

3

3 2 1

1 2 3 1 2 3

1 2 3 0... 0 3 6 0

0 0 0 0

(anything), c 2 ,

, , are linearly dependent, i.e. 2

c

c

c

c t t c t

t t t

v v v v v v 0

Each one of1 2 3, ,v v v is a linear combination of the other two.




- 67 -

EXAMPLE 48 – LINEARLY DEPENDENT VECTORS

1

1 2 3 1 1 2 2 3 3 1 2 3 2

3

1 2 3 1 2 3 1 2 3 0

2 , 4 , 6 2 4 6 2 4 6 0

3 6 9 3 6 9 3 6 9 0

c

c c c c c c c

c

v v v v v v

1

2

3

3 2 1

1 2 3 1 2 3

1 2 3 0... 0 0 0 0

0 0 0 0

(anything), c (anything), 2 3

, , are linearly dependent, i.e. ( 2 3 )

c

c

c

c p q c q p

q p q p

v v v v v v 0

Pick one of1 2 3, ,v v v , then determine the remaining two.

• Given1,...,

k v v in n , how many are linearly independent?

EXAMPLE 49 – SOLUTION OF HOMOGENEOUS SYSTEM OF LINEAR ALGEBRAIC EQUATIONS

1

2

3

3 2 1

1

2

3

1

2

3

1 2 3 0

2 4 6 0 ...

3 6 9 0

(anything), c (anything), 2 3

2 3 3 2

0 1

1 0

c

c

c

c p q c q p

c q p

c q p q

c p

c

c

c

3 2

span 0 , 1

1 0

Theorem 10 – Null space of a matrix

Solutions of homogeneous systems of linear algebraic equations form a vector space, called the null

space of A with corresponding basis vectors.

HWNTHI: What is a vector space and a vector space basis? (Hint: A set, closed under vector additionand scalar multiplication.)




- 68 -

2.9.2 Basic facts about matrix rank

Definition 9 – Matrix rank (column)

For matrix A inm n

(column) rank( ) maximum number of LI column vectors ofA A (68)

Definition 10 – Matrix rank (row)

For matrix A inm n

(row) rank( ) maximum number of LI row vectors ofA A (69)

Theorem 11 – Matrix rank

For matrix A in m n

(row) rank( ) (column) rank( )A A (70)

•

rank( )A determined through forward elimination (row reduction of A )Theorem 12 – Matrix rank through Gauss elimination

For matrix A in m n

rank( ) Number of non-zero rows in row-reduced form ofA A (71)

Theorem 13 – Sub-additivity of rank

For matrices A in m n , B in n k

rank( ) rank( ) rank( ) A B A B (72)

Theorem 14 – Sylvester inequality

For matrices A in m n , B in n k

rank( ) rank( ) rank( ) min{rank( ), rank( )}n A B AB A B (73)

Definition 11 –Matrix nullity

For matrix A in m n

nullity( ) rank( )n A A (74)

A m

n

rank( )A

A

m

n

rank( )A




- 69 -

2.9.3 Matrix rank and solution of systems of linear equations

Theorem 15 – Consistency of linear system of equations

Ax b ( A in m n , b in m ) is consistent iff

rank([ | ]) rank( )A b A (75)

Sketch of Proof:

rank([ | ]) rank( )A b A


A b

m

n


A b



m rank([ | ]) rank( )A b A

n




- 70 -

Theorem 16 – Uniqueness of solution of linear system of equations

Ax b (A in m n , b in m ) has unique solution iff

rank([ | ]) rank( ) n A b A (76)

Sketch of Proof:

rank( ) n A

rank([ | ]) rank( ) n A b A A b

m

n

A b m

rank( ) n A n

rank([ | ]) rank( ) n A b A




- 71 -

EXAMPLE 50 – INDEPENDENT REACTIONS IN OXIDATION OF ETHYLENE

Partial and total oxidation of ethylene:

2 4 2 2 4

2 4 2 2 2

2 4 2 2 2

1C H O C H O

25

C H O O 2CO 2H O2C H 3O 2CO 2H O

• How many and which of the above reactions are independent?

Write reactions as equations:

2 4 2 4 2 2 2

2 4 2 4 2 2 2

2 4 2 4 2 2 2

11C H 1C H O O 0CO 0H O 0

25

0C H C H O O 2CO 2H O 02

1C H 0C H O+3O 2CO 2H O 0

or

2 4

12 42

522

2

2

C H

C H O1 1 0 0 0

O0 1 2 2 0

CO1 0 3 2 2 0

H O

A

Are the rows of A linearly independent? If linearly dependent, how?

Let

1

1 25

2 2

3

1 1 0 00 1 2 2ˆ

1 0 3 2 2

T

T

T

vv

v

A

Then, to check linear independence, solve

1 1 2 2 3 3

1 1

1 5

2 2 2 2

3 3

1 0 1 0 1 0 1 0

1 1 0 0 0 1 1 0

3 0 0 0 0 0...

0 2 2 0 0 0 0 0

0 2 2 0 0 0 0 0

c c c

c c

c c

c c

v v v 0

3

2

1

(anything)c p

c p

c p

or

1 2 3 v v v 0

Two reactions are independent; the third depends on the other two.




- 72 -

EXAMPLE 51 – DIMENSIONAL ANALYSIS FOR PRESSURE DROP IN A PIPE

Physical (dimensional) variables associated with fluid flow in a pipe:

pressure drop per unit length in a pipe

pipe diameter

mean velocity of fluid in pipefluid density

viscosity

p

D

v

Dimensions of above variables ( M mass, L length, T time):

2 2

1

3

1 1

[ ]

[ ]

[ ]

[ ]

[ ]

p ML T

D L

v LT

ML

ML T

(Unknown) relationship among above variables

( , , , , ) 0 f p D v

• Construct dimensionless variables1 2, , ... such that

1 2( , ,...) 0 f

Let

1 2 3 4 5

1 1 1 2 3 3 4 4 5 5 5

1 4 5 1 2 3 4 5 1 3 5

2 2 3

2 3 2

[ ] [ ]

[ ] ( )( )( )( )( )

a a a a a

a a a a a a a a a a a

a a a a a a a a a a a

p D v

M L T L L T M L M L T

M L T

Because is dimensionless

1

21 4 5

31 2 3 4 5

41 3 5

5

0 1 0 0 1 1 0

2 3 0 2 1 1 3 1 0

2 0 2 0 1 0 1 0

a

a a a a

a a a a a a

a a a a

a

1 0 0 1 1 0 1 0 0 1 1 0

2 1 1 3 1 0 0 1 1 1 1 0

2 0 1 0 1 0 0 0 1 2 1 0

nullity( ) rank( ) 5 3 2n A A




- 73 -

There are two basis vectors for the solution space:

1

2

3

4

5

1 1

2 2 1

2 1 2

0 11 0

a

a

a

a a

Therefore there are two dimensionless variables,1 2, , which can be defined in an infinite number of

ways, depending on the choice of two pairs ( , ) . E.g.,

1 2

1

1 1 2

2

( , ) (1,0)

( , ) (0,1)

p D v

p D v

or

1

1

1 1 2

2

( , ) ( 1, 1)

( , ) (0,1)

Dv

p D v

or

...

• 1

familiar?




- 74 -

3. EIGENVALUE/EIGENVECTOR PROBLEMS

3.1 Review

Given square matrix A , find scalar/vector pair ( , ) x , such that

Ax x (77)

iff det( ) 0 (Why?)( )

iff det( ) 0 (Why?)

x 0 I AAx x x Ax 0 I A x 0

x 0 I A

Definition 12 – Characteristic equation

det( ) 0 I A (78)

Theorem 17 – Characteristic (polynomial) equation

1 2

1 2 1

Characteristic polynomial

det( ) 0 ... 0n n n

n n

I A

(79)

Definition 13 – Eigenvalues (characteristic values)

Eigenvalues of A : Roots , 1, ...,i

i n of characteristic (polynomial) equation (79).

Definition 14 – Eigenvectors (characteristic vectors)

Eigenvectors of A : Non-zero solutions of( )( ) , 1,...,i

i i n I A x 0 .

•

How many distinct eigenvalues, eigenvectors?

HWNTHI: Prove or disprove: The eigenvalues of a triangular or a diagonal matrix are equal to its

diagonal elements.




- 75 -

EXAMPLE 52 – EIGENVALUES AND EIGENVECTORS OF A MATRIX

1 2

3 4

A

Find eigenvalues:

2

1 2 1 0det det

3 4 0 1

1 2det

3 4

(1 )(4 ) (2)(3)

5 2 0

A I

1,2

5.372

0.372

Find eigenvector corresponding to1

:

1 1 1

2 2 2 normalized

1 5.372 2 0 1 0.4160

3 4 5.372 0 2.186 0.9094

v v v c

v v v

HWNTHI: Verify above result.

HWNTHI: Is1

Av v ?

Find eigenvector corresponding to2

:

1 1 1

2 2 2 normalized

1 0.372 2 0 1 0.8246

3 4 0.372 0 0.686 0.5658

w w w d

w w w

HWNTHI: Verify above result.

HWNTHI: Is2

Aw w ?




- 76 -

Fi gur e 7 – Gr aphi cal r epr esent ati on of ei genval ue/ ei genvect or pai r f or EXAMPLE 51. For al l

vect ors1 2

[ ]T x x x wi t h2

1x , r epr esent ed by poi nt s on t he ci r cl e, t he corr espondi ng

vect ors Ax ar e r epr esent ed by poi nt s on t he el l i pse. For t he choi ce of x on t he l ef t , t he

vect ors x and Ax ar e not al i gned. For t he choi ce of x on t he r i ght , t he vect or s x and

Ax are al i gned, i . e. Ax x , whi ch means t hat t hi s x i s an ei genvect or and acorr espondi ng ei genval ue.

2 x 1

2

2

x 2

2 2 4 x 1

x 2




- 77 -

EXAMPLE 53 – EIGENVALUES AND EIGENVECTORS OF A MATRIX

2 0 0

0 1 1

0 1 1

A

Find eigenvalues:

2 3

2 0 0 1 0 0

det det 0 1 1 0 1 0

0 1 1 0 0 1

2 0 0

det 0 1 1

0 1

4

1

04

A I

1,2,3

2

2

0

Find eigenvectors corresponding to1,2

:

1 1

2 2

3 3

3

2

1

0 0

0 0

0 0

0

0 0 0 0 1 1

0 1 1 0 1 1

0 1 1 0 0 0

1 1 0

1 1 0

0 0 0

1 1 0

0 0

0

0

0

0

v v

v v

v v

v

v

v

3 1

2 2 1 2

31

0

0 0

1 0

0 0 1

0 0 1

v v a

v v b a b a b

v bv

v v

Find eigenvectors corresponding to 3 :

1 1 1

2 2 2

3 3 3

2 0 0 0 2 0 0 0 0 0

0 1 1 0 0 1 1 0 1

0 1 1 0 0 0 0 0 1

v v v

v v v c c c

v v v c

v3

HWNTHI: Verify above results.




- 78 -

EXAMPLE 54 – MATRIX A WITH REPEATED EIGENVALUES

1 2 3

0 1 4

0 0 1

A

Eigenvalues:

1 2 3 1

Eigenvector v :

1

2

3

0 2 3 0 1

0 0 4 0 0

0 0 0 0 0

v

v

v

A I v 0 v




- 79 -

Theorem 18 – Eigenvalues and determinant

1 2det( )

n A (80)

Quick Proof: Theorem 17

01 2

1 2 1Characteristic polynomial

det( ) ...

det( ) ( 1)

n n n

n n

n

n

I A

A

det( ) ( 1)n A1 2

(Why?)n

Theorem 19 – Eigenvalues and matrix invertibility

A square matrix is invertible iff all of its eigenvalues are non-zero.




- 80 -

3.1.1 Solution of polynomial equations

Theorem 20 – Fundamental theorem of Algebra

The polynomial equation 1

0 1 1... 0n n

n n a x a x a x a

with , 0, ...,

i a i n has exactly n

roots in the complex plane.

• Polynomial equations can be solved relatively easily using specialized methods.

• Eigenvalues usually not computed as solutions of the characteristic equation, but through direct

numerical methods.

3.1.2 Software for polynomial equations

•

Matlab:>> hel p roots

•

Mathematica:RootsSol ve

•

Fortran, C

Check www.netlib.org

HWNTHI: Execute the following Mathematica commands, to check for the general solutions of

quadratic, cubic, quartic, and quintic polynomial equations.

Root s[ a*x 2̂ + b*x + c == 0, x]

Root s[ a*x 3̂ + b*x 2̂ + c*x + d == 0, x]Root s[ a*x 4̂ + b*x 3̂ + c*x 2̂ + d*x + e == 0, x]Root s[ a*x 5̂ + b*x 4̂ + c*x 3̂ + d*x 2̂ + e*x + f == 0, x]

3.1.3 Software for eigenvalues

•

Matlab:>> hel p ei g

• Mathematica:Ei genval ues

•

Fortran, C

Check www.netlib.org












- 81 -

3.2 Why eigenvalues and eigenvectors?

EXAMPLE 55 – DYNAMICS OF CSTR

A B

Liquid volume = V

Feed flowrate = F i Feed temperature = T i Feed concentration of A = C Ai

Effluent flowrate = F

Effluent temperature = T Effluent concentration of A = C A

Coolant

Fi gur e 8 – Cont i nuous st i r r ed- t ank react or ( CSTR)

CSTR dynamics around steady state:

11 12 1

21 22 2

( ) ( )

( ) ( )

( ) ( )

( ) ( )

A AdC a a C t f t

d dt t t dT dt

a a T t f t dt

t t d

dt

xAx f

A x f x

•

When is this CSTR stable?

• What is the dominant time constant of this CSTR?

EXAMPLE 56 – DYNAMICS OF CSTR IN DETAIL

Consider the exothermic reaction AB in a jacket-cooled CSTR (Figure 8). Assuming that the volume

V of the liquid in the reactor remains constant, the reactor can be modeled as

0

0

( )( ( )) exp ( )

( )

( )( ( )) exp ( ) ( ( ) ( ))

( )

A

Ai A A

t

i A c

p

dC F t E C C t k C t

dt V RT t

UAdT F t E T T t Jk C t T t T t

dt V RT t c V

(81)

where

• Manipulated inputs:1 2

( , ) ( , )c

u u F T

• Measured outputs:1 2

( , ) ( , )A

y y C T

• States:1 2

( , ) ( , )A

x x C T

In vector form

At




- 82 -

( )( ( ), ( ))

( ) ( ( ), ( ))

d t t t

dt t t t

xf x u

y h x u

(82)

where

1( ) ( )

( ) ˆ

( ) ( )

A

n

x t C t

t

x t T t

x ,

1( ) ( )

( ) ˆ

( ) ( )m c

u t F t

t

u t T t

u ,

1( ) ( )

( ) ˆ

( ) ( )

A

q

y t C t

t

y t T t

y (83)

1 1 1

1 1

( )

0 ( )

( )

0 ( )

( ( ),..., ( ), ( ),..., ( ))

( ( ), ( )) ˆ

( ( ),..., ( ), ( ),..., ( ))

( ( )) exp[ ] ( )

( ( )) exp[ ] ( ) ( ( ) ( ))t

p

n m

n n m

F t E

Ai A AV RT t

UAF t E

i A c V RT t c V

f x t x t u t u t

t t

f x t x t u t u t

C C t k C t

T T t Jk C t T t T t

f x u

(84)

1 1 1

1 1

( ( ), ..., ( ), ( ),..., ( )) ( )

( ( ), ( )) ˆ

( ( ), ..., ( ), ( ), ..., ( )) ( )

n m A

q n m

h x t x t u t u t C t

t t

h x t x t u t u t T t

h x u (85)

CASE I (Single steady state):31.36mV (volume of liquid in reactor)

7

0 7.08 10 1/hk (reaction rate constant)

30.02775m K/molJ (heat of reaction per mole per density per specific heat)

/ 8375KE R (activation energy over gas constant)

32.8 m /ht

p

UA

c (heat transfer coefficient x heat exchange area per density per specific heat)

38008 mol/mAi

C (inlet concentration of A)

373.3 Ki

T (inlet temperature)

The steady state corresponding to the input values for flowrate and coolant temperature3

1.133m /h532.6 K

s

cs

F T

(86)

is3393.2 mol/m

547.6 KAs

s

C

T

(87)

for the concentration of A and temperature in the CSTR.




- 83 -

(HWNTHI: Verify the above steady state. Is it unique? Combining eqns. (81) at steady state yields a

single eqn. fors

T :

0

0

0

0

0 ( ) exp

exp

0 ( ) exp ( )

exp

s

Ai s

Ai As As As s s

s

s

Ai s t

i s s cs

s ps

s

F C F E V C C k C C

V RT F E k V RT

F C F UAE V T T Jk T T

V RT c V F E k

V RT

(88)

0

/

0

( ) ( ) 0exp[ ]

s

s

s

s

F

F Ai V t

i s s cs V F E R

pV T

Jk C UAT T T T

C V k

(89)

To visualize the solution of eqn. (89)fors

T , plot 0

/

0exp[ ]

s

s

s

F

Ai V

F E R

V T

Jk C

k and

( ) ( )s t

i s s cs

p

F UAT T T T

V C V vs.

s T ; find intersection. See Figure 9.)

Fi gur e 9 – Pl ot of heat gener at i on r at e (S- cur ve,0

/

0exp[ ]

s

s

s

F

Ai V

F E R

V T

Jk C

k ) and heat i nput mi nus

out put r ate ( ( ) ( )s t

i s s cs

p

F UAT T T T

V C V ) f or t he cont i nuous st i r r ed- t ank react or ( CSTR)

st udi ed.

Taylor-series approximate linearization of the state and output equations around a steady state

yields the following equations




- 84 -

, ,

, ,

( )( )

( )

( ) ( )

( ) ( ) ( )

s s

s s s s

s

s s s s

t t

t

d t t

dt

t t t

x u x u

x u x u

f f u ux xx u

h hy yx u

xA x B u

y C x D u (90)

where

f

x is the Jacobian matrix of the vector-valued function f of the vector x defined as

1 1

1

1

ˆn

n n

n

f f

x x

f f

x x

f

x

(91)

and similarly for

f

u

,

g

x

,

g

u

; and the deviation variables are defined as

( )( ) ( )

( ), ( ), ( )

s s s t t t

t t t

u ux x y y

x u y (92)

and , , ,A B C D are (Jacobian) matrices of appropriate dimensions defined in eqn. (90).

Therefore

00 2

00 2

16.974035 177.2558

0.4479112 2.0269367

s s

s s

E E

RT RT s

As

s E E

RT RT s t As

ps

F k E k e e C

V RT

F Jk E UAJk e e C V c V RT

A (93)

0 5599.118 0

128.162 2.05882

Ai As

i s t

p

C C

V T T UA

V c V

B (94)

1 0

0 1

C (95)

0 00 0

D (96)

• Eigenvalues of 10.76, 4.178 A .




- 85 -

Fi gur e 10 – Temperat ur e and concent r at i on r esponse of CSTR t o st ep change i n i nl et f l ow r ateby +10% of i t s st eady st ate val ue ( dashed l i ne = appr oxi matel y l i near i zed syst em; cont i nuousl i ne = nonl i near eqns. ( 81) ) . Note the i nver se r esponse of t emperat ur e t o f l owr ate.

CASE II (Multiple steady states):31.36mV (volume of liquid in reactor)

70

7.08 10 1/hk (reaction rate constant)

30.02775m K/molJ (heat of reaction per mole per density per specific heat)

/ 8375KE R (activation energy over gas constant)

30.5 m /ht

p

UA

c (heat transfer coefficient heat exchange area per density per specific heat)

38008 mol/mAi

C (inlet concentration of A)

373.3 Ki

T (inlet temperature)

The steady states for concentration of A and temperature in the CSTR corresponding to31.133m /h

400 Ks

cs

F

T

(97)

are multiple (Figure 11):




- 86 -

Fi gur e 11 – Pl ot of heat gener at i on rate ( S- cur ve,0

/

0exp

s

s

s

F

Ai V

F E R

V T

Jk C

k

) and heat i nput mi nus

out put r ate ( ( ) ( )s F

t i s s cs V

p

UAT T T T

C V ) f or t he cont i nuous st i r r ed- t ank react or ( CSTR)

st udi ed.

• Steady state 1 (Stable)37754 mol/m

386.4 KAs

s

C

T

(98)

0.8604 11.88

0.0007575 0.8712

A ,186.8 0

9.608 0.3676

B ,1 0

0 1

C ,0 0

0 0

D (99)

Eigenvalues of 0.8658 0.09469 j A (both in the left half-plane).

Period of CSTR-induced oscillations is2

66 h0.09469

, too long to observe before decay due

to exp[ 0.8658 ]t .

Fi gur e 12 – Temperat ur e and concent r at i on r esponse of CSTR t o st ep change i n i nl et f l ow r ateby +10% of i t s st eady st ate val ue ( dashed l i ne = appr oxi matel y l i near i zed syst em; cont i nuousl i ne = nonl i near eqns. ( 81) ) .

(HWNTHI: Verify the above.)




- 87 -

• Steady state 2 (Unstable)33983 mol/m

459.0 KAs

s

C

T

(100)

1.675 133.3

0.02336 2.498

A ,

2959 0

62.99 0.3676

B ,

1 0

0 1

C ,

0 0

0 0

D (101)

Eigenvalues of 1.525, 0.7019A (one in the right half-plane).

Fi gur e 13 – Temperat ur e and concent r at i on r esponse of CSTR t o st ep change i n i nl et f l ow r ateby +10% ( t op r ow) and - 10% ( bot t om r ow) of i t s st eady st at e val ue ( dashed l i ne =appr oxi matel y l i near i zed syst em; cont i nuous l i ne = nonl i near eqns. ( 81) ) . The CSTR escapesei t her t owards st eady st ate 1 or 3.





- 88 -

• Steady state 3 (Stable)3mol/m

519.3 K

850.7As

s

C

T

(102)

7.842 185.2

0.1945 3.938

A ,

5262 0

107.3 0.3676

B ,1 0

0 1

C ,0 0

0 0

D (103)

Eigenvalues of 1.952 1.151 j A (both in the left half-plane).

Period of CSTR-induced oscillations is2

5.5 h/cycle1.151

, too long to observe before decay

due to exp[ 1.952 ]t .

Fi gur e 14 – Temperat ur e and concent r at i on r esponses of CSTR t o st ep change i n i nl et f l ow r ateby +10% of i t s st eady st ate val ue ( dashed l i ne = appr oxi matel y l i near i zed syst em; cont i nuousl i ne = nonl i near eqns. ( 81) ) . Note t he i nver se r esponses of bot h t emperat ur e andconcent r at i on.





- 89 -

Theorem 21 – Diagonalization of a square matrix

The matrix n n A has distinct eigenvalues1,...,

n and eigenvectors

1,...,

n v v . Then

1

1 1 1 ...T T

n n n A P P v w v w (104)

where

1ˆ

n

P v v ,1

ˆ

n

,1

1 ˆ

T

T

n

w

P

w

,

• Why diagonalize?

− Solve systems of ordinary differential equations

− Solve optimization problems

−

Multivariate statistics

− Principal-stress analysis

− …

A

P

1P




- 90 -

EXAMPLE 57 – MODES OF CSTR MODEL THROUGH MATRIX DIAGONALIZATION

EXAMPLE 55

Steady state 1

16.974035 177.2558

0.4479112 2.0269367

-0.9994 0.9974 -10.7697 0 -1.9424 -26.9051

0.0350 -0.0720 0 -4.1774 -0.9436 -26.9586

A

Steady state 2

1.0000 0.9997 0.7019 0 1.4371 59.8649

0.0073 0.0240 0 1.525 0.4371

1.675 133.3

0.02336 2.49

59.8805

8

A

Steady state 3

0.9995 0.9995

0.0318 0.0062 0.0318 0.0062

1.952 1.153 0

7.842 18

5.2

0.1945 3.938

. 0 1.952 1.153

0..

i i

i

i

A

5003 2.5557 0 80.35770.5003 2.5557 0 80.3577

i i i i

• What is this good for?

Hint: EXAMPLE 55

1

1 1 1

( ) ( )

( ) ( )

( ) ( )

( ) ( )

( ) ( )

( ) ( ), 1,...,i

i i i

d t t

dt t t

d t t dt

t t

d t t

dt dz

z t g t i n dt

xAx f

P P x f

xP P x P f

z g

zz g




- 91 -

3.3 Quadratic forms

• Why?

- Frequent appearance in some optimization problems

- Can help understand basic concepts in collective properties of systems

• What is a quadratic form?

-

Loosely speaking: A multivariable extension of a quadratic polynomialCompare:

- Scalar2

22 2

f f Hx fx c H x c

H H

(105)

2

22

2opt

opt

min{ 2 } min{ }

(for ) if 0

(for ) else

f f Hx fx c H x c

H H x x

f f c x H

H H x

(106)

- Vector/matrix1 1 12 ( ) ( )T T T T c c x Hx f x x H f H x H f f H f

(107)

Note: 2 2T T f x x f (Why?)

1 1 1

opt 1

opt

min{ 2 } min{( ) ( ) }

(for ) if ( ) 0

(for ) else

T T T T

T

c c

c

x Hx f x x H f H x H f f H f x x

f Hf x H f H

x

(108)

EXAMPLE 58 – Q UADRATIC EXPRESSIONS AS Q UADRATIC FORMS

1 12 2

1 1 2 2 1 2 1 22 2

4 14 2 4 2 2 [ ] 2[1 1]

1 4T T

x x x x x x x x x x

x x

x f H x x

12 21 1 2 2 1 2

2

2 [ ] a b x ax bx x cx x x b c x

11 12 13 12 2 2

11 1 22 2 33 3 12 1 2 13 1 3 23 2 3 1 2 3 12 22 23 2

13 23 33 3

2 2 2

a a a x

a x a x a x a x x a x x a x x x x x a a a x

a a a x




- 92 -

Definition 15 – Quadratic form

Quadratic form is defined asT x Ax (109)

where n n A symmetric, n x

Note:n n

A in eqn. (109) can be assumed symmetric without loss of generality. Because, ifn n A is not symmetric, then it can be shown (how?)

2

T T T T

A Ax Ax x x x Hx

with n n H symmetric ( T H H ).

Theorem 22 – Eigenvalues of symmetric matrix

n n A symmetric eigenvalues of A are real.

Quick proof: Let , v be complex eigenvalue, eigenvector of n n A . Then

( )

0 0

(Why?) (Why?)

T T

T T T

T T T

Av v v Av v vv Av v Av v v

Av v v Av v v v v

Theorem 23 – Eigenvectors of symmetric matrix

n n A symmetric eigenvectors ,i j

v v of A corresponding to eigenvaluesi j

are

orthogonal (orthonormal) to each other.Quick proof:

( ) 0

0 (Why?) 0

T T

i i i j i i j i

T T T

j j j i j j i j j j i

T T T T

j i i j i j j i j i

Av v v Av v v

Av v v Av v v v v

v Av v Av v v v v

,i j

v v are orthogonal to each other; orthonormal to each other if, in addition,

2 2

1, 1ˆ ˆT T

i i i j j j v v v v v v .

Theorem 24 – Diagonalization of symmetric matrix

n n A symmetric T A P P (110)

where is diagonal, P orthonormal.




- 93 -

Theorem 25 – Eigenvalues ofT M M

T n n A M M eigenvalues of A are non-negative.

Quick proof:

Let , v be real (why?) eigenvalue, eigenvector of T n n A M M . Then

( ) 0 (Why?)

0

(Why?)

T T T

T T T T

T T

v M Mv Mv MvM Mv v v M Mv v vv v v v

Theorem 26 – Quadratic form with linear term

If H is invertible, the quadratic-plus-linear expression 2T T c x Hx f x can be brought to the pure

quadratic form T d z Hz .

Quick proof: (Cf. eqn. (106).)

Define6 10 ˆ x H f

0 0 0 0

0 0

2 ( ) ( ) (Why?)

( ) ( ) (Why?)

T T T T

T

T

d

x Hx x f x x H x x x Hx

x x H x x

zz

, (111)

6 This is the vector/matrix version of “completing the square” that you learned in junior high school for quadratic equations:

2 2 22 ( )Hx xf H x f H f H




- 94 -

3.3.1 Visualize quadratic forms

2 2

1 1 ...T T T T

n n

T

c c c z z c x Hx x P P x z z

zz

(112)

0i

eqn. (112) defines ellipsoid in rotated orthogonal z -coordinates, with semi-axes

1

,...,n

c c

.

Rotated coordinates have eigenvectors of H as basis vectors:

1 1 ...T

n n z z z P x x Pz v v

• 1D: Eq. (112) 2xHx c Hx c

Fi gur e 15 – Poi nt s cor r espondi ng t o t he quadr ati c- f orm equati on2Hx c .

• 2D: Eq. (112)

2 22 2 1 2

1 1 2 2 2 2

1 2

1T T T T

T

z z c c c z z c

c c

x Hx x P P x z z

zz

Fi gur e 16 – El l i pt i cal cont our s cor r espondi ng t o the quadr at i c- f or m equat i onT c x Hx .

c1

c2

c3

c4

c5

Hx

1z

2z

1c

2c

1x

2x




- 95 -

EXAMPLE 59 – VISUALIZE Q UADRATIC FORM

1 1 1 11 12 2 2 2

1 1 1 12 2 2 2 2

1 2

4 1 3 0

1 4 5 0T

H P P P

v v

2 22 2 1 21 2 2 2

3 5 1

3 5

T z z

c z z c

c c

x Hx ellipse on z -plane

Fi gur e 17 – El l i pt i cal cont our s i n t he z - pl ane cor r espondi ng t o t he quadr ati c- f orm equati on

2 2

1 23 5z z c .

Connection between z and x coordinates:

Every point 1

2

ˆx

x

x can be expressed as a linear combination of the unit vectors1 2,e e in the x -plane

(Figure 18) as

1 1

1 2 1 1 2 22 2

1 0 1 0

0 1 0 1

x x x x x x

x x

e e

Fi gur e 18 – Vect or r epr esent ati on i n x - pl ane

1x

1

2

x

x

1

1ˆ

0

e

2

0ˆ

1

e

2x

1z

2z

3c

5c




- 96 -

1 1

12 21 1 2 21 1

22 2

1 2

T z

z z z

z P x x Pz v v

v v

where 1 2,v v are orthonormal, and produced from rotation of 1 2,e e (Why?) every point 1

2

x x

x

can be expressed as a linear combination of the vectors1 2,v v in the z -plane using weights

1 2,z z

(Figure 19, Figure 20). But points in the z -plane fall on an ellipse for a given c (Figure 17).

Fi gur e 19 – Vect or r epr esent ati on i n z and x - pl anes.

Fi gur e 20 – El l i pt i cal cont our s cor respondi ng t o t he quadr at i c- f or m equat i on2 2

1 1 2

2 2

1 22 34 2 4 5T T x x x x c z z c x Hx z z i n 2D and 3D.

1z

2z

3c

5c

1x

2x

2e

1e

2z

1

2

z

z

2

v

1v

1x

2x

1z




- 97 -

3.3.2 Optimization with quadratic forms and linear inequalities (quadratic programming)

EXAMPLE 60 – Q UADRATIC PROGRAMMING FOR A DISTILLATION COLUMN

Fi gur e 21. Low- pur i t y di st i l l at i on col umn.

For the low-purity binary distillation column shown in Figure 21, the following model captures the

effect of the reflux and boil-up flow rates (manipulated inputs) on the top and bottom concentrations(controlled outputs).

0.7 0.9

1.0 0.9D

B

y L

x V

y G m

or

1 1

2 2

0.7 0.9

1.0 0.9

y m

y m

y G m

(113)

(All variables are shown in deviation from the normal operation steady state).

It is desired to select values for the manipulated inputs, namely reflux and boil-up flow rates

L and V , respectively, such that the output vector1 2

[ ] [ ]ˆ ˆT T

D B y x y y y of top and

bottom concentrations approaches the setpoint vectorSP [1 1]T y as close as possible.

Eqn. (113)

SP

1 0.7 0.9 6.67

1 1.0 0.9 6.30

L L

V V

G my

(114)

Distillate, D

Bottoms, B

Condenser

ReboilerBoil-up

V

Reflux

L y D

x B

Feed




- 98 -

3.3.2.1 Case 1

The flow rates L and V cannot be moved arbitrarily, but must satisfy the constraints

5 20

20 10

L

V

(115)

Unfortunately, the above 6.67L in eqn. (114) does not satisfy the constraints, eqn. (115)

( 6.30V does). What is the best that can be achieved in this case?

- Mathematical problem formulation:

Select L and V to minimize distance of [ ]ˆ T

D B y x y from setpoint SP SP SP[ ]ˆ T

D B y x y

[1 1]T :

1 2

2SP SP 2 SP 2

1 1 2 2,2min min[( ) ( ) ]

m m y y y y

my y

(116)7

subject to the constraints in eqn. (115).

Using eqn. (113), write2

SP

2y y in terms of [ ]ˆ T L V m

2SP SP SP

2SP SP

SP SP

SP SP SP

2 2

11 22 12 1 2

( ) ( )

( ) ( )

( ( ) )( )

2 ( )

2( 2 2 2 )

T

T

T T T

T T T T T

T T

c h L h V h L V f L f V c

y y y y y y

Gm y Gm y

m G y Gm y

m G Gm m G y y y

m Hm m f

(117)

where

1.49 1.53

1.53 1.62ˆ T

H G G

(118)

SP 0.3

ˆ0.

T

f G y

(119)

andSP SP( )ˆ T c y y

is not important (Why?)

Theorem 26 2T T

m Hm m f in eqn. (117) can be brought to the form T d x Hx where1ˆ x m H f (120)

2

SP

2y y represents elliptical contours in

1 2( , )x x -plane (Figure 21, Figure 22).

7 Recall that the 2-norm of a vector x inn is defined as

2 2

12... T

n x x x x x ,




- 99 -

- Graphical solution

Fi gur e 22. Sur f ace pl ot of ( , ) 2ˆ T T g L V m Hm m f .

Fi gur e 23. Cont our pl ot of ( , ) 2T T g L V m Hm m f and gr aphi cal sol ut i on of eqn. f or

opt

1 5ˆm L and

opt

2 4.7222ˆm V , resul t i ng i n

opt

1 0.75y ( i nst ead of 1) and

opt

1 0.75y ( i nstead of - 1) .

-

Numerical solution using Excel Solver in Appendix A.




- 100 -

3.3.2.2 Case 2

The flow rates L and V cannot be moved arbitrarily, but must satisfy the constraints

10 20

20 10

L

V

(121)

Luckily, the above L and V satisfy the constraints, eqn. (121).

-

Mathematical problem formulation

The same formulation as in Case 1 can be used again:

1 2

SP 2 SP 2 SP 2

1 1 2 2, ,2min ( , ) min( ) min[( ) ( ) ]ˆ

D B L V y y f y x y y y y

yy y

(122)

subject to the constraints in eqn. (121).

- Graphical solution

o The surface plot of ( , ) 2T T g L V m Hm m f is the same as in Figure 22.

Fi gur e 24. Cont our pl ot of ( , ) 2T T g L V m Hm m f and const r ai nt s. Opt i mum i s at

opt

1 6.6667ˆm L and

opt


opt

1 1y and

opt

1 1y as

desi r ed. Not e si mi l ari t i es wi t h and di f f er ences fr om Fi gur e 23.

- Numerical solution using Excel Solver in Appendix B.




- 101 -

3.3.3 Quadratic programming in general

min( 2 )T T x

x Hx x f (123)

subject to

Ax b (124)

or, in detail,

1

2 2

11 1 21 2 1 , 1 1 1 1,...,min( ... 2 ... 2 2 ... 2 )

2n

nn n n n n n n n x x

T T

h x h x h x x h x x f x f x

x f x Hx

(125)

subject to

11 1 1 1

1 1

...

...

n n

m mn n m

a x a x b

a x a x b

(126)

( ) 0i

H for all i Numerical solution of eqns. (123) and (124) is easy.

( ) 0i

H for some i Numerical solution of eqns. (123) and (124) is extremely difficult for large

problems.

- General numerical solution via a number of methods.

- Software readily available.

-

What does the optimum look like?




- 102 -

• To visualize, define8 1

0 ˆ x H f

0 0 0 0

0 0

0

min{ 2 } min{( ) ( ) } (Why?)

min ( ) ( ) (Why?)ˆ

= min{ }

T T T T

T

T

T

d

d

x Hx x f x x H x x x Hxx x

x x H x xy x x

yyy Hy

y

, (127)

subject to

0 0 Ax c Ax Ax c Ax Ay b

b

, (128)

•

To minimize, without constraints in eqn. (124)

min{ 2 }= min{ }T T T d d x Hx x f y Hyx y

, (129)

obtained foropt opt 1

0 ˆ y 0 x x H f (130)

(Why?)

8 This is the vector/matrix version of “completing the square” familiar since junior high school for quadratics:2 2 22 ( )Hu uf H u f H f H . The vector/matrix version can also be untangled to a sum of quadratics:

2 21 1 1 1

2 2

1 1 1 1

2 2

1

1

min{ 2 } min{ 2 }

min{ 2 }

min{ ... 2 ... 2 }

min{( 2 ) ... ( 2 )}

( ... )

T T T T T T

T T

n n n n

n n n n

n

n

z z g z g z

z g z z g z

g g

x Hx x f x P P x x P P f x x z g

z z z gz

z

z

attained at

opt , 1, ...,k

k

k

g z k n

.




- 103 -

Fi gur e 25 – El l i pt i cal cont our s cor r espondi ng t o t he quadr at i c- f or m equat i on T c y Hy

subj ect t o Ay b .

1z

2z

1c

2c

1y

2y

Ay b

1x

2x

0,1x

0,2x

Optimum




- 104 -

4. SINGULAR-VALUE DECOMPOSITION9 (SVD)

4.1 What is singular-value decomposition (SVD)?

Theorem 27 – Diagonalization of a general non-square matrix

The matrix m n A can be factorized as

1 1 1 ...T T T r r r A USV u v u v (131)

where

1ˆ m m

m

U u u ,

1

ˆ m n

r

S

,1ˆ n n

n

V v v ,

with

•

rank( )ˆr A

• 1

{ ,..., }m

u u = eigenvectors of T AA

• 1

{ , ..., }n

v v = eigenvectors of T A A

• 1 1 1

{ ,..., } { ( ),..., ( )} { ( ),..., ( )}ˆ T T T T

r r r A A A A AA AA =

= Nonzero singular values of A = nonzero e-values of T AA = nonzero e-values of T A A (Why?)

9 “…SVD is not nearly as famous as it should be”

- Strang, G. Linear Algebra and its Applications, 2nd edition, Academic Press, New York, 1980, p. 142.

1

T v + 1

1

u + T

r v

r

r

u

A

U

T V

S




- 105 -

Note: Because bothi

u are orthonormal eigenvectors of T AA and bothi

v are orthonormal

eigenvectors of T A A , the sign ofi

u (for 0i

) must be adjusted to match the sign of the

correspondingi

Av , or, equivalently, the sign ofi

v must be adjusted to match the sign of the

corresponding T i

A u .

(Why? Hint:

...T T

i i i i i i i i i A Av v AA Av Av Av u and similarly T

i i i A u v .)

Theorem 28 – Matrix condition number (based on the induced 2-norm)

n n A

1 max

2 2min

cond( )î i

A A A (132)

Theorem 29 – Matrix inversion via SVD

m n A

1

1 1

1p pT T

i i i i i i i

i

A u v A v u (133)

Note:

Theorem 30 – Rank property

m n A rank( ) rank( ) rank( ) rank( )T T T AA A A A A (134)

4.2 Why SVD?

• Least squares parameter estimation

• Multivariate statistics

• Model reduction (approximation)

• Others

4.3 Software for SVD

• Matlab:>> hel p svd

• Mathematica:Si ngul arVal ueDecomposi t i on




- 106 -

EXAMPLE 61 – SVD OF THE HILBERT MATRIX

>> f or i =1: 5, f or j =1: 5, Hi l ( i , j ) =1/ ( i +j - 1) ; end, end>> [ U, S, V] = svd( Hi l )

U =- 0. 7679 0. 6019 - 0. 2142 0. 0472 0. 0062

- 0. 4458 - 0. 2759 0. 7241 - 0. 4327 - 0. 1167- 0. 3216 - 0. 4249 0. 1205 0. 6674 0. 5062- 0. 2534 - 0. 4439 - 0. 3096 0. 2330 - 0. 7672- 0. 2098 - 0. 4290 - 0. 5652 - 0. 5576 0. 3762

S =1. 5671 0 0 0 0

0 0. 2085 0 0 00 0 0. 0114 0 00 0 0 0. 0003 00 0 0 0 0. 0000

V =- 0. 7679 0. 6019 - 0. 2142 0. 0472 0. 0062

- 0. 4458 - 0. 2759 0. 7241 - 0. 4327 - 0. 1167- 0. 3216 - 0. 4249 0. 1205 0. 6674 0. 5062- 0. 2534 - 0. 4439 - 0. 3096 0. 2330 - 0. 7672- 0. 2098 - 0. 4290 - 0. 5652 - 0. 5576 0. 3762

>> U*S*V'

ans =1. 0000 0. 5000 0. 3333 0. 2500 0. 20000. 5000 0. 3333 0. 2500 0. 2000 0. 16670. 3333 0. 2500 0. 2000 0. 1667 0. 14290. 2500 0. 2000 0. 1667 0. 1429 0. 12500. 2000 0. 1667 0. 1429 0. 1250 0. 1111

>> Hi l 3 = U( : , 1: 3) *S( 1: 3, 1: 3) *V( : , 1: 3) '

Hi l 3 =1. 0000 0. 5000 0. 3333 0. 2500 0. 20000. 5000 0. 3333 0. 2501 0. 2000 0. 16660. 3333 0. 2501 0. 1999 0. 1666 0. 14300. 2500 0. 2000 0. 1666 0. 1428 0. 12500. 2000 0. 1666 0. 1430 0. 1250 0. 1110

>> Hi l - Hi l 3

ans =

1. 0e- 003 *0. 0007 - 0. 0062 0. 0096 0. 0033 - 0. 0080

- 0. 0062 0. 0573 - 0. 0885 - 0. 0305 0. 07370. 0096 - 0. 0885 0. 1371 0. 0463 - 0. 11320. 0033 - 0. 0305 0. 0463 0. 0185 - 0. 0407

- 0. 0080 0. 0737 - 0. 1132 - 0. 0407 0. 0956




- 107 -

1 1 1 5 5 5 1 1 1 3 3 3... ...T T T T T H USV u v u v u v u v

1 1

1 1 5 5

1 5

1 1...T T T

H VS U v u v u

cond( ) ?H

1

T v + 1

1

u + 5

T v 5

5

u

H

U

T V S

1

T v + 1

1

u + 3

T v 3

3

u




- 108 -

Theorem 31 – Matrix approximation using SVD

Consider the matrix

1ˆ m n

n

X x x (135)

Then the optimal approximation of X by a matrix ̂ of lower rank, p , in the sense

2ˆ arg min

rank( ) min( , )p m n

X (136)

where the matrix norm in eqn. (136) is either the induced 2-norm10 or the Frobenius norm,11 can be

shown12 to be

1 1ˆ p pT T

i i i i i i i

i

u v y v

y

(137)

where1

0p

are the p ( p r ) largest singular values of1

n T T

i i i i

X u v USV , and

,i i

u v are the corresponding standardized singular vectors. (The vectors î i i

y u in eqn. (137) are

called principal components).

In addition,

22

2 1ˆ

i p

X (138)

and

22

1ˆ n

F i i p

X . (139)

- Also known as

o

principal component analysis (PCA),

o

Karhunen-Loéve expansion,

o proper orthogonal decomposition

- Applications in control, statistics, model reduction…

10 22 max

2

maxî σ

≠= =

v 0

AvA

v, where 2

2 iiv= ∑v is the Euclidean vector norm.

11 2 2ˆF ij k i j k

a σ = =∑ ∑ ∑A . Both the induced 2-norm and the Frobenius norm are frequently called Euclidean norm

in literature. The reason is that the induced-2 norm is induced by a Euclidean vector norm, and the Frobenius norm would

be a Euclidean vector norm if the matrix A were re-organized as a vector.12P. Dewilde and E. F. Deprettere, "Singular Value Decomposition: An Introduction," in SVD and signal processing:

algorithms, applications, and architectures, E. F. Deprettere, Ed.: North-Holland, 1988, pp. 3-41.




- 109 -

EXAMPLE 62 – SVD AND CONDITION NUMBER FOR BINARY DISTILLATION COLUMN

Fi gur e 26 – Hi gh- pur i t y bi nar y di st i l l at i on col umn13.

0.78

0.71 0.710.6

0.71 0.

0

710.63 2.0

0.78

.878 0.8

3 0.01

64

1.082 1.09

0 4

6

0

D

B

T

y L

x V

y G m

VU S

1 2

1

1

22

2

1

2

1

2

1

0.780.01

0.632.0

4 (0.71 0.71 )0.6

( 0.71 0.71 )0.78

2.

3

ˆ

ˆT

T

L

V

m m

m m

m

m

v

vu

mu

1

11

1

1 2

0.630 ( 0.71 0.

ˆ

71 )0.78

T

m m

umv

13M. Morari and E. Zafiriou, Robust Process Control , Prentice-Hall, 1989.

Distillate, D

Bottoms, B

Condenser

ReboilerBoilup

V

Reflux

Ly D

x B

Feed

Strong

direction

Weak

direction




- 110 -

Fi gur e 27 – Response of hi gh- pur i t y bi nary di st i l l ati on col umn t o changes i n r ef l ux and boi l -up r at es.

cond( ) 142G

• What does this condition number mean?

− Desired1

0D

B

y

x

.

RequiredL

V

:

1 0.878 0.864 39.9417

0 1.082 1.096 39.4315

L L

V V

y G m

What ifreal

G G ?

real real

0.88 0.86 0.88 0.86 39.9417 1.24

1.08 1.10 1.08 1.10 39.4315 0.24

L

V

G G

Percent error = ________________________________________________

The sensitivity of a solution to data is as important as the solution itself.




- 111 -

Eqn. (133)

1

1

0.78

0.71 0.710.63 0.

0.71 0.7100.878 0.864 0

1.082 1.0

.63 2.0

0.78

0.632.0

96

0.78

0 014

T

u

VG U S

1

1

1

2

1

22

21

0.780.014 0.71 0.7100.71 0.71

0.7110.63 0.78

0.712.0

1 /

.63

0.711

0.014

1 /T

T T

v

uv

G

vu

22

222

0.78 0.630.71

0.711 0.78 0.630.710.014

1 /

T

T

uv

uv




- 112 -

Appendix A. Numerical solution of EXAMPLE 59 – Quadratic Programming for a Distillation Column, using Excel Solver

Fi gur e 28. Excel sheet t o sol ve t he quadr at i c pr ogr ammi ng pr obl em. Opt i mum i s atopt

1 5ˆm L and

opt


opt

1 0.75y ( i nst ead of 1) and

opt

1 0.75y ( i nstead of - 1) .

.

Fi gur e 29. Excel Sol ver f or sol ut i on of t he quadr at i c pr ogr ammi ng pr obl em.




Appendix B. Numerical solution of EXAMPLE 59 – Quadratic Programming for a Distillation Column, using Excel Solver

Fi gur e 30. Excel sheet t o sol ve t he quadr at i c pr ogr ammi ng pr obl em. Opt i mum i s at Opt i mum i s

at opt1

6.6669ˆm L and opt2

6.2965ˆm V , resul t i ng i n opt

1 1y and opt

1 1y as

desi r ed. Not e si mi l ari t i es wi t h and di f f er ences fr om Fi gur e 28.

Documents

CHEE6330 LectureNotes Matrix Algebra(16)