Numerical Methods Notes

Approximation and Errors Significant Figures

• 4 significant figures

– 1.845

– 0.01845

– 0.0001845

• 43,500 ? confidence

• 4.35 x 104 3 significant figures



Accuracy and Precision

• Accuracy - how closely a computed or

measured value agrees with the true value

• Precision - how closely individual

computed or measured values agree with

each other

– number of significant figures

– spread in repeated measurements or

computations

increasing accuracy

incr

easi

ng p

reci

sion

Error Definitions

• Numerical error - use of approximations to

represent exact mathematical operations and

quantities

• true value = approximation + error

– absolute error, Et= true value - approximation

– subscript t represents the true error

– shortcoming....gives no sense of magnitude

– normalize by true value to get true relative error

Error definitions cont.

t

true error true value estimate100% 100%

true value true value

• True relative percent error

• But we may not know the true answer a priori


• May not know the true answer a priori

a

approximate error100%

approximation

• This leads us to develop an iterative

approach of numerical methods

a

approximate error100%

approximation

present approx. previous approx.100%

present approx.


• Usually not concerned with sign, but with

tolerance

• Want to assure a result is correct to n

significant figures

%105.0 n2

s

sa

Example

Consider a series expansion to estimate trigonometric

functions

x.....!7

x

!5

x

!3

xxxsin

753

Estimate sin / 2 to three significant figures

Error Definitions cont.

• Round off error - originates from the fact

that computers retain only a fixed number

of significant figures

• Truncation errors - errors that result from

using an approximation in place of an exact

mathematical procedure

Computer Storage of Numbers

• Computers use a binary base system for logic

and storage of information

• This is convenient due to on/off circuitry

• The basic unit of computer operation is the bit

with value either 0 (off) or 1 (on)

• Numbers can be represented as a string of bits

• 1102 = 0*20 + 1*21 + 1*22 = 0+2+4 = 610

• A string of 8 bits is called a byte

Computer Storage of Integers

• It is convenient to think of an integer as being

represented by a sign bit and m number bits

• This is not quite correct

• Perhaps you remember that underflow & overflow

are -4,294,967,296 & +4,294,967,295 for a 4 byte

(32 bit or single precision) integer

• With a sign bit this should be 2(32-1) - 1 =

4,294,967,295

• Because +0 and -0 are redundant, a method of

storage called 2’s complement is used

2’s Complement

• The easiest way to understand 2’s

complement is in terms of a VCR counter

• When you rewind a tape and the counter goes

past zero, does it register negative values?

0 0 2

2’s Complement & -1

• Use a 1 byte (8 bit) integer as an example

Use all 1’s (our highest digit in base 2) for -1

Thus we use 11111111 to represent -1

• We furthermore represent 0 as all zero’s

What happens we we add -1 and 1 using these?

• 1 1 1 1 1 1 1 1

+ 0 0 0 0 0 0 0 1

1 0 0 0 0 0 0 0 0

Carried Bit is Lost

8 Bit Integer Example, Cont.

• With our system we don’t need separate

algorithms for addition and subtraction.

We simply add the negative of a number

• So -1 plus -1 is 1 1 1 1 1 1 1 1

+ 1 1 1 1 1 1 1 1

(-2) 1 1 1 1 1 1 1 1 0

• And 1 less than -2 is 1 1 1 1 1 1 1 0

+ 1 1 1 1 1 1 1 1

(-3) 1 1 1 1 1 1 1 0 1

8 Bit Integer Example, Cont.

• It is not hard to see from the trend that the

most negative number is 10000002= -128 (-27)

and the most positive is 01111112 = 127 (27-1)

• The algorithm for negation is to reverse

all bits and add 1, thus two’s complement.

So that - 01111112 = 10000012= -127

• This even works for 0 1 1 1 1 1 1 1 1

+ 0 0 0 0 0 0 0 1

1 0 0 0 0 0 0 0 0

8 Bit Real Variable Example

• Real numbers must be represented by a

mantissa and an exponent (each with a sign)

• Computers use 0.XXX E2 +ZZ rather than

the X.XX E10 +YY of scientific notation

• Divide the 8 bits up into 3 bits for the

exponent and 5 bits for the mantissa, with

each using a sign bit (not really, but nearly)

So we have Z Z X X X X

Exponent Mantissa

8 Bit Real, Machine

• Machine is the error relative to real #

representation of 1 0 1 1 0 0 0

• Addition of two reals requires that their

exponents be the same 0 1 0 0 0 1

• So the smallest number which can be added

to real 1without losing the mantissa bit is

shown above (0x2-1 + 0x2-2 + 0x2-3

+1x2-4 ) x 21 = 0.125 Machine here

Exponent Mantissa

Exponent Mantissa

32 Bit Real Representations

• IEEE standard real numbers have 8 bit

exponents and 24 bit mantissas

• Machine for this system is then

.000000000000000000000012 = 2-23

= 1.192093 x 10-7

• Overflow for this system is then

.111111111111111111111112 x E2 21111111

= (1- machine ) x 2127+1 = 3.40282 x 10+38

(note that the first mantissa bit after . is

assumed as 1--thus 127+1 factors of 2)

TAYLOR SERIES

• Provides a means to predict a function value

at one point in terms of the function value

and its derivative at another point

• Zero order approximation

i1i xfxf

This is good if the function is a constant.

Taylor Series Expansion

• First order approximation

{

slope multiplied by distance

Still a straight line but capable of predicting

an increase or decrease - LINEAR

i1iii1i xxx'fxfxf


• Second order approximation - captures

some of the curvature

2

i1ii

i1iii1i xx!2

x''fxxx'fxfxf


1ii

1n1n

n

i1i

n

ni

n

......3i

2iiii1i

xxh!1n

fR

xxsizestephwhere

Rh!n

xfh

!3

x'''f

h!2

x''fhx'fxfhxfxf

Example

Use zero through fourth order Taylor series expansion

to approximate f(1) from x = 0 (i.e. h = 1.0)

2.1x25.0x5.0x15.0x1.0xf 234

-1

-0.5

0

0.5

1

1.5

0 0.5 1 1.5

x

f(x)

Note:

f(1) = 0.2

-1.5

-1

-0.5

0

0.5

1

1.5

0 0.5 1 1.5

x

f(x

)

n = 3

n = 0

n = 1

n = 2

actual value

Functions with infinite number of

derivatives

• f(x) = cos x

• f `(x) = -sin x

• f ``(x) = -cos x

• f ```(x) = sin x

• Evaluate the system where xi = /4 and

xi+1 = /3

• h = /3 - /4 = /12

Functions with infinite number of

derivatives

• Zero order

f( /3) cos ( /4 ) = 0.707 t = 41.4%

• First order

f( /3) cos ( /4 ) - sin ( /4 )( /12)

t = 4.4%

• Second order

f( /3) 0.4978 t = 0.45%

• By n = 6 t = 2.4 x 10-6 %

Roots (Quadratic Formula)

0cbxax)x(f

a2

ac4bbx

2

2

This equation gives us the roots of the algebraic function

f(x)

i.e. the value of x that makes f(x) = 0

How can we solve for f(x) = e-x - x?

Roots of Equations

• Plot the function and determine where it

crosses the x-axis

• Lacks precision

• Trial and error

-10

-5

0

5

10

15

-5 0 5 10

x

f(x

)

f(x) = e-x

- x

Overview of Methods

• Bracketing methods

– Graphing method

– Bisection method

– False position

• Open methods

– One point iteration

– Newton-Raphson

– Secant method

Bracketing Methods

• Graphical

• Bisection method

• False position method

Graphical (limited practical value)

x

f(x)

x

f(x)

x

f(x)

x

f(x)

consider lower

and upper bound

same sign,

no roots or

even # of roots

opposite sign,

odd # of roots

Bisection Method

• Takes advantage of sign changing

• f(xl)f(xu) < 0 where the subscripts refer to

lower and upper bounds

• There is at least one real root

x

f(x)

x

f(x)

x

f(x)

-10

-5

0

5

10

15

-5 0 5 10

x

f(x

)f(x) = e

-x - x

•f(x) = e-x - x

•xl = -1

•xu = 1

PROBLEM STATEMENT

Use the bisection method

to determine the root

SOLUTION

• f(x) = e-x - x

• xl = -1 xu = 1

– check if f(xl) f(xu) < 0

– f(-1) f(1) = (3.72)(-0.632) < 0

• xr = (xl + xu) / 2 = 0

• f(0) = 1 exchange so that xl = 0

• xl = 0 xu = 1

Solution cont.

• xl = 0 xu = 1 SWITCH LOWER LIMIT

check if f(xl) f(xu) < 0

f(0) f(1) = (1) (-0.632) < 0

• xr = (xl + xu) / 2 = (0 + 1)/2 = 0.5

• f (0.5) = 0.1065

• xl = 0.5 xu = 1 SWITCH LOWER LIMIT

• xr = (xl + xu) / 2 = (0.5 + 1)/2 = 0.75

• f (0.75) = -0.2776

Solution cont.

• xl = 0.5 xu = 0.75 SWITCH UPPER LIMIT

• xr = (xl + xu) / 2 = (0.5 + 0.75)/2 = 0.625

• f (0.625) = -0.090

• xl = 0.5 xu = 0.625 SWITCH UPPER LIMIT

• xr = (0.5 + 0.625) / 2 = 0.5625

• f (0.5625) = 0.007

Solution cont.

• Here we consider an error that is not

contingent on foreknowledge of the root

• a = f (present and previous approx.)

-10

-5

0

5

10

15

-5 0 5 10

x

f(x

)f(x) = e

-x - x

False Position Method

• “Brute Force” of bisection method is

inefficient

• Join points by a straight line

• Improves the estimate

• Estimating the curve by a straight line gives

the “false position”

xl

xu

f(xl)

f(xu) next estimate

real root

DEVELOP METHOD BASED ON

SIMILAR TRIANGLES

xl

xu

f(xl)

f(xu) next estimate, xr

ul

uluur

ur

u

lr

l

xfxf

xxxfxx

xx

xf

xx

xf

Based on

similar

triangles

Example

• f(x) = x3 - 98

∙ x = 981/3 = 4.61

• xl = 4.55 f(xl) = -3.804

• xu= 4.65 f(xu) = 2.545

• xr = 4.65 - (2.545)(4.55-4.65)/(-3.804-2.545)

• xr= 4.6099 f(xr) = -0.03419

– if f(xl)f(xr) > 0 xl = xr

– if f(xl)f(xr) < 0 xu = xr

Example (continued)

• xl = 4.61 f(xl) = -0.034

• xu= 4.65 f(xu) = 2.545

• xr = 4.610

• xr= 4.6104 f(xr) = -0.0004

• a = 0.011%

Pitfalls of False Position Method

f(x) = x10

- 1

-5

0

5

10

15

0 0.5 1 1.5

x

f(x

)

Open Methods

• Fixed point iteration

• Newton-Raphson method

• Secant method

• Multiple roots

• In the previous bracketing methods, the root

is located within an interval prescribed by

an upper and lower boundary

Open Methods cont.

• Such methods are said to be convergent

– solution moves closer to the root as the

computation progresses

• Open method

– single starting value

– two starting values that do not necessarily

bracket the root

• These solutions may diverge

– solution moves farther from the root as the

computation progresses

Pick initial

estimate xi

xi

f(x)

x

f(xi)

draw a tangent

at f(xi)

xi

f(x)

x

f(xi)

At the intersection

with the x-axis

we get xi+1 and

f(xi+1 )

f(x)

x xi+1

f(xi+1 )

The tangent

gives next

estimate. xi

f(x)

x

f(xi)

xi+1

f(xi+1 )

Solution can “overshoot”

the root and potentially

diverge

x0

f(x)

x

x1

Solution can “overshoot”

the root and potentially

diverge

x0

f(x)

x

x1 x2

Fixed point iteration

• Open methods employ a formula to predict

the root

• In simple fixed point iteration, rearrange the

function f(x) so that x is on the left hand

side of the equation

i.e. for f(x) = x2 - 2x + 3 = 0

x = (x2 + 3) / 2

Simple fixed point iteration

• In simple fixed point iteration, rearrange the

function f(x) so that x is on the left hand

side of the equation

i.e. for f(x) = sin x = 0

x = sin x + x

• Let x = g(x)

• New estimate based on

x i+1 = g(xi)

Example

• Consider f(x) = e-x -3x

• g(x) = e-x / 3

• Initial guess x = 0 f(x) = e-x

-3x

-15

-10

-5

0

5

10

15

-4 -2 0 2 4 6

x

f(x

)

Initial guess 0.000

g(x) f(x) a

0.333 -0.283

0.239 0.071 39.561

0.263 -0.018 9.016

0.256 0.005 2.395

0.258 -0.001 0.612

0.258 0.000 0.158

0.258 0.000 0.041

f(x) = e-x

-3x

-15

-10

-5

0

5

10

15

-4 -2 0 2 4 6

x

f(x

)

Newton Raphson most widely used

xi+2

f(x)

x

Note how the

new estimates

converge

i.e. xi+2

is closer to the

root, f(x) = 0.

Newton Raphson

i

ii1i

1ii

ii

x'f

xfxx

rearrange

xx

0xfx'f

'fdx

dygenttan

f(xi)

xi

tangent

xi+1

Newton Raphson Pitfalls

f(x)

(x)

Newton Raphson Pitfalls

f(x)

(x)

solution diverges

Example

• f(x) = x2 - 11

• f '(x) = 2x

• initial guess xi = 3

• f(3) = -2

• f '(3) = 6

f(x) = x2 - 11

-20

0

20

40

60

80

100

0 5 10

xf(

x)

Secant method

i1i

i1i

xx

xfxfx'f

Approximate derivative using a finite divided difference

What is this? HINT: dy / dx = lim y / x

Substitute this into the formula for Newton Raphson

Secant method

i1i

i1iii1i

i

ii1i

xfxf

xxxfxx

x'f

xfxx

Substitute finite difference

approximation for the

first derivative into this

equation for Newton

Raphson

Secant method

• Requires two initial estimates

• f(x) is not required to change signs, therefore this

is not a bracketing method

i1i

i1iii1i

xfxf

xxxfxx

Secant method

new estimate

{

initial estimates

slope

between

two

estimates

f(x)

x

Example

• Let’s consider f(x) = e-x - x

-10

-5

0

5

10

15

-5 0 5 10

x

f(x

)

f(x) = e-x

- x

Example cont.

• Choose two starting points

x0 = 0 f(x0 ) =1

x1 = 1.0 f(x1) = -0.632

• Calculate x2

x2 = 1 - (-0.632)(0 - 1)/(1+0.632) = 0.6127

x

f(x)

1

2

new est.

x

f(x)

1

new est.

2

FALSE POSITION

SECANT METHOD

The new estimate

is selected from the

intersection with the

x-axis

Multiple Roots

• Corresponds to a point where

a function is tangential to the

x-axis

• i.e. double root

f(x) = x3 - 5x2 + 7x -3

f(x) = (x-3)(x-1)(x-1)

i.e. triple root

f(x) = (x-3)(x-1)3

f(x) = (x-3)(x-1)2

-4

-2

0

2

4

6

8

10

0 1 2 3 4 5

x

f(x

)

double root

Difficulties

• Bracketing methods won’t work

• Limited to methods that may diverge

f(x) = (x-3)(x-1)2

-4

-2

0

2

4

6

8

10

0 1 2 3 4 5

x

f(x

)

double root

• f(x) = 0 at root

• f '(x) = 0 at root

• Hence, zero in the

denominator for

Newton-Raphson

and Secant

Methods

f(x) = (x-3)(x-1)3

-3

-2

-1

0

1

2

3

4

5

6

0 1 2 3 4

x

f(x

)

triple root

Multiple Roots

ii

2

i

iii1i

x''fxfx'f

x'fxfxx

f(x) = (x-3)(x-1)4

-4

-2

0

2

4

6

8

10

12

14

16

0 1 2 3 4

x

f(x

)

quadruple

root

Systems of Non-Linear Equations

• We will later consider systems of linear

equations

f(x) = a1x1 + a2x2+...... anxn - C = 0

where a1 , a2 .... an and C are constant

• Consider the following equations

y = -x2 + x + 0.5

y + 5xy = x3

• Solve for x and y

Systems of Non-Linear Equations cont.

• Set the equations equal to zero

y = -x2 + x + 0.5

y + 5xy = x3

• u(x,y) = -x2 + x + 0.5 - y = 0

• v(x,y) = y + 5xy - x3 = 0

• The solution would be the values of x and y

that would make the functions u and v equal

to zero

Recall the Taylor Series

i1i

n

ni

n

......

3i2iii1i

xxsizestephwhere

Rh!n

xf

h!3

x'''fh

!2

x''fhx'fxfxf

Write 2 Taylor series with respect

to u and v

HOTyyy

vxx

x

vvv

HOTyyy

uxx

x

uuu

i1ii

i1ii

i1i

i1ii

i1ii

i1i

The root estimate corresponds to the point where

ui+1 = vi+1 = 0

First Order Taylor Series

Approximation

ii1ii

i1ii

ii1ii

i1ii

vyyy

vxx

x

v

uyyy

uxx

x

u

Defining x = xi+1 – xi & y = yi+1 – yi

Then in matrix form, the equations are

i

i

ii

ii

v

u

y

x

y

v

x

vy

u

x

u

This can be solved for xi+1 & yi+1

This is a 2 equation version of Newton-Raphson

x

v

y

u

y

v

x

ux

vu

x

uv

yy

x

v

y

u

y

v

x

u

y

uv

y

vu

xx

iiii

ii

ii

i1i

iiii

ii

ii

i1i

Therefore

THE DENOMINATOR

OF EACH OF THESE

EQUATIONS IS

FORMALLY

REFERRED TO

AS THE DETERMINANT

OF THE

JACOBIAN

This is a 2 equation version of Newton-Raphson

x

v

y

u

y

v

x

ux

vu

x

uv

yy

x

v

y

u

y

v

x

u

y

uv

y

vu

xx

iiii

ii

ii

i1i

iiii

ii

ii

i1i

Example

• Determine the roots of the following

nonlinear simultaneous equations

y = -x2 + x + 0.5

y + 5xy = x3

• u(x,y) = -x2 + x + 0.5 - y = 0

• v(x,y) = y + 5xy - x3 = 0

• Use an initial estimate of x = 0, y =1

Or alternately

i ii i

i 1 i ii i i i

i ii i

i 1 i ii i i i

v uu v

y yx x x

u v u v

x y y x

u vv u

x xy y yu v u v

x y y x

THE

SAME

DETERMINANT

OF THE

JACOBIAN

This is the Δ form of the equations

Example cont.

x51y

vx3y5

x

v

1y

u1x2

x

u

2

First iteration: [J] = 6

x = -0.08333

y = 0.41667

System of Linear Equations

• We have focused our last lectures on

finding a value of x that satisfied a single

equation

f(x) = 0

• Now we will deal with the case of

determining the values of x1, x2, .....xn, that

simultaneously satisfy a set of equations

System of Linear Equations

• Simultaneous equations

f1(x1, x2, .....xn) = 0

f2(x1, x2, .....xn) = 0

.............

f3(x1, x2, .....xn) = 0

• Methods will be for linear equations

a11x1 + a12x2 +...... a1nxn =c1

a21x1 + a22x2 +...... a2nxn =c2

.............. an1x1 + an2x2 +...... annxn =cn

Mathematical Background

Matrix Notation

• a horizontal set of elements is called a row

• a vertical set is called a column

• first subscript refers to the row number

• second subscript refers to column number

mn3m2m1m

n2232221

n1131211

a...aaa

....

a...aaa

a...aaa

A

This matrix has m rows an n column.

It has the dimensions m by n (m x n)

note

subscript

mn3m2m1m

n2232221

n1131211

a...aaa

....

a...aaa

a...aaa

A

mn3m2m1m

n2232221

n1131211

a...aaa

....

a...aaa

a...aaa

Arow 2

column 3

Note the consistent

scheme with subscripts

denoting row,column

Row vector: m=1

Column vector: n=1 Square matrix: m = n

n21 b.......bbB

m

2

1

c

.

.

c

c

C

333231

232221

131211

aaa

aaa

aaa

A

333231

232221

131211

aaa

aaa

aaa

A

The diagonal consists of the elements

a11 a22 a33

• Symmetric matrix

• Diagonal matrix

• Identity matrix

• Upper triangular matrix

• Lower triangular matrix

• Banded matrix

Symmetric Matrix

aij = aji for all i’s and j’s

872

731

215

A Does a23 = a32 ?

Yes. Check the other elements

on your own.

Diagonal Matrix

A square matrix where all elements off

the main diagonal are zero

44

33

22

11

a000

0a00

00a0

000a

A

Identity Matrix

A diagonal matrix where all elements on

the main diagonal are equal to 1

1000

0100

0010

0001

A

The symbol [I] is used to denote the identify matrix.

Upper Triangle Matrix

Elements below the main diagonal are

zero

33

2322

131211

a00

aa0

aaa

A

Lower Triangular Matrix

All elements above the main diagonal

are zero

872

031

005

A

Banded Matrix

All elements are zero with the exception

of a band centered on the main diagonal

4443

343332

232221

1211

aa00

aaa0

0aaa

00aa

A

Matrix Operating Rules

• Addition/subtraction

add/subtract corresponding terms

aij + bij = cij

• Addition/subtraction are commutative

[A] + [B] = [B] + [A]

• Addition/subtraction are associative

[A] + ([B]+[C]) = ([A] +[B]) + [C]


• Multiplication of a matrix [A] by a scalar g

is obtained by multiplying every element of

[A] by g

mn2m1m

n22221

n11211

ga...gaga

......

......

......

ga...gaga

ga...gaga

AgB


• The product of two matrices is represented as

[C] = [A][B]

n = column dimensions of [A]

n = row dimensions of [B]

n

1k

kjikij bac

Simple way to check whether

matrix multiplication is possible

[A] m x n [B] n x k = [C] m x k

interior dimensions

must be equal

exterior dimensions conform to dimension of resulting matrix

Matrix multiplication

• If the dimensions are suitable, matrix

multiplication is associative

([A][B])[C] = [A]([B][C])

• If the dimensions are suitable, matrix

multiplication is distributive

([A] + [B])[C] = [A][C] + [B][C]

• Multiplication is generally not commutative

[A][B] is not equal to [B][A]

Inverse of [A]

Transpose of [A]

IAAAA11

mnn2n1

2m2212

1m2111

t

a...aa

......

......

......

a...aa

a...aa

A

bcaddc

ba

Determinants

Denoted as det A or A

for a 2 x 2 matrix

bcaddc

ba

Determinants cont.

There are different schemes used to compute the determinant.

Consider cofactor expansion

- uses minor and cofactors of the matrix

Minor: the minor of an entry aij is the determinant of the

submatrix obtained by deleting the ith row and the jth

column

Cofactor: the cofactor of an entry aij of an n x n matrix

A is the product of (-1)i+j and the minor of aij

333231

232221

131211

32

aaa

aaa

aaa

aofminor

Minor: the minor of an entry aij is the determinant of the

submatrix obtained by deleting the ith row and the jth

column.

Example: the minor of a32 for a 3x3 matrix is:

13222312

2322

1312

333231

232221

131211

31 aaaaaa

aa

aaa

aaa

aaa

aofminor

Cofactor: Aij, the cofactor of an entry aij of an n x n matrix

A is the product of (-1)i+j and the minor of aij

i.e. Calculate A31 for a 3x3 matrix

First calculate the minor a31

Minors and cofactors are used to calculate the determinant

of a matrix.

Consider an n x n matrix expanded around the ith row

ini2i1 AAA in2i1i a......aaA

Consider expanding around the jth column

nj2j1j AAA njj2j1 a......aaA

(for any one value of i)

(for any one value of j)

3231

2221

13

3331

2321

12

3332

2322

11

333231

232221

131211

aa

aaa

aa

aaa

aa

aaa

aaa

aaa

aaa

D

3122322113

31

3123332112

21

3223332211

11

aaaaa1

aaaaa1aaaaa1det

Example: Calculate the determinant of the following 3x3 matrix.

First, calculate it using the 1st row (the way you

probably have done it all along).

Then try it using the 2nd row.

516

234

971

Properties of Determinants

• det A = det AT

• If all entries of any row or column are zero,

then det A = 0

• If two rows or two columns are identical,

then det A = 0

How to represent a system of

linear equations as a matrix

[A]{X} = {C}

where {X} and {C} are both column vectors

44.0

67.0

01.0

x

x

x

5.03.01.0

9.115.0

152.03.0

}C{}X{A

44.0x5.0x3.0x1.0

67.0x9.1xx5.0

01.0xx52.0x3.0

3

2

1

321

321

321

Graphical Method

2 equations, 2 unknowns

22

21

22

212

12

11

12

112

2222121

1212111

a

cx

a

ax

a

cx

a

ax

cxaxa

cxaxa

x2

x1

( x1, x2 )

1x2

1x

9x2

3x

2x2x

18x2x3

12

12

21

21

x2

x1

( 4 , 3 )

3

2

2

1

9

1

Check: 3(4) + 2(3) = 12 + 6 = 18

Special Cases

• No solution

• Infinite solution

• Ill-conditioned

x2

x1

( x1, x2 )

a) No solution - same slope f(x)

x b) infinite solution

f(x)

x

-1/2 x1 + x2 = 1

-x1 +2x2 = 2

c) ill conditioned

so close that the points of

intersection are difficult to

detect visually

f(x)

x

Let’s consider how we know if the system is

ill-conditions. Start by considering systems

where the slopes are identical

• If the determinant is zero, the slopes are

identical

2222121

1212111

cxaxa

cxaxa

Rearrange these equations so that we have an

alternative version in the form of a straight

line: i.e. x2 = (slope) x1 + intercept

22

21

22

212

12

11

12

112

a

cx

a

ax

a

cx

a

ax

If the slopes are nearly equal (ill-conditioned)

0aaaa

aaaa

a

a

a

a

12212211

12212211

22

21

12

11

Adetaa

aa

2221

1211

Isn’t this the determinant?

If the determinant is zero the slopes are equal.

This can mean:

- no solution

- infinite number of solutions

If the determinant is close to zero, the system is ill

conditioned.

So it seems that we should use check the determinant

of a system before any further calculations are done.

Let’s try an example.

Example

Determine whether the following matrix is ill-conditioned.

12

22

x

x

5.22.19

7.42.37

2

1

Solution

76.2

2.197.45.22.375.22.19

7.42.37

What does this tell us? Is this close to zero? Hard to say.

If we scale the matrix first, i.e. divide by the largest

a value in each row, we can get a better sense of things.

-80

-60

-40

-20

0

0 5 10 15

x

y

This is further justified

when we consider a graph

of the two functions.

Clearly the slopes are

nearly equal

1 0126

1 01300 004

.

..

Cramer’s Rule

• Not efficient for solving large numbers of

linear equations

• Useful for explaining some inherent

problems associated with solving linear

equations.

bxA

b

b

b

x

x

x

aaa

aaa

aaa

3

2

1

3

2

1

333231

232221

131211

Cramer’s Rule

to solve for

xi - place {b} in

the ith column

33323

23222

13121

1

aab

aab

aab

A

1x

33323

23222

13121

1

aab

aab

aab

A

1x

Cramer’s Rule

to solve for

xi - place {b} in

the ith column

Cramer’s Rule

to solve for

xi - place {b} in

the ith column

33231

22221

11211

3

33331

23221

13111

2

33323

23222

13121

1

baa

baa

baa

A

1x

aba

aba

aba

A

1x

aab

aab

aab

A

1x

Example: Use of Cramer’s Rule

5

5

x

x

11

32

5xx

5x3x2

2

1

21

21

Note the substitution

of {c} in [A]

15

51552

5

1

51

52

5

1x

45

205315

5

1

15

35

5

1x

5321312A

5

5

x

x

11

32

2

1

2

1

Elimination of Unknowns ( algebraic approach)

2112221111121

1212122111121

112222121

211212111

2222121

1212111

caxaaxaa

SUBTRACTcaxaaxaa

acxaxa

acxaxa

cxaxa

cxaxa

Elimination of Unknowns

( algebraic approach)

21122211

2121221

11222112

2111212

1122112112222112

2112221111121

1212121̀211121

aaaa

cacax

aaaa

cacax

acacxaaxaa

caxaaxaa

SUBTRACTcaxaaxaa

NOTE: same result as

Cramer’s Rule

Gauss Elimination

• One of the earliest methods developed for

solving simultaneous equations

• Important algorithm in use today

• Involves combining equations in order to

eliminate unknowns

Blind (Naive) Gauss Elimination

• Technique for larger matrix

• Same principles of elimination

– manipulate equations to eliminate an unknown

from an equation

– Solve directly then back-substitute into one of

the original equations

Two Phases of Gauss Elimination

''

3

''

33

'

2

'

23

'

22

1131211

3333231

2232221

1131211

c|a00

c|aa0

c|aaa

c|aaa

c|aaa

c|aaa

Forward

Elimination

Note: the prime indicates

the number of times the

element has changed from

the original value. Also,

the extra column for c’s

makes this matrix form

what is called augmented.

Two Phases of Gauss Elimination

11

31321211

'

22

3

1

23

'

22

''

33

''

33

''

3

''

33

'

2

'

23

'

22

1131211

a

xaxacx

a

xacx

a

cx

c|a00

c|aa0

c|aaa

Back substitution

Example

3x9x5x2

1x7x4x4

1x3xx2

321

321

321

1xx2x0

21x67x24x44

321

321

2 3 1

4 4 7 1

2 5 9 3

1 2 3

1 2 3

1 2 3

x x x

x x x

x x x

3x9x5x2

1x7x4x4

2x6x2x4

321

321

321

a21 / a11 = 4/2 = 2 (called pivot factor for row 2)

Subtract the (temporary) revised first equation from

the second equation

Multiply equation 1 by a31/a11 = 2/2 = 1

and subtract it from equation 3

2x6x4x0

13x39x15x22

321

321

The equivalent matrix algorithm for calculating

the revised entries in row 3 (a31, a32, a33, & a34) is

note that a31 can be assumed 0

kk

ki

jkji

'

jia

aaaa

Where i is the row being modified 3 in this case

j is the column in row i 1,2,3, & 4 in this case

k is index of the x eliminated 1 in this case

2x6x4

1xx2

1x3xx2

3x9x5x2

1x7x4x4

1x3xx2

32

32

321

321

321

321

We now replace equations 2 and 3 in the revised matrix

(Note that 1st eq. is still original)

Continue the

computation

by multiplying

the second equation

by a32’/a22’ = 4/2 =2

4x4

1xx2

1x3xx2

2x6x4

1xx2

1x3xx2

3

32

321

32

32

321

THIS DERIVATION OF

AN UPPER TRIANGULAR MATRIX

IS CALLED THE FORWARD

ELIMINATION PROCESS

From the system we immediately calculate:

14

4x3

Continue to back substitute

4x4

1xx2

1x3xx2

3

32

321

2

1

2

131x

12

11x

1

2

THIS SERIES OF

STEPS IS THE

BACK

SUBSTITUTION

Pitfalls of the Elimination

Method

• Division by zero

• Round off errors

– magnitude of the pivot element is small

compared to other elements

• Ill conditioned systems

Division by Zero

• When we normalize i.e. a12/a11 we need to

make sure we are not dividing by zero

• This may also happen if the coefficient is

very close to zero

5x6xx2

3x7x6x4

8x3x2

321

321

32

Techniques for Improving the

Solution

• Use of more significant figures

• Pivoting

• Scaling

bxA

b

b

b

x

x

x

aaa

aaa

aaa

3

2

1

3

2

1

333231

232221

131211

Use of more significant figures

• Simplest remedy for ill conditioning

• Extend precision

– computational overhead

– memory overhead

Pivoting

• Problems occur when the pivot element is

zero - division by zero

• Problems also occur when the pivot element

is smaller in magnitude compared to other

elements (i.e. round-off errors)

• Prior to normalizing, determine the largest

available coefficient

Pivoting

• Partial pivoting

– rows are switched so that the largest element is

the pivot element

• Complete pivoting

– columns as well as rows are searched for the

largest element and switched

– rarely used because switching columns changes

the order of the x’s, adding unjustified

complexity to the computer program

Division by Zero - Solution

Pivoting has been developed

to partially avoid these problems

5x6xx2

8x3x2

3x7x6x4

321

32

321

5x6xx2

3x7x6x4

8x3x2

321

321

32

Scaling

• Minimizes round-off errors for cases where some

of the equations in a system have much larger

coefficients than others

• In engineering practice, this is often due to the

widely different units used in the development of

the simultaneous equations

• As long as each equation is consistent, the system

will be technically correct and solvable

Scaling

998,49x999,49

000,100x000,100x2

2xx2xx

1xx00002.0000,100x000,100x2

2

21

2121

2121

1xx00.1x00.0x 2121

Piv

ot

row

s to

put

the

gre

ate

st

valu

e on t

he

dia

gonal

EXAMPLE

Use Gauss Elimination to solve the following set

of linear equations

4x8x4

45xx6x2

50x13x3

31

321

32

4x8x4

45xx6x2

50x13x3

31

321

32

SOLUTION

First write in matrix form, employing short hand

presented in class.

0 3 13 50

2 6 1 45

4 0 8 4

We will clearly run into

problems of division

by zero.

Use partial pivoting

501330

43360

4804

501330

45162

4804

4804

45162

501330

Begin developing

upper triangular matrix

okay50966.113149.83

CHECK

931.24

966.184x

149.86

966.1343x966.1

5.14

5.28x

5.285.1400

43360

4804

501330

43360

4804

1

23

...end of

problem

Matrix Inversion with Gauss-

Jordan Method

• Gauss-Jordan (Like Gauss Elim.--Direct Sol.)

– primary motive for introducing this method is that

it provides and simple and convenient method for

computing the matrix inverse

• Gauss-Seidel (Ch. 11)

– fundamentally different from Gauss elimination

– this is an approximate, iterative method

– particularly good for large number of equations

Gauss-Jordan

• Variation of Gauss elimination

• When an unknown is eliminated, it is eliminated

from all other equations, rather than just the

subsequent one

• All rows are normalized by dividing them by their

pivot elements

• Elimination step results in and identity matrix

rather than an UT matrix

33

2322

131211

a00

aa0

aaa

A

1000

0100

0010

0001

A

n

33

n

22

n

1

n

3

n

2

n

cx

cx

cx

c|100

c|010

c|001

1

1

Graphical depiction of Gauss-Jordan

n

3

n

2

n

''

3

''

333231

'

2

'

23

'

2221

1131211

c|100

c|010

c|001

c|aaa

c|aaa

c|aaa

1

Matrix Inversion

• [A] [A] -1 = [A]-1 [A] = I

• One application of the inverse is to solve

several systems differing only by {c}

– [A]{x} = {c}

– [A]-1[A] {x} = [A]-1{c}

– [I]{x}={x}= [A]-1{c}

• One quick method to compute the inverse is

to augment [A] with [I] instead of {c}

Graphical Depiction of the Gauss-Jordan

Method with Matrix Inversion

1

1

33

1

32

1

31

1

23

1

22

1

21

1

13

1

12

1

11

333231

232221

131211

AI

aaa100

aaa010

aaa001

100aaa

010aaa

001aaa

IA

Note: the superscript

“-1” denotes that

the original values

have been converted

to the matrix inverse,

not 1/aij

EXAMPLE

Use the Gauss-Jordan Method to solve the

following problem.

Note: You can use this solution to practice Gauss

elimination method.

65x5x5

29x7x2

28x6x5x4

21

31

321

65055

29702

28654

65x5x5

29x7x2

28x6x5x4

21

31

321

SOLUTION

First, write the coefficients

and the right hand vector

as an augmented matrix

4 5 6 28

2 0 7 29

5 5 0 65

1 125 15 7

2 0 7 29

5 5 0 65

. . Normalize the first row by

dividing by the pivot

element 4

Example: -6/4 = -1.5

1 125 15 7

2 0 7 29

5 5 0 65

. .

The x1 term in the second

row can be eliminated by

subtracting 2 times the

first row from the second row

In other words, we want this

term to be zero

1 125 15 7

2 0 7 29

5 5 0 65

1 125 15 7

5 5 0 65

. .

. .

? ? ? ?





All values

in this row

will change

1 125 15 7

2 0 7 29

5 5 0 65

1 125 15 7

5 5 0 65

. .

. .

?





2 - (1)

1 125 15 7

2 0 7 29

5 5 0 65

1 125 15 7

5 5 0 65

. .

. .

?





2

1 125 15 7

2 0 7 29

5 5 0 65

1 125 15 7

5 5 0 65

. .

. .

?





2 - (1)(2)

1 125 15 7

2 0 7 29

5 5 0 65

1 125 15 7

0

5 5 0 65

. .

. .





2 - (1)(2) = 0

1 125 15 7

2 0 7 29

5 5 0 65

1 125 15 7

0

5 5 0 65

. .

. .

?





0

65055

?0

75.125.11

65055

29702

75.125.11





0 - (2)

65055

?0

75.125.11

65055

29702

75.125.11





0 - (2)(1.25)

65055

5.20

75.125.11

65055

29702

75.125.11





0 - (2)(1.25) = -2.5

65055

1545.20

75.125.11

65055

29702

75.125.11





These term are derived

following the same algorithm.

305.725.10

1545.20

75.125.11

65055

1545.20

75.125.11

The x1 term in the third


subtracting -5 times the


Now we need to reduce the x2 term from

the 1st and 3rd equation

305.725.10

1545.20

75.125.11



305.725.10

1545.20

75.125.11



Normalize the second row by a22

305.725.10

1545.20

75.125.11




305.725.10

1545.20

75.125.11

1 125 15 7

0 1 16 6

0 125 7 5 30

. .

.

. .




1 125 15 7

0 1 16 6

0 125 7 5 30

. .

.

. .

-2.5/(-2.5) = 1

305.725.10

1545.20

75.125.11

1 125 15 7

0 1 16 6

0 125 7 5 30

. .

.

. .




15/(-2.5) = -6

305.725.10

1545.20

75.125.11

305.725.10

66.110

01

305.725.10

66.110

75.125.11

1.25 - 1.25 (1) = 0

305.725.10

66.110

?01

305.725.10

66.110

75.125.11

Before going any further

calculate the new coefficient

for a13

305.725.10

66.110

?01

305.725.10

66.110

75.125.11

Before going any further

calculate the new coefficient

for a13

Your answer should

follow the following scheme

305.725.10

66.110

?01

305.725.10

66.110

75.125.11

-1.5

305.725.10

66.110

?01

305.725.10

66.110

75.125.11

-1.5 - 1.6

-1.5 - 1.6 (1.25)

The element to get the

zero at a12

305.725.10

66.110

?01

305.725.10

66.110

75.125.11

305.725.10

66.110

?01

305.725.10

66.110

75.125.11

-1.5 - 1.6 (1.25)

Recall, 1.25

was need to establish

a zero at a12

305.725.10

66.110

5.301

305.725.10

66.110

75.125.11

-1.5 - 1.6 (1.25) = -3.5

Follow the same scheme for c1

and for the third row

5.225.900

66.110

5.145.301

305.725.10

66.110

5.145.301

305.725.10

66.110

75.125.11

5.225.900

66.110

5.145.301

305.725.10

66.110

5.145.301

305.725.10

66.110

75.125.11

1 0 35 14 5

0 1 16 6

0 0 9 5 22 5

1 0 0 22 79

0 1 0 9 79

0 0 1 2 37

. .

.

. .

.

.

.

Now we need

to reduce x3

from the 1st and

2nd equation

In addition, we

will complete the

identity matrix

......end of problem

LU Decomposition Methods

Chapter 10

• Two fundamentally different approaches for solving

systems of linear algebraic equations

• Elimination methods

– Gauss elimination

– Gauss Jordan

– LU Decomposition Methods

• Iterative methods

– Gauss Seidel

– Jacobi

Naive LU Decomposition

• [A]{x}={c}

• Suppose this can be rearranged as an upper

triangular matrix with 1’s on the diagonal

• [U]{x}={d}

• [A]{x}-{c}=0 [U]{x}-{d}=0

• Assume that a lower triangular matrix exists

that has the property

[L]{[U]{x}-{d}}= [A]{x}-{c}

Naive LU Decomposition

• [L]{[U]{x}-{d}}= [A]{x}-{c}

• Then from the rules of matrix multiplication

• [L][U]=[A]

• [L]{d}={c}

• [L][U]=[A] is referred to as the LU

decomposition of [A]. After it is

accomplished, solutions can be obtained

very efficiently by a two-step substitution

procedure

Consider how Gauss elimination can be

formulated as an LU decomposition

U is a direct product of forward

elimination step if each row is scaled by

the diagonal

100

a10

aa1

U 23

1312

Although not as apparent, the matrix [L] is also

produced during the step. This can be readily

illustrated for a three-equation system

3

2

1

3

2

1

333231

232221

131211

c

c

c

x

x

x

aaa

aaa

aaa

The first step is to multiply row 1 by the factor

11

2121

a

af

Subtracting the result from the second row eliminates a21

3

2

1

3

2

1

333231

232221

131211

c

c

c

x

x

x

aaa

aaa

aaa

Similarly, row 1 is multiplied by

11

3131

a

af

The result is subtracted from the third row to eliminate a31

In the final step for a 3 x 3 system is to multiply the modified

row by

22

3232

'a

'af Subtract the results from the third

row to eliminate a32

The values f21 , f31, f32 are in fact the elements

of an [L] matrix

1ff

01f

001

L

3231

21

CONSIDER HOW THIS RELATES TO THE

LU DECOMPOSITION METHOD TO SOLVE

FOR {X}

[A] {x} = {c}

[U][L]

[L] {d} = {c}

{d}

[U]{x}={d} {x}

Crout Decomposition

• Gauss elimination method involves two

major steps

– forward elimination

– back substitution

• Efforts in improvement focused on

development of improved elimination

methods

• One such method is Crout decomposition

Crout Decomposition

Represents and efficient algorithm for decomposing [A]

into [L] and [U]

333231

232221

131211

23

1312

333231

2221

11

aaa

aaa

aaa

100

u10

uu1

0

00

Recall the rules of matrix multiplication.

The first step is to multiply the rows of [L] by the

first column of [U]

3131

2121

111111

a

a

00001a

Thus the first

column of [A]

is the first column

of [L]

333231

232221

131211

23

1312

333231

2221

11

aaa

aaa

aaa

100

u10

uu1

0

00

Next we multiply the first row of [L] by the columns

of [U] to get

131311

121211

1111

au

au

a

333231

232221

131211

23

1312

333231

2221

11

aaa

aaa

aaa

100

u10

uu1

0

00

n,.....3,2jfora

u

au

au

au

au

a

11

j1

j1

11

1313

11

1212

131311

121211

1111

Once the first row of [U] is established

the operation can be represented concisely

333231

232221

131211

23

1312

333231

2221

11

aaa

aaa

aaa

100

u10

uu1

0

00

n,.....3,2jfora

u11

j1

j1

Schematic

depicting

Crout

Decomposition

333231

232221

131211

23

1312

333231

2221

11

aaa

aaa

aaa

100

u10

uu1

0

00

Schematic

depicting

Crout

Decomposition

1n

1k

knnknnnn

jj

1j

1i

ikjijk

jk

1j

1k

kjikijij

11

j1

j1

il1i

ua

n....2j,1jkfor

ua

u

n,.....1j,jiforua

1n......3,2jFor

n,.....3,2jfora

u

n,.....,2,1ifora

The Substitution Step

• [L]{[U]{x}-{d}}= [A]{x}-{c}

• [L][U]=[A]

• [L]{d}={c}

• [U]{x}={d}

• Recall our earlier graphical depiction of the

LU decomposition method

[A] {x} = {c}

[U][L]

[L] {d} = {c}

{d}

[U]{x}={d} {x}

1,.....2,1

,......3,2

1

1

1

11

11

nniforxudx

dx

dxUrecallutionsubstitBack

nifor

dc

d

cd

n

ij

jijii

nn

ii

i

j

jiji

i

Example

Use LU decomposition to solve the following matrix

3a

2a

1a

n,.....,2,1ifora

39

22

15

x

x

x

743

342

231

3131

2121

1111

il1i

3

2

1

2a

u

31

3au

n,.....3,2jfora

u

11

1313

11

1212

11

j1

j1

5334ua

2324ua

n,.....1j,jiforua

12

1k

2kk33232

12

1k

2kk22222

1j

1k

kjikijij

5.0

2

223ua

u

n....2j,1jkfor

ua

u

22

12

1i

3ii223

23

jj

1j

1i

ikjijk

jk

5.35.05237ua

ua

13

1k

3kk33333

1n

1k

knnknnnn

Therefore the L and U matrices are

100

5.010

231

U

5.353

022

001

L

151

15cd

cdLsolvingand

39

22

15

c

c

c

11

11

3

2

1

Recall the original column matrix

4

5.3

4515339dc

d

42

15222dc

d

n,......3,2ifor

dc

d

ii

13

1j

jj33

3

ii

12

1j

jj22

2

ii

1i

1j

jiji

i

Forward substitution

Backsubstitution

1422315xudx

245.04xudx

1,.....2n,1niforxudx

4dx

dx

3

11j

jj111

3

12j

jj322

n

1ij

jijii

33

nn

...end of example

Matrix Inversion

• [A] [A] -1 = [A]-1 [A] = I

• One application of the inverse is to solve

several systems differing only by {c}

– [A]{x} = {c}

– [A]-1[A] {x} = [A]-1{c}

– [I]{x}={x}= [A]-1{c}

• One quick method to compute the inverse is

to augment [A] with [I] instead of {c}

Graphical Depiction of the Gauss-Jordan

Method with Matrix Inversion

1

1

33

1

32

1

31

1

23

1

22

1

21

1

13

1

12

1

11

333231

232221

131211

AI

aaa100

aaa010

aaa001

100aaa

010aaa

001aaa

IA

Note: the superscript

“-1” denotes that

the original values

have been converted

to the matrix inverse,

not 1/aij

Matrix Inversion with LU

Decomposition Method

11 1(1) 11 1(2)

21 22 2(1) 21 22 2(2)

31 32 33 3(1) 31 32 33 3(2)

11 1(3)

21 22 2(3)

31 32 33 3(3)

0 0 d 1 0 0 d 0

0 d 0 0 d 1

d 0 d 0

0 0 d 0

0 d 0

d 1

Solving for unknowns using {d}(1) gives the first column of [I]

and so forth for {d}(2) through {d}(n)

Chapter 11:Iterative Methods

• Solution by Gauss-Seidel Iteration

• Solution by Jacobi Iteration

Gauss Seidel Method

• An iterative approach

• Continue until we converge within some

pre-specified tolerance of error

• Round off is no longer an issue, since you

control the level of error that is acceptable

Gauss-Seidel Method

• If the diagonal elements are all nonzero, the

first equation can be solved for x1

• Solve the second equation for x2, etc.

11

nn131321211

a

xaxaxacx

To assure that you understand this, write the equation for x2

n,n

1n1n,n23n11nn

n

33

nn323213133

22

nn232312122

11

nn131321211

a

xaxaxacx

a

xaxaxacx

a

xaxaxacx

a

xaxaxacx

Gauss-Seidel Method

• Start the solution process by guessing

values of x

• A simple way to obtain initial guesses is to

assume that they are all zero

• Calculate new values of xi starting with

x1 = c1/a11

• Progressively substitute through the

equations

• Repeat until tolerance is reached

33323213133

2222312122

111

111131211

3323213133

2232312122

1131321211

'xa/'xa'xacx

'xa/0a'xacx

'xa

ca/0a0acx

a/xaxacx

a/xaxacx

a/xaxacx

Gauss-Seidel Method for 3 eq. System

Example Given the following augmented matrix,

complete one iteration of the Gauss

Seidel method.

2 3 1 2

4 1 2 2

3 2 1 1

2 3 1 2

4 1 2 2

3 2 1 1

10

1

1231

1

62131x

'xa/'xa'xacx

61

42

1

02142x

'xa/0a'xacx

12

2

2

01032x

'xa

ca/0a0acx

3

33323213133

2

2222312122

1

111

111131211

GAUSS SEIDEL

Gauss-Seidel Method

convergence criterion

sj

i

1j

i

j

ii,a 100

x

xx

as in previous iterative procedures in finding the roots,

we consider the present and previous estimates.

As with the open methods we studied previously with one

point iterations

1. The method can diverge

2. May converge very slowly

Convergence criteria for two

linear equations

0x

v

a

a

x

v

a

a

x

u0

x

u

vanduofsderivativepartialtheconsider

xa

a

a

cx,xv

xa

a

a

cx,xu

222

21

1

11

12

21

1

22

21

22

221

2

11

12

11

121

Class question:

where do these

formulas come from?

Convergence criteria for two linear

equations cont.

1y

v

y

u

1x

v

x

u

Criteria for convergence

in class text material

for nonlinear equations.

Noting that x = x1 and

y = x2

Substituting the previous equation:


equations cont.

1a

a1

a

a

11

12

22

21

This is stating that the absolute values of the slopes must

be less than unity to ensure convergence.

Extended to n equations:

ijexcludingn,1jwhereaa ijii


equations cont.

ijexcludingn,1jwhereaa ijii

This condition is sufficient but not necessary; for convergence.

When met, the matrix is said to be diagonally dominant.

x2

x1

Review the concepts

of divergence and

convergence by graphically

illustrating Gauss-Seidel

for two linear equations

99x9x11:v

286x13x11:u

21

21

x2

x1

Note: initial guess is

x1 = x2 = 0

286x13x11:u

99x9x11:v

21

21

x2

x1

Note: initial guess is

x1 = x2 = 0

286x13x11:u

99x9x11:v

21

21

u

v

x2

x1

286x13x11:u

99x9x11:v

21

21

Solve 2nd eq. for x2

v

u

x2

x1

286x13x11:u

99x9x11:v

21

21

Solve 1st eq. for x1

v

u

x2

x1

286x13x11:u

99x9x11:v

21

21


v

u

x2

x1

286x13x11:u

99x9x11:v

21

21


v

u

x2

x1

286x13x11:u

99x9x11:v

21

21


v

u

x2

x1

Note: we are converging

on the solution

286x13x11:u

99x9x11:v

21

21

v

u

x2

x1

Change the order of

the equations: i.e. change

direction of initial

estimates

99x9x11:v

286x13x11:u

21

21


u

v

x2

x1

99x9x11:v

286x13x11:u

21

21


u

v

x2

x1

99x9x11:v

286x13x11:u

21

21


u

v

x2

x1

99x9x11:v

286x13x11:u

21

21


u

v

x2

x1

This solution is diverging!

99x9x11:v

286x13x11:u

21

21

u

v

Improvement of Convergence

Using Relaxation

This is a modification that will enhance slow convergence.

After each new value of x is computed, calculate a new value

based on a weighted average of the present and previous

iteration.

old

i

new

i

new

i x1xx

Improvement of Convergence Using

Relaxation

• if = 1 unmodified

• if 0 < < 1 underrelaxation

– nonconvergent systems may converge

– hasten convergence by dampening out oscillations

• if 1< < 2 overrelaxation

– extra weight is placed on the present value

– assumption that new value is moving to the correct

solution by too slowly

old

i

new

i

new

i x1xx

Jacobi Iteration

• Iterative like Gauss Seidel

• Gauss-Seidel immediately uses the value of

xi in the next equation to predict x i+1

• Jacobi calculates all new values of xi’s to

calculate a set of new xi values

FIRST ITERATION

x c a x a x a x c a x a x a



SECOND ITERATION


x c a x a x a x c a x

1 1 12 2 13 3 11 1 1 12 2 13 3 11

2 2 21 1 23 3 22 2 2 21 1 23 3 22

3 3 31 1 32 2 33 3 3 31 1 32 2 33

1 1 12 2 13 3 11 1 1 12 2 13 3 11

2 2 21 1 23 3 22 2 2 21 1

/ /

/ /

/ /

/ /

/

a x a


23 3 22

3 3 31 1 32 2 33 3 3 31 1 32 2 33

/

/ /

Graphical depiction of difference between Gauss-Seidel and Jacobi

FIRST ITERATION




SECOND ITERATION



1 1 12 2 13 3 11 1 1 12 2 13 3 11

2 2 21 1 23 3 22 2 2 21 1 23 3 22

3 3 31 1 32 2 33 3 3 31 1 32 2 33

1 1 12 2 13 3 11 1 1 12 2 13 3 11

2 2 21 1 23 3 22 2 2 21 1

/ /

/ /

/ /

/ /

/

a x a


23 3 22

3 3 31 1 32 2 33 3 3 31 1 32 2 33

/

/ /


FIRST ITERATION




SECOND ITERATION



1 1 12 2 13 3 11 1 1 12 2 13 3 11

2 2 21 1 23 3 22 2 2 21 1 23 3 22

3 3 31 1 32 2 33 3 3 31 1 32 2 33

1 1 12 2 13 3 11 1 1 12 2 13 3 11

2 2 21 1 23 3 22 2 2 21 1

/ /

/ /

/ /

/ /

/

a x a


23 3 22

3 3 31 1 32 2 33 3 3 31 1 32 2 33

/

/ /


FIRST ITERATION




SECOND ITERATION



1 1 12 2 13 3 11 1 1 12 2 13 3 11

2 2 21 1 23 3 22 2 2 21 1 23 3 22

3 3 31 1 32 2 33 3 3 31 1 32 2 33

1 1 12 2 13 3 11 1 1 12 2 13 3 11

2 2 21 1 23 3 22 2 2 21 1

/ /

/ /

/ /

/ /

/

a x a


23 3 22

3 3 31 1 32 2 33 3 3 31 1 32 2 33

/

/ /


2 3 1 2

4 1 2 2

3 2 1 1

10

1

1231

1

62131x

'xa/'xa'xacx

61

42

1

02142x

'xa/0a'xacx

12

2

2

01032x

'xa

ca/0a0acx

3

33323213133

2

2222312122

1

111

111131211

RECALL

GAUSS SEIDEL

2 3 1 2

4 1 2 2

3 2 1 1

1

1

1

1

02031x

'xa/0a0acx

21

2

1

02042x

'xa/0a0acx

12

2

2

01032x

'xa

ca/0a0acx

3

333323133

2

222232122

1

111

111131211

... end of problem

JACOBI

Optimization Methods

Overview

• Unconstrained Searches

▀ 1-D (one independent variable)

▲Golden Search

▲Quadratic Interpolation

▲Newton’s Method

▀ 2-D (two or more independent variables)

▲Steepest Ascent (Descent)

▲Conjugate Gradients

▲Newton and Quasi-Newton Methods

• Constrained Optimization

▀ Linear Programming

Optimization Uses in Engineering

● Maximizing Strength/Weight

● Minimizing Cost or Maximizing Profit

● Minimizing Wait Time

● Maximizing Efficiency

● Minimizing Virtual Work

1-D Searches

• Objective is to maximize a function Objective Function

• 1-D searches have objective functions of one independent

variable f(x)

• Analogy to root finding :

►Bracketed Method Interval Search or Limited

Search

►Open Method Unlimited or Free Search

Golden Section Search

• Defining 2 / 1 as R and taking the reciprocal of the above

equation

1

2

21

1

1

2

0

1 or

• Define the search interval as [x ,xu ]

• Define the search interval length as 0 = |xu - x|

• Divide 0 up into 1 and 2 such that 0 = 1 + 2

• Also make sure that the ratio of the first sub-length

to the original length is the same as the first

sub-length to second sub-length ratio

618.02

15R1RR

R

1R1 2

Golden Section Search, Continued

• Once subdivided, we now evaluate values of f(x)

• Since the designations 1 and 2 are arbitrary, test f(x + 1)

and f(xu - 1) to determine the new search interval

• So let x1 = (x + d) and x2 = (xu - d), where d = 1

• Then if f(x1) > f(x2) the maximum must lie in the interval

[xu - d, xu ] call xu - d (= x2 ) the new x (Case 1)

otherwise call x + d (= x1 ) the new xu (Case 2)

• With a little thinking you can convince yourself that for

(Case 1), x2 for this new search interval is x1 from the

previous, because 0 / 1 = 1 / 2 (Hint: d = R 0 )

And for (Case 2) Old x1 x2

x xu

d d

x2 x1

First Step

x xu

Old x2 Old x1

Second Step

x1 x x2

d’

f (x)

f (x)

Text Example 13.1

● Use Golden Search to find max (2 sinx -x2/10) on [0,4]

● First Step: Calculate d = R 0 = 0.61803*4 = 2.4721

● Second Step: Calculate x2 = 4 - 2.4721 = 1.5279

and x1 = 0 +2.4721 = 2.4721

● Third Step: Calculate f(x2) = 1.7647

and f(x1) = 0.6300

● Now because f(x1) is not > f(x2), set xu to 2.4721

& set new x1 = 1.5279

Graphically

Text Example 13.1 f(x)

-4

-3

-2

-1

0

1

2

3

0 1 2 3 4 5

x

f(x)

<-------- l0 = 4 ------------>

x1 x2

f=1.7647

f=0.6300

Newton’s Method for Optimization in 1-D

• Basis: Maximum (or minimum) of f(x) will occur where

f ’(x) = 0

• If we call f ’(x) by the name g(x) and apply Newton-

Raphson to find where g(x) = 0

xi +1 = xi - g (x) / g ’(x)

or

xi +1 = xi - f ’ (x) / f ”(x)

• Method converges to nearest max or min (local max/min)

• Method can “blow up” near points where f ”(x) = 0

if f ’(x) is not going to 0 as fast (or at all) as f ”(x)

0.0 0.5 1.0 1.5 2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

f

g

f (x)

& g

(x)

x

f(x) and its Derivative in Examples on [0,2]

2-D Search Methods

• The Multi-D Equivalent of f ’ is the

Gradient f or [f/x1 i, f/x2 j, f/x3 k...]T

• All 2-D Methods Use f

• We can Conceptualize the Gradient as the

Topographic Elevation Slope in x-y-z space

x1 x

x2 y

f z

-50

-40

-30

-20

-10

0

10

-3 -2

-1 0

1 2

3 4

5 6

-2 -1

0 1

2 3

3D Elevation Plot

3-D ‘Elevations’ Contour Plot

f

Contour Graph of f (x,y)

X

-2 -1 0 1 2 3 4 5

Y

-1

0

1

2

3

-5

-5

-5

-10

-10

-15

-20

0

0

0

0

0

0

0

-5

-5

-5

-5

-10

-10

-10

-15

-15

-20

-20

-25

-30

-35

Steepest Ascent/Descent

• Follow f = g direction (- g for descent)

• How “far” should we go in this direction?

• Call the magnitude of the step h

• Now maximize f along the g direction as a

function of h called a line search

• Step x1i+1 = x1

i + g1h & x2i+1 = x2

i + g2h

Text Example 14.4

• Search for max of f = 2 x1x2 + 2x1 - x12 - 2x2

2

from the point x1 = -1 and x2 = 1

• Calculate g (= 6 i - 6 j), because

g1 = 2 x2 + 2 - 2 x1 = 2 (1) + 2 - 2 (-1) = 6

g2 = 2 x1 - 4 x2 = 2 (-1) - 4 (1) = -6

Text Example 14.4, Continued

• Now search for max of f (x11, x2

1) subject to

x11 = x1

0 + h g1 = -1+6h & x21 = x2

0 + h g2= 1 - 6h

f (x11, x2

1) = -180 h2 +72 h - 7 = g(h)

Max {g(h)} g’(h) =0 -360 h +72 = 0 h=0.2

• Now stepping along g (= 6 i - 6 j) gives

x1i+1 = x1

i + g1h = - 1 +(6)(0.2) = 0.2

x2i+1 = x2

i + g2h = 1 +(-6)(0.2) = -0.2

Graphical Depiction of Steepest Ascent

Steps in Example 14.4


X

-2 -1 0 1 2 3 4 5

Y

-1

0

1

2

3

-5

-5

-5

-10

-10

-15

-20

0

0

0

0

0

0

0

-5

-5

-5

-5

-10

-10

-10

-15

-15

-20

-20

-25

-30

-35

2-D Newton Method

• The 2-D Version of Newton Maximization

Gradient 0 or f = 0 g1 = g2 = 0

• Applying Multi-variate Newton-Raphson

2

1

i

2

1

1i

2

1

2

1

2

1

2

2

1

2

2

1

1

1

x

x

x

x

x

xand

g

g

x

x

x

g

x

g

x

g

x

g

• The first matrix is the Jacobian of the Gradient

. AKA the Hessian

2

2

2

21

221

2

2

1

2

2

2

1

2

2

1

1

1

x

f

xx

f

xx

f

x

f

x

g

x

g

x

g

x

g

H

2-D Newton Method

• Text describes Newton’s equation as

0g

g

x

x

HH

HH

2

1

2

1

2221

1211

• The system is then solved by matrix inversion

2

1

1

2221

1211

i

2

1

1i

2

1

g

g

HH

HH

x

x

x

x

Newton Method for Text Example 14.4

• Search for max of f(x1,x2) = 2 x1x2 + 2x1 - x12 - 2x2

2

from the point x1 = -1 and x2 = 1

• g1 = 2 x2 + 2 - 2 x1 = 2 (1) + 2 - 2 (-1) = 6

• g2 = 2 x1 - 4 x2 = 2 (-1) - 4 (1) = -6

• H11= -2, H12= 2, H21= 2, H22= -4

• Solving,

0

3

x

xgetwe

6

6

x

x

42

22

2

1

2

1

• So x1 = 2, x2 = 1 f = 2 Maximum (Why?)

Graphical Depiction of Newton Step


X

-2 -1 0 1 2 3 4 5

Y

-1

0

1

2

3

-5

-5

-5

-10

-10

-15

-20

0

0

0

0

0

0

0

-5

-5

-5

-5

-10

-10

-10

-15

-15

-20

-20

-25

-30

-35

Constrained Optimization

• Search for maximum is limited by constraints

• Problem related constraints (resource limits)

Plant operates no more than 80 hours per week

Raw materials can not be purchased for less than

$30/ton

• Feasibility constraints

Efficiency can not be greater than unity

Cost can not be negative

Mass can not be negative

Linear Programming

• Multi-D Objective Function Z(x1, x2, x3…)

is a linear function of x1, x2, x3…

• Constraints must be formulated as linear

inequality statements in terms of x1, x2, x3…

• Multi-D Problem Statement

Z = c1 x1 + c2 x2 +… cn xn

a i1 x1 + a i2 x2 +... a in xn bi ( i = 1, m)

Text Example • Maximize profit (Z) in the production of

two products (x1, x2)

• 2-D Problem

Z = 150 x1 + 175 x2

7 x1 + 11 x2 77 (1)

10 x1 + 8 x2 80 (2)

x1 9 (3)

x2 6 (4) (m=4)

x1 0 (5) (Note that these are

x2 0 (6) not constraints)

Graphical Depiction of Constraints

x1

x2

Graphical Depiction Z(x1 , x2) Subject to Constraints

x1

x2

Z = 0 Z = 600 Z = 1400

175

Zx

175

150x 12

Things to Note from Graphical Solution

• Region of possible solutions which meet all

constraints (feasible solutions) is a polygon

• Brute force approach could find max Z by

evaluating Z at all vertices

• Vertices are simultaneous solution of 2

constraints (inequalities) converted to 2

equations (equalities) x1= x1* & x2 = x2

*

• Z evaluation is of Z (x1*, x2

*)

Simplex Method • Converts relations to m equations in m+n

unknowns we must have at least m = 1

• Constraint inequalities are converted to equalities

with new variables called slack variables Si

• a i1x1+ a i2 x2+... + a inxn + Si = bi (i = 1, m)

• Inequality i will then be at its limit when Si = 0

• The independent variables (x’s) are called

structural variables

Simplex Method, Continued

• Slack variables will have positive values if the

values of structural variables meet the inequality

with room to spare

• There will be Comb(n+m,m) different possible

constraint intersections (including non-negativity

constraints)

• Simplex method uses Gauss Jordan elimination

to check only vertices on the feasible polygon

starting with all x’s = 0

Simplex Tableau for Text Example

Basic Z x1 x2 S1 S2 S3 S4 bi

___________________________________________________________________________________

61000100S

9

80

77

0

0100

0010

0001

0000

010S

8100S

1170S

1751501Z

4

3

2

1

• Note that we never do Gauss Jordan elimination on Z row

• 1st row keeps track of current estimate of Z in b column

• Basic variables start out as S’s

• Non-basic variables start out as x’s and are assumed zero

Simplex Elimination


___________________________________________________________________________________

61000100S

9

80

77

0

0100

0010

0001

0000

010S

8100S

1170S

1751501Z

4

3

2

1

• Arbitrarily choose x1for Gauss Jordan elimination (not standard)

• This makes it a basic variable (‘enters’ basic list: entering variable)

• Pivot about S2 row, because 80/10 is smallest bi / ai1 -- any other

choice results in negative S’s after elimination

• This choice makes S2 non-basic (it is the leaving variable)

First Simplex Step - Eliminate x1

61000100S

1

8

21

1200

011.00

001.00

007.01

00150

8.000S

8.010x

4.500S

5501Z

4

3

1

1

• S1 row can be read as S1 = 21 (reduced from 77), because x2 and

S2 are non-basic variables (their values can be taken as zero)

• Similarly x1 row is read x1 = 8

• Equivalent to moving from [0,0] to [8,0] for location of max

• This moves our estimate of max Z from 0 to 1200 (see Z eq.)


___________________________________________________________________________________

Next Elimination Choice

61000100S

1

8

21

1200

011.00

001.00

007.01

00150

8.000S

8.010x

4.500S

5501Z

4

3

1

1

• Now choose x2 as the entering variable (most neg. Z eq. Coef.)

• The S1 row is chosen as pivot row (note: 1/(-0.8) not allowed-why?)

• So now S1 is our leaving variable


___________________________________________________________________________________

Next Simplex Step (Elimination)

11.21013.018.0000S

11.4

89.4

89.3

1414

0120.015.0

0020.015.0

0013.018.0

0087.718.10

000S

010x

100x

001Z

4

3

1

2

• We know we are done, because all coefficients in Z eq. are positive

• The non- basic variables are S1 & S2 , so vertex is at the intersection

of the equations from constraint (1) and constraint (2)

• The optimal Z is then 1414


___________________________________________________________________________________

Graphical Depiction of Simplex Steps

x1

x2

Actual Simplex Path

We want to find the best “fit” of a curve through the data.

Here we see :

a) Least squares fit

b) Linear interpolation

Can you suggest another?

f(x)

x

Curve Fitting


• The prerequisite mathematical background

for interpolation is found in the material on

the Taylor series expansion and finite

divided differences

• Simple statistics

– average

– standard deviation

– normal distribution

Normal Distribution

A histogram used

to depict the distribution

of exam grades.

x

x 2

95%

x

68%

Material to be Covered in Curve

Fitting • Linear Regression

– Polynomial Regression

– Multiple Regression

– General Linear Least Squares

– Nonlinear Regression

• Interpolation

– Newton’s Polynomial

– Lagrange Polynomial

– Coefficients of Polynomials

Least Squares Regression

• Simplest is fitting a straight line to a set of

paired observations

– (x1,y1), (x2, y2).....(xn, yn)

• The resulting mathematical expression is

– y = ao + a1x + e

• We will consider the error introduced at

each data point to develop a strategy for

determining the “best fit” equations

f(x)

x

2n

1i

i1oi

n

1i

2

ir xaayeS

i1oi xaay

To determine the values for ao and a1, differentiate

with respect to each coefficient

ii1oi

1

r

i1oi

o

r

xxaay2a

S

xaay2a

S

Note: we have simplified the summation symbols.

What mathematics technique will minimize Sr?

2n

1i

i1oi

n

1i

2

ir xaayeS

ii1oi

1

r

i1oi

o

r

xxaay2a

S

xaay2a

S

Setting the derivative equal to zero will minimizing Sr.

If this is done, the equations can be expressed as:

2

i1ioii

i1oi

xaxaxy0

xaay0

2

i1ioii

i1oi

xaxaxy0

xaay0

If you recognize that:

you have two equations, with two simultaneous equations

with two unknowns, ao and a1.

What are these equations? (hint: only place terms with

ao and a1 on the LHS of the equations)

What are the final equations for ao and a1?

oo naa

xaya

xxn

yxyxna

yxxaxa

yxana

1o

2

i

2

i

iiii

1

ii

2

i1io

ii1o

These first two

equations are called

the normal equations

n

yy&

n

xxthatnoteAlso

yx

y

a

a

xx

xn

i_

i_

ii

i

1

0

2

ii

i

These second two

result from solving

Error

2n

1i

i1oi

n

1i

2

r xaayeSi

Recall:

f(x)

x

1n

Ss

yyS

ty

2

it

The most common measure of the “spread” of a sample is the

standard deviation about the mean:

Introduce a term to measure the standard error of the estimate:

2|

n

Ss r

xy

Coefficient of determination r2:

t

rt2

S

SSr

r is the correlation coefficient

t

rt2

S

SSr

The following signifies that the line explains 100 percent

of the variability of the data:

Sr = 0

r = r2 = 1

If r = r2 = 0, then Sr = St and the fit is invalid.

Data 1 Data 2 Data 3 Data 4

10 8.04 10 9.14 10 7.46 8 6.58

8 6.95 8 8.14 8 6.77 8 5.76

13 7.58 13 8.74 13 12.74 8 7.71

9 8.81 9 8.77 9 7.11 8 8.84

11 8.33 11 9.26 11 7.81 8 8.47

14 9.96 14 8.10 14 8.84 8 7.04

6 7.24 6 6.13 6 6.08 8 5.25

4 4.26 4 3.10 4 5.39 19 12.50

12 10.84 12 9.13 12 8.15 8 5.56

7 4.82 7 7.26 7 6.42 8 7.91

5 5.68 5 4.74 5 5.73 8 6.89

Consider the following four sets of data

Data Set 1

0.00

2.00

4.00

6.00

8.00

10.00

12.00

0 5 10 15

x

f(x

)

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

0 5 10 15 20

f(x

)

Data Set 2

0.00

2.00

4.00

6.00

8.00

10.00

0 5 10 15

x

f(x

)

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

0 5 10 15

0.00

2.00

4.00

6.00

8.00

10.00

0 5 10 15

Data Set 3

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

0 5 10 15

x

f(x

)

0.00

2.00

4.00

6.00

8.00

10.00

12.00

0 5 10 15

Data Set 4

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

0 5 10 15 20

x

f(x

)

GRAPHS OF FOUR DATA SETS

Data Set 1

0.00

2.00

4.00

6.00

8.00

10.00

12.00

0 5 10 15

x

f(x

)

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

0 5 10 15 20

f(x

)

Data Set 2

0.00

2.00

4.00

6.00

8.00

10.00

0 5 10 15

x

f(x

)

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

0 5 10 15

0.00

2.00

4.00

6.00

8.00

10.00

0 5 10 15

Data Set 3

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

0 5 10 15

x

f(x

)

0.00

2.00

4.00

6.00

8.00

10.00

12.00

0 5 10 15

Data Set 4

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

0 5 10 15 20

xf(

x)

Linear Regression Data Set 1

0.50 3.00

0.117906 1.124747

0.666542 1.236603

17.98994 9


0.50 3.00

0.117964 1.125302

0.666242 1.237214

17.96565 9


0.50 3.00

0.117878 1.124481

0.666324 1.236311

17.97228 9


0.50 3.00

0.117819 1.123921

0.666707 1.235695

18.00329 9

All equations are y = 0.5x + 3 R2 = 0.67

GRAPHS OF FOUR DATA SETS

Linearization of non-linear

relationships

Some data is simply ill-suited for

linear least squares regression....

or so it appears.

f(x)

x

P

t

ln P

t

slope = r

intercept = ln P0

rt

oePP

Lin

eari

ze

why?

EXPONENTIAL

EQUATIONS

rtPln

elnPln

ePlnPln

ePP

0

rt

0

rt

0

rt

0

slope = r intercept = ln Po

Can you see the similarity

with the equation for a line:

y = b + mx

where b is the y-intercept

and m is the slope? lnP

t

trPln

elnPln

ePlnPln

ePP

0

tr

0

tr

0

tr

0

ln P0

t

slope = r intercept = ln P0

After taking the natural log

of the y-data, perform linear

regression.

From this regression:

The value of b will give us

ln (P0). Hence, P0 = eb

The value of m will give us r

directly.

POWER EQUATIONS

log Q

log H

Q

H

aHcQ

Here we linearize

the equation by

taking the log of

H and Q data.

What is the resulting

intercept and slope?

(Flow over a weir)

Hlogaclog

Hlogclog

HclogQlog

HcQ

a

a

a

log Q

log H

slope = a

intercept = log c

m

S 1/m

1/ S

SATURATION-GROWTH

RATE EQUATION

SK

S

s

max

mm

slope = Ks/mmax

intercept = 1/mmax

Here, m is the growth rate of

a microbial population,

mmax is the maximum

growth rate, S is the

substrate or food

concentration, Ks is the

substrate concentration at a

value of m = mmax/2

General Comments of Linear

Regression

• You should be cognizant of the fact that

there are theoretical aspects of regression

that are of practical importance but are

beyond the scope of this book

• Statistical assumptions are inherent in the

linear least squares procedure


Regression

• x has a fixed value; it is not random and is

measured without error

• The y values are independent random

variable and all have the same variance

• The y values for a given x must be normally

distributed


Regression

• The regression of y versus x is not the same

as x versus y

• The error of y versus x is not the same as x

versus y f(x)

x

y-d

irec

tion

x-direction

Polynomial Regression

• One of the reasons you were presented with

the theory behind linear regression was to

allow you the insight into similar

procedures for higher order polynomials

• y = a0 + a1x

• mth - degree polynomial

y = a0 + a1x + a2x2 +....amxm + e

Based on the sum of the squares

of the residuals

2m

im

2

i2i1oir xa......xaxaayS

1. Take the derivative of the above equation with respect to

each of the unknown coefficients: i.e. the partial with respect

to a2

m

im

2

i2i1oi

2

i

2

r xa.....xaxaayx2a

S

2. These equations are set to zero to minimize Sr., i.e.

minimize the error.

3. Set all unknowns values on the LHS of the equation.

Again, using the partial of Sr. wrt a2

4. This set of normal equations result in m+1 simultaneous

equations which can be solved using matrix methods to

determine a0, a1, a2......am

i

2

i

2m

im

4

i2

3

i1

2

io yxxa.....xaxaxa

Multiple Linear Regression

• A useful extension of linear regression is

the case where y is a linear function of two

or more variables

y = ao + a1x1 + a2x2

• We follow the same procedure

y = ao + a1x1 + a2x2 + e

Multiple Linear Regression

For two variables, we would solve a 3 x 3 matrix

in the following form:

ii2

ii1

i

2

1

0

2

i2i2i1i2

i2i1

2

i1i1

i2i1

yx

yx

y

a

a

a

xxxx

xxxx

xxn

[A] and {c} are clearly based on data given for x1, x2 and y

to solve for the unknowns in {x}.

Interpolation

• General formula for an n-th order

polynomial

y = a0 + a1x + a2x2 +....amxm

• For m+1 data points, there is one, and only

one polynomial of order m or less that

passes through all points

• Example: y = a0 + a1x

fits between 2 points

1st order

Interpolation

• We will explore two mathematical methods

well suited for computer implementation

• Newton’s Divided Difference Interpolating

Polynomials

• Lagrange Interpolating Polynomial

Newton’s Divided Difference

Interpolating Polynomials

• Linear Interpolation

• Quadratic Interpolation

• General Form

• Errors

Linear Interpolation Temperature, C Density, kg/m3

0 999.9

5 1000.0

10 999.7

15 999.1

20 998.2

How would you approach estimating the density at 17 C?

Temperature, C Density, kg/m3

0 999.9

5 1000.0

10 999.7

15 999.1

20 998.2

r

T

15 20

998.2

999.1

???

999.1 > r > 998.2

0

01

01o1 xx

xx

xfxfxfxf

Note: The notation f1(x) designates that this is a first order

interpolating polynomial

1517

1.999

1520

1.9992.998

r

f(x)

x

true solution

smaller intervals

provide a better estimate

1

2

f(x)

x

true solution

Alternative approach would be to

include a third point and estimate

f(x) from a 2nd order polynomial.

Quadratic Interpolation

1020102 xxxxbxxbbxf

Prove that this a 2nd order polynomial of

the form:

2

210 xaxaaxf

1202102

2

201102 xxbxxbxxbxbxbxbbxf

First, multiply the terms

2

210 xaxaaxf

1020102 xxxxbxxbbxf

Collect terms and recognize that:

22

120211

1020100

ba

xbxbba

xxbxbba

f(x)

x

x2, f(x2)

x1, f(x1)

x0, f(x0)

x, f(x)

Procedure for

Quadratic

Interpolation

02

01

01

12

12

2

01

011

00

xx

xx

xfxf

xx

xfxf

b

xx

xfxfb

xfb

Procedure for Quadratic

Interpolation

02

01

01

12

12

2

01

011

00

xx

xx

xfxf

xx

xfxf

b

xx

xfxfb

xfb

1020102 xxxxbxxbbxf

Example

998

998.5

999

999.5

1000

1000.5

0 10 20 30

Temp

Den

sit

y


0 999.9

5 1000.0

10 999.7

15 999.1

20 998.2

Include 10 degrees in

your calculation of the

density at 17 degrees.

Example


0 999.9

5 1000.0

10 999.7

15 999.1

20 998.2

Include 10 degrees in

your calculation of the

density at 17 degrees.

1020102 xxxxbxxbbxf

02

01

01

12

12

2

01

011

00

xx

xx

xfxf

xx

xfxf

b

xx

xfxfb

xfb

General Form of Newton’s

Interpolating Polynomials

for the nth-order polynomial

1n10n010n xxxxxxb....xxbbxf

To establish a methodical approach to a solution define

the first finite divided difference as:

ji

ji

jixx

xfxfx,xf

ji

ji

jixx

xfxfx,xf

if we let i=1 and j=0 then this is b1

01

011

xx

xfxfb

Similarly, we can define the second finite divided difference,

which expresses both b2 and the difference of the first two

divided difference

Similarly, we can define the second finite divided difference,

which expresses both b2 and the difference of the first two

divided difference

ki

kjji

kji

02

01

01

12

12

2

xx

x,xfx,xfx,x,xf

xx

xx

xfxf

xx

xfxf

b

Following the same scheme, the third divided difference is

the difference of two second finite divided difference.

This leads to a scheme that can easily lead to the

use of spreadsheets

i xi f(xi) first second third

0 x0 f(x0) f[x1,x0] f[x2,x1,x0] f[x3,x2,x1,x0]

1 x1 f(x1) f[x2,x1] f[x3,x2,x1]

2 x2 f(x2) f[x3,x2]

3 x3 f(x3)


These difference can be used to evaluate the b-coefficient s.

The result is the following interpolation polynomial called

the Newton’s Divided Difference Interpolating Polynomial

1n1001nn

0010n

xxxxxxx,,x,xf

....xxx,xfxfxf

To determine the error we need an extra point.

The error would follow a relationship analogous to the error

in the Taylor Series.

Lagrange Interpolating

Polynomial

n

ij0j ji

j

i

n

0i

iin

xx

xxxL

xfxLxf

where P designates the “product of”

The linear version of this expression is at n = 1

1

01

00

10

11

n

ij0j ji

j

i

n

0i

iin

xfxx

xxxf

xx

xx)x(f

xx

xxxL

xfxLxf

Linear version: n = 1

Your text shows you how to do n = 2 (second order).

What would third order be?

.......

xfxxxxxx

xxxxxx)x(f

xx

xxxL

xfxLxf

0

302010

3213

n

ij0j ji

j

i

n

0i

iin

Note:

x0 is

not being subtracted

from the constant

term x

.......

xfxxxxxx

xxxxxx)x(f

xx

xxxL

xfxLxf

0

302010

3213

n

ij0j ji

j

i

n

0i

iin

Note:

x0 is


from the constant

term x

or xi = x0 in

the numerator

or the denominator

j= 0

.......

xfxxxxxx

xxxxxx)x(f

xx

xxxL

xfxLxf

0

302010

3213

n

ij0j ji

j

i

n

0i

iin

3

231303

210

2

321202

310

1

312101

320

0

302010

3213

n

ij0j ji

j

i

n

0i

iin

xfxxxxxx

xxxxxx

xfxxxxxx

xxxxxx

xfxxxxxx

xxxxxx

xfxxxxxx

xxxxxx)x(f

xx

xxxL

xfxLxf

Note:

x3 is


from the constant

term x or xi = x3 in

the numerator

or the denominator

j= 3

Example

998

998.5

999

999.5

1000

1000.5

0 10 20 30

Temp

Den

sit

y


0 999.9

5 1000.0

10 999.7

15 999.1

20 998.2

Determine the density

at 17 degrees.

In fact, you can derive Lagrange directly from

Newton’s Interpolating Polynomial

776.998496.279244.839964.11917f2

74.99817f

776.99817f

1

2

Using Newton’s

Interpolating Polynomial

Coefficients of an Interpolating

Polynomial


y = a0 + a1x + a2x2 +....amxm

HOW CAN WE BE MORE STRAIGHT

FORWARD IN GETTING VALUES?

2

222102

2

121101

2

020100

xaxaaxf

xaxaaxf

xaxaaxf

This is a 2nd order polynomial.

We need three data points.

Plug the value of xi and f(xi)

directly into equations.

This gives three simultaneous equations

to solve for a0 , a1 , and a2

Example

998

998.5

999

999.5

1000

1000.5

0 10 20 30

Temp

Den

sit

y


0 999.9

5 1000.0

10 999.7

15 999.1

20 998.2

Determine the density

at 17 degrees.

2.998

1.999

7.999

a

a

a

20201

15151

10101

2

1

0

2

2

2


0 999.9

5 1000.0

10 999.7

15 999.1

20 998.2

78.998

17006.01703.01000

006.0

03.0

1000

a

a

a

2

17

2

1

0

r

Numerical Differentiation and

Integration

• Calculus is the mathematics of change.

• Engineers must continuously deal with

systems and processes that change, making

calculus an essential tool of our profession.

• At the heart of calculus are the related

mathematical concepts of differentiation

and integration.

Differentiation

• Dictionary definition of differentiate - “to

mark off by differences, distinguish; ..to

perceive the difference in or between”

• Mathematical definition of derivative - rate

of change of a dependent variable with

respect to an independent variable

x

xfxxf

x

y ii

x

xfxxf

x

y ii

ixf

xxf i

y

x

f(x)

x

Integration

• The inverse process of differentiation

• Dictionary definition of integrate - “to bring

together, as parts, into a whole; to unite; to

indicate the total amount”

• Mathematically, it is the total value or summation

of f(x)dx over a range of x. In fact the

integration symbol is actually a stylized capital S

intended to signify the connection between

integration and summation.

f(x)

x

b

a

dxxfI

?dxe

?x

dx

?dxa

?duu

?udv

ax

bx

n


?)uv(dx

d?u

dx

d

xoffunctionsarevanduif

?xlndx

d

?xdx

d?xtan

dx

d

?adx

d?xcos

dx

d

?edx

d?xsin

dx

d

n

n

x

x

Newton-Cotes Integration

• Common numerical integration scheme

• Based on the strategy of replacing a complicated

function or tabulated data with some

approximating function that is easy to integrate






n

n10n

b

a

n

b

a

xa....xaaxf

dxxfdxxfI






n

n10n

b

a

n

b

a

xa....xaaxf

dxxfdxxfI

fn(x) is an nth order

polynomial

The approximation of an integral by the area under

- a first order polynomial

- a second order polynomial

We can also approximated the integral by using a

series of polynomials applied piece wise.

An approximation of an integral by the area under

five straight line segments.

Newton-Cotes Formulas

• Closed form - data is at the beginning and

end of the limits of integration

• Open form - integration limits extend

beyond the range of data.

Trapezoidal Rule

• First of the Newton-Cotes closed integration

formulas

• Corresponds to the case where the

polynomial is a first order

xaaxf

dxxfdxxfI

10n

b

a

1

b

a

xaaxf

dxxfdxxfI

10n

b

a

1

b

a

A straight line can be represented as:

axab

afbfafxf1

Integrate this equation. Results in the trapezoidal rule.

2

bfafabI

b b

1

a a

b

a

I f x dx f x dx

f b f af a x a dx

b a

2

bfafabI

Recall the formula for computing the area of a trapezoid:

height x (average of the bases) hei

ght

base

base

The concept is the same but the trapezoid is on its side.

width

hei

ght

hei

ght

hei

ght

base

base

2

bfafabI

Error of the Trapezoidal Rule

bawhere

ab''f12

1E

3

t

This indicates that is the function being integrated is

linear, the trapezoidal rule will be exact.

Otherwise, for section with second and higher order

derivatives (that is with curvature) error can occur.

A reasonable estimate of is the average value of

b and a

Multiple Application of the

Trapezoidal Rule

• Improve the accuracy by dividing the

integration interval into a number of smaller

segments

• Apply the method to each segment

• Resulting equations are called multiple-

application or composite integration

formulas

h

n

xxgiven

2

xfxfh

2

xfxfh

2

xfxfhI

dx)x(fdx)x(fdx)x(fI

0n

n1n2110

x

x

x

x

x

x

n

1n

2

1

1

0

Multiple Application of the

Trapezoidal Rule

where there are n+1 equally spaced base points.

n2

xfxf2xf

ab2

xfxf2xf

n

abI

2

xfxfh

2

xfxfh

2

xfxfhI

dx)x(fdx)x(fdx)x(fI

1n

1i

ni0

1n

1i

ni0

n1n2110

x

x

x

x

x

x

n

1n

2

1

1

0

We can group terms to express a general form

}

}

width average height

n2

xfxf2xf

abI

1n

1i

ni0

}

}


The average height represents a weighted average

of the function values

Note that the interior points are given twice the weight

of the two end points

''f

n12

abE

2

3

a

Example Evaluate the following integral using the

trapezoidal rule and h = 0.1

n2

xfxf2xf

abI

1n

1i

ni0

6.1

1

x dxeI2

Example Problem

0

10

20

30

40

50

60

0 0.5 1 1.5 2 2.5

x

f(x

)

n

abh

Solution 6.1f5.1f24.1f23.1f22.1f21.1f21f

n2

abI

Example Problem

0

10

20

30

40

50

60

0 0.5 1 1.5 2 2.5

x

f(x

)

x f(x)

1 2.718

1.1 3.353

1.2 4.221

1.3 5.419

1.4 7.099

1.5 9.488

1.6 12.936

741.3816.74

62

16.1dxeI

6.1

1

x2

.....end of example

Simpson’s 1/3 Rule

• Corresponds to the case where the function

is a second order polynomial

2

210n

b

a

2

b

a

xaxaaxf

dxxfdxxfI


• Designate a and b as x0 and x2, and

estimate f2(x) as a second order Lagrange

polynomial

dx.......xfxxxx

xxxx

dxxfdxxfI

2

0

x

x

0

2010

21

b

a

2

b

a


• After integration and algebraic

manipulation, we get the following

equations

6

xfxf4xfab

02

xxhwherexfxf4xf

3

hI

210

02210

}

}


Error

bawhere

ab''f12

1E

3

t

Single application of Trapezoidal Rule.

Single application of Simpson’s 1/3 Rule

5)4(

t abf2880

1E

Multiple Application of


4

4

5

a

1n

..5,3,1i

2n

..6,4,2j

nji0

x

x

x

x

x

x

fn180

abE

n3

xfxf2xf4xf

abI

dx)x(fdx)x(fdx)x(fIn

1n

2

1

1

0

n3

xfxf2xf4xf

abI

1n

..5,3,1i

2n

..6,4,2j

nji0

The odd points represent the middle term for each application.

Hence carry the weight 4

The even points are common to adjacent applications

and are counted twice.


• Corresponds to the case where the function

is a third order polynomial

03

xxhwhere

xfxf3xf3xf8

h3I

xaxaxaaxf

dxxfdxxfI

03

3210

3

3

2

210n

b

a

3

b

a

Integration of Unequal Segments

• Experimental and field study data is often

unevenly spaced

• In previous equations we grouped the term (i.e. hi)

which represented segment width.

2

xfxfh

2

xfxfh

2

xfxfhI

n2

xfxf2xf

abI

n1n2110

1n

1i

ni0

Integration of Unequal Segments

• We should also consider alternately using

higher order equations if we can find data in

consecutively even segments

trapezoidal

rule

1/3

rule

3/8

rule

trapezoidal

rule

Example Integrate the following using the trapezoidal rule,

Simpson’s 1/3 Rule and a multiple application of

the trapezoidal rule with n=2. Compare results with

the analytical solution.

4

0

x2 dxxe

0

5000

10000

15000

20000

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x

f(x

)

Solution

927.52161x24

edxxe

4

0

x24

0

x2

First, calculate the analytical solution for this problem.

Consider a single application

of the trapezoidal rule.

f(4) = 11923.83

f(0) = 0

%35710093.5216

66.2384793.5216

66.238472

83.11923004

2

bfafabI

t

0

5000

10000

15000

20000

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x

f(x

)

%13310093.5216

22.1214293.5216

22.1214222

83.11923196.1092004

n2

xfxf2xf

abI

t

1n

1i

ni0

Multiple Application of

the Trapezoidal Rule

We are obviously not

doing very well on our

estimates.

Lets consider a scheme

where we “weight” the

estimates ....end of example

0

5000

10000

15000

20000

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x

f(x

)

Integration of Equations

• Integration of analytical as opposed to

tabular functions

• Romberg Integration

– Richardson’s Extrapolation

– Romberg Integration Algorithm

• Gauss Quadrature

• Improper Integrals

Richardson’s Extrapolation

• Use two estimates of an integral to compute a third more

accurate approximation

• The estimate and error associated with a multiple

application trapezoidal rule can be represented generally

as:

I = I(h) + E(h)

where I is the exact value of the integral

I(h) is the approximation from an n-segment application

E(h) is the truncation error

h is the step size (b-a)/n

Make two separate estimates using step sizes

of h1 and h2 .

I(h1) + E(h1) = I(h2) + E(h2)

Recall the error of the multiple-application of the trapezoidal

rule

''fh12

abE 2

Assume that is constant regardless of the step size ''f

2

2

2

1

2

1

h

h

hE

hE

2

2

121

2

2

2

1

2

1

h

hhEhE

h

h

hE

hE

Substitute into previous equation:

I(h1) + E(h1) = I(h2) + E(h2)

2

2

1

212

h

h1

hIhIhE

Thus we have developed an estimate of the truncation

error in terms of the integral estimates and their step

sizes. This estimate can then be substituted into:

I = I(h2) + E(h2)

to yield an improved estimate of the integral:

122

2

1

2 hIhI

1h

h

1hII

2

2

1

212

h

h1

hIhIhE

122

2

1

2 hIhI

1h

h

1hII

What is the equation for the special case where

the interval is halved?

i.e. h2 = h1 / 2

12

1222

2

2

2

1

hI3

1hI

3

4I

termscollecting

hIhI12

1hII

2h

h2h

h

Example

Use Richardson’s extrapolation to evaluate:

4

0

x2 dxxe

0

5000

10000

15000

20000

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x

f(x

)

Solution

Recall our previous example where we calculated the integral

using a single application of the trapezoidal rule, and then

a multiple application of the trapezoidal rule by dividing the

interval in half.

0

5000

10000

15000

20000

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x

f(x

)

SINGLE APPLICATION

OF TRAPEZOIDAL RULE

MULTIPLE APPLICATION

OF TRAPEZOIDAL RULE (n=2)

RICHARDSON’S

EXTRAPOLATION

357%

133%

57.96%

....end of example

14

II4I

hI63

1hI

63

64I

hI15

1hI

15

16I

hI3

1hI

3

4I

1k

1k,j1k,1j

1k

k,j

m

m

12

We can continue to improve the estimate by successive

halving of the step size to yield a general formula:

k = 2; j = 1

ROMBERG INTEGRATION

Note:

the subscripts

m and refer to

more and less

accurate estimates

k = 3

k = 4

Following a similar pattern to Newton divided differences,

Romberg’s Table can be produced

Error orders for j values

i = 1 i = 2 i = 3 i = 4

j O(h2) O(h4) O(h6) O(h8)

1 h I1,1 I1,2 I1,3 I1,4

2 h/2 I2,1 I2,2 I2,3

3 h/4 I3,1 I3,2

4 h/8 I4,1

Trapezoidal Simpson’s 1/3 Boole’s

Rule Rule Rule

hI3

1hI

3

4I m


i = 1 i = 2 i = 3 i = 4

j O(h2) O(h4) O(h6) O(h8)

1 h I1,1 I1,2 I1,3 I1,4

2 h/2 I2,1 I2,2 I2,3

3 h/4 I3,1 I2,3

4 h/8 I4,1


Rule Rule Rule



hI3

1hI

3

4I m


i = 1 i = 2 i = 3 i = 4

j O(h2) O(h4) O(h6) O(h8)

1 h I1,1 I1,2 I1,3 I1,4

2 h/2 I2,1 I2,2 I2,3

3 h/4 I3,1 I3,2

4 h/8 I4,1


Rule Rule Rule



hI15

1hI

15

16I m


i = 1 i = 2 i = 3 i = 4

j O(h2) O(h4) O(h6) O(h8)

1 h I1,1 I1,2 I1,3 I1,4

2 h/2 I2,1 I2,2 I2,3

3 h/4 I3,1 I3,2

4 h/8 I4,1


Rule Rule Rule



hI3

1hI

3

4I m


i = 1 i = 2 i = 3 i = 4

j O(h2) O(h4) O(h6) O(h8)

1 h I1,1 I1,2 I1,3 I1,4

2 h/2 I2,1 I2,2 I2,3

3 h/4 I3,1 I3,2

4 h/8 I4,1


Rule Rule Rule



hI15

1hI

15

16I m


i = 1 i = 2 i = 3 i = 4

j O(h2) O(h4) O(h6) O(h8)

1 h I1,1 I1,2 I1,3 I1,4

2 h/2 I2,1 I2,2 I2,3

3 h/4 I3,1 I3,2

4 h/8 I4,1


Rule Rule Rule



hI63

1hI

63

64I m

Gauss Quadrature

f(x)

x

f(x)

x

Extend the area

under the straight

line

Method of Undetermined

Coefficients

Recall the trapezoidal rule

2

bfafabI

This can also be expressed as

bfcafcI 10

where the c’s are constant

Before analyzing

this method,

answer this

question.

What are two

functions that

should be evaluated

exactly

by the trapezoidal

rule?

The two cases that should be evaluated exactly

by the trapezoidal rule: 1) y = constant

2) a straight line

f(x)

x

y = 1

(b-a)/2 -(b-a)/2

f(x)

x

y = x

(b-a)/2

-(b-a)/2

Thus, the following equalities should hold.

2ab

2ab10

2ab

2ab10

10

xdx2

abc

2

abc

dx1cc

bfcafcI

For y = 1

since f(a) = f(b) =1

For y = x

since f(a) = x =-(b-a)/2

and

f(b) = x =(b-a)/2

Evaluating both integrals

02

1bc

2

abc

abcc

10

10

For y = 1

For y = x

Now we have two equations and two unknowns, c0 and c1.

Solving simultaneously, we get :

c0 = c1 = (b-a)/2

Substitute this back into:

bfcafcI 10

2

bfafabI

We get the equivalent of the trapezoidal rule.

DERIVATION OF THE TWO-POINT

GAUSS-LEGENDRE FORMULA

1100 xfcxfcI

Lets raise the level of sophistication by:

- considering two points between -1 and 1

- i.e. “open integration”

f(x)

x -1 x0 x1 1

Previously ,we assumed that the equation

fit the integrals of a constant and

linear function.

Extend the reasoning by assuming that

it also fits the integral of a parabolic and a cubic function.

1

1

3

1100

1

1

2

1100

1

1

1100

1

1

1100

0dxxxfcxfc

3/2dxxxfcxfc

0xdxxfcxfc

2dx1xfcxfc We now have four

equations and four

unknowns

c0 c1 x0 and x1

What equations are

you solving?

0xcxc0dxxxfcxfc

3

2xcxc

3

2dxxxfcxfc

0xcxc0xdxxfcxfc

21c1c2dx1xfcxfc

3

11

3

00

1

1

3

1100

2

11

2

00

1

1

2

1100

`1100

1

1

1100

10

1

1

1100

Solve these equations simultaneously

f(xi) is either 1, xi, xi2 or xi

3

3

1f

3

1fI

3

1x

3

1x

1cc

1

0

10

This results in the following

The interesting result is that the simple addition

of the function values at

3

1and

3

1

However, we have set the limit of integration at -1 and 1.

This was done to simplify the mathematics. A simple

change in variables can be use to translate other limits.

Assume that the new variable xd is related to the

original variable x in a linear fashion.

x = a0 + a1xd

Let the lower limit x = a correspond to xd = -1 and the upper

limit x=b correspond to xd=1

a = a0 + a1(-1) b = a0 + a1(1)

a = a0 + a1(-1) b = a0 + a1(1)

SOLVE THESE EQUATIONS

SIMULTANEOUSLY

2

aba

2

aba 10

d

dd10

x2

ab

2

abxor

2

xababxaax

substitute

d

dd10

dx2

abdx

x2

ab

2

abxaax

These equations are substituted for x and dx respectively.

Let’s do an example to appreciate the theory

behind this numerical method.

Example

Estimate the following using two-point Gauss Quadrature:

4

0

x2 dxxe

0

5000

10000

15000

20000

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x

f(x

)

1

1

d

x222

d dx2ex22 d

Now evaluate the integral

55.347738.346817.9

2e3

1222e

3

122

3

1f

3

1fI

3

1222

3

1222

....end of problem

SINGLE APPLICATION

OF TRAPEZOIDAL RULE

MULTIPLE APPLICATION

OF TRAPEZOIDAL RULE (n=2)

RICHARDSON’S

EXTRAPOLATION

2-POINT GAUSS

QUADRATURE

357%

133%

58%

33 %

Higher-Point Gauss Formulas

1n1n1100 xfcxfcxfcI

For two point, we determined that c0 =c1 = 1

For three point:

c0 = 0.556 (5/9) x0=-0.775 = - (3/5)1/2

c1 = 0.889 (8/9) x1=0.0

c2 = 0.556 (5/9) x2=0.775 = (3/5)1/2

Higher-Point Gauss Formulas

For four point:

c0 = {18-(30)1/2 }/36 x0= -{525+70(30)1/2 } 1/2/35

c1 = {18+(30)1/2 }/36 x1= -{525-70(30)1/2 } 1/2/35

c2 = {18+(30)1/2 }/36 x2= +{525-70(30)1/2 } 1/2/35

c3 = {18-(30)1/2 }/36 x3= +{525+70(30)1/2 } 1/2/35

Your text goes on to provide additional weighting

factors (ci’s) and function arguments (xi’s)

in Table 22.1 p. 626.

Numerical Differentiation

• Forward finite divided difference

• Backward finite divided difference

• Center finite divided difference

• All based on the Taylor Series

.........h!2

x''fhx'fxfxf 2i

ii1i

Forward Finite Difference

2ii1i

i

i1ii

3i2iii1i

hOh2

x''f

h

xfxfx'f

hOh

xfxfx'f

.........h!3

x'''fh

!2

x''fhx'fxfxf

Forward Divided Difference

f(x)

x

(xi, yi)

(x i+1,y i+1)

hOh

fxxO

xx

xfxfx'f i

i1i

i1i

i1ii

first forward divided difference

Error is proportional to

the step size

O(h2) error is proportional to the square of the step size

O(h3) error is proportional to the cube of the step size

hOh

fx'f i

i

f(x)

x

(xi,yi)

(xi-1,yi-1)

h

f

h

xfxfx'f

.....h!2

x''fhx'fxfxf

......h!2

x''fhx'fxfxf

i1iii

2iii1i

2iii1i

Backward Difference Approximation of the

First Derivative

Expand the Taylor series backwards

The error is still O(h)

Centered Difference Approximation of the

First Derivative and Second Derivative

Subtract and add backward Taylor expansion

from and to the forward Taylor series expansion

2

2

1ii1ii

21i1ii

3i2iii1i

3i2iii1i

hOh

xfxf2xfx''f

hOh2

xfxfx'f

h6

x'''fh

2

x''fhx'fxfxf

....h6

x'''fh

2

x''fhx'fxfxf

Subtracting

Adding

Forward

Backward

f(x)

x

(xi,yi)

(xi-1,yi-1)

(xi+1,yi+1)

21i1i

i hOh2

xfxfx'f

Numerical Differentiation

• Forward finite divided differences Fig. 23.1

• Backward finite divided differences Fig. 23.2

• Centered finite divided differences Fig. 23.3

• First - Fourth derivative

• Error: O(h), O(h2), O(h4) for centered only

Derivatives with Richardson

Extrapolation

• Two ways to improve derivative estimates

¤ decrease step size

¤ use a higher order formula that employs more

points

• Third approach, based on Richardson

extrapolation, uses two derivatives

estimates to compute a third, more accurate

approximation

Richardson Extrapolation

2 2 12

1

2

12

2 1

2 1

1I I h I h I h

h1

h

hSpecial case where h

2

4 1I I h I h

3 3

In a similar fashion

4 1D D h D h

3 3

For a centered difference

approximation with

O(h2) the application of

this formula will yield

a new derivative estimate

of O(h4)

Example

Given the function:

f(x) = -0.1x4 - 0.15x3 - 0.5x2 - 0.25x +1.2

Use centered finite divided difference to estimate

the derivative at 0.5.

f(0) = 1.2

f(0.25) =1.1035

f(0.75) = 0.636

f(1) = 0.2

Example Problem

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 0.2 0.4 0.6 0.8 1 1.2

x

f(x

)

Using centered finite divided difference for h = 0.5

f(0) = 1.2

f(1) = 0.2

D(0.5) = (0.2 - 1.2)/1 = -1.0 t = 9.6%

Using centered finite divided difference for h = 0.25

f(0.25) =1.1035

f(0.75) = 0.636

D(0.25) = (0.636 - 1.1035)/0.5 =-0.934 t = 2.4%

Derivatives of Unequally Spaced

Data • Common in data from experiments or field studies

• Fit a second order Lagrange interpolating polynomial to

each set of three adjacent points, since this polynomial

does not require that the points be uniformly spaced

• Differentiate analytically

1i1ii1i

1ii1i

1ii1ii

1i1ii

1i1ii1i

1ii1i

xxxx

xxx2xf

xxxx

xxx2xf

xxxx

xxx2xfx'f

Derivative and Integral Estimates

for Data with Errors

• In addition to unequal spacing, the other problem related to

differentiating empirical data is measurement error

• Differentiation amplifies error

• Integration tends to be more forgiving

• Primary approach for determining derivatives of imprecise

data is to use least squares regression to fit a smooth,

differentiable function to the data

• In absence of other information, a lower order polynomial

regression is a good first choice

0

100

200

300

0 10 20

ty

0

10

20

30

0 5 10 15

t

dy

/dt

0

50

100

150

200

250

0 5 10 15

t

y

0

10

20

30

40

0 5 10 15

0

50

100

150

200

250

0 5 10 15

0

10

20

30

40

0 5 10 15

t

dy

/dt

Ordinary Differential Equations

• A differential equation defines a

relationship between an unknown function

and one or more of its derivatives

• Physical problems using differential

equations

– electrical circuits

– heat transfer

– motion


• The derivatives are of the dependent

variable with respect to the independent

variable

• First order differential equation with y as

the dependent variable and x as the

independent variable would be:

dy/dx = f(x,y)


• A second order differential equation would

have the form:

}

does not necessarily have to include

all of these variables

dx

dy,y,xf

dx

yd2

2


• An ordinary differential equation is one

with a single independent variable.

• Thus, the previous two equations are

ordinary differential equations

• The following is not:

y,x,xfdx

dy21

1


• The analytical solution of ordinary

differential equation as well as partial

differential equations is called the “closed

form solution”

• This solution requires that the constants of

integration be evaluated using prescribed

values of the independent variable(s).


• An ordinary differential equation of order n

requires that n conditions be specified.

• Boundary conditions

• Initial conditions

consider this beam where the

deflection is zero at the boundaries

x= 0 and x = L

These are boundary conditions

consider this beam where the

deflection is zero at the boundaries

x= 0 and x = L

These are boundary conditions

a

yo

P

In some cases, the specific behavior of a system(s)

is known at a particular time. Consider how the deflection of a

beam at x = a is shown at time t =0 to be equal to yo.

Being interested in the response for t > 0, this is called the

initial condition.


• At best, only a few differential equations

can be solved analytically in a closed form.

• Solutions of most practical engineering

problems involving differential equations

require the use of numerical methods.

Review of Analytical Solution

C3

x4y

dxx4dy

x4dx

dy

3

2

2

At this point lets consider

initial conditions.

y(0)=1

and

y(0)=2

2Cand

C3

042

20yfor

1Cthen

C3

041

10yfor

C3

x4y

3

3

3

What we see are different

values of C for the two

different initial conditions.

The resulting equations

are:

23

x4y

13

x4y

3

3

0

4

8

12

16

0 0.5 1 1.5 2 2.5

x

y

y(0)=1

y(0)=2

y(0)=3

y(0)=4

One Step Methods

• Focus is on solving ODE in the form

y

x

slope = f yi

yi+1

h

This is the same as saying:

new value = old value + slope * step size

hyy

y,xfdx

dy

i1i f

Euler’s Method

• The first derivative provides a direct

estimate of the slope at xi

• The equation is applied iteratively, or one

step at a time, over small distance in order

to reduce the error

• Hence this is often referred to as Euler’s

One-Step Method

Example

4413.1y

4413.0x3

41y

dxx4dy

solutionAnalytical

1.0sizestep

1xat1y.C.I

x4dx

dy

1.1

1

3

1.1

1

2

y

1

2

1.0141f1.1f

:Note

4.11.0141f1.1f

hyy

x4dx

dy

2

2

i1i

2

f

I.C. Slope step size

Recall the analytical solution was 1.4413

If we instead reduced the step size to to 0.05 and

apply Euler’s twice

Error Analysis of Euler’s Method

• Truncation error - caused by the nature of

the techniques employed to approximate

values of y

– local truncation error (from Taylor Series)

– propagated truncation error

– sum of the two = global truncation error

• Round off error - caused by the limited

number of significant digits that can be

retained by a computer or calculator

Example

0

2

4

6

8

10

12

0 1 2 3

x

y

Analytical

Solution

Numerical

Solution

....end of example

Higher Order Taylor Series

Methods

• This is simple enough to implement with

polynomials

• Not so trivial with more complicated ODE

• In particular, ODE that are functions of both

dependent and independent variables require

chain-rule differentiation

• Alternative one-step methods are needed

2ii

iii1i h2

y,x'fhy,xfyy

Modification of Euler’s Methods

• A fundamental error in Euler’s method is

that the derivative at the beginning of the

interval is assumed to apply across the

entire interval

• Two simple modifications will be

demonstrated

• These modification actually belong to a

larger class of solution techniques called

Runge-Kutta which we will explore later.

Heun’s Method

• Consider our Taylor expansion

2ii

iii1i h2

y,x'fhy,xfyy

• Approximate f’ as a simple forward difference

h

y,xfy,xfy,x'f ii1i1i

ii

• Substituting into the expansion

h2

ffy

2

h

h

ffhfyy i1i

i

2

i1iii1i

Heun’s Method Algorithm

• Determine the derivatives for the interval @

– the initial point

– end point (based on Euler step from initial point)

• Use the average to obtain an improved

estimate of the slope for the entire interval

• We can think of the Euler step as a “test” step

y

xi xi+1

h

y

xi xi+1

Take the average of these

two slopes

y

xi xi+1

y

x

xi xi+1

h

2

y,xfy,xfyy 1i1iii

i1i

Improved Polygon Method

• Another modification of Euler’s Method

(sometimes called the Midpoint Method)

• Uses Euler’s to predict a value of y at the

midpoint of the interval

• This predicted value is used to estimate the

slope at the midpoint

2

hy,xfyy iii2/1i

2/1i2/1i2/1i y,xf'y

Improved Polygon Method

• We then assume that this slope represents a valid

approximation of the average slope for the entire

interval

hy,xfyy 2/1i2/1ii1i

• Use this slope to extrapolate linearly from xi to

xi+1 using Euler’s algorithm

• We could also get this algorithm from

substituting a forward difference in f to i+1/2 into

the Taylor expansion for f’, i.e.

hfy2

h

2/h

ffhfyy 2/1ii

2

i2/1iii1i

y

x xi xi+1/2

f(xi+1/2)

y

x xi xi+1/2

f’(xi+1/2)

y

x xi xi+1/2 xi+1

h

Extend your slope

now to get f(x i+1)

Runge-Kutta Methods

• RK methods achieve the accuracy of a Taylor

series approach without requiring the calculation

of a higher derivative

• Many variations exist but all can be cast in the

generalized form:

hh,y,xyy iii1i f{

f is called the incremental function

f , Incremental Function

can be interpreted as a representative slope

over the interval

hkqhkqhkqy,hpxfk

hkqhkqy,hpxfk

hkqy,hpxfk

y,xfk

:ares'ktheandttanconsares'athewhere

kakaka

1n1n,1n22,1n11,1ninin

222121i2i3

111i1i2

ii1

nn2211

f

NOTE:

k’s are recurrence relationships,

that is k1 appears in the equation for k2

and each appear in the equation for k3

This recurrence makes RK methods efficient for

computer calculations

hkqhkqhkqy,hpxfk

hkqhkqy,hpxfk

hkqy,hpxfk

y,xfk


kakaka


222121i2i3

111i1i2

ii1

nn2211

f

Second Order RK Methods

hkqy,hpxfk

y,xfk

where

hkakayy

111i1i2

ii1

2211i1i

hkqhkqhkqy,hpxfk

hkqhkqy,hpxfk

hkqy,hpxfk

y,xfk


kakaka


222121i2i3

111i1i2

ii1

nn2211

f

Second Order RK Methods

• We have to determine values for the constants a1,

a2, p1 and q11

• To do this consider the Taylor series in terms of

yi+1 and f(xi,yi)

2

hy,x'fhy,xfyy

hkakayy

2

iiiii1i

2211i1i

2

h

dx

dy

y

f

x

fhy,xfyy

ansionexptoinsubstitute

dx

dy

y

f

x

fy,x'f

2

iii1i

ii

Now, f’(xi , yi ) must be determined by the

chain rule for differentiation

The basic strategy underlying Runge-Kutta methods

is to use algebraic manipulations to solve for values

of a1, a2, p1 and q11

2

hy,x

dx

dyy,x

y

fy,x

x

fhy,xfyy

hkakayy

2

iiiiiiiii1i

2211i1i

By setting these two equations equal to each other and

recalling:

hkqy,hpxfk

y,xfk

111i1i2

ii1

we derive three equations to evaluate the four unknown

constants

2

1qa

2

1pa

1aa

112

12

21

Because we have three equations with four unknowns,

we must assume a value of one of the unknowns.

Suppose we specify a value for a2.

What would the equations be?

2

111

21

a2

1qp

a1a

Because we can choose an infinite number of values

for a2 there are an infinite number of second order

RK methods.

Every solution would yield exactly the same result

if the solution to the ODE were quadratic, linear or a

constant.

Lets review three of the most commonly used and

preferred versions.

2

1qa

2

1pa

1aa

hkqy,hpxfk

y,xfk

where

hkakayy

112

12

21

111i1i2

ii1

2211i1i

Consider the following:

Case 1: a2 = 1/2

Case 2: a2 = 1

These two methods

have been previously

studied.

What are they?

hky,hxfk

y,xfk

where

hk2

1k

2

1yy

1a2

1qp

2

1qa

2

1pa

2/12/11a1a

1ii2

ii1

21i1i

2

111

112

12

21

Case 1: a2 = 1/2

This is Heun’s Method with

a single corrector.

Note that k1 is the slope at

the beginning of the interval

and k2 is the slope at the

end of the interval.

hkqy,hpxfk

y,xfk

where

hkakayy

111i1i2

ii1

2211i1i

hk2

1y,h

2

1xfk

y,xfk

where

hkyy

2

1

a2

1qp

2

1qa

2

1pa

011a1a

1ii2

ii1

2i1i

2

111

112

12

21

hkqy,hpxfk

y,xfk

where

hkakayy

111i1i2

ii1

2211i1i

Case 2: a2 = 1

This is the Improved Polygon

Method.

Ralston’s Method

Ralston (1962) and Ralston and Rabinowitiz (1978)

determined that choosing a2 = 2/3 provides a minimum

bound on the truncation error for the second order RK

algorithms.

This results in a1 = 1/3 and p1 = q11 = 3/4

hk4

3y,h

4

3xfk

y,xfk

where

hk3

2k

3

1yy

1ii2

ii1

21i1i

Example

1.0hsizestep

11y.e.i1xat1y:.C.I

yx4dx

dy 2

As a class problem, lets

consider two steps.

Some of you folks do the

analytical solution,

others do either:

•Ralstons’s

•Heun’s

•Improved Polygon

Third Order Runge-Kutta Methods

• Derivation is similar to the one for the second-order

• Results in six equations and eight unknowns.

• One common version results in the following

21ii3

1ii2

ii1

321i1i

hk2hky,hxfk

hk2

1y,h

2

1xfk

y,xfk

where

hkk4k6

1yy

Note the third term

NOTE: if the derivative is a function of x only, this reduces to Simpson’s 1/3 Rule

Fourth Order Runge Kutta • The most popular

• The following is sometimes called the classical

fourth-order RK method

3ii4

2ii3

1ii2

ii1

4321i1i

hky,hxfk

hk2

1y,h

2

1xfk

hk2

1y,h

2

1xfk

y,xfk

where

hkk2k2k6

1yy

• Note that for ODE that are a function of x alone

that this is also the equivalent of Simpson’s 1/3

Rule

3ii4

2ii3

1ii2

ii1

4321i1i

hky,hxfk

hk2

1y,h

2

1xfk

hk2

1y,h

2

1xfk

y,xfk

where

hkk2k2k6

1yy

Example

Use 4th Order RK to solve the following differential equation:

11y.C.Ix1

xy

dx

dy2

using an interval of h = 0.1

Solution

3ii4

2ii3

1ii2

ii1

4321i1i

hky,hxfk

hk2

1y,h

2

1xfk

hk2

1y,h

2

1xfk

y,xfk

where

hkk2k2k6

1yy

We will determine

different estimates

of the slope

i.e. k1, k2, k3 and k4

05119.1

1.052323.051219.0251189.025.06

11

hkk2k2k6

1yy 4321i1i

.....end of problem

Higher Order RK Methods

• When more accurate results are required,

Bucher’s (1964) fifth order RK method is

recommended

• There is a similarity to Boole’s Rule

• The gain in accuracy is offset by added

computational effort and complexity

Systems of Equations

• Many practical problems in engineering and

science require the solution of a system of

simultaneous differential equations

n21nn

n2122

n2111

y,,y,y,xfdx

dy

y,,y,y,xfdx

dy

y,,y,y,xfdx

dy

• Solution requires n initial conditions

• All the methods for single equations can be used

• The procedure involves applying the one-step

technique for every equation at each step before

proceeding to the next step

n21nn

n2122

n2111

y,,y,y,xfdx

dy

y,,y,y,xfdx

dy

y,,y,y,xfdx

dy

• Note that higher order ODEs can be reformulated

as simultaneous first order ODEs

212

21

21

2

2

y,y,xgdx

dy

ydx

dy

so,dx

dyy&yydefiningby

dtransformebecandx

dy,y,xg

dx

yd

Partial Differential Equations

• An equation involving partial derivatives of an

unknown function of two or more independent

variables

• The following are examples. Note: u depends on

both x and y

xy

uxu

x

uy5u8

y

ux

yx

u

xyx

u6

x

u1u

y

uxy2

x

u

2

2

2

22

2

33

2

2

2

2

2

2

Partial Differential Equations

• Because of their widespread application in

engineering, our study of PDE will focus on linear,

second-order equations

• The following general form will be evaluated for

B2 - 4AC (note that text does not list E, F, G terms)

0Guy

uF

x

uED

y

uC

yx

uB

x

uA

2

22

2

2

B2-4AC Category Example

< 0 Elliptic Laplace equation (steady state with

2 spatial dimensions

= 0 Parabolic Heat conduction equation (time variable

with one spatial dimension

>0 Hyperbolic Wave equation (time-variable with one

spatial dimension

0y

T

x

T2

2

2

2

t

T

x

Tk

2

2

2

2

22

2

t

y

c

1

x

y

y or t

x

set up a grid

estimate the dependent

variable at the center

or intersections

of the grid

• Typically used to characterize steady-state

boundary value problems

• Before solving, the Laplace equation will be

solved from a physical problem

0y

u

x

u

0Dy

uC

yx

uB

x

uA

2

2

2

2

2

22

2

2

Finite Difference:

Elliptic Equations

B2- 4AC < 0

The Laplace Equation

• Models a variety of problems involving the

potential of an unknown variable

• We will consider cases involving

thermodynamics, fluid flow, and flow through

porous media

0y

u

x

u2

2

2

2

The Laplace equation

• Let’s consider the case of a plate heated from the

boundaries

• How is this equation derived from basic concepts

of continuity?

• How does it relate to flow fields?

0y

T

x

T2

2

2

2

Consider the plate below, with thickness z.

The temperatures are known at the boundaries.

What is the temperature throughout the plate?

T = 200 T= 200

T = 400

T = 200

T = 200 T= 200

T = 400

T = 200

y

x

Divide into a grid, with increments by x and y

T = 200 T= 200

T = 400

T = 200

y

x

What is the temperature here, if

using a block centered scheme?

T = 200 T= 200

T = 400

T = 200

y

x

What is the temperature here, if

using a grid centered scheme?

Consider the element shown below on the face of

a plate z in thickness.

The plate is illustrated everywhere by at its edges or

boundaries, where the temperature can be set.

y

x x

y

q(y + y)

q(x + x) q(x)

q(y)

By continuity, the flow of heat in must equal the flow of heat

out.

tzxyyqtzyxxq

tzxyqtzyxq

Consider the heat flux q

in and out of the elemental volume.

tzxyyqtzyxxq

tzxyqtzyxq

Divide by z and t and collect terms, this equation

reduces to:

0y

q

x

q

Again, this is our

continuity equation

0y

q

x

q

The link between flux and temperature is provided by Fourier’s

Law of heat conduction

i

TCkqi

r

Where qi is the heat flux in the direction i.

Substitute B into A to get the Laplace equation

Equation A

Equation B

0y

q

x

q

i

TCkqi

r

Equation A

Equation B

0y

T

x

T

y

TCk

yx

TCk

xy

q

x

q

2

2

2

2

r

r

Consider Fluid Flow

In fluid flow, where the fluid is a liquid or a gas, the

continuity equation is:

0y

V

x

V yx

The link here can by either of the following sets of equations:

The potential function:

Stream function:

yV

xV yx

f

f

xV

yV yx

0yx

or0yx 2

2

2

2

2

2

2

2

f

f

The Laplace equation is then

0y

V

x

V yx

yV

xV yx

f

f

xV

yV yx

Flow in Porous Media

0y

q

x

q

i

HKqi

Darcy’s Law

The link between flux and the pressure head is

provided by Darcy’s Law

0y

h

x

h2

2

2

2

)y,x(fy

u

x

u2

2

2

2

For a case with sources and sinks within the 2-D

domain, as represented by f(x,y), we have the

Poisson equation.

Now let’s consider solution techniques.

Evaluate these equations based on the grid and

central difference equations

(i,j)

(i+1,j) (i-1,j)

(i,j+1)

(i,j-1)

2

1j,ij,i1j,i

2

2

2

j,1ij,ij,1i

2

2

y

uu2u

y

u

x

uu2u

x

u

0y

uu2u

x

uu2u2

1j,ij,i1j,i

2

j,1ij,ij,1i

(i,j)

(i+1,j) (i-1,j)

(i,j+1)

(i,j-1)

If x = y

we can collect the terms

to get:

0u4uuuu j,i1j,i1j,ij,1ij,1i

(i,j)

(i+1,j) (i-1,j)

(i,j+1)

(i,j-1)

0u4uuuu j,i1j,i1j,ij,1ij,1i

This equation is referred

to as the Laplacian difference

equation.

It can be applied to all

interior points.

We must now consider

what to do with the

boundary nodes.

Boundary Conditions

• Dirichlet boundary conditions: u is specified at the

boundary

Temperature

Head

• Neumann boundary condition: the derivative is specified

qi

• Combination of both u and its derivative (Mixed BC)

ii x

hor

x

T

The simplest case is where the boundaries are

specified as fixed values.

This case is known as the Dirichlet boundary

conditions.

u1

u2

u3

u4

u1

u2

u3

u4

Consider how we can deal with the lower node shown, u1,1

-4u1,1 +u1,2+u2,1+u1 +u4 = 0

1,2

1,1 2,1

Note:

This grid would result

in nine simultaneous

equations.

Let’s consider how to model the Neumann boundary condition

x2

uu

x

u j,1ij,1i

centered finite divided difference

approximation

2

2

2

20

h

x

h

y

h

x 0

h

x 0

h

y 0

suppose we wanted to consider this end

grid point

1,2

1,1 2,1

0x

h

0y

h

The two boundaries are consider

to be symmetry lines due to the

fact that the BC translates

in the finite difference form

to:

h i+1,j = h i-1,j

and

h i,j+1 = h i,j-1

1,2

1,1 2,1

h1,1 = (2h1,2 + 2 h2,1)/4

h1,2 = (h1,1 + h1,3+2h22)/4 2,2 0

x

h

0y

h

Example

The grid on the next slide is designed to solve the LaPlace

equation

0yx 2

2

2

2

Write the finite difference equations for the nodes (1,1),

(1,2), and (2,1). Note that the lower boundary is a

Dirichlet boundary condition, the left boundary is a

Neumann boundary condition, and x = y.

Solution

0x

20

(1,1) (2,1)

(1,2)

4

20

4

202

4

2

31221121

211211

11221312

The Liebmann Method

• Most numerical solutions of the Laplace

equation involve systems that are much

larger than the general system we just

evaluated

• Note that there are a maximum of five

unknown terms per line

• This results in a significant number of terms

with zero’s

The Liebmann Method

• In addition to the fact that they are prone to

round-off errors, using elimination methods

on such sparse systems wastes a great

amount of computer memory storing zeros

• Therefore, we commonly employ

approaches such as Gauss-Seidel, which

when applied to PDEs is also referred to as

Liebmann’s method.

The Liebmann Method • In addition the equations will lead to a matrix that

is diagonally dominant.

• Therefore the procedure will converge to a stable

solution.

• Over relaxation is often employed to accelerate the

rate of convergence

old

j,i

new

j,i

new

j,i

j,i1j,i1j,ij,1ij,1i

u1uu

0u4uuuu

old

j,i

new

j,i

new

j,i

j,i1j,i1j,ij,1ij,1i

u1uu

0u4uuuu

As with the conventional Gauss Seidel method, the iterations

are repeated until each point falls below a pre-specified

tolerance:

%100u

uunew

j,i

old

j,i

new

j,i

s

Groundwater Flow Example

2

2

2

20

h

x

h

y

h

x 0

h

x 0

h

y 0

Modeling 1/2 of the system shown, we can develop the following

schematic where x = y = 20 m

The finite difference equations can be solved using a

a spreadsheet.

100 =A1+0.05*20 =B1+0.05*20 =C1+0.05*20 =D1+0.05*20 =E1+0.05*20 =F1+0.05*20 =G1+0.05*20 =H1+0.05*20 =I1+0.05*20 =J1+0.05*20

=(A1+2*B2+A3)/4 =(B1+C2+B3+A2)/4 =(C1+D2+C3+B2)/4 =(D1+E2+D3+C2)/4 =(E1+F2+E3+D2)/4 =(F1+G2+F3+E2)/4 =(G1+H2+G3+F2)/4 =(H1+I2+H3+G2)/4 =(I1+J2+I3+H2)/4 =(J1+K2+J3+I2)/4 =(K1+K3+2*J2)/4




=(2*A5+2*B6)/4 =(2*B5+C6+A6)/4 =(2*C5+D6+B6)/4 =(2*D5+E6+C6)/4 =(2*E5+F6+D6)/4 =(2*F5+G6+E6)/4 =(2*G5+H6+F6)/4 =(2*H5+I6+G6)/4 =(2*I5+J6+H6)/4 =(2*J5+K6+I6)/4 =(2*K5+2*J6)/4

100 =A1+0.05*20 =B1+0.05*20 =C1+0.05*20 =D1+0.05*20 =E1+0.05*20 =F1+0.05*20 =G1+0.05*20 =H1+0.05*20 =I1+0.05*20 =J1+0.05*20






100

=(A1+2*B2+A3)/4

=(A2+2*B3+A4)/4

=(A3+2*B4+A5)/4

=(A4+2*B5+A6)/4

=(2*A5+2*B6)/4

You will get an

error message in

Excel that states that

it will not resolve

a circular reference.

CAN USE EXCEL DEMONSTRATION

100 =A1+0.05*20 =B1+0.05*20 =C1+0.05*20 =D1+0.05*20 =E1+0.05*20 =F1+0.05*20 =G1+0.05*20 =H1+0.05*20 =I1+0.05*20 =J1+0.05*20






=B1+0.05*20

=(C1+D2+C3+B2)/4

=(C2+D3+C4+B3)/4

=(C3+D4+C5+B4)/4

=(C4+D5+C6+B5)/4

=(2*C5+D6+B6)/4

After selecting the

appropriate command,

EXCEL with perform

the Liebmann method

for you.

In fact, you will be able

to watch the iterations.

Table 2: Results of finite difference model.

A B C D E F G H I J K

1 100 101 102 103 104 105 106 107 108 109 110

2 101.6 102 102.6 103.4 104.2 105 105.8 106.6 107.4 108 108.4

3 102.5 102.7 103.1 103.7 104.3 105 105.7 106.3 106.9 107.3 107.5

4 103 103.1 103.4 103.9 104.4 105 105.6 106.1 106.6 106.9 107

5 103.3 103.3 103.6 104 104.5 105 105.5 106 106.4 106.7 106.7

6 103.3 103.4 103.7 104 104.5 105 105.5 106 106.3 106.6 106.7

...end of problem.

Secondary Variables

• Because its distribution is described by the

Laplace equation, temperature is considered

to be the primary variable in the heated

plate problem

• A secondary variable may also be of interest

• In this case, the second variable is the rate

of heat flux across the place surface

i

TCkqi

r

r

y

x1

2

y

2

xn

1j,i1j.i

y

j,1ij.1i

x

i

q

qtan

qqq

y2

TT'kq

x2

TT'kq

i

TCkq

FINITE DIFFERENCE

APPROXIMATION

BASED ON RESULTS

THE RESULTING

FLUX IS A VECTOR

WITH MAGNITUDE

AND DIRECTION

Finite Difference: Parabolic

Equations B2- 4AC = 0

0Dy

uC

yx

uB

x

uA

2

22

2

2

These equations are used to characterize

transient problems.

We will first study this in one spatial direction (1-D).

t

T

x

Tk

2

2

Consider the heat-conduction equation

As with the elliptic PDEs, parabolic equations can be solved

by substituting finite difference equations for the

partial derivatives.

However we must now consider changes in time

as well as space

t

x

y

x

l

iu

spatial

{ temporal

{

Centered finite divided difference

Forward finite divided

difference

T

TT

x

TT2Tk

t

T

x

Tk

l

i

1l

i

2

l

1i

l

i

l

1i

2

2

2

l

1i

l

1i

l

1i

l

i

1l

i

l

i

1l

i

2

l

1i

l

i

l

1i

xtk

where

TT2TTT

T

TT

x

TT2Tk

We can further reduce the equation:

NOTE:

Now the temperature

at a node is estimated

as a function of

the temperature at the

node, and surrounding

nodes, but at a previous time

Example

Consider a thin insulated rod 10 cm long with

k = 0.835 cm2/s

Let x = 2 cm and t = 0.1 sec.

At t=0 the temperature of the rod is zero.

hot

cold

Now subject the two ends to temperatures of

100 and 50 degrees

Set the nodes with these boundary and initial conditions.

This is what we consider as the conditions

at t=0

100 0 0 0 0 50

i= 0 1 2 3 4 5

x= 0 2 4 6 8 10

100

0

0

0

0

50

t = 0

Consider the temperature

at this node at time t+t

lllll

1ii1ii

1

i TT2TTT

0

2

4

6

8

10

x

0021 0 00T tt

1

0

1

2

3

4

5

i

= 0.020875

t 0 0.1 0.2

x

0 100 100 100

2 0 =B5+$B$1*(B6-2*B5+B4) =C5+$B$1*(C6-2*C5+C4)

4 0 =B6+$B$1*(B7-2*B6+B5) =C6+$B$1*(C7-2*C6+C5)

6 0 =B7+$B$1*(B8-2*B7+B6) =C7+$B$1*(C8-2*C7+C6)

8 0 =B8+$B$1*(B9-2*B8+B7) =C8+$B$1*(C9-2*C8+C7)

10 50 50 50

A B C D 1

2

3

4

5

6

7

8

9

2

1ii1ii

1

i

x

tk

where

TT2TTT

lllll

0

10

20

30

40

50

60

70

80

90

100

0 5 10

x (cm)

t (s

ec)

5 sec

10 sec

15 sec

...end of example

Convergence and Stability

• Convergence means that as x and t

approach zero, the results of the numerical

technique approach the true solution

• Stability means that the errors at any stage

of the computation are attenuated, not

amplified, as the computation progresses

• The explicit method is stable and

convergent if 2

1

Derivative Boundary Conditions T

o

TL

In our previous example To and TL were constant values.

However, we may also have derivative boundary conditions

i

1

i

0

i

1

i

0

1i

0 TT2TTT

Thus we introduce an imaginary point at i = -1

This point provides the vehicle for providing the derivative BC

Derivative Boundary Conditions

q0=

0

TL

For the case of qo = 0, so T-1 = T1 .

So in this case the balance at node 0 is:

llll

010

1

0 T2T2TT

Derivative Boundary Conditions

q0=

10

TL

For the case of qo = 10, we need to know k’ [= k/(rC)].

Assuming k’ =1, then 10 = - (1) dT/dx,

or dT/dx = -10

llll

ll

ll

010

1

0

11

11

T21

x20T2TT

k

x20TT

x2

TTk10

Implicit Method

• Explicit methods have problems relating to

stability

• Implicit methods overcome this but at the

expense of introducing a more complicated

algorithm

• In this algorithm, we develop simultaneous

equations

Explicit Implicit

grid point involved with space difference

grid point involved with time difference

With the implicit method, we develop a set of simultaneous

equations at step in time

2

1l

1i

1l

i

1l

1i

x

TT2T

2

l

1i

l

i

l

1i

x

TT2T

1l

0

1l

0

l

i

1l

1i

1l

i

1l

1i

l

i

1l

i

2

1l

1i

1l

i

1l

1i

tfT

TTT21T

t

TT

x

TT2Tk

which can be expressed as:

For the case where the temperature level is given at the end

by a function f0 i.e. x = 0

1l

0

l

i

1l

1i

1l

i

1l

0

1l

0

l

i

1l

1i

1l

i

1l

1i

tfTTT21

tfT

TTT21T

Substituting

In the previous example problem, we get a 4 x 4 matrix

to solve for the four interior nodes for each time step

Documents

Numerical Methods Notes