Process Model Formulation and Solution, 3E4Section E, Part A: Numerical Differentiation

Instructor: Kevin Dunn dunnkg@mcmaster.ca

Department of Chemical Engineering

Course notes: © Dr. Benoıt Chachuat01 November 2010

1

Why calculate derivatives numerically?

Consider the modelling of an isothermal batch reactor, where anirreversible, first-order reaction A→ B is taking place.

Experimental data:

t [min] cA [mol L−1]

0 1.005 0.8410 0.7220 0.5730 0.4745 0.3760 0.30

Problem:Estimate the kinetic rateconstant k

2

How calculate derivatives numerically?

Problem statement:

Given a real-valued function f : IR → IR, calculate its derivative f ′(x0) ata point x0 ∈ IR, using function evaluations only!

Two possible scenarios

1. Empirical function: you are free to choose the points at which toevaluate the function f , but:

I The derivatives f ′ of f is extremely difficult or inconvenient toevaluate

I An algebraic expression for f is not available, e.g. a functionevaluated via a computer program

2. Experimental data: function values available at discrete (fixed)data points only!

3

Outline and recommended readings

Numerical differentiation for black-box functionsFirst- and second-order approximationsRichardson extrapolationSecond-order derivatives

Numerical differentiation for discrete data pointsMethods based on polynomial interpolationMethods based on spline interpolation

(Strongly) recommended readings:

I Chapters 4.1.3, and 23.1-23.3 in S. C. Chapra,and R. P. Canale, “Numerical Methods forEngineers”, McGraw Hill, 5th/6th edition.

4

First-order derivatives for black-box functions

I Consider a function f : IR → IR and a point x0 ∈ IR

I Assume that f ′ exists and is continuous around x0

First finite divided differenceGiven a finite step-size h > 0:

f ′(x0) ≈f (x0 + h)− f (x0)

h

f ′(x0) ≈f (x0)− f (x0 − h)

h

Critical questions:

I What is the error associated to such an approximation?

I How to choose the step-size h?

5

Refresher: the O(hp) notation

Consider the following term:

Rp(h) = αphp + αp+1h

p+1 + · · ·

I Rp(h) is said to be of the order of hp

as h→ 0, denoted by Rp(h) = O(hp)

I The behaviour of Rp(h) is similar tothat of hp as h→ 0

Properties:

I Let f (h) = O(hp) and g(h) = O(hq)

I Product: f (h)× g(h) = O(hp+q)

I Examples: f (h)× h3 = O(hp+3), g(h)

h2 = O(hq−2)

I Sum: f (h) + g(h) = O(hmin(p,q))I Example: −0.01h2 + 100h5 = O(h2)

6

First-order derivatives: O(h) approximations

What is the truncation error in the formula f ′(x0) ≈ f (x0+h)−f (x0)h ?

I By Taylor theorem:

f (x0 + h) = f (x0) + f ′(x0)h + f ′′(x0)h2

2+ O(h3)

I Therefore,

E (h) = f ′(x0)−f (x0 + h)− f (x0)

h= O(h) Why?

The truncation error is O(h)

How about in the formula f ′(x0) ≈ f (x0)−f (x0−h)h ?

f (x0 − h) = f (x0)− f ′(x0)h + f ′′(x0)h2

2+ O(h3)

E (h) = f ′(x0)−f (x0)− f (x0 − h)

h= O(h)

7

First-order derivatives: O(h) approximations (cont’d)

Consider the function f (x) = exp(−x2). Estimate the value of f ′(1)using forward and backward finite divided differences.

How to choose the step-size? What is the best achievableestimate?

h |f ′(1)− f (1+h)−f (1)h | |f ′(1)− f (1)−f (1−h)

h |1e-01 3.89e-02 3.40e-021e-02 3.70e-03 3.65e-031e-03 3.68e-04 3.68e-041e-04 3.68e-05 3.68e-051e-05 3.68e-06 3.68e-061e-06 3.68e-07 3.68e-071e-07 3.63e-08 3.64e-081e-08 1.18e-08 4.81e-091e-09 7.14e-08 3.96e-081e-10 3.49e-07 3.49e-071e-11 2.57e-06 2.98e-061e-12 9.69e-05 1.41e-051e-13 7.91e-04 3.19e-041e-14 3.01e-03 3.01e-03

8

First-order derivatives: O(h) approximations (cont’d)

Guidelines for step-size selection:

I Trade-off between

Truncation error:Gets smaller as hdecreases to 0

←→Round-off error:Gets larger as h

comes closer to 0

I There exists an optimal step-size that minimize the estimation error

I Bad news: A ‘good’ step-size must be chosen on a case-by-case basis

O(h) methods offer limited accuracy, which motivatesthe development of higher-order formulas

9

First-order derivatives: O(h2) approximations

Can a higher-order formula be developed?

I By Taylor theorem:

f (x0 + h) = f (x0) + f ′(x0)h + f ′′(x0)h2

2+ O(h3)

f (x0 − h) = f (x0)− f ′(x0)h + f ′′(x0)h2

2+ O(h3)

I Subtracting these expressions yields thecentered finite-divided-differenceapproximation:

f ′(x0) =f (x0 + h)− f (x0 − h)

2h+ O(?)

I Can you think of other O(h2) formulas?

10

First-order derivatives: O(h2) approximations (cont’d)

Consider the function f (x) = exp(−x2). Estimate the value of f ′(1)using centered finite divided differences.

How to choose the step-size? Are O(h2) formulas better?

h |f ′(1)− f (1+h)−f (1)h | |f ′(1)− f (1+h)−f (1−h)

2h |1e-01 3.89e-02 2.45e-031e-02 3.70e-03 2.45e-051e-03 3.68e-04 2.45e-071e-04 3.68e-05 2.45e-091e-05 3.68e-06 2.31e-111e-06 3.68e-07 4.67e-121e-07 3.63e-08 8.79e-111e-08 1.18e-08 3.52e-091e-09 7.14e-08 1.59e-081e-10 3.49e-07 3.49e-071e-11 2.57e-06 2.06e-071e-12 9.69e-05 4.14e-051e-13 7.91e-04 2.36e-041e-14 3.01e-03 3.01e-03

11

Richardson extrapolation: principles

Concept:

Combine two estimates of a given order O(hp) in order to obtain a newestimate of higher-order O(hp+1)

First-order case, p = 1I Consider the forward-finite-divided-difference approximation:

f ′(x0) =f (x0 + h)− f (x0)

h+ O(h) = Q1(h) + c1h + O(h2)

f ′(x0) =f (x0 + 1

2h)− f (x0)12h

+ O(h) = Q1(12h) + 1

2c1h + O(h2)

I How can these expressions be combined to yield an O(h2) formula?

f ′(x0) =

12

Richardson extrapolation: principles (cont’d)

General case, p ≥ 1

I Consider the order-p approximation:

f ′(x0) = Qp(h) + cphp + O(hp+1)

f ′(x0) = Qp(12h) + 1

2p cphp + O(hp+1)

I Combine these expressions to yield an O(hp+1) formula:

f ′(x0) =2pQp(

12h)− Qp(h)

2p − 1+ O(hp+1) How?

Application to p = 2

f ′(x0) =

13

Richardson extrapolation: recursive application

O(h) O(h2) O(h3)

Q1(h) =f (x0+h)−f (x0)

h → Q2(h) =2Q1( 1

2h)−Q1(h)

1 → Q3(h) =4Q2( 1

2h)−Q2(h)

3 · · ·↗ ↗

Q1(12 h) =

f (x0+ 12

h)−f (x0)

12

h→ Q2(

12 h) =

2Q1( 14

h)−Q1( 12

h)

1 · · ·

↗

Q1(14 h) =

f (x0+ 14

h)−f (x0)

14

h· · ·

.

.

.

Remarks:I Start with O(h) formula

I e.g., forward- or backward-finite-divided-difference approximation

I Increasing by one order O(hp)→ O(hp+1) requires one extrafunction evaluation f (x0 + 1

2p h) only

I Convenient to automate in a computer program

I For black-box/empirical functions only - incompatible with discretedata points

14

Richardson extrapolation: example

Consider the function f (x) = exp(−x2). Estimate the value of f ′(1)using Richardson extrapolation up to O(h5) for h = 10−1

Recursive application of Richardson extrapolation:

O(h) O(h2) O(h3) O(h4) O(h5)h -6.96821617e-01 -7.36758216e-01 -7.35796420e-01 -7.35758947e-01 -7.35758881e-01

h/2 -7.16789917e-01 -7.36036869e-01 -7.35763631e-01 -7.35758886e-01h/4 -7.26413393e-01 -7.35831941e-01 -7.35759479e-01h/8 -7.31122667e-01 -7.35777594e-01h/16 -7.33450130e-01

p |E(h)|O(h) 3.8937e-02

O(h2) 9.9933e-04

O(h3) 3.7538e-05

O(h4) 6.4523e-08

O(h5) 8.5059e-10

15

Interlude: who was Richardson?

WikipediA:

“Lewis Fry Richardson (1881 – 1953) was anEnglish mathematician, physicist,meteorologist, psychologist and pacifist whopioneered modern mathematical techniques ofweather forecasting, and the application ofsimilar techniques to studying the causes ofwars and how to prevent them. He is alsonoted for his pioneering work on fractals.”

16

Second-order derivatives for black-box functions

I Consider a function f : IR → IR and a point x0 ∈ IR

I Assume that f ′ and f ′′ exist and are continuous around x0

Can a formula that approximates f ′′(x0) be developed?

I By Taylor theorem:

f (x0 + h) = f (x0) + f ′(x0)h + f ′′(x0)h2

2+ O(h3)

f (x0 − h) = f (x0)− f ′(x0)h + f ′′(x0)h2

2+ O(h3)

I Subtracting these expressions yields the centeredfinite-divided-difference approximation:

f ′′(x0) =f (x0 + h)− 2f (x0) + f (x0 − h)

h2+ O(?)

I Approximations for higher derivatives f (3)(x), f (4)(x), . . ., can beobtained in a similar manner

17

First-order derivatives from discrete data points

Problem:

Given a set of data points (x1, f (x1)),(x2, f (x2)), . . ., (xn, f (xn)), estimate thederivative f ′(x) at a point x

At a data point x1, x2, . . . , xn:

I Possibility to use forward/backward finite-divided-differenceapproximations,

f ′(xk) ≈f (xk+1)− f (xk)

xk+1 − xkor f ′(xk) ≈

f (xk)− f (xk−1)

xk − xk−1

But, poor accuracy in case of distant data points

I In general, centered and higher-order formula not applicable forunevenly spaced data

18

First-order derivatives from discrete data points

Problem:

Given a set of data points (x1, f (x1)),(x2, f (x2)), . . ., (xn, f (xn)), estimate thederivative f ′(x) at a point x

At a point x that is not a data point:

I None of the techniques seen so far is applicable!

Concept:

1. Fit the polynomial pn−1 of degree n − 1 through n data points

2. Approximate the derivative f ′(x) by that of the interpolant p′n−1(x)

Caution: Interpolation should be local!

I Polynomial interpolation (involving no more than a few data points)

I Cubic spline interpolation

19

First-order derivatives using polynomial interpolation

Approximate derivatives at discretedata point x1, . . . , xn:

1. Consider each set of 3 adjacent points(xk−1, xk , xk+1)

2. Fit a 2nd-order polynomial

3. Calculate its derivative at xk

With Newton interpolating polynomials:

p2(x) = f (xk−1) + [f (xk), f (xk−1)] (x − xk−1)

+ [f (xk+1), f (xk), f (xk−1)] (x − xk−1)(x − xk)

p′2(x) = [f (xk), f (xk−1)] + [f (xk+1), f (xk), f (xk−1)] (2x − xk−1 − xk)

Therefore,

p′2(xk) = [f (xk), f (xk−1)] + [f (xk+1), f (xk), f (xk−1)] (xk − xk−1)

20

First-order derivatives using polynomial interpolation

With Lagrange interpolating polynomials:

p2(x) = f (xk−1)(x − xk)(x − xk+1)

(xk−1 − xk)(xk−1 − xk+1)+ f (xk)

(x − xk−1)(x − xk+1)

(xk − xk−1)(xk − xk+1)

+ f (xk+1)(x − xk−1)(x − xk)

(xk+1 − xk−1)(xk+1 − xk)

p′2(x) = f (xk−1)2x − xk − xk+1

(xk−1 − xk)(xk−1 − xk+1)+ f (xk)

2x − xk−1 − xk+1

(xk − xk−1)(xk − xk+1)

+ f (xk+1)2x − xk−1 − xk

(xk+1 − xk−1)(xk+1 − xk)

Therefore,

p′2(xk) = f (xk+1)xk − xk−1

(xk+1 − xk−1)(xk+1 − xk)+ f (xk)

(xk+1 − xk)− (xk − xk−1)

(xk − xk−1)(xk+1 − xk)

−f (xk−1)xk+1 − xk

(xk − xk−1)(xk+1 − xk−1)

What if xk+1 − xk = xk − xk−1 = ∆x? p′2(xk) =

21

First-order derivatives using polynomial interpolation

Consider the data points (0,−1), (2, 1), (3,−2), and estimate the firstderivative at x = 2 using polynomial interpolation

Newton interpolating polynomial:

0 −1 1−(−1)2−0 = 1 (−3)−1

3−0 = − 43

2 1 (−2)−13−2 = −3

3 −2

p2(x) =− 1 + 1x − 4

3x(x − 2)

p′2(x) = 1− 4

3(2x − 2) =

11

3− 8

3x

Estimated derivative at x=2:

p′2(2) =− 5

3

22

First-order derivatives using spline interpolation

Approximate derivatives at discretedata point x1, . . . , xn:

1. Fit a cubic spline s3(x) passingthrough x1, . . . , xn

2. Calculate its derivative at the desiredxk

Interpolating cubic splines:

s3(x) =

8><>:a1 + b1x + c1x

2 + d1x3, x1 ≤ x < x2

...an−1 + bn−1x + cn−1x

2 + dn−1x3, xn−1 ≤ x ≤ xn

s ′3(x) =

8><>:b1 + 2c1x + 3d1x

2, x1 ≤ x < x2

...bn−1 + 2cn−1x + 3dn−1x

2, xn−1 ≤ x ≤ xn

Therefore,

s ′3(xk) = bk−1 + 2ck−1xk + 3dk−1x2k = bk + 2ckxk + 3dkx

2k

23

First-order derivatives using spline interpolation

Consider the data points (0,−1), (2, 1), (3,−2), and estimate the firstderivative at x = 2 using spline interpolation

Interpolating spline:

s3(x) =

−1 + 7

3x − 1

3x3, 0 ≤ x < 2

−9 + 433x − 6x2 + 2

3x3, 2 ≤ x < 3

s ′3(x) =

73− x2, 0 ≤ x < 2

433− 12x + 2x2, 2 ≤ x < 3

Estimated derivative at x=2:

s ′3(2) =− 5

3

24