sensitivity analysis

Sensitivity Analysis: Sensitivity Analysis: a Validation and a Validation and Verification ToolVerification Tool

Terry BahillTerry BahillSystems and Industrial EngineeringSystems and Industrial EngineeringUniversity of ArizonaUniversity of ArizonaTucson, AZ 85721-0020Tucson, AZ [email protected]@sie.arizona.eduCopyright ©, 1993-2009 BahillCopyright ©, 1993-2009 BahillThis file is located atThis file is located athttp://www.sie.arizona.edu/sysengr/slides/http://www.sie.arizona.edu/sysengr/slides/

04/08/23 2

ReferencesReferencesSmith, E. D., Szidarovszky, F., Karnavas, W. J. and Bahill, A. T., Sensitivity analysis, a powerful system validation technique, The Open Cybernetics and Systemics Journal,http://www.bentham.org/open/tocsj/openaccess2.htm, 2: 39-56, 2008, doi: 10.2174/1874110X00802010039

W. J. Karnavas, P. Sanchez and A. T. Bahill, Sensitivity analyses of continuous and discrete systems in the time and frequency domains, IEEE Trans. Syst. Man. Cybernetics, SMC-23(2), 488-501, 1993.

© 2009 Bahill

04/08/23 3

You should perform a You should perform a sensitivity analysis anytime yousensitivity analysis anytime you create a model write a set of requirements design a system make a decision do a tradeoff study originate a risk analysis want to discover the cost drivers

© 2009 Bahill

04/08/23 4

In a sensitivity analysisIn a sensitivity analysis change

the values of inputs parameters

architectural features measure changes in

outputs performance indices

© 2009 Bahill

04/08/23 5

A sensitivity analysis can be used A sensitivity analysis can be used toto validate a model, warn of unrealistic model behavior, point out important assumptions, help formulate model structure, simplify a model, suggest new experiments, guide future data collection efforts, suggest accuracy for calculating parameters, adjust numerical values of parameters, choose an operating point, allocate resources, detect critical criteria, suggest manufacturing tolerances, identify cost drivers.

© 2009 Bahill

04/08/23 6

History: the earliest sensitivity History: the earliest sensitivity analysesanalyses The genetics studies on the pea by

Gregor Mendel, 1865. The statistics studies on the Irish hops

crops by Gosset (reported under the pseudonym Student), ca 1890.

© 2009 Bahill

04/08/23 7

Classes of sensitivity functionsClasses of sensitivity functions Analytic

for well defined systems usually partial derivatives

Empirical show sensitivity to parameters observe system changes when

parameters are changed works for an unmodeled system

© 2009 Bahill

04/08/23 8

Types of sensitivity functionsTypes of sensitivity functions Analytic

absolute relative semirelative

Empirical direct observation sinusoidal variation of parameters design of experiments Excel, using what-if analysis

© 2009 Bahill

04/08/23 9

The absolute-sensitivity functionThe absolute-sensitivity functionThe absolute-sensitivity of the function F to variations in the parameter is

It should be evaluated at the normal operating point (NOP).

NOP

F FS

© 2009 Bahill

04/08/23 10

ExamplesExamples Absolute-sensitivity functions are used

to calculate changes in the output due to changes in the inputs or system parameters

to see when a parameter has its greatest effect

in adaptive control systems

© 2009 Bahill

04/08/23 11

A process model exampleA process model example

x and y are inputs and A to F model system parameters. The output z is love potion number 9. The normal operating point is

What is the easiest way to increase the quantity of z? This sounds like a problem for absolute-sensitivity functions.

2 2z Ax By Cxy Dx Ey F

0 0 0 0

0 0 0 0

( , ) (1,1), 1, 2,3, 5, 7, 8,

x y A BC D E F

0 2.z

© 2009 Bahill

Absolute-sensitivity functions*Absolute-sensitivity functions*

04/08/23 12 © 2009 Bahill

0 0 0 0 0NOP

0 0 0 0 0NOP

2 0,

2 0.

zx

zy

zS A x C y Dx

zS B y C x Ey

20

NOP

20

NOP

0 0NOP

0NOP

0NOP

NOP

1,

1,

1,

1,

1,

1,

zA

zB

zC

zD

zE

zF

zS xAzS yBzS x yCzS xDzS yEzSF

04/08/23 13

What about interactions?What about interactions? Change two parameters at the same time. Interactions can be bigger than the first-order

effects. Non-steroidal, anti-inflammatory drugs (NSAID)

such as Ibupropren and Aleve have dangerous interactions with angiotensin converting enzymes, which effect the kidneys and lower blood pressure. No pharmacists would allow you to take both.

My mother once cleaned the toilet with sodium hypochlorite (Clorox bleach) and ammonia. It produced chorine gas.

Alcohol and barbiturates are much more dangerous if mixed.

© 2009 Bahill

CooperationCooperation The performance of a system could be

greater than the sum of its subsystems (cooperation).

Alone, neither a human nor a knife can slice bread.

Together a blind person and a Seeing Eye dog do better than either alone.

A pair of chopsticks performs more than twice as well as an individual chopstick.

Two lions chasing a Thompkins Gazelle are more than twice as likely to catch it, than a single lion.

04/08/23 © 2009 Bahill14

04/08/23 15

Interactions in the process control Interactions in the process control modelmodel Change two parameters at the same

time. Mixed partial derivatives can be bigger

than first-order partial derivatives. Of the 64 possible second-partial

derivatives, only the following are nonzero.

© 2009 Bahill

Interactions*Interactions*

04/08/23 16 © 2009 Bahill

2

2

2

NOP

2

0NOP

2

02NOP

2

02NOP

1,

3,

2 2,

2 4.

zy E

zx y

zx

zy

zSy E

zS Cx y

zS Ax

zS By

2

0NOP

2

0NOP

2

NOP

2

0NOP

2

0NOP

2 2,

1,

1,

2 2,

1,

zx A

zx C

zx D

zy B

zy C

zS xx A

zS yx C

zSx D

zS yy B

zS xy C

04/08/23 © 2009 Bahill17

Table 1. Effects of Individual and Combined Parameter Changes for some second-order interaction terms with delta of 1.0, where

0x x x etc.

Functions Normal values

Values increased

by one unit

New z values z

Total change

in z

( , )f x A A=1 x=1

A=2 x=2 7 5 5

0( , )f x A A=1 A=2 3 1 2z

0( , )f x A x=1 x=2 3 1

0 0( , )f x A A=1 x=1 2 0

( , )f y B B=2 y=1

B=3 y=2 8 6 6

0( , )f y B B=2 B=3 3 1 3z

0( , )f y B y=1 y=2 4 2

0 0( , )f y B B=2 y=1 2 0

The purpose of this slide is to show the affects of interactions, without using mathematics.

We will now use these data to estimate the value of one of the mixed-partial derivatives

Estimate the Estimate the mixed-second-partial derivativemixed-second-partial derivative

04/08/23 © 2009 Bahill18

20 0 0 0 0 0( , ) ( , ) ( , ) ( , ) ( , )f f f f f

2 7 3 3 2 31

zx A

This is the wrong answer. Analytically we found that the correct value is 2.

A smaller step size, 0.01A smaller step size, 0.01

04/08/23 © 2009 Bahill19

Use the following general equation from Smith, Szidarovszky, Karnavas and Bahill [2008].

0 0 0 0 0 0 0 0

2 20 0 0 0

( , ) ( , ) ( , )( ) ( , )( )

1 ( , )( ) 2 ( , )( )( ) ( , )( )2!

x y

xx xy yy

f x y f x y f x y x x f x y y y

f x x f x x y y f y y

Converting to find the value if we change x and A yields

0 0 0 0 0 0 0 0

2 20 0 0 0

( , ) ( , ) ( , )( ) ( , )( )1 ( , )( ) 2 ( , )( )( ) ( , )( )2!

x A

xx xA AA

f x A f x A f x A x x f x A A A

f x x f x x A A f A A

Now, using the symbols that we used in our absolute sensitivity functions, we can write

2 2

0 0 0 0

2 2 2

( , ) ( , )1 ( , ) 2 ( , ) ( , )2!

z zx A

z z zx Ax A

z S x A S x A

S S S

Inserting numbers we get

10*0.01 1*0.01 2*0.0001 2* 2*0.0001 0.01 30 02

z

This delta z will now be put into row 2 column 5 of the following table.

z

04/08/23 © 2009 Bahill20

Table 2. Effects of Individual and Combined Parameter Changes for some second-order interaction terms with delta of 0.01, where

0x x x etc.


Values increased by delta

New z values z

Total change in z

( , )f x A A=1 x=1

A=1.01 x=1.01 2.0103 0.0103 0.0103

0( , )f x A A=1 A=1.01 2.0100 0.0100 0.0101z

0( , )f x A x=1 x=1.01 2.0001 0.0001

0 0( , )f x A A=1 x=1 2.0000 0

( , )f y B B=2 y=1

B=2.01 y=2.01 2.0104 0.0104 0.0104

0( , )f y B B=2 B=2.01 2.0100 0.0100 0.0102z

0( , )f y B y=1 y=2.01 2.0002 0.0002

0 0( , )f y B B=2 y=1 2.0000 0

04/08/23 © 2009 Bahill21

Estimate the mixed-second-partial derivative, according to this formula from Smith, Szidarovszky, Karnavas and Bahill [2008].

20 0 0 0 0 0( , ) ( , ) ( , ) ( , ) ( , )f f f f f

Table 3. Values to be used in estimating the second partial derivative

Terms Parameter values with a 0.01 step size

Function values

( , )f A =1.01 x =1.01 2.0103

0( , )f A =1.00 x =1.01 2.0100

0( , )f A =1.01 x =1.00 2.0001

0 0( , )f A =1.00 x =1.00 2.0000

2 2.0103 2.0100 2.0001 2.0000 0.0002 20.01*0.01 0.0001

zx A

This is the same value that we computed analytically.

04/08/23 © 2009 Bahill22

Table 4. Effects of Individual and Combined Parameter Changes for some third-order interaction terms, delta = 0.001


Values increased by delta

New z values z Total change in z

( , )f x A A=1 x=1

A=1.001 x=1.002 2.0108 0.0108 0.0108

0( , )f x A A=1 A=1.001 2.0100 0.0100 0.0102z 0( , )f x A x=1 x=1.001 2.0001 0.0001

0( , )f x A x=1 x=1.001 2.0001 0.0001

0 0( , )f x A A=1 x=1 2.0000 0

( , )f y B B=2 y=1

B=2.001 y=1.002 2.0112 0.0112 0.0112

0( , )f y B B=2 B=2.001 2.0100 0.0100 0.0104z 0( , )f y B y=1 y=1.001 2.0002 0.0002

0( , )f y B y=1 y=1.001 2.0002 0.0002

0 0( , )f y B B=2 y=1 2.0000 0

04/08/23 23

Third-order partial derivativesThird-order partial derivatives Once again the interaction affect is larger

than the sum of the individual changes. But at least the third-order terms are

smaller than the first and second-order terms.

Three of the third-order partial derivatives are greater than zero.

All of the fourth-order partial derivatives are zero.

© 2009 Bahill

MinisummaryMinisummary The purposes of this section were

to show the bad affects of too large of a step size

to show how to calculate derivatives analytically and to estimate derivatives numerically

to show that interactions are important to show how to consider third- and

forth-order derivatives.

04/08/23 © 2009 Bahill24

04/08/23 25

A pendulum clockA pendulum clock11I have a grandfather clock in Tucson that I would like to move to a cabin up on Mount Lemmon. But I’ve been told that the changes in temperature and altitude will make it inaccurate. Which will be the bigger culprit?The period of oscillation of a pendulum is

A one-meter pendulum has a two-second period. If the temperature changes by T, then the length becomes

Use the absolute-sensitivity function of P with respect to T to calculate how many seconds per day the clock will gain.

glP /2

0 (1 )Tl l k T

© 2009 Bahill

04/08/23 26

A pendulum clockA pendulum clock22The absolute-sensitivity function is

The coefficient of expansion of a brass rod is

At the normal operating point T=0 and l=l0, so

Mt. Lemmon is 2000 meters higher than Tucson and temperature changes 5ºC per 1000 m. So T=-10ºC. Therefore, the change in period is

The pendulum will gain 8.6 seconds per day*.

NOP NOP

2 /(1

P TT

T

l g l kT g l k TS

50 0/ 2 10 sec/ CP

TTk glS

52 10 /°CTk

42 10 secPTP S T

© 2009 Bahill

04/08/23 27

A pendulum clockA pendulum clock33However, the gravitational acceleration constant depends on altitude (H)

Therefore the period becomes

and the absolute-sensitivity of P with respect to H is

For this equation the normal operating point is sea level, so H=0 and g0=9.78. So,

60 3 10 , where is in meters.g g H H

0

2

H

lPg k H

30 NOP

P HH

H

k lSg k H

© 2009 Bahill

04/08/23 28

A pendulum clockA pendulum clock44

Going from Tucson to Mount Lemmon H=2000, so

The clock loses 26 seconds per day.

Although changes due to temperature and altitude are in the opposite direction, they do not cancel each other out, because changes due to altitude are bigger.

0 7

30

3 10 s/mHPH

k lS

g

46 10 secPHP S H

© 2009 Bahill

04/08/23 29

Single pole with time delaySingle pole with time delay11

Use an absolute-sensitivity function to find when the parameter K has the greatest effect on the step response of the system. The step response is

( )( )( ) 1

sY s KeM sR s s

( )

1

s

srKesY

s s

© 2009 Bahill

04/08/23 30

Single pole with time delaySingle pole with time delay22

The absolute-sensitivity function of the step-response with respect to K is

which transforms into

K has its greatest effect when the response reaches steady-state.

1τ)(

0

θ0

ssesY

s

KS sr

0 0( ) /( ) 1sr tKy t eS

© 2009 Bahill

04/08/23 31




© 2009 Bahill

04/08/23 32

The relative-sensitivity functionThe relative-sensitivity functionThe relative-sensitivity of the function F to variations in the parameter is

Relative-sensitivity functions are used to compare parameters.

% change in% change in

F F F FS

0

0NOP

F FFS

© 2009 Bahill

04/08/23 33

Process control modelProcess control modelWhat is the easiest way (smallest percent change in an operating point parameter) to increase the quantity of z that is being produced? Now this problem seems appropriate for relative-sensitivity functions.

© 2009 Bahill

Relative-sensitivity functionsRelative-sensitivity functions

04/08/23 34 © 2009 Bahill

0 00

NOP 0 0

0 0

NOP 0 0

0 00 0 0 0 0

NOP 0 0

0 00 0 0 0 0

0 0NOP

3.5,

4,

2 0,

2 0.

zE

zF

zx

zy

z E ES yE z zz F FSF z zz x xS A x C y Dx z z

z y yS B y C x Ey z z

20 00

NOP 0 0

20 00

NOP 0 0

0 00 0

NOP 0 0

0 00

NOP 0 0

0.5,

1,

1.5,

2.5,

zA

zB

zC

zD

z A AS xA z zz B BS yB z zz C CS x yC z zz D DS xD z z

04/08/23 35

FF is most important is most important Therefore, we should increase F if we wish

to increase z. What about interactions? Could we do

better by changing two parameters at the same time?

© 2009 Bahill

Interactions*Interactions*

04/08/23 36 © 2009 Bahill

2 2 20 0 0 0

2 2 20 0NOP

20 0 0 0 0

2 20 0NOP

20 0

20NOP

2 20 0 0 0

2 20 0NOP

20 0 0 0 0

2 20 0NOP

2 2*1 *1 0.5,2

0.75,

1.25,

2 1.0,

0.75,

zx A

zx C

zx D

zy B

zy C

z x A x ASx A z z

z x C x y CSx C z z

z x DSx D z

z y B y BSy B z z

z y C x y CSy C z z

2

2

20 0

20NOP

20 0 0 0 0

2 20 0NOP

2 2 20 0 0

2 2 20 0NOP

2 2 20 0 0

2 2 20 0NOP

1.75,

0.75,

2 0.5,

2 1.0.

zy E

zx y

zx

zy

z y ESy E z

z x y x y CSx y z z

z x x ASx z z

z y y BSy z z

04/08/23 37

Mini-summaryMini-summary Using absolute-sensitivity functions, the

second- and third-order terms, e. g.

were the most important, but using relative-sensitivity functions, F was the most important parameter.

The absolute-sensitivity functions show the most important parameters for a fixed size change in the parameters

The relative-sensitivity functions show the most important parameters for a certain percent change in the parameters.

3

2and zx A

2

2

zy

© 2009 Bahill

04/08/23 38

Relatively the most importantRelatively the most important0 0

NOP 0 0

4zF

F FzSF z z

© 2009 Bahill

04/08/23 39

The operating point*The operating point* In the process control example, if the

operating point is changed from (1, 1) to (10, 10), then the output z becomes most sensitive (relatively) to the input y.

At the operating point (1, 1), the output is not sensitive to the inputs, which means we could twiddle with the inputs forever and not be able to control the output. Therefore, this is not a desirable operating point.*

© 2009 Bahill

More examples of More examples of relative sensitivity functionsrelative sensitivity functions

04/08/23 © 2009 Bahill40

04/08/23 41

Double pole with time-delayDouble pole with time-delay

Which of the parameters is most important?

2

( )( )( ) (τ 1)

sY s KeM sR s s

0

0NOP

1M

K

M KMKS

0

00θ

NOP

θ θθ

M M sMS

0

0

0

NOP 0

2ττ τ 1M sM

M sS

© 2009 Bahill

04/08/2342

Frequency domain output

04/08/23 43

ResultsResults For low frequencies, K is biggest For mid-frequencies, is biggest For high frequencies, is biggest

© 2009 Bahill

A simple closed loop systemA simple closed loop system

04/08/23 44 © 2009 Bahill

Transfer function ( )( )( )

Y s KM sR s s K

Time-domain step-response ( ) 1 Kt

sry t e Time-domain relative-sensitivity function

0

0 0

0 0

1 1sr

K ty Kt

K K t K tNoP

K K teS tee e

Frequency-domain step-response

( )( )sr

KY ss s K

Frequency-domain relative-sensitivity function

0

ˆ ( )srYK

sS ss K

Take the inverse Laplace transform 0

0ˆ ( )srY K t

KS t K e

where is the unit impulse.

04/08/23 45 © 2009 Bahill

Time domain output

04/08/23 46

An expert systemAn expert systemSensitivity functions need not be

functions of time or frequency.If premise1 = true (CF1)and premise2 = true (CF2)or premise3 = true (CF3)then conclusion = true CF4.

The certainty of the rule uses the minimum of the certainties of the AND clauses.

© 2009 Bahill

Certainty factor domain output

04/08/23 47 © 2009 Bahill

If CF1 < CF2, then the final certainty factor becomes

3 41 4 1 4100100 100 10,000f

CF CFCFCF CFCFCF

The relative-sensitivity functions are

1

3 4 104

0NOP

110,000 100

fCF

CFf

CF CF CFCFSCF

20fCF

CFS

3

301 4 4

0NOP

110,000 100

fCF

CFf

CFCFCF CFSCF

4

3 1 3 4 401

0NOP

2100 100 1,000,000

fCF

CFf

CF CFCF CF CFCFSCF

04/08/23 48 © 2009 Bahill

Assume 1 3 4 280 and 81CF CF CF CF Then 0 87fCF and

10.26fCF

CFS

20fCF

CFS

30.26fCF

CFS

40.53fCF

CFS Changes in CF4 are twice as important as changes in CF1 or CF3 and increases in CF2 have no effect.

04/08/23 49

Limitations of the Limitations of the relative-sensitivity functionrelative-sensitivity function

L x t y t Lx t Ly t

0

NOP 0

( ) ( ) ( )( )

f t f t f t f tSf t

0

NOP 0

( ) ( ) ( )ˆ( )

F s F s F s F sSF s

Because ˆf t F sS S we have two functions

© 2009 Bahill

04/08/2350

Disadvantages ofDisadvantages ofrelative-sensitivity functionsrelative-sensitivity functions different in time and frequency

domains cannot use Laplace transforms to get

time-domain solution division by zero problem

0

0NOP

F FFS

© 2009 Bahill

04/08/23 51




© 2009 Bahill

04/08/23 52

Double pole with time delayDouble pole with time delay

© 2009 Bahill

04/08/23 53

The semirelative-sensitivity functionThe semirelative-sensitivity functionThe semirelative-sensitivity of the function F to variations in the parameter is

0NOP

F FS

© 2009 Bahill

Tradeoff studyTradeoff study

04/08/23 54 © 2009 Bahill

A Generic Tradeoff Study Criteria Weight of

Importance Alternative-1 Alternative-2

Criterion-1 Wt1 S11 S12 Criterion-2 Wt2 S21 S22 Alternative Rating Sum1 Sum2

A Numeric Example of a Tradeoff Study Alternatives Criteria Weight of

Importance Umpire’s Assistant

Seeing Eye Dog

Accuracy 0.75 0.67 0.33 Silence of Signaling 0.25 0.83 0.17

Sum of weight times score

0.71 The

winner 0.29

1 1 11 2 21 2 1 12 2 22andSum Wt S Wt S Sum Wt S Wt S

04/08/23 55

Which parameters could change Which parameters could change the recommendations?the recommendations?Use this performance index

Compute the semirelative-sensitivity functions.

1 1 2

1 11 2 21 1 12 2 22 0.420PI Sum Sum

Wt S Wt S Wt S Wt S

© 2009 Bahill

Semirelative-sensitivity functions*Semirelative-sensitivity functions*

04/08/23 56 © 2009 Bahill

1

2

11

21

12

22

11 12 1

21 22 2

1 11

2 21

1 12

2 22

0.26

0.16

0.50

0.21

-0.25

-0.04

FWt

FWt

FS

FS

FS

FS

S S S Wt

S S S Wt

S Wt S

S Wt S

S Wt S

S Wt S

A sensitivity matrixA sensitivity matrix

04/08/23 57 © 2009 Bahill

Analytic semirelative-sensitivity function values for PI1, the difference of the alternative ratings Alternatives

Criteria Weight of Importance

Umpire’s Assistant

Seeing Eye Dog

Accuracy 1

1

PIWtS = 0.26 1

11

PISS = 0.50 1

12

PISS = -0.25

Silence of Signaling

1

2

PIWtS = 0.16 1

21

PISS = 0.21 1

22

PISS = -0.04

A nice way to display the sensitivities

04/08/23 58

What about interactions?What about interactions?The semirelative-sensitivity function of PI1 for the interaction of Wt1 and S11 is

which is as big as the first-order terms.

1 11 0 0 0 0

2

1 11 1 111 11 NOP

0.5025FWt S

FS Wt S Wt SWt S

© 2009 Bahill

04/08/23 59

Interactions for PIInteractions for PI11

So interactions are important.

Semirelative Sensitivity Values Showing Interaction Effects

Function Normal values

Values increased by 10%

New F values F Total change

in z

1

FWtS 1Wt =0.75 1Wt =0.82 0.446 0.026

11

FSS 11S =0.67 11S =0.74 0.470 0.050

0.076F

1 11

FWt SS 1Wt =0.75

11S =0.67 1Wt =0.82

11S =0.74 0.501 0.081 0.081

© 2009 Bahill

04/08/23 60

A new performance indexA new performance index A problem with performance index PI1 is that

if s11=s12, then the sensitivity with respect to Wt1 becomes equal to zero.

Mathematically this is correct, but logically it is wrong.

Another problem is that the sensitivity with respect to Wt1 does not depend on scores for the nonwinning alternatives and we do want the sensitivities to depend on the other parameters.

The following performance index solves both of these problems.

3

1 1

1 n m

i iji j

PI Wt Sm

© 2009 Bahill

04/08/23 61

Semirelative sensitivity functionsSemirelative sensitivity functions

3 3

1NOP

1i

mPI

Wt i i ijji

PIS Wt Wt S

Wt m

3

ij

i ijPIS

Wt SS

m

© 2009 Bahill

Sensitivity matrix for PISensitivity matrix for PI33

Table VI. Analytic semirelative-sensitivity function values for PI3, the sum of all weight times scores Alternatives



Seeing Eye Dog

Accuracy of the call

3

10.38PI

WtS 3

110.25PI

SS 3

120.12PI

SS


3

20.13PI

WtS 3

210.10PI

SS 3

220.02PI

SS

04/08/23 © 2009 Bahill62

04/08/23 63

It is not difficultIt is not difficultAlthough these equations may look formidable, they are easy to compute with a spreadsheet. For example

is merely the sum of the weight times scores in column k and this is already in the spreadsheet. Furthermore, because

and the rest of the second order sensitivities are zero, Table VI is complete: it has all of the sensitivities in it.

1

n

k kjk

Wt S

3 3

ij i ij

PI PIS Wt SS S

© 2009 Bahill

04/08/23 64

What about interactions?What about interactions?

Yes, We do have to worry about interactions, because this is bigger than most of the first order sensitivities.

3

1 11 0 0

23 1 11

1 111 11 NOP

0.25PIWt S

PI Wt SS Wt SWt S m

© 2009 Bahill

04/08/23 65

Interaction Sensitivity MatrixInteraction Sensitivity MatrixTable VII. Analytic semirelative-sensitivity function values for the interactions of PI3 Alternatives



Seeing Eye Dog

Accuracy of the call 3

1 110.25PI

Wt SS 3

1 120.12PI

Wt SS

Silence of Signaling 3

2 210.10PI

Wt SS 3

2 220.02PI

Wt SS

These cells contain the same numerical values as Table VI.

© 2009 Bahill

Tradeoff studies are hierarchicalTradeoff studies are hierarchical

04/08/23 66 © 2009 Bahill

The structure of a hierarchical tradeoff study

Criteria Normalized Criteria Weights

Subcriteria Normalized Subcriteria Weights

Scores for Alternative-1

Scores for Alternative-2

Performance (1)CW Subcriteria-1 (1)

1Wt (1)11S (1)

12S Subcriteria-2 (1)

2Wt (1)21S (1)


3Wt (1)31S (1)


4Wt (1)41S (1)

42S Cost (2)CW Subcriteria-1 (2)

1Wt (2)11S (2)


2Wt (2)21S (2)

22S Schedule (3)CW Subcriteria-1 (1)

1Wt (3)11S (3)


2Wt (3)21S (3)

22S Risk (4)CW Subcriteria-1 (4)

1Wt (4)11S (4)


2Wt (4)21S (4)


3Wt (4)31S (4)

32S Alternative Ratings

1Sum 2Sum

04/08/23 67

A new performance index, PI5A new performance index, PI5Because most of tradeoff studies are hierarchical, in the Spin Coach and the PopUp Coach I used this performance index

( )( ) ( ) ( )

51 1 1

1 n lk ml l l

i ijl i j

PI CW Wt Sm

5( )

( )( ) ( ) ( )

1 1

1l

n l mPI l l l

i ijCWi j

S CW Wt Sm

5 ( ) ( ) ( )1ij

PI l l lS i ijS CW Wt S

m

© 2009 Bahill

04/08/23 68

Single pole with time delay (when)Single pole with time delay (when)Transfer function

( )( )( ) 1

sY s KeM sR s s

Step-response

( )( 1)

s

srKeY s

s s

Semirelative-sensitivity functions 0

0

0( 1)sr

sYK

K eSs s

0

0 0

0( 1)sr

sY K eS

s

00 0

20( 1)

sr

sY K eS

s

Which (for 0t ) transform to 0 0( )

0( ) (1 )sry tKS t K e

0 0( )0 0( )sry tS t K e

0 0( )

0 0 0( ) ( )sry tS t K t e

© 2009 Bahill

04/08/23 69 © 2009 Bahill

04/08/23 70

What does this teach us?What does this teach us? If the model does not match the physical

system in the early part of the step response, then adjust the time-delay of the model.

For steady state … In the middle …

© 2009 Bahill

04/08/23 71



We have just examined the analytic sensitivity functions. We are now ready to look at the empirical sensitivity functions.


© 2009 Bahill

04/08/23 72

Empirical sensitivity functionsEmpirical sensitivity functions11 The method of direct

observation can be preformed on real-world systems models of those systems simulations of those

models

© 2009 Bahill

04/08/23 73 © 2009 Bahill

04/08/23 74 © 2009 Bahill

RealSystem

Model ofReal

System

ComputerSimulationof Model

Modelers Modelers

GoodModelers

GoodModelers

Mathem

aticiansEx

perim

enta

lists

04/08/23 75

Estimating derivativesEstimating derivatives

are small, then the second term on the right can be neglected.

0If (x-x ) and ( )f

© 2009 Bahill

04/08/23 76

Tradeoff study exampleTradeoff study exampleFor a +10% parameter change, the semirelative-sensitivity function is

This is very easy to compute.

Tradeoff Study Matrix with S11 Increased by 5%



Seeing Eye dog

Accuracy 0.75 0.74 0.33 Silence of Signaling 0.25 0.83 0.17 Sum of weight times score 0.76 0.29

0 0

0

100.1

F F FS F

© 2009 Bahill

Sensitivity matrixSensitivity matrix

04/08/23 © 2009 Bahill77

Table X. Numerical estimates for semirelative-sensitivity function for PI3, the sum of the alternative ratings squared, for a plus 10% parameter perturbation Alternatives



Seeing Eye Dog

Accuracy of the call

3

10.38PI

WtS

3

110.25PI

SS

3

120.12PI

SS


3

20.13PI

WtS 3

210.10PI

SS

3

220.02PI

SS

These are the same results that were obtained in the analytic semirelative sensitivity section.

04/08/23 78

But what about the second-order But what about the second-order terms? terms? Namely

When using the sum of weighted scores combining function

the second derivatives are all zero. So our estimations are all right. This is not true for the product combining function,

most other combining functions (See Daniels, Werner and Bahill [2001] for explanations of other combining functions.) or other performance indices.In particular let’s try PI3.

20

( ) ( )2!

f x x

1 1 11 2 21 2 1 12 2 22andSum Wt S Wt S Sum Wt S Wt S

1 2 1 21 11 21 2 12 22 and Wt Wt Wt WtF S S F S S

© 2009 Bahill

04/08/23 79

Derivative of a function of two Derivative of a function of two variablesvariables

Let us examine the second-order terms, those inside the { }, for two reasons to see if they are large and must be

included in computing the first derivative

to estimate the effects of interactions on the sensitivity analysis

0 0 0 0 0 0 0 0

2 20 0 0 0

( , ) ( , ) ( , )( ) ( , )( )

1 ( , )( ) 2 ( , )( )( ) ( , )( )2!

x y

xx xy yy

f x y f x y f x y x x f x y y y

f x x f x x y y f y y

© 2009 Bahill

04/08/23 80

InteractionsInteractionsPreviously we derived the analytic semirelative-sensitivity function for the interaction of Wt1 and S11 as,

which is as big as the first-order semirelative-sensitivity functions.

0 03

1 11 0 0

21 113

1 111 11 NOP

0.25PIWt S

Wt SPIS Wt S

Wt S m

© 2009 Bahill

04/08/23 81

InteractionsInteractionsFor a 10% change in parameter values, a simple-minded approximation is

using our tradeoff study values we get

This does not match the analytic value.

What went wrong?

2

20 0 0 0

0 0

100.1 0.1

F FF FS F

3

1 11

210 0.424PIWt SS F

© 2009 Bahill

04/08/23 82

Maybe the step size is too bigMaybe the step size is too big Let’s reduce the perturbation step size to

0.1%?

This is closer, but it is still too big.

2

20 0 0 0

0 0

10000.001 0.001

F FF FS F

3

1 11

21000 0.393PIWt SS F

© 2009 Bahill

04/08/23 83

What went wrong?What went wrong?In the previous computations, we changed both parameters at the same time and then compared the value of the function to the value of the function at its normal operating point. However, this is not the correct estimation for the second-partial derivative.

© 2009 Bahill

04/08/23 84

Estimating the second partialsEstimating the second partials11

To estimate the second-partial derivatives we should start with

20 0 0 0 0( , ) ( , ) ( , )f f f

0 0 0 02

0 0

( , ) ( , ) ( , ) ( , )( , )

f f f ff

20 0 0 0 0 0( , ) ( , ) ( , ) ( , ) ( , )f f f f f

0

© 2009 Bahill

Estimating the second partialsEstimating the second partials22

04/08/23 85 © 2009 Bahill

Values to be Used in Estimating the Second Derivative

Terms Parameter values with a 0.1% step size, that is 1Wt =0.00075 and 11S =0.00067

Function values

( , )f 1Wt =0.75075

11S =0.67067 0.50063

0( , )f 11S =0.67067 0.50025

0( , )f 1Wt =0.75075 0.50038

0 0( , )f 1Wt =0.75000

11S =0.67000 0.50000

23

1 11

0.50063 0.50038 0.50025 0.50000 0.5*0.00075*0.00067

PIWt S m

04/08/23 86

Estimating the sensitivity Estimating the sensitivity functionsfunctionsTo get the semirelative-sensitivity function we multiply the second-partial derivative by the normal values of Wt1 and S11 to get

Now, this is the same result that we derived in the analytic semirelative sensitivity section.

3

1 11 0 0 0 0

23

1 11 1 111 11 NOP

0.5 0.25PIWt S

PIS Wt S Wt SWt S

© 2009 Bahill

04/08/23 87

Lessons learnedLessons learned For a tradeoff study using the sum

combining function and a simple performance index, anything works.

Otherwise, the perturbation step size should be small. Five and 10% perturbations are not acceptable.

It is incorrect to estimate the second partial derivative by changing two parameters at the same time and then comparing that value of the function to the value of the function at its normal operating point. Estimating second derivatives requires evaluation of four not two numerator terms.

© 2009 Bahill

04/08/23 88

Sensitivity analysis of a risk Sensitivity analysis of a risk analysisanalysisLet be the probability of occurrence, the severity, and the risk, for the jth failure mode. Risk is

Use the performance index* andcalculate the semirelative-sensitivity functions

The largest sensitivities are always those for the largest risk. This means that we should spend extra time and effort estimating the probability and severity of the highest ranked risk, which seems eminently sensible.

, and j j jP S R

j j jR P S

1

n

jj

PI R

0j

PIP j j jS S P R

0j

PIS j j jS R S R

© 2009 Bahill

04/08/23 89

LinearityLinearityAlthough the model is linear, the sensitivity functions are not.

© 2009 Bahill

04/08/23 90 © 2009 Bahill

04/08/23 91




© 2009 Bahill

04/08/23 92

Empirical sensitivity functionsEmpirical sensitivity functions22 Sinusoidal variation of parameters, also called

frequency-domain experiments response-surface methodology

© 2009 Bahill

04/08/23 93

Sinusoidal variation of Sinusoidal variation of parametersparameters Make two runs of the system

1. all parameters are set at their normal values

2. all parameters are modulated sinusoidally

Compute the power spectrum of each Form the ratio of the two spectra at each

frequency Spikes will be observed

at the modulation frequencies at frequencies related to nonlinearities at frequencies related to product effects

© 2009 Bahill

04/08/23 94

A first-order, negative-feedback A first-order, negative-feedback systemsystem

K______s + A

H

R(s) Y(s)+

-

© 2009 Bahill

04/08/23 95

A first-order, negative-feedback, A first-order, negative-feedback, control system (continued)control system (continued)

Input: one Hertz, unit-amplitude sinusoid Duration: one second Sampling: 2048 evenly spaced samples Modulation frequencies: 5, 30 and 170 Hz

HKAsK

sRsYsM

)()()(

© 2009 Bahill

04/08/23 96

Modulation equationsModulation equations A = 0.1 (1 + 0.5 sin (5 2t)) H = 50 (1 + 0.5 sin (30 2t)) K = 1.0 (1 + 0.5 sin (170 2t))

© 2009 Bahill

Step response with modulated Step response with modulated parametersparameters

04/08/23 97 © 2009 Bahill

04/08/23 98

Location of spikesLocation of spikes If a parameter is modulated at a we expect

to see power at a If there is a parabolic nonlinearity we also

expect power at 2a because 2 sin2 x = 1 - cos 2x

If the system is sensitive to the product of two parameters modulated at a and b, then we expect power at a b because 2 sin x sin y = cos(x-y) - cos(x+y) These product terms are called

interactions.

© 2009 Bahill

04/08/23 99 © 2009 Bahill

04/08/23 100

Split peaks are due to the one Hertz input.

This technique probably produces relative-sensitivities.

© 2009 Bahill

The semirelative-sensitive The semirelative-sensitive functions*functions*

04/08/23 101

© 2009 Bahill

1.50

~22

00 1.0

000

sHKAs

AKSssM

K

1.50~

220 50

000

20

sHKAs

HKS

M

H

1.50~

2200 1.0

000

sHKAs

AKSM

A

04/08/23 102

An M/M1 queueAn M/M1 queue service rate =

= 0.8 + 0.2 sin(46/4096 2t) arrival rate =

= 0.4 + 0.2 sin(4/4096 2t)

© 2009 Bahill

04/08/23 103

An M/M1 queueAn M/M1 queue service rate =

= 0.8 + 0.2 sin(46/4096 2t) arrival rate =

= 0.4 + 0.2 sin(4/4096 2t)

© 2009 Bahill

04/08/23 104

© 2009 Bahill

04/08/23 105

An M/M1 queue is a low pass filter The interaction peaks are not the same

height.

© 2009 Bahill

Bode diagram of a low-pass filterBode diagram of a low-pass filter

04/08/23 106

© 2009 Bahill

04/08/23 107

Some problems with sinusoidal Some problems with sinusoidal variation of parametersvariation of parameters There must be an input signal. Shape of the spikes depends on parameters

of the input signal. The output cannot be stationary. The frequency response of the system (e.g.

low-pass, high-pass, resonance, etc.) must be known.

The range of linearity of the system must be known.*

Parameters of the FFT and the windows must be understood.

© 2009 Bahill

04/08/23 108

© 2009 Bahill

04/08/23 109



Empirical direct observation sinusoidal variation of parameters design of experiments (DoE) Excel, using what-if analysis

© 2009 Bahill

04/08/23 110

Empirical sensitivity functionsEmpirical sensitivity functions33 Design of experiments (DoE) should be used

when experiments are expensive. When doing a Taguchi 3-level design pick a

normal value, a high value and a low value. If the high and low are some percentage

change, then you are doing a relative sensitivity analysis.

If the high and low are plus and minus a unit, then you are doing an absolute sensitivity analysis.

Alternatively, the high and low could be realistic design options, in which case it does not correspond to any of our sensitivity functions.

© 2009 Bahill

04/08/23 111

© 2009 Bahill

Low High

Altitude

Temperature

Relative Humidity

Barometric Pressure

Perc

ent C

hang

e in

Air

Den

sity

0 ft

70 ºF 2600 feet85 ºF50%760 mm Hg

90%745 mm Hg

10%775 mm Hg

100 ºF

5200 feet

1%

3%

5%

7%

9%

-1%

-3%

-5%

-7%

-9%

Percent change in air density over the parameter ranges to be expected for a typical July

afternoon in United States ballparks.

Medium

04/08/23 112




© 2009 Bahill

The Pinewood DerbyThe Pinewood Derby

04/08/23 113

© 2009 Bahill

04/08/23 114

The most important parameters The most important parameters in the Pinewood Derbyin the Pinewood Derby baseline for Overall Happiness

scoring function baseline for Percent Happy

Scouts scoring function importance weight for Overall

Happiness evaluation criterion baseline for Number of Repeat

Races scoring function input value for Percent Happy

Scouts evaluation criterion input value for Number of

Repeat Races evaluation criterion

© 2009 Bahill

04/08/23 115

Of the 89 parameters only 3 could Of the 89 parameters only 3 could change the preferred alternativechange the preferred alternative 1. The Tradeoff Function

with 90% for Performance and 10% for Utilization of Resources the preferred alternative was a round robin with best time scoring

with 57% for Performance and 43% for Resources the preferred alternative switched to the double elimination tournament

© 2009 Bahill

04/08/23 116

© 2009 Bahill

Sensitivity Analysis of Pinewood Derby (simulation data)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Performance Weight

Ove

rall

Sco

re

Single eliminationDouble eliminationRound robin, mean-timeRound robin, best-time Round robin, points

04/08/23 117

© 2009 Bahill

Sensitivity of Pinewood Derby (prototype data)

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Performance Weight

Ove

rall

Sco

re

Double eliminationRound robin, best-time Round robin, points

04/08/23 118

Of the 89 parameters only 3 could Of the 89 parameters only 3 could change the preferred alternativechange the preferred alternative22 2. The slope of the

Percent Happy Scouts scoring function 3. The baseline for the

Percent Happy Scouts scoring function

© 2009 Bahill

04/08/23 119

© 2009 Bahill

04/08/23 120

ValidationValidation11 If a system (or its model) is very sensitive to

parameters over which the customer has no control, then it may be the wrong system for that customer.

If the sensitivity analysis reveals the most important parameters and that result is a surprise, then it may be the wrong system.

If a system is more sensitive to its parameters than to its inputs, then it may be the wrong system or the wrong operating point.

If the sensitivities of the model are different from the sensitivities of the physical system, then it may be the wrong model.

© 2009 Bahill

04/08/23 121

ValidationValidation22 If you delete a requirement, then your

completeness measure (a traceability matrix) should show a vacuity.

After you make a decision, do a sensitivity analysis and see if changing a parameter would change your decision.

Domain experts should agree with the sensitivity analysis about which criteria in a tradeoff study are the most important.

Domain experts should agree with the sensitivity analysis about which risks are the most important.

Do a sensitivity analysis of prioritized lists: see if changing the most important criteria would change the prioritization.

© 2009 Bahill

04/08/23 122

VerificationVerification Unplanned excessive sensitivity to any

parameter is a verification mistake. Sensitivity to interactions should be

flagged and studied: such interactions may be unexpected and undesirable.

© 2009 Bahill

04/08/23 123

RésuméRésumé After you

build a model, or write a set of requirements, or do a tradeoff study, or design a system,

you should study that thing to see if it makes sense.

One of the best ways to study a thing is with a sensitivity analysis.

© 2009 Bahill

Describe this talk to your Vice Describe this talk to your Vice PresidentPresident Professor Bahill modeled our potion production

process and did a sensitivity analysis of it. So we now have a better understanding of our process. His sensitivity analysis accounts for nonlinearities and parameter interactions. His equations for estimating parameters are correct (because Szidarovszky derived them). This analysis shows which parameters are the most important for making our potent.

I recommend that we name our potion Love Potion Number Nine. We should buy the copyright for that song and play it in the background of our TV commercials.

Play an audio clip.

04/08/23 © 2009 Bahill124

04/08/23 125

© 2009 Bahill

Documents

sensitivity analysis