Systems Realization Laboratory

Criteria for evaluating uncertainty representations

ASME DETC/CIE 2006 Philadelphia, PA

Workshop on Uncertainty Representation in Robust and Reliability-Based Design

September 10, 2006

Jason Matthew Aughenbaugh

jason@arlut.utexas.eduhttp://www.srl.gatech.edu/Members/jaughenbaugh/

2

Outline: basic questions discussed

• What is an “uncertainty representation”?

• What requirements must an uncertainty representation meet to be valid?

• How can uncertainty representations be compared?

3

Basic definition

• “Uncertainty representation” is used here as an all encompassing term▪ Underlying model of uncertainty▪ Definition of mathematical operations▪ Computational implementations▪ Associated decision rules

• Conjecture▪ Uncertainty representations are merely models of a

decision-maker’s information state▪ As such, there is no absolute validation of a single

“true” representation, only relative validation

4

Basic criteria for comparing uncertainty representation in engineering design

• How easy to implement in practice?• Can it be applied consistently?• Does it help lead to better designs?

▪ Better = ???• More robust?• More reliable?• Better optima?

▪ How measure quality external to the formalism?

• e.g.: Robustness = f(x,y), then f*(x,y) clearly the most robust

5

Fundamental requirements for theories of uncertainty

• A mathematical representation of that uncertainty▪ e.g., the axioms of probability

• A calculus for manipulating that uncertainty▪ e.g., P(A or B) = P(A)+P(B)-P(A and B)

• A meaningful way of measuring the uncertainty▪ e.g., P(A)=(time A occurred)/(total events)

• Mature methodological aspects of the theory▪ including procedures for making the various uncertainty principles

operational within the theory

• In design, also need a method for decision makingKlir, G. J. and R. M. Smith (2001). "On measuring uncertainty and uncertainty-based information: recent developments." Annals of Mathematics and Artificial Intelligence 32.1-4: 5-33.

??

Adapted from (Klir and Smith 2001)

6

Measurement example: a meter

• What is a “meter”?• Is there ambiguity?• What are the formal definitions?

▪ the length of pendulum with period of 2 seconds▪ one ten-millionth of the length of the earth’s meridian

along one-fourth the polar circumference▪ a particular platinum metre bar placed in the National

Archives in France▪ the length of the path traveled by a particular

wavelength of light in vacuum during a time interval of 1/299,792,458 of a second

• So what is a meter? …It depends on the def’n

7

Operational definitions in science

• How can we define quantities?• Bridgeman writes the following:

▪ “We evidently know what we mean by length if we can tell what the length of any and every object is....”

▪ “To find the length of an object, we have to perform certain physical operations.”

▪ The concept of length is therefore fixed when the operations by which length is measured are fixed…

Bridgeman, P. W. (1927). Logic of modern physics. New York: The Macmillan Company.

8

Operational definitions in uncertainty modeling

• Example (adapted from Cooke)▪ I ask Dr. Paredis, “What is the fadizzle that it rains

tomorrow?”▪ He answers: “First tell me what a fadizzle is.”▪ Right answer. But instead of telling him, I said: “Well, just

use your own idea of what you think a fadizzle is, and tell what the fadizzle that it rains tomorrow is.”

• An uncertainty representation must include a clearly defined procedure for measuring uncertainty

Can the outcome of this procedure be dependent on the individual?

Cooke, R. (2004). "The anatomy of the squizzel - the role of operational definitions in representing uncertainty." Reliability Engineering & System Safety 85.1-3: 313.

9

The role of the observer in measurement

• Example: relativity▪ Observations of length and time depend on state of the

observer• Two people observing the same object can measure different lengths

• However, two observers in the same state will measure the same length

• Can measurements of uncertainty be based on the state of the observer?

• e.g., on his or her preferences and beliefs?

• I believe “yes”, but there is still some debate▫ (e.g., frequentist versus subjective/personalist probabilities)

• Still requires a defined procedure for eliciting uncertainty measurements

10

Comparing uncertainty representations

• Why compare them?▪ To see which one is the one to use in

engineering design

• How compare them?▪ Start with theoretical criteria▪ Proceed to practical criteria

11

Theoretical comparisons

• Does each meet the main requirements?▪ Do any meet all of the main requirements?

• What is uncertainty?▪ Do we even know what we are trying to

represent?• an inherent property of the natural universe?• a psychological state in the brain?• a purely philosophical construct?

▪ Is there only one true uncertainty?

• Some theoretical considerations may be unanswerable

12

Practical comparisons

• How are they used in practice?

• Which one works better?▪ Easier to measure?▪ Faster to compute with?▪ Lead to better design decisions?

• Practical analysis▪ Empirical results▪ Theoretical performance bounds

Cost Benefit

13

Summary: Questions to ask when evaluating uncertainty formalisms

What is X?• How does one represent uncertainty in X?• How does X support decision making in design?

▪ How does one measure/elicit uncertain information in X?▪ What is the inference formalism in X?▪ How does one implement such inference numerically?▪ How does one make decisions based on uncertain information

represented in X?

• What are the advantages and limitations of X?• For which design scenarios is X most appropriate?

For formalism “X” (a model and its associated methods)

14

Thank you for your attention• Acknowledgements

▪ Office of Naval Research • Contract No. N00014-06-G-0218-01

▪ NSF Graduate Research Fellowship

• Questions?