Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Unit 4 - 1 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
SYST 542Decision Support Systems
EngineeringInstructor: Kathryn Blackmond Laskey
Fall Semester, 2006
Unit 4: DSS Elements:The Model Subsystem (2)
Other Model System Technologies
Unit 4 - 2 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Outline
• Other Model System Technologies– Heuristic methods for solving optimization
problems– Artificial Intelligence and Expert Systems– Statistical Methods– Some Additional Methods
• Matching Methods to Problems
Unit 4 - 3 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Models for Supporting Decisions• Models can support decisions in a number of
ways:– Assist with problem formulation– Find optimal or approximately optimal (according to model)
solution– Assist in composing solutions to subproblems– Portray decision-relevant information in a way that makes
decision implications clear– Draw conclusions from data (data --> information ->
knowledge)– Predict results of proposed solution(s)– Evaluate proposed solution(s)– Can you think of others?
• Different modeling technologies are useful fordifferent kinds of support
Unit 4 - 4 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Example Heuristics• Greedy hill climber
– Begin with a candidate solution– Change in direction that most improves solution– Never go downhill
• Decomposition– Break problem into simpler subproblems– Solve subproblems separately– Recompose solutions
• Heuristic search– Search space can be constructed as tree– Depth first, breadth first, best first: policies for deciding how to expand the
tree
• Approximate and adjust– Use cheap / fast / available approximation method– Adjust solution– e.g., use linear programming on integer problem and move to nearest
integer solution
Unit 4 - 5 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Natural Analogy Heuristics• Nature is an efficient optimizer
– Apply methods based on analogy to natural systems
• Simulated annealing– Modify current solution randomly and evaluate objective function– Accept new solution if better than old. Otherwise, accept with probability
depending on system "temperature"– Gradually decrease temperature (make it harder to accept worse solutions)
• Evolutionary algorithms– Maintain "population" of solutions– Solutions reproduce with # offspring depending on objective function
(survival of fittest)– Apply evolutionary operators to change solutions from generation to
generation (e.g., crossover, mutation)
Unit 4 - 6 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Knowledge-Based Systems• Basic premise: intelligent problem solving
requires flexible application of knowledge• Key feature: separate representation of
knowledge from application to solve problems• Advantages:
– opportunistic application of knowledge– flexibility to try different solution strategies– facilitates modularity, maintainability, ability to explain
reasoning to users
Unit 4 - 7 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Elements of a Generic Knowledge Based System
• Knowledge base– Generalizations about the domain– R1: “IF GPA of <x> is less than 3.0 THEN <x> is not
eligible for combined BS/MS program”• Long term memory
– Facts about the world– F1: “GPA of G12345678 is 2.75”
• Inference engine– Uses R1 and F1 and the allowable proof steps to derive
the conclusion “G12345678 is not eligible forcombined BS/MS program”
• Working memory– Holds on to intermediate results of computations– Receives inputs from the outside world– Things can be read from and written into LTM and KB
• Control strategy– Sets priorities– Decides what order to do which tasks
LTM KB
WM
Data KnowledgeStructures(e.g. rules)
InferencesInferenceEngine
Unit 4 - 8 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Representing Domain Knowledge
• Knowledge is different from data or information– Knowledge structures reflect structure of domain– Reasoners access and manipulate knowledge structures– Representation serves as a surrogate for represented entities
• Problems that are very difficult in one representation canbecome straightforward in a better representation
– e.g., arithmetic in Roman vs Arabic numerals• A knowledge base contains knowledge structures that
represent:– Types of entities that can exist in the world– Attributes of entities of each type– Relationships entities can have to each other– Processes in which entities can participate– Individual entities: their types, attributes, relationships and behavior
• A formal representation of knowledge about a domain isalso called a domain ontology
– Ontologies represent semantics to facilitate reuse, sharing, andsemantic interoperability
• Reasoners manipulate stored knowledge and incominginformation to derive new knowledge
Unit 4 - 9 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Example Knowledge Structures• Rules
– IF-THEN knowledge– Allows system to derive consequences of knowledge– Rule-based system
» Represents knowledge as rules and reasons by applying rules» One of the first and most popular technologies of AI
• Object Classes / Instances– Represent types of entities and type-specific attributes– Subtypes can inherit structure and properties from parent
types– Instances represent specific entities of a given type– (Frames/Slots <=> Objects/Attributes)
• Semantic networks– Organizes knowledge about relationships between entities– Nodes represent entities; arcs represent relationships; arcs
are labeled by relationship type
Unit 4 - 10 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Organizing Knowledge• Knowledge about things
– Knowledge about things» Types of things» Features a type of thing can have» Possible values for features
– Frames (artificial intelligence), objects (computer science), and relations(information systems)
• Knowledge about allowable transformations– Declarative knowledge: State axioms for allowable transformations and use
theorem proving techniques to do the transformation– Procedural knowledge: Set up triggers to execute sequences of
transformations• Applying knowledge to achieve goal
– Forward chaining: apply rules in forward direction to achieve goal– Backward chaining: look for antecedents of goal and declare achieving them
as sub-goal
Unit 4 - 11 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Three-Legged Stool• Knowledge representation - formal representation for domain
semantics:– Entity types and instances, attributes, relationships, processes
• Decision theory– Goals are formalized as utility functions– Uncertainty is represented as probability distributions– “Ideal” inference and problem solving are formalized as Bayesian reasoning
and decision making– Practical inference and problem solving are formalized as “bounded
rationality”
• Algorithmic computer science– Find efficient algorithms for solving optimization and inference problems– Find efficient algorithms for approximating solutions to optimization and
inference problems and characterizing the quality of solutions– Apply heuristic methods to decompose problem, solve sub-problems,
compose results
• There is increasing cross-fertilization among OR andtraditional artificial intelligence
Unit 4 - 12 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Statistics• Statistics is about using data to draw
inferences about uncertain phenomena– Many methods from OR, decision analysis, and AI use
probability distributions to represent uncertain outcomes– Distributions are specified by some combination of theory,
expert knowledge, and observations– Statistics provides methods for drawing well-justified
inferences from data
• Statistics is also about creating summaries ofobservations that will be useful for decisionmakers– Graphical displays– Numerical summaries
Unit 4 - 13 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Statistical Methods• Exploratory data analysis
– graphical tools for data display– data transformation methods– outlier detection methods– robust methods for exploratory identification of trends and
patterns– clustering
• Statistical model construction and evaluation– Parameter estimation– Hypothesis testing and model evaluation– Model selection and model averaging– Statistical pattern recognition
• Sampling design methods– experimental design– survey design– design of observational studies
Unit 4 - 14 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Parametric Models• Parametric statistical model:
y ~ f(y | θ), or y ~ f(y | x, θ), where~ means “distributed as”y denotes an uncertain variable or vector of variablesx denotes observed variables (covariates)θ denotes an unobserved parameter
• Goals of modeling and data analysis:– Determine functional form of parametric model– Estimate value of parameter
» “Best” point estimate» Interval of plausible values» Predict likely future values» Draw substantive conclusions (whether a variable influences another;
whether data support a theoretically interesting value of parameter)
Unit 4 - 15 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Types of Statistical Models(some examples)
• Regression– Estimate an equation relating a dependent variable to
one or more independent variables– Example: examine relationship between students’
college GPA and high school grades• Analysis of variance
– Evaluate whether average value of a response isdifferent for different groups of individuals
– Example: evaluate whether patients taking a drug dobetter than patients taking a placebo
• Time series models– Examine trends and/or cycles in data over time– Example: predict price of a stock
Unit 4 - 16 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Linear Regression in Excel• Data on tar, nicotine, weight and CO in 25 brands of cigarettes
downloaded from http://www.mste.uiuc.edu/regression/cig.html
Tar vs Carbon Monoxide
0
5
10
15
20
25
30
0 5 10 15 20 25 30 35
Tar (mg)
CO (mg)
Linear regression linecan be computedusing slope andintercept functions inExcelExcel statistics add-inhas more sophisticatedstatistical capabilityCan add trendline tochart without creatingnew column of data
Unit 4 - 17 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Data Analysis Process• If you are collecting data, consider analyses to be run in the design of
your data collection– Simulate “dummy data” and analyze to test your procedures– How much data will you need to make the case you need to make?
• First step: clean and explore– Look for errors and problems– Check for outliers– Plot data and look for patterns
• Next step: confirmatory analysis– Select statistical model– Estimate parameters– Evaluate confidence levels or other indicators of strength of conclusion– Evaluate sensitivity to outliers and violations of assumptions
• Final step: report results– Conveying complex data understandably is an important human factors issue– A picture is worth a thousand words– Include information about uncertainty in result– Interface between model and dialog subsystems
Unit 4 - 18 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Connectionist Modelsor Neural Networks
• Connectionist philosophy– Complex behavior comes from interactions among simple
computational units– Natural analogy: simulate intelligent behavior using process
modeled after human brains
• A neural network consists of– a large set of computationally simple units or nodes– links or connections between nodes
• Learning occurs by adjusting strengths ofconnections
– supervised learning: regression– unsupervised learning: clustering
Unit 4 - 19 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
A Simple Feedforward Network
Input layer
Hidden layer
Output layer
We can think of a feedforward neural network as anonlinear regression model:
y ~ f(y | x, w) y denotes the output node(s)w denotes the weight vectorx denotes the input node(s)
Learning the weights corresponds to estimating theparameters of the regression model
A good discussion of the relationship between neural networks andstatistical methods can be found at:http://www.faqs.org/faqs/ai-faq/neural-nets/part1/section-15.html
Unit 4 - 20 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Machine Learning• Machine learning is the discipline devoted to
development of methods that allow computersto “learn” (improve performance based onresults of past performance)
• Machine learning draws from artificialintelligence, traditional computer science, andstatistics– Extract regularities from samples of data– Construct knowledge structures (typically rules) that
characterize the regularities– Evaluate performance against samples not seen before
Unit 4 - 21 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Data Mining• The IT revolution has created vast archives of data• Data mining is a collection of methods from statistics,
computer science, engineering, and artificial intelligence forsifting through large stores of data to identify interestingpatterns
• There is a great deal of overlap with machine learning– In machine learning the emphasis is on using data to improve performance
on a well-defined task according to some performance measure (induction)– In data mining the emphasis is on identifying interesting patterns in large
volumes of data (discovery)– Both machine learning and data mining make heavy use of statistical
methods• The term data mining is sometimes used pejoratively to mean
fishing for spurious patterns and concocting post-hocexplanations
Unit 4 - 22 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Knowledge Discovery and Data Mining
• Source: From Data Mining to Knowledge Discovery in Databases (PDF),Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth, AIMagazine 17(3): Fall 1996, 37-54
• Available from http://www.kddnuggets.com
Unit 4 - 23 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Economic Methods• Microeconomic models
– Analyze economic systems in which firms / agents are modeled as utilitymaximizers
– Static: analyze equilibrium– Dynamic: analyze behavior over time
• Game theory– Multiple players each have possible actions and objective functions– An economy is a many-person game
• Macroeconomic models (econometrics)– Statistical estimation of relationships between economic variables
• Cost / benefit analysis– Benefits of proposed policy option are quantified in dollar terms and
evaluated against cost
Unit 4 - 24 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Cost-Benefit Analysisvs Multiattribute Utility
• Conceptually these methods are concerned with the sameproblem
• Both translate costs and benefits to common units– CBA: common units are dollars– MAU: common units are utilities
• Concerns with CBA as typically applied:– tendency to ignore "nonmeasurable" attributes– measuring social costs by "willingness to pay"– equity concerns due to maximizing net dollar benefit
• Objective:– CBA usually maximizes Benefit/Cost ratio (good for allocating limited budget
among multiple projects)– MAU chooses option with highest expected utility (good for selecting single best
option)
Unit 4 - 25 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Management Science Methods• Project planning and scheduling methods
– Milestone charts– Gantt charts– Critical Path Method (CPM) charts
• Project monitoring methods– Earned value analysis
Unit 4 - 26 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Methods from Accounting and Finance
• Not really a separate category of modelingmethods– Draws from optimization, statistics, economics, and other
disciplines• Examples of common model types
– Return on investment analysis» Identify time period for investment to pay back
– Break even analysis» Identify revenues needed for product to cover costs
– Forecasting supply, demand, interest rates, etc– Cost estimation
» Example: COCOMO is a model for estimating cost of softwareprojects
» See http://www.softstarsystems.com/overview.htm for adescription
Unit 4 - 27 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Problem Structuring Methods• Group techniques
– Brainstorming– Nominal group– Delphi
• Methods for organizing large sets of elements– Checklists– Interaction matrices– Trees
• Structured modeling methods– Cognitive maps– Causal loop diagrams– Semantic networks
Unit 4 - 28 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Some Philosophy• The most valuable output of a good model is
insight• Blind acceptance of "the computer's answer"
is almost always a mistake– sometimes closer examination of unintuitive result reveals
unmodeled asepct of the problem
• Often much insight can be gained from a less-than-complete analysis
Unit 4 - 29 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Sensitivity Analysis• Sensitivity analysis means varying the inputs
to a model to see how the results change• Sensitivity analysis is a very important
component of exploratory use of models– model is not regarded as “correct”– sensitivity analysis helps user explore implications of
alternate assumptions– human computer interface for sensitivity analysis is
difficult to design well• In many models we need to make assumptions
we cannot test– Sensitivity analysis examines dependence of results on
these assumptions
Unit 4 - 30 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Model Evaluation• Verification: evaluate whether model is
implemented correctly and produces outputs itwas designed to produce
• Validation: evaluate whether model accuratelyreproduces the phenomenon being modeled
• Why are verification and validation bythemselves insufficient for model evaluation?
Unit 4 - 31 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Some Approaches to Model Evaluation
• Compare model output to ground truth– Quality of result (according to what metric?)– How many cases?– Coverage of all relevant conditions?– Can we instrument all relevant variables?– Fidelity of test conditions to real problem
• Compare model output to expert judgment– Same questions as above
• Model “walk through”– Have we included the right variables?– Are the methods appropriately matched with problems?– Are the methods applied correctly?
Unit 4 - 32 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Model Evaluation for DSS• In a decision support environment we are
concerned with the quality of the Model +Decision Maker combination
• We need to evaluate whether supporteddecision maker performs better than– Decision maker alone– Model alone
• Metrics include– Quality of result– Time to solution– Understanding of solution by stakeholders– Buy-in by organization– Other measure?
Unit 4 - 33 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Method Selection• Problem Type 1: General decision support for
poorly structured problems– Brainstorming or cognitive mapping to define the problem– Rule-based expert system to suggest solution approaches
and assist in constructing model» no uncertainty, multiple objectives --> MAU» uncertainty, simple objective --> influence diagram» uncertainty, multiple objectives --> MAU combined with
influence diagram– Rule-based expert system to suggest default values for
parameters (reduces elicitation burden)– Influence diagram solver– MAU model– Explanation generation expert system for interpreting
results– Sensitivity analysis methods
Unit 4 - 34 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Method Selection• Problem Type 2: Optimization problem with
complex constraints and objective function– Use simplified mathematical programming model which does not
account for all aspects of problem– Use heuristic search (deterministic or stochastic) to modify
model output– Use statistical parameter estimation and/or expert system and/or
user input for values of parameters (constraints, objectivefunction)
• Problem Type 3: Plan design or enhancement ofcomplex system
– Build causal model of existing system– Use statistical methods to estimate parameters of model– Use simulation to predict system evolution– Use designed experiments to test cause and effect assumptions– Use sensitivity analysis to investigate how different values of
input parameters affect result
Unit 4 - 35 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Method Selection• Problem Type 4: Identify patterns in large
database of high-dimensional data– neural network (unsupervised learning)– cluster analysis– graphical exploratory data analysis
• Problem Type 5: Classify new observationsinto categories, where categories are definedaccording to a large sample of exemplars
– discriminant analysis– neural network– genetic algorithm– simulated annealing– Bayesian network learning
Unit 4 - 36 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Method Selection• Problem Type 6: Classify observations into
categories, when there is no pre-defined databaseof exemplars
– expert system– Bayesian network– neural network (trained by human-generated exemplars)– policy capture (this is a regression method in which the
dependent variable is human judgment and the independentvariables are cues used by human to classify)
• Problem Type 7: Select option from largedatabase classified on many dimensions
– multiattribute utility– screening– policy capture
Unit 4 - 37 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Method Selection
• Problem Type 8: Select the best option froman implicitly defined option space with givenmeasure of performance
– mathematical programming– heuristic search– genetic algorithm– simulated annealing
• Problem Type 9: Forecast future values ofsome variable or variables
– time series methods– macroeconomic methods– simulation methods
Unit 4 - 38 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Model Management• DBMS allows user to store and retrieve data
independent of physical representation of data incomputer
• MBMS allows user to invoke and run modelsindependent of particular implementation of models
– Paradigm of models as data is gaining favor– Parameters and assumptions represented explicitly rather than
being buried in computer codes– Facilitates flexible application and reuse
• Unlike DBMS, no standardized MBMS have emerged– Many different kinds of models, each with specialized structures,
uses, and methods– Building and applying models requires specialized expertise
Unit 4 - 39 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Activities Supported by MBMS• Access and retrieve models• Exercise models
– specify model parameters– link models together– control execution of models
• Manipulate models– change assumptions and/or parameter values
• Store models and model results• Maintain models• Create new models
Unit 4 - 40 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
Simulation Methods• Given:
– Executable model of how a system behaves– Inputs representing policy options, environmental variables,
initial state of system, assumptions about system andenvironment
• Output:– Behavior of system state over time
• We use a simulation to conduct computerexperiments on “virtual reality” to evaluate options
– Can be exploratory, consolidative, or combination• Examples:
– Simulate future conflict to help decide which weapons systemsto procure
– Simulate economy to predict impact of economic policy options• Issues:
– Accuracy of simulation– Sensitivity of results to input assumptions– Design of experiment to maximize information from simulation
Unit 4 - 41 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
In Summary...
Unit 4 - 42 -
Department of Systems Engineering and Operations Research
Copyright © 2006, Kathryn Blackmond LaskeySYST 542
References• .