View
32
Download
0
Category
Tags:
Preview:
DESCRIPTION
Improving Group Decision Making Under Uncertain Circumstances: Applications in Defense Acquisition. Dennis Goldenson & Bob Stoddard (SEI) Ricardo Valerdi (University of Arizona ) COCOMO 2013 23 October 2013. Information Flow for Early Lifecycle Estimation (QUELCE). - PowerPoint PPT Presentation
Citation preview
© 2013 Carnegie Mellon University
Improving Group Decision Making Under UncertainCircumstances: Applications in Defense Acquisition
Dennis Goldenson & Bob Stoddard (SEI)Ricardo Valerdi (University of Arizona)
COCOMO 201323 October 2013
3
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
• Mission / CONOPS• Capability Based Analysis ...
• KPP selection• Systems Design• Sustainment issues
...
• Production Quantity• Acquisition Mgt• Scope definition/responsibility• Contract Award
Technology Development Strategy
Operational CapabilityTrade-offs
System CharacteristicsTrade-offs
Proposed Material Solution & Analysis of Alternatives
Information from Analogous Programs/Systems
Program Execution Change Drivers
Probabilistic Modeling (BBN) &
Monte Carlo Simulation
Expe
rt J
udge
men
tsInformation Flow for Early Lifecycle Estimation (QUELCE)
Plans, Specifications, Assessments
• analogy• parametri
c
Cost EstimatesProgram Execution
Scenarios with conditional probabilities of drivers/states
Driver States & Probabilities
• engineering
• CERs
4
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Expert Judgment: Dependency Structure Matrix
5
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
5
Most people are significantly overconfident and overoptimistic in their judgment!
Issues with Expert Judgment
An Estimate of SW Size
Calibrated = more realistic size and
wider range to reflect true expert
uncertainty
6
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
6
Studies Confirm Expert Judgment Issues
7
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Cost Estimation Research
Previous calibration research
Current research in progress
Future research & applications
8
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Calibration Training
A series of training exercises•Typically 3 or 4 in sequence
Each exercise includes:•A battery of factual questions
–Asking for upper and lower bounds within which people are 90 percent certain the correct answer lies
–Sometimes true false questions where people provide their confidence in their answers
•Brief reviews of the correct answers–Group discussions of why the participants answered as they did–Guidance with heuristics about ways to explicitly consider
interdependencies among related factors ... that might affect the basis of one’s best judgments under uncertain circumstances
9
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Example Open Source Software Questions
10
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Example Open Source Reference Points
11
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Example Open Source Reference Points
12
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
A Study of Accuracy versus Precision
Which would you rather have?•Someone whose recognized bounds of uncertainty include the correct
answer...•Someone who’s a little overconfident but is closer to being accurate...
13
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Relative Accuracy Improves
Generic Tests N=14
Domain Specific Tests n=29
Experiments confirm that expert judgment can be calibrated.
14
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Training Leads to Better Recognition of Uncertainty
Generic Tests
Domain Specific Tests
15
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Experts Improved with Training
Test 1: Inaccurate & imprecise
Test 2: Accurate & imprecise
Test 3: Accurate & Precise
16
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Cost Estimation Research
Previous calibration research
Current research in progress
Future research & applications
17
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
“Change Drivers” Explain Program Execution
Categories of unanticipated change events that often occur in MDAPs over the acquisition lifecycle:•Often a result of previous changes•Leading to subsequent changes•Or affecting program outcomes (which themselves can be drivers of further
change.•Status of MDAP activities that are proceeding as planned are not change
drivers.Intended use•To enable DoD domain specific expert judgment training
– Initially in QUELCE workshops•Other uses may be possible if we are successful in populating a larger DoD
domain-specific reference point repository, e.g.:– “Deep dives” earlier in pre-Milestone A–Program planning & risk analysis throughout the Acquisition lifecycle
18
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Domain Reference Points Aid Judgment
“There is a 90% probability that MDAPs with certain characteristics will experience off nominal change drivers A, B and C.”
“When change driver A goes off nominal, there is a 75% probability change driver B will go off nominal”
“When change drivers A, B, and C go off nominal, there is a 90% probability that change driver D will go off nominal.”
“When specific change drivers go off-nominal, specified impacts have occurred.”
“When specific change drivers go off-nominal, other change drivers are influenced or impacts felt within a certain amount of calendar time.”
19
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
A Reference Point Repository for DoD
Categorizing & tagging textual information about change events•From program documents such as SAR & DAES• Identify DoD domain specific reference points mapped to QUELCE change
drivers
Joining the tags & text excerpts with existing data•MDAP domain characteristics•Program performance outcomes, e.g., cost, schedule &scope of deliverables
Using the categories & text excerpts:•To assist judgments by QUELCE workshop teams based on experiences in
analogous programs•For use in individual expert calibration experiments & group resolution of
differences among team members
If we’re successful: Also used to support other activities•Both earlier in Milestone A & throughout the program lifecycle
20
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Cost Estimation Research
Previous calibration research
Current research in progress
Future research & applications
21
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
What’s Next for Expert Judgment Research?
A focus on DoD domain-specific questions & reference points
Seed a queryable reference point repository with DoD data
Shift our focus to experiments on resolution of differences among members of expert groups•Quantify benefit of access to domain reference points•Comparing algorithmic & group consensus methods with respect to accuracy,
recognition of uncertainty, & time required to resolve differences among team members
Upgrade our existing software support:•To capture individual judgments & eventually resolve differences without the
need for face-to-face meetings
22
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Leveraging the Delphi Planning Process
Given historical work•Wideband Delphi applied to cost estimation enabling discussion & a broader
communications channel to produce more accurate results (Boehm 1981)
•Recent research in software project estimation shows that estimates that benefit from group discussion tend to be more accurate (Cohn, 1997; Moløkken & Jørgensen, 2004).
We will research improved group decision-making judgment•Leverage expertise to forecast uncertainties related to costs and risks of
program execution
•Revisit conventional Delphi discouragement of discussion between rounds, introducing discussion of domain reference points
23
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Additional Considerations in Judgment Experiments
Heuristics such as anchoring & adjustment, overconfidence, blind spot bias, and others commonly bias individual experts’ judgments
•An individual estimator may first make a “best estimate” of duration for a program element ... then adjust it to form long-duration and short-duration estimates giving a range of likely outcomes
•Such adjustments are commonly known to be too small (Fischhoff, 1994)
•Resulting in too-tight range estimates & hugely over-frequent 1% and 5% tail occurrences
•However, explicit prompting of the estimator’s imagination can substantially reduce this tightness (Connolly & Deane, 1997)
24
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Summary
The eventual target is to apply these & related group reconciliation methods in our current research on QUELCE•QUELCE works by codifying expert judgment for cost estimates prior to
Milestone A (Ferguson et al., 2011 )•However improving group decision making is equally important for program
planning and risk analysis throughout the lifecycle.
We will validate & enhance our previous research on calibrating individual judgment (Goldenson & Stoddard, 2013) by: •Developing DoD domain-specific questions for a series of test batteries &
associated training exercises• Investigating the value of DoD domain-specific reference points that provide
more detailed contextual background about programs analogous to the programs being considered in calibration test questions
We welcome collaborators for the expert judgment experiments!
25
Improving Group Decision Making Under Uncertain CircumstancesCOCOMO 2013, 23 October© 2013 Carnegie Mellon University
Contact Information
Dennis R. GoldensonSenior Member of the Technical StaffSoftware Engineering InstituteTelephone: +1 412-268-8506Email: dg@sei.cmu.edu
U.S. MailSoftware Engineering InstituteCustomer Relations4500 Fifth AvenuePittsburgh, PA 15213-2612USA
Webwww.sei.cmu.eduwww.sei.cmu.edu/contact.cfm
Customer RelationsEmail: info@sei.cmu.eduTelephone: +1 412-268-5800SEI Phone: +1 412-268-5800SEI Fax: +1 412-268-6257
Recommended