Embed Size (px)
Citation preview
VV&A ofDIME/PMESII TOOLS
Dean Hartley
January 21, 2008
Building and Using a Model
Real WorldProxy for orPerception ofReal World
Data
Conceptual Model Coded
ModelCoded Model
Hard Data
SoftData
CMCreation
Model Creation
Data Use
Risk 1
Risk 2
Risk 3
Risk 4 Risk 5
Risk 6
ModelUse
RiskRisk 77
Most PMESII valuesare not observable
Verifying and Validating a Model
Real WorldProxy for orPerception ofReal World
Data
Conceptual Model Coded
ModelCoded Model
Model Verification
CMValidation
Data Use
Data V&V
DataVerification Data
V&VModelValidation
Theory, Data & SMEValidation
Changed-Tool Processes*
Entrenched VV&A
Standard Model-Creation ProcessesDesign Walkthroughs, Code Review, Debugging, etc.
Standard Model-Acceptance ProcessesAlpha testing, Beta testing, Acceptance testing, etc.
Multi-model-System Processes*Developm
ental testing
Periodic Supplemental TestsTests with different data sets, Tests of different functionality
Periodic testing
New-Tool Processes*
Changed-Environment Processes*
* e.g., Interface testing, Special input testing, Full system testing
Formal Process
Accreditat
ionT
riggered testing
U
DB
T U
DB
T U
DB
TUDB
T UDB
T UDB
TUDB
T
Entrenched VV&A as Part of the Lifecycle
Understand the Requirements
Design the System
Build the System
Test and Evaluate
P P PP
D
T
T TT T
D
Use the System
A A
Accreditation=A
Triggered testing
=T
Periodic testing
=PDevelopmental
testing=D
Legend
P
A A A A A A
Simplified Entrenched VV&A Process Flow
Initiate VV&A Process
Define D Tests
Execute D Tests
Evaluate D Tests
Support Accreditation
Define P Tests
Execute P Tests
Evaluate P Tests
Define T Tests
Execute T Tests
Evaluate T Tests
Type of V&V
Development TriggeredPeriodic
Code Changes Time Passes Trigger Event
M&S Use
Use the Model
Evaluate Use
Final D Test?YesNo
Manage Residual Risk
Revisit Initiation Process
Verification Metrics ExampleDate 06/01/06 11/01/06 12/01/06 07/01/07 11/01/07 12/01/07Event Alpha Test Beta Test Final Test Alpha Test Beta Test Final TestVersion Number 1 1 1 2 2 2Successful Tests 81 106 109 99 116 119Unresolved Problems 6 3 0 3 3 0Total Tests 87 109 109 102 119 119New Tests 87 68 0 65 79 0Repeated Tests 0 41 109 37 40 119
System TestsSuccessful Tests 20 25 25 25 30 31Unresolved Problems 1 0 0 2 1 0Total Tests 21 25 25 27 31 31New Tests 21 10 0 17 19 0Repeated Tests 15 25 10 12 31
Name 1Version Number 2 2 2 3 3 3Successful Tests 30 40 42 35 41 42Unresolved Problems 3 2 0 1 1 0Total Tests 33 42 42 36 42 42New Tests 33 34 0 24 29 0Repeated Tests 8 42 12 13 42
Name 2Version Number 3.1 3.1 3.1 3.7 3.7 3.7Successful Tests 31 41 42 39 45 46Unresolved Problems 2 1 0 0 1 0Total Tests 33 42 42 39 46 46New Tests 33 24 24 31 0Repeated Tests 18 42 15 15 46
Validation Metrics for Conceptual Models• Labels for individual model sub-components and for entire ensemble of models• “5” label meets the most stringent standards, “4” next most stringent, etc.• In general, the label for an ensemble will be lower than the labels for its components• Expectations for PMESII components and ensembles shaded in yellow
Label Component Ensemble
5
Expresses fully validated theory, e.g., Newtonian physics with caveats on operations near light speed or in regimes subject to quantum effects
Expresses multiple Level 5 theories with fully engineered interfaces, e.g., fly-out model of ground-to-air rocket, involving, chemical reactions of propellants, ballistics, air flow, electronics, etc.
4Expresses well researched theory involving considerable data checking and peer review, e.g., economic theory earning Nobel prize
Expresses multiple Level 4 or Level 5 theories with well researched interfaces, e.g., economic model ensemble used by Federal Reserve in setting U.S. interest and discount rates
3Expresses theory supported by data and published in peer-reviewed literature
Expresses multiple Level 3, 4 or 5 theories with considerable peer-reviewed interfaces, e.g., some U.S. combat models
2Expresses theory with rational basis, accepted by some experts as plausible (SWAG)
Expresses multiple Level 2 - 5 theories with plausible interfaces (SWAG)
1 Expresses a codified theory (WAG)Expresses multiple Level 1 - 5 theories with codified interfaces (WAG)
0Uncodified, mental model of uncertain consistency and completeness
Uncodified processes for connecting models of uncertain consistency and completeness
Coded Model Validation Preliminaries
Name 2 Use A
PMESII DIME PMESII Variables and DIME Functions
Relevant for this model? (0=no, 1=yes)
Theory (1=WAG, 2=SWAG, 3=Peer Rvw, 4=Accepted, 5=Proved)
Model Raw Score
Suitability for Use Fraction from Testing (granularity match, model match, right direction of change, right order of magnitude change)
Model Use Score
Political-Gov DIME Conducting nationwide elections 1 1 0.20 0.90 0.18Political-Gov DIME Create local governments 1 1 0.20 1.00 0.20Political-Gov DIME Educate local governments 1 1 0.20 0.90 0.18Political-Gov DIME Establishing a mechanism for constitutional reform 1 1 0.20 0.90 0.18Political-Gov DIME Establishing, staffing & funding effective transition national govt 1 1 0.20 0.90 0.18Political-Gov DIME Providing advisors to national govt officials 1 1 0.20 1.00 0.20Political-Gov DIME Supply local governments 1 1 0.20 0.90 0.18Political-Gov DIME Training newly elected national political leaders 1 1 0.20 0.90 0.18Political-Gov Central authority is effective 1 1 0.20 0.90 0.18Political-Gov Central government exists 1 1 0.20 0.95 0.19Political-Gov Social services are adequate 1 1 0.20 0.90 0.18
Sums 11 2.20 2.03Scaled Scores 1.00 0.92
Political-Pol DIME Maintaining compliance with peace accord milestones & conditions 1 1 0.20 0.90 0.18Political-Pol DIME Mediating & negotiating w/ conflicting parties 1 1 0.20 0.90 0.18Political-Pol DIME Monitoring government powersharing arrangements 1 1 0.20 0.90 0.18Political-Pol DIME Transferring control of government functions to host nation officials 1 1 0.20 1.00 0.20Political-Pol Opposition party does not espouse force 1 1 0.20 0.90 0.18Political-Pol Opposition party doesn't attempt to dominate by force 1 1 0.20 1.00 0.20Political-Pol There are charismatic leaders advocating peace and stability 1 1 0.20 0.90 0.18Political-Pol There are no charismatic leaders advocating dissension 1 1 0.20 0.90 0.18Political-Pol There are no external forces advocating conflict 1 1 0.20 0.90 0.18
Sums 9 1.80 1.66Scaled Scores 1.00 0.92
Political-ROL DIME (Re)building & monitoring new police force 0 3 0.00 0.00 0.00Political-ROL DIME Assisting in establishing humane penal systems 0 1 0.00 0.00 0.00Political-ROL DIME Assisting in establishing/reforming legitimate legal system 0 1 0.00 0.00 0.00Political-ROL DIME Conducting constabulary operations 0 1 0.00 0.00 0.00Political-ROL DIME Conducting war crimes investigations, tribunals, etc. 0 1 0.00 0.00 0.00Political-ROL DIME Monitoring and reporting on corruption by govt officials 1 1 0.20 1.00 0.20Political-ROL DIME Monitoring human rights practices 1 1 0.20 0.90 0.18
Coded Model Validation DisplaySystem Validation Metrics
0.00
1.00
2.00
3.00
4.00
5.00Political
Military
Economic
Social
InformationInfrastructure
DIME
User Issues
Connect
PMESII System
Model Validation Metrics
0.00
1.00
2.00
3.00
4.00
5.00Political
Military
Economic
Social
Information
Infrastructure
DIME
User Issues
Name 1
Model Validation Metrics
0.00
1.00
2.00
3.00
4.00
5.00Political
Military
Economic
Social
Information
Infrastructure
DIME
User Issues
Name 2
Inter-Model Connections
0.00
1.00
2.00
3.00
4.00
5.00Model 1-2
Model 1-3
Model 1-4
Model 1-5
Model 1-6
Model 2-3
Model 2-4Model 2-5Model 2-6
Model 3-4
Model 3-5
Model 3-6
Model 4-5
Model 4-6
Model 5-6
PMESII System
Experimental Design
Create Experimental
Design
DIME/PMESII System DIME/PMESII System
SMEInput
Outputs
Outputs
Exploratory Space Analysis
Risk Analysis
Save SelectInput Sets andOutput Sets
Filter to select presetDataTests
Time
Past Now
Modified
Embedded Results-Traceback System
Internal, Time-Coded
Data
Automated Results-Traceback Program
Extract data duringmodel execution to support later tracebackof the results
Time
Now
DIME/PMESII SystemModified
Embedded Instruments
Model Comparison
SME
Calibrate DIME/PMESII System
Time
Past Now
DIME/PMESII SystemModified
DIME/PMESII SystemModified
Prior Situation
Current Situation
Future Situation
Future
Validation?
DataExtraction
Extrapolation
Yes
NoOutput Comparison
ISSM
Backup
Dean Hartley
January 21, 2008
CM Risks and Mitigations (1 of 2)
Intended use CONOPS maturity
User experience SME availability
SME quality Bad input data Mis-interpreting results
Wrong format Content nature
Level-5 Medium M: Train users
Medium M: Train users
Minor Minor Minor Major M: V&V data
Major M: Train users; Use peer review
Medium M: Train users
Minor
Level-4 Medium M: Train users
Medium M: Train users
Medium M: Train users
Medium M: Train users
Medium M: Train users
Major M: V&V data
Major M: Train users; Use peer review
Medium M: Train users
Minor
Level-3 Major M: Enforce caveats
Major M: Train users; enforce caveats
Medium M: Train users
Medium M: Train users
Medium M: Train users
Medium M: Use parametric data exploration
Medium M: Use peer review
Medium M: Train users
Medium M: Use parametric data exploration
Level-2 Major M: Enforce caveats
Major M: Train users; enforce caveats
Major M: Train users
Major M: Train users
Major M: Train SMEs; train users
Medium M: Use parametric data exploration
Medium M: Use peer review
Medium M: Train users
Major M: Use parametric data exploration
Level-1 Major M: Enforce caveats
Major M: Train users; enforce caveats
Major M: Train users
Major M: Train users
Major M: Train SMEs; train users
Medium M: Use parametric data exploration
Medium M: Use peer review
Medium M: Train users
Major M: Use parametric data exploration; use peer review
Risk
CMVal Lev
Methodology Intellectual Capital DataParticular Data General Data
CM Risks and Mitigations (2 of 2)
Model is biased Model is incomplete
Model is wrong Interactions are not traceable
Interaction is biased
Interactions are incomplete
Interaction is wrong
Maturity of component
Maturity of composed M&S
Prior V&V Configuration management
Level-5 Minor Minor Minor Minor Minor Minor Minor Minor Minor Minor Medium M: Institute Conf Mgmt
Level-4 Minor Medium M: Add SME opinions
Minor Minor Minor Medium M: Add SME opinions
Minor Minor Minor Medium M: Institute V&V
Medium M: Institute Conf Mgmt
Level-3 Medium M: Use parametric data exploration
Major M: Add SME opinions
Medium M: All to the left
Medium M: Use parametric data exploration
Medium M: Use parametric data exploration
Major M: Add SME opinions
Medium M: All to the left
Medium M: All to the left
Medium M: All to the left
Medium M: Institute V&V
Major M: Enforce Rigorous Conf Mgmt
Level-2 Major M: Use parametric data exploration; try weighting results
Major M: Add SME opinions
Major M: All to the left
Major M: Use parametric data exploration
Major M: Use parametric data exploration; try weighting results
Major M: Add SME opinions
Major M: All to the left
Major M: All to the left
Major M: All to the left
Major M: Enforce entrenched V&V
Major M: Enforce Rigorous Conf Mgmt
Level-1 Major M: Use parametric data exploration; try weighting results; use peer review
Major M: Add SME opinions
Major M: All to the left
Major M: Use parametric data exploration
Major M: Use parametric data exploration; try weighting results ; use peer review
Major M: Add SME opinions
Major M: All to the left
Major M: All to the left
Major M: All to the left
Major M: Enforce entrenched V&V
Major M: Enforce Rigorous Conf Mgmt
Risk
CMVal Lev
General ToolTool
Individual Model Model Interactions
Level Validation Criteria
Referent Conceptual Model
Development Products
Simulation Results
Documentation Consistency
Testing Standards
Configuration Management
Standards
0none derived none chosen none constructed verified enough to
support development
not validated at all little or no documentation
no standard practices
no standard practices
1represented by SME opinion
represented by SME opinion
none constructed verified enough to support development
validated by SME observing simulation results
little or no documentation
no standard practices
no standard practices
2
defined from user statements as entities represented, their attributes & the dependencies among them
represented solely by SME opinion, based on intended use of the model/system
validated against the validation criteria by the SME
verified against the conceptual model inventory
validated by SME against the validation criteria
standard documentation produced, but limited in scope
standard practices with developer performing the V&V tests
manual configuration management, but not standardized
3
defined from user statements as Level 2 terms plus attribute ranges, domains & errors
derived from a single source, based on intended use of the model/system
validated by objective party from validation criteria & referent
verified against the conceptual model
validated by objective party from validation criteria & referent
standard documentation produced, with nearly complete scope
standard practices with someone other than the developer performing the V&V tests at acceptance or accreditation time
standardized manual configuration management
4
defined from user statements as Level 3 terms plus confidences
sampled from multiple independent sources, based on intended use of the model/system, & correlated statistically with estimates of uncertainties
validated by objective party from validation criteria & referent; analyzed to suggest results sampling space & estimate the confidence associated with the sampling
verified against the conceptual model; provides information to guide results sampling & estimate the confidence associated with that sampling
sampled from guidance developed from CM & verification analyses; validated by objective party from validation criteria & referent
standard documentation produced, with complete scope and database
standard practices with a disinterested party performing the V&V tests at acceptance or accreditation time
standard computer-based configuration management
5
Level 4 terms formally derived from user statements using causality arguments
rigorously derived from multiple independent sources, based on intended use of the model/system, & characterized statistically with estimates of uncertainties
formally stated & validated automatically from validation criteria & referent: analyzed to define results validation sample space
verified against conceptualo model & used to define results validation sample space & the confidence associated with that sampling
automatically sampled from guidance developed from CM & verification analyses; validated automatically from validation criteria & referent
standard documentation produced, with complete scope and database and complete retrieval system
standard practices with a disinterested party performing the V&V tests at acceptance or accreditation time
standard computer-based configuration management with check-out, check-in safeguards
VV&A Process Maturity Metric
Probable Current Level
Probable Achievable Level
Red font indicates additions to the Youngblood VPMM
