WE 0201 – PKIT2Date : November 2004Slide : 1 / 74 WEAG – THALES – JP 11.20 REVVAREVVA

WE 0201 – PKIT2 Date : November 2004 Slide : 1 / 74

WEAG – THALES – JP 11.20WEAG – THALES – JP 11.20WEAG – THALES – JP 11.20WEAG – THALES – JP 11.20

REVVAREVVAREVVAREVVA


Presentation Outline

Project facts ; the consortium

Background and rationale

The REVVA concepts

The REVVA generic process

The REVVA products

Techniques and support

REVVA technical achievements

REVVA future


Project Facts

Purpose:The THALES JP 11.20 is a Western European Union / WEAG research program funded by 5 nations: France (lead nation), Denmark, Italy, the Netherlands, Sweden.

Objectives:To develop a basis for a common methodological framework for the Verification, Validation and Accreditation (VV&A) of data, models and simulations. To be a starting point for a draft standard defining VV&A taxonomy, process and methodology.

Project timeline:March 2003 – September 2004

Common Validation, Verification and Accreditation Framework for Simulation (REVVA)


The consortium

The THALES JP 11.20 is conducted by a multi-national consortium led by ONERA (France).

UNI-C (DK)

TNO (NL)

FOI – FMV(SW)

DATAMAT (IT)

ONERA – ALEKTO(FR)


REVVA background and rationale


REVVA Background (1)

JP 11.20REVVA

ITOP

AFDRG

DMSO

Capitalization

ExchangeSISO

NATO

RTP 11.13

Exchange

CapitalizationInformation


REVVA Background (2)

• VV&A state-of-the-art:– DMSO, SISO, NATO, ITOP.

• M&S scientific contributions:– Systems Theory (B. Zeigler)

• Contributions on the concept of validation coming from the epistemology of simulation


VV&A rationale (1)

Doing (only) what makes sense for customers

and users in their search for trust.

• Trust:– identification of trust components,

– support (based on project knowledge, consolidation by shared domain knowledge)


VV&A rationale (2)

Trust

M&S supplier skillsand reputation

Audit trail of experiments and uses

Formal analysis ofproducts, processes andorganisation

VV&A

Informal

Formal


VV&A rationale (3)

Processes

Organisation ProductsNeedsPurpose

FormalAnalysis

Acceptance

Validation

Verification


VV&A rationale (4)

Activities driven and prioritised by:

• Requirements (acceptance requirements)

• Identification / perception of risks

• Organisational needs, risks and benefits

• Costs


REVVA Concepts


REVVA Terminology (1)

Focus on Acceptance and Validation

• Decomposition of VV&A in Verification, Validation

and Acceptance; (temporal) independence of each

concept.– Verification: Model Verification.– Validation: Model Validation within an Experimental Frame (EF)

– Acceptance: Before use / After use



• Correctness:– The property of a simulation model to comply with

formal rules and bodies of reference information for its representation and for the transformation of its representation into another one.

• Verification:– The process which is used to construct, under a set of

time, cost, skills, and organisational constraints a justified belief about model correctness.



• Validity:– The property of a simulation model to have, within

a specific experimental frame, a behaviour which is indistinguishable under a set of validation criteria from the behaviour of the System of Interest.

• Validation:– The process which is used to construct, under a set

of time, cost, skills, and organisational constraints a justified belief about model validity.



• Acceptance:

– The informed decision of a M&S customer to use

the results of a simulation for a specific purpose.

– The informed decision of a M&S customer to use

a simulation, within an experimental frame, for a

specific purpose


REVVA Assumptions

• Acceptance (and Validation) should:

– be purpose oriented,

– be development neutral,

– be technology neutral,

– recognise the impact of organisations,

– recognise the impact of timeframe of M&S and VV&A,

– rely on data and knowledge management.


A comprehensive view

The"Real World"

ObjectivesNeeds

DecisionsActions

ProblemFormalisation

Problem solution

M&SQuestions

M&S Answers

SystemRequirements

M&S Enabling Products

The"Problem World"

The"M&S World"

The"M&S Product

World"


Methodological hints

• Independence

• Incremental nature of activities and results

• Focus on the intended purpose

• Objective assessment (Acceptance Criteria)

• Careful construction of a justified belief

• Tailoring


Tailoring

The EF used for Acceptance (and for Validation) is derived from the Acceptance Requirements All the activities are planed, designed and controlled by them. The Acceptance Requirements are not (necessarily) the M&S Requirements ; they are specific of each use and purpose

EF

M

Question

Answer


Impact of time

t| | |

Before M&S use1

BeforeM&S use2

AfterM&S use1

t1 t2 t3

Validation t1 Validation t2 Validation t3

• Verification is (mostly/hopefully) a one-shot process

• Validation is a permanent/recurrent issue

• Acceptance and Validation are purpose oriented and tailored


The REVVA Generic Process


Acquire Information

Develop ToVV

Conduct V&V

Assess Evidence

Assess Evidence

Integration

Evaluate V&V Report

DevelopToA

ToA

ToVV

V&V Items

Assessed items of evidence

V&V report

Acceptance Recommendation

Model information and system knowledge

Phase

Product

Intended purpose

Product flow

Back step


ToA: Target of Acceptance

ToVV: Target of Verification and Validation


Phase Overlaps

Develop ToA

Acquire Information

Develop ToVV

Conduct V&V

Assess Items of Evidence

Assess Evidence Integration

Evaluate V&V Report

VV&A Endeavour DurationMain effort


Develop ToA

• Performed by the Acceptance Agent in close cooperation with the user

• Develop Acceptability Criteria (AC), including Validity Criteria (VC)

• Document rationale for development


Intended Purpose

Objectives and Sub-Objectives(”Questions”)

(Fictive) Experimental Frames

Precision Requirements for the Behavior Characteristics

Develop ToA (2)


Intended Purpose

AC1

AC2

Acceptability Criterion m

Sub-Objective 1 Sub-Objective 2 Sub-Objective n

AC3

ToASO1.1 SO1.1 SO1.1


Acquire Information

• Led by the V&V Agent, performed by the V&V Team

• Get overview over the data, information, and knowledge (likely to become) available– about the Simulation Executable Model– about the System of Interest

• Create data, information, and knowledge repository

• Identify additional sources


Develop ToVV

• Performed by the V&V Agent, supported by the V&V Team

• Develop an approach to demonstrate that the Validation Criteria are passed or failed, with the data, information, and knowledge available

• Identify data, information, and knowledge desirable for more efficient V&V


Intended Purpose

AC1

AC2



AC3

ToA

ToVV

SO1.1 SO1.1 SO1.1

TaskTask

Task

Sub-Criterion m.1

SC m.1.1 SC m.1.2 SC m.1.3


Conduct V&V

• Led by the V&V Agent, performed by the V&V Team

• Choose techniques and tools

• Run the technique, e.g.,– Use 2-test to measure deviation– Use SME to assess structural decomposition

• Record technique, referrenced information, and result as V&V item

• Modify ToVV, if creation of V&V item fails


Assess Evidence

• Performed by the V&V Agent

• Review documented V&V items

• Assess the possibility of an erroneous result

• Accept V&V item as assessed item of evidence, or reject it

• Direct feedback loop


Assess Evidence Integration

• Performed by the Acceptance Agent

• Assemble items of evidence according to most

recent ToVV

• Derive which VC are passed or failed– Intuitively, subjectivly– Based on methods for dealing with uncertainty

• Assess possibility of misperception


Intended Purpose

AC1

AC2



AC3

ToA

ToVV

Item ofEvidence IoE IoE

SO1.1 SO1.1 SO1.1

TaskTask

Task

Sub-Criterion m.1

SC m.1.1 SC m.1.2 SC m.1.3


Evaluate V&V Report

• Performed by the Acceptance Agent

• Review expressiveness and completeness of AC with newly gained knowledge

• Weigh impact of misperception according to possibility of misperception

• Make acceptance recommendation


The REVVA Products


Acquire Information

Develop ToVV

Conduct V&V

Assess Evidence

Assess Evidence

Integration

Evaluate V&V Report

DevelopToA

ToA

ToVV

V&V Items


V&V report



Phase

Product

Intended purpose

Product flow

Back step



Target of Acceptance (ToA)

• Precise specification of the Acceptability Criteria (coming from validity related functional requirements)

• Stakeholder oriented• “What exactly needs to be assessed?”• Important elements in the ToA nodes:

requirement, MoE, constraint, level of impact, decomposition argument

Acquire Information

Develop ToVV

DevelopToA

ToA

Intended purpose


ToA (2)

• Building– Getting the requirements– Deriving MoE and constraints– Determining the level of impact– Validation of the ToA – Reuse

• Using– Part of contract– Basis for assessment report

• Maintaining– Due to changes in science, stakeholder changing mind,

corrections in ToA found during V&V work, V&V findings may indicate more limited validity range



• Information used to determine the V&V strategy for demonstrating the model’s correctness and validity

• Identifies all sources of information and knowledge

Acquire Information

Develop ToVV

Conduct V&V

Assess Evidence

Integration


• Important elements: – design documents, code,

modeller’s perception of the real world, behaviour data (of model and sub-models), conceptual model.

– Real world data (referent): data on (related or similar) systems, expert opinions


Target of Verification and Validation (ToVV)

• V&V items: evidence required to substantiate the ToA Acceptability Criteria

• When necessary: further decomposition of criteria (extension of ToA)

Develop ToVV

Conduct V&V

Assess Evidence

IntegrationToVV

• Important additional elements in the ToVV: method to use for collecting evidence, required rigor, cost estimate, aggregation instructions

• Tasks


V&V Items

• A V&V item consists of:– relevant information, – the evaluation objective, – reference information, – an evaluation technique, – evaluation result.

• Must be reproducible

Conduct V&V

Assess Evidence

V&V Items



• Assessed items of evidence:– V&V agent assessment of the

V&V result, – the acceptance or rejection of

each V&V item, – probative force (depending on

method or technique and reference information)

Assess Evidence

Assess Evidence

Integration



V&V report

• The assessed items of evidence assembled and integrated based on the ToVV

• Links the assessed items of evidence to the Acceptability Criteria

• Indicates if– evidence is considered sufficient to pass an

acceptability criterion

– V&V items for a criterion are missing or too weak,

– disproving evidence was found

Assess Evidence

Integration

Evaluate V&V Report

V&V report



• The final recommendation on acceptance• Based on the ToA• Residual uncertainty after V&V

– from probative forces,

– use level of impact to determine recommendation

• The recommendation states (with consequences for M&S product’s utility):– if required evidence is missing or too weak,

– if a weakness in the ToA was detected

Evaluate V&V Report



A VV&A project management view

The "3 pillars" concept

Process

Organisation ProductsPurpose

Objectives


The M&S and VV&A roles

• M&S stakeholders

• M&S suppliers

• VV&A agents

• Experts


Levels of independence

• DV&V:– dependent V&V; V&V performed by suppliers and

accepted as it is

• IA:– independent assessment; V&V performed by supplier

and assessed by the V&V agent

• IV&V:– independent V&V; V&V performed under the control

of the V&V agent


Feasibility assumptions

• High-level organisational decision

• Standards of V&V documentation

• Incremental accumulation of knowledge:– project vs corporate knowledge management

• Process for knowledge revision


The REVVA Techniquesand support


REVVA point of view

• Identification of the most important techniques and support

• Techniques to support decision making, …

• Next direction: develop techniques to support the whole process


Most important techniques and support

• Most importance of Requirements Engineering– Develop ToA & ToVV

• Design of Experiments (DOE) – Conduct V&V: calibration, sensitivity analysis & validation

• Computation of probative force and convincing force– Assess evidence – Assess argument

• Uncertainty analysis– Assess evidence– Evaluate V&V Report

• Decision tools– Evaluate V&V Report


Claim-Argument-Evidence

Types of argument

Evidence

Evidence

Evidence

Claim

Argument• Deterministic – relying upon axioms, logic and proof

• Probabilistic – relying upon probabilities, statistical analysis

• Qualitative – relying upon adherence to standards, design codes


Decision Tools

• The model is fit for purpose

• The model is not fit for purpose

Fact

Perception

Valid Not valid

Accepted OK Type II error

Rejected Type I error OK


Uncertainty Analysis

• Integrated approach for uncertainty estimation based on– Behaviour Characteristics– Propositional Logic– Hierarchical Breakdown Structures– Possibility Theory– Basic Model of Decision Theory

• Increases quantitative share in decision making• Leaves ”subjectivity gaps” with

– Probative Forces – Impressions of confirmation and denial


p1

p1’: p1.1 p1.2 p1.3

+(--)

p1.1 p1.2 p1.3

Composition and Inference

+ : inference relation (p1’ p1 : p1’ is a sufficient condition, but not a necessary one)

: composition relation (p1’ is a conjunction

of p1.1 through p1.3 )

(--) : quantification of denial impression (from (-) to (----))

(if p1’ is not verified, how strongly would it mean that p1 is not verified?)


Completeness and consistency checking

• The dependencies and consistency between the sponsor needs and requirements, any structured problem description, the model (conceptual, formal or executable), the ToA and ToVV and the final simulation results are checked

• Contents (evidences, specifications, etc.) without requirements and requirements without contents need to be resolved


REVVA technical achievements


REVVA Project Breakdown

The project includes six technical Work Packages:

WP 1 : Problem Analysis

WP 2 : Definition of AccreditationTarget

WP 3 : VV&A Techniques

WP 4 : Optimisation of Testing

WP 5 : Outline for a VV&A Methodological Framework

WP 6 : Evaluation of Results


Technical achievements (1)

• Taxonomy: Initial contribution dealing with V&V state of the art

• VV&A problem statement

• Review of acceptance criteria and levels of confidence

• Process models



• Emphasis on Acceptance Agent role: Target of Acceptance (ToA) and assessment of ToA satisfaction

• Integration of qualitative and quantitative confidence in a logical framework of argumentation

• Guidance for ToA and ToVV definition



• Review of existing techniques and tools resources

• V&V techniques state-of-the-art

• User guidance



• Capitalisation of data and knowledge

• Experience optimisation

• Relation with field trial data

• Identification of data/knowledge management issues


• Deep understanding of the terminology

• Definition of the VV&A scope

• Identification of VV&A organisations depending on level of impact

• Definition of VV&A products and processes



REVVA Outcomes

• Generic VV&A process, described by:

– Methodology Reference manual (METHGU)

– User’s Manual (PROSPEC)

• Illustration by case studies

• Draft contributions to guidance

• REVVA Website (www.JP1120-REVVA.com)


Lessons learnt at methodological level

• Compliant with the mainstream idea that every euro spent in requirements generates high Returns on Investment (ROI)

• The cost driver is the criticality of the use, not the quality of the process

• Generic, versatile methodology: impact on the easiness of tailoring

• Ability to comply with formal supports for guidance or automation of the process


Lessons learnt from the case studies

The REVVA application on Case Studies has shown capital points:

• To answer the right question:– It is important to formalise at early steps the problem to be

solved by M&S use AND the question asked to VV&A.

• To answer the question right:– It is enabled by the derivation of the ToVV from the ToA.

• To construct a justified belief:– The confidence is substantiated by gathering and assessing the

items of evidence.


Practical experience on case studies

• REVVA method helps to balance between purpose, model and data

• Difficult to gather the right information

• ToA seems to be particularly useful for large

projects


REVVA communications

• 04S-SIW-153 : A Common Validation, Verification and

Accreditation Framework for Simulations.

• 04S-SIW-005 : T&E Based on M&S VV&A:

The Importance of Experience Capitalization.

• 04S-SIW-152 : Assessment Criteria and V&V Levels in

JP11.20.

Three papers have been presented in a special session during the Spring '04 SIW


REVVA Future


Follow-up

REVVA is a first methodological step;the work will continue with

a EUROPA project (REVVA2) which is to start during Spring 05.


REVVA2: Conceptual challenges

• Organisation and roles clarification

• Incremental knowledge management (i.e. repository …)

• Formalisation of VV&A products

• Formalisation of the methodology

• Tailoring

• Costs model and project management


REVVA2: Technical challenges

• VV&A requirements formalisation

• V&V evidences : collection, organisation

• V&V argumentation: formalisation and support

• V&V and decision-making under uncertainty

• Formal foundations for V&V tools

• V&V tools selection and guidance (tailoring)

• V&V actual processes support


REVVA2: Current status

• Objective: to produce draft documents for standardisation.

• The project timeline will be 36 months, expected start in March 2005.

• EUROPA MoU• FR, DK, SW and NL intend to join REVVA2.• The UK may consider to join after CConVV&A.• The WBS are refined taking into account the roadmap

defined in Munich.

Documents

WE 0201 – PKIT2Date : November 2004Slide : 1 / 74 WEAG – THALES – JP 11.20 REVVAREVVA