Verification and Validation— An OSD Perspective —

Fred MyersDeputy Director, Test InfrastructureTest Resource Management Center

November 4, 2009

2

OutlineOutline

• Test Resources• Test Organizations within the DoD• Definitions• Acquisition Time-line for the DoD• Systems Engineering (SE)• T&E Life Cycle (TES/TEMP)• Relationship between SE and T&E• DoD T&E Best Practices• Summary

3

T&E Resources: A collective term that encompasses therequisite Workforce, Infrastructure and Funding resulting

in a T&E Capability, by means of the T&E Processes

T&E Resources: A collective term that encompasses therequisite Workforce, Infrastructure and Funding resulting

in a T&E Capability, by means of the T&E Processes

T&E ResourcesT&E Resources

T&ECapabilities

T&ET&EProcessesProcesses

T&ET&EWorkforceWorkforce

T&E InfrastructureT&E InfrastructureFacilitiesFacilitiesBuildingsBuildings

EquipmentEquipmentNetworksNetworks

Range SpaceRange Space::

FundingFunding

DMSDMS MFMF SILSIL HWILHWIL ISTFISTF OAROAR

4

Test Organizations within the DoDTest Organizations within the DoD

Secretary of DefenseSecretary of DefenseSecretary of DefenseSecretary of Defense

Under Secretary of DefenseUnder Secretary of DefenseAcquisition, Technology & LogisticsAcquisition, Technology & Logistics

Under Secretary of DefenseUnder Secretary of DefenseAcquisition, Technology & LogisticsAcquisition, Technology & Logistics

DirectorDirectorOperational Test & Operational Test &

EvaluationEvaluation• Certify weapon systems for:Certify weapon systems for:

– Operational Effectiveness Operational Effectiveness and Suitabilityand Suitability

– Live-fire System Live-fire System Vulnerability and LethalityVulnerability and Lethality

DirectorDirectorOperational Test & Operational Test &

EvaluationEvaluation• Certify weapon systems for:Certify weapon systems for:

– Operational Effectiveness Operational Effectiveness and Suitabilityand Suitability

– Live-fire System Live-fire System Vulnerability and LethalityVulnerability and Lethality

DirectorDirectorTest ResourceTest ResourceManagement Management

CenterCenter• DoD Strategic Plan DoD Strategic Plan

for T&E Resourcesfor T&E Resources• Certification of Certification of

Service / Agency Service / Agency T&E BudgetsT&E Budgets

• Oversight of T&E Oversight of T&E InfrastructureInfrastructure

DirectorDirectorTest ResourceTest ResourceManagement Management

CenterCenter• DoD Strategic Plan DoD Strategic Plan

for T&E Resourcesfor T&E Resources• Certification of Certification of

Service / Agency Service / Agency T&E BudgetsT&E Budgets

• Oversight of T&E Oversight of T&E InfrastructureInfrastructure

DirectorDirectorDefense ResearchDefense Researchand Engineeringand Engineering

DirectorDirectorDefense ResearchDefense Researchand Engineeringand Engineering

DirectorDirectorDevelopmental TestDevelopmental Test

and Evaluationand Evaluation• DT&E PolicyDT&E Policy• Acquisition T&E Career Acquisition T&E Career

WorkforceWorkforce• DT&E Program DT&E Program

OversightOversight

DirectorDirectorDevelopmental TestDevelopmental Test

and Evaluationand Evaluation• DT&E PolicyDT&E Policy• Acquisition T&E Career Acquisition T&E Career

WorkforceWorkforce• DT&E Program DT&E Program

OversightOversight

5

DefinitionsDefinitionsSystems Engineering is defined by ANSI/EIA-632-1999, “Processes for Engineering a System” as "An interdisciplinary approach encompassing the entire technical effort to evolve and verify an integrated and total life cycle balanced set of system, people, and process solutions that satisfy customer needs" Integrated Testing is defined by OSD Memo, “Definition of Integrated Testing,” dated 25 April 2008 as “The collaborative planning and collaborative execution of test phases and events to provide shared data in support of independent analysis, evaluation, and reporting by all stakeholders, particularly the developmental (both contractor and government) and operational test and evaluation communities” Verification is defined by DoD Directive 5000.59, "DoD Modeling and Simulation (M&S) Management" as "The process of determining that a model implementation and its associated data accurately represents the developer's conceptual description and specifications"Validation is also defined by DoDD Directive 5000.59 as "The process of determining the degree to which a model and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model"

Verify: To establish the truth, accuracy, and reality of a system's performanceValidate: To corroborate on a sound or authoritative bases--confirm Verify: To establish the truth, accuracy, and reality of a system's performanceValidate: To corroborate on a sound or authoritative bases--confirm

6

Operations & SupportOperations & SupportProduction &Production &DeploymentDeployment

FRPFRPDecisionDecisionReviewReview

LRIP / IOT&ELRIP / IOT&ELRIP / IOT&ELRIP / IOT&E

Engineering &Engineering &ManufacturingManufacturingDevelopmentDevelopment

PostPostPostPostPDRPDRCDRCDRAA AA

TechnologyTechnologyDevelopmentDevelopment

MaterialMaterialSolutionSolutionAnalysisAnalysis

MaterialMaterialDevelopmentDevelopmentDecisionDecision

Full OperationalFull OperationalCapability (FOC)Capability (FOC)

Initial OperationalInitial OperationalCapability (IOC)Capability (IOC)

CCCCAAAA BBBB

Pre-Systems AcquisitionPre-Systems AcquisitionPre-Systems AcquisitionPre-Systems Acquisition Systems AcquisitionSystems AcquisitionSystems AcquisitionSystems Acquisition SustainmentSustainmentSustainmentSustainment

= Decision Point = Milestone Review = Decision Point if PDR is not conducted before Milestone B

(ProgramInitiation)

User NeedsUser NeedsUser NeedsUser Needs

Technology Opportunities & ResourcesTechnology Opportunities & ResourcesTechnology Opportunities & ResourcesTechnology Opportunities & Resources

• The Material Developer Decision precedesentry into any phase of the acquisition Management System

• Entrance Criteria met before entering phase

• Evolutionary Acquisition or Single Step toFull Capability

Acquisition Time-line for the DoD

Acquisition Time-line for the DoD

PDR: Preliminary Design ReviewCDR: Critical Design ReviewLRIP: Low-Rate Initial ProductionFPR: Full-Rate Production

PDR: Preliminary Design ReviewCDR: Critical Design ReviewLRIP: Low-Rate Initial ProductionFPR: Full-Rate Production

7

Scope: Enable systems engineering across acquisition life cycle

Systems Engineering (SE)Systems Engineering (SE)

User Requirements& Concept

of Operations

User Requirements& Concept

of Operations

SystemRequirements &

Architecture

SystemRequirements &

Architecture

ComponentDesign

ComponentDesign

ComponentIntegration &

Test

ComponentIntegration &

Test

SystemIntegration &

Test

SystemIntegration &

Test

SystemDemonstration &

Validation

SystemDemonstration &

Validation

Procure, Fabricate,& Assemble Parts Procure, Fabricate,& Assemble Parts

Design

Feedback

RecursiveProcesses

RecursiveProcesses

Operations & SupportOperations & SupportProduction &Production &DeploymentDeployment

FRPFRPDecisionDecisionReviewReview

LRIP / IOT&ELRIP / IOT&ELRIP / IOT&ELRIP / IOT&E

Engineering &Engineering &ManufacturingManufacturingDevelopmentDevelopment

PostPostPostPostPDRPDRCDRCDRAA AA

TechnologyTechnologyDevelopmentDevelopment

MaterialMaterialSolutionSolutionAnalysisAnalysis

MaterialMaterialDevelopmentDevelopmentDecisionDecision

FOCFOCIOCIOCCCCCAAAA BBBB

"V&V Occurs throughout the SE Process""V&V Occurs throughout the SE Process"

8

Operations & SupportOperations & SupportProduction &Production &DeploymentDeployment

FRPFRPDecisionDecisionReviewReview

LRIP / IOT&ELRIP / IOT&ELRIP / IOT&ELRIP / IOT&E

Engineering &Engineering &ManufacturingManufacturingDevelopmentDevelopment

PostPostPostPostPDRPDRCDRCDRAA AA

T&E Life Cycle (TES/TEMP)T&E Life Cycle (TES/TEMP)

Early T&E involvement in technical planning is criticalEarly T&E involvement in technical planning is critical

TechnologyTechnologyDevelopmentDevelopment

MaterialMaterialSolutionSolutionAnalysisAnalysis

MaterialMaterialDevelopmentDevelopmentDecisionDecision

FOCFOCIOCIOCCCCCAAAA BBBB

9

Relationship between SE and T&E(Defense Acquisition Guidebook)

Relationship between SE and T&E(Defense Acquisition Guidebook)

• Systems engineering is a process to transform required operational and sustainment capabilities into a system design solution

• As the design solution evolves, a verification component of the systems engineering process must provide confidence that the design solution properly addresses the desired capabilities, as intended

• T&E is the mechanism for accomplishing the verification loop in the systems engineering process and characterizing technical risk of achieving a proper final design solution

10

DoD T&E Best PracticesDoD T&E Best Practices

• Recognize T&E value: – Use T&E to understand risk and identify problems early– Provides V&V for system design and development

• Embed T&E in Acquisition Strategy: – T&E Strategy must be part of the acquisition strategy

• Develop a sound T&E strategy and plan:– Ensure both contain realistic schedules and adequate

resources

• Employ event-driven T&E planning and execution:– Use event-driven T&E plan to reduce technical risk– Do not sacrifice testing to meet schedules

11

DoD T&E Best Practices (Cont.)DoD T&E Best Practices (Cont.)

• Ensure early involvement by all stakeholders in T&E planning and execution

• Integrate T&E in contracts:– Develop T&E strategy along with RFP development– Ensure key requirements can be tested and evaluated– Use T&E as a source selection criteria– Ensure Government has access to all test data– Ensure Government can conduct an independent

evaluation and assessment

• Focused early testing — Identify biggest risks and test them first

• "Build-a-Little — Test-a-Little"• Evaluation is the key to successful testing

– Provide feedback to the system designer and developer

12

SummarySummary

• T&E can reduce cost, schedule and technical risk

• T&E results figure prominently in the decisions reached at design and milestone reviews

• The fact that T&E results are required does not presuppose the results will be favorable

• Only the decision-makers can determine the weight and importance of a system's diverse capabilities and shortcomings — the degree of risk they are willing to accept

• Good T&E information enables informed decisions