Upload
heller
View
130
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Integrating Developmental and Operational Testing. Dr. William (Dave) Bell 2 December 2009. Agenda/Overview. What/Why Integrated Testing? When Should We Do It? What Can/Should Testers Do? Summary. Recent DoD Integrated Testing Policy . - PowerPoint PPT Presentation
Citation preview
© 2009 The MITRE Corporation. All rights reserved.
Integrating Developmental andOperational Testing
Dr. William (Dave) Bell2 December 2009
© 2009 The MITRE Corporation. All rights reserved.
Agenda/Overview
■What/Why Integrated Testing?■When Should We Do It?■What Can/Should Testers Do?■Summary
2
© 2009 The MITRE Corporation. All rights reserved.
Recent DoD Integrated Testing Policy
3
Integrated testing is the collaborative planning and collaborative execution
of test phases and events to provide shared data in support of independent
analysis, evaluation and reporting by all stakeholders particularly the
development (both contractor and government) and operational test and
evaluation communities
T&E expertise must be brought to bear at the beginning
of the system life cycle to provide earlier learning about
the strengths and weaknesses of the system under development
© 2009 The MITRE Corporation. All rights reserved.
Integrated Testing Defined
4
■ Defense Acquisition Guidebook, Chapter 9– Whenever feasible, DT&E and OT&E events should be combined, if that supports technical
and operational test objectives to gain the optimum amount of testing benefit for reasonable cost and time. The user community should be involved early in test planning to ensure the statement of desired capabilities is interpreted correctly and tested realistically. Certain events can be organized to provide information useful to developmental and operational evaluators and lend themselves to the DT and OT approach. The concept is to conduct a single combined test program that produces credible, qualitative and quantitative information that can be used to address developmental and operational issues. Examples of this approach include: combined DT/OT events or piggybacking an operational assessment onto a developmental test. Likewise, developmental testing data requirements can be accommodated by an operational test. This approach can reduce the time and expense of conducting dedicated OT events that replicate DT events, or vice versa, yet still provide adequate technical risk reduction. The developmental and operational testers can develop a test management structure to share control of the combined events. Combined DT and OT events and test data requirements must be identified early to prevent unnecessary duplication and to control costs. It is important that neither the DT&E nor OT&E objectives are compromised in designing combined events.
Combined DT & OT eventssupport technical and operational objectives
User community involved early to ensure desired capabilities
interpreted correctly and tested realistically
© 2009 The MITRE Corporation. All rights reserved.
DT/OT to IDT/OT Evolution - Past
5
Contractors
Responsible Test Organizations
Find & fix technical problems
Operational TestAgencies
Director, OT&E
Discover ops issues& confirmation
Developmental Test
Operational Test
Certification
© 2009 The MITRE Corporation. All rights reserved.
IntegratedTest
DT/OT to IDT/OT Evolution - Present
6
Developmental Test
Operational Test
Certification
OTAs
DOT&E
Finish discovery &
perform confirmation
Ktrs
RTOs
Find & fix technical problems
KtrsRTOsOTAs
DOT&ECertification
Discoveryof ops/tech
issues
Ktrs ContractorsRTO Responsible Test OrganizationOTA Operational Test Agency
© 2009 The MITRE Corporation. All rights reserved.
DT/OT to IDT/OT Evolution - Future
7
Developmental Test
Operational Test
IntegratedTest
OTAs/DOT&ETitle 10 confirmation
Final certifications
KtrsRTOsOTAs
DOT&EJITC
Develop/refinerequirements
Discover/fixops/techissues
Contractors/RTOs
Early technologydemo & selection
WIN - WIN: Capability in warfighters’ hands ASAP
Reduces cost, schedule, and risks
© 2009 The MITRE Corporation. All rights reserved.
Agenda/Overview
■What/Why Integrated Testing?■When Should We Do It?■What Can/Should Testers Do?■Summary
8
© 2009 The MITRE Corporation. All rights reserved.
Testers Informing Early Decision Making…
■ Requirements/specification impact– Disciplined testing
mentality helps discovery process; prior to large funding commitment
9
© 2009 The MITRE Corporation. All rights reserved.
Integrated Testing - Part of Systems Engineering
© 2009 The MITRE Corporation. All rights reserved.
12
ComponentTesting
• Unit testing• CI testing
Stand-Alone SystemTesting
• CI-to-CI testing
System-of-SystemsTesting
• Field testing• Exercise
Integrated SystemTesting
• I&I testing
FQT I&ICert
OTCert
Contractor DT
Government DT
Government OT
Accreditation andCertification Testing
Accreditation andCertification Testing
Integrated Testing Environment
• ••
Integrated Testing Execution Period
© 2009 The MITRE Corporation. All rights reserved.
Agenda/Overview
■What/Why Integrated Testing?■When Should We Do It?■What Can/Should Testers Do?■Summary
13
© 2009 The MITRE Corporation. All rights reserved.
14
During Requirements Development
■ Early T&E influence during requirement development :– Ensures requirements are testable and unambiguous
■ Can you readily imagine a verification technique for this requirement?■ Your requirement may be ambiguous if interpretation is required to
design and build – Ensures that the test measures address/answer the Critical
Operational Issues– Requirements that are high risk/push the state-of-the-art are
identified/addressed
Getting the requirements right is a pothole-filled road
© 2009 The MITRE Corporation. All rights reserved.
T&E During SOW, RFP, and Contract Development
■ T&Eers ensure that test-related scheduling, budgeting, infrastructure, other resources and personnel are included in program’s contractual documents– SOW/RFP, proposal evaluation, contract
■ T&Eers help answer the question, “How much and what type of data need to be collected?” – To inform early decisions– Assure continuing involvement
■ Test design– Test events /Operational/Mission scenarios– Distributed/Joint/LVC– Statistical sample size determination– Analysis planning
15
© 2009 The MITRE Corporation. All rights reserved.
Integrated Test Execution
■ Early, continuous IT serves both DT and OT objectives – Provides adequate test time ensuring all test objectives are met – Allows for better/more thorough data collection– Allows PM’s an early look at and time to fix operational issues
■ Provides opportunity for assessment of reliability growth– Maintenance activities on early test units yield valuable
information■ Initial maintainability assessments■ Evolve maintenance approach or design at this relatively
inexpensive stage
16
Earlier insight = cheaper & easier fixes
© 2009 The MITRE Corporation. All rights reserved.
17
Opportunities for Integrated Testing■ Integrated DT/OT supported in SOW and RFP■ Integrated DT/OT supported in Test Strategy and Planning documents■ Requirements unambiguous and testable■ DT&E and OT&E personnel working together from Program’s outset ■ Contractor and Government DT Data augments OT&E data■ OT&E personnel influence Operational Realism of DT&E scenarios■ Test Planning Working Groups develop common data collection and reporting formats ■ The DT&E System Under Test (SUT) reflects Operational System and is operated by typical/actual users■ OT&E personnel participate in DT Testing and/or receive test documentation■ OT&E personnel assist Program Managers to develop “Acceptable Work-arounds” mitigating serious deficiencies
© 2009 The MITRE Corporation. All rights reserved.
Recommendations for Integrating DT/OT
1. Build a culture of Mutual Trust and cooperation ■ Demonstrate to PMs OT is not simply “Pass or Fail” ■ Assist PMs in lowering overall program risk by providing
SE and T&E involvement early in the development process2. Develop a “Partnership” not an adversarial relationship
between the Test Community and Contractors or PMs■ Identify ways Integrated DT/OT is of mutual benefit to
PM’s, Contractors and OTA’s■ Demonstrate how early detection of deficiencies reduces
overall risk and cost ■ Show how schedule test time can be reduced by
integrating DT/OT■ Assist PM’s in developing and presenting an acceptable
case for correcting deficiencies to the MDA
© 2009 The MITRE Corporation. All rights reserved.
3. Encourage adoption of Common data Format/Standards for all Joint and Distributed■ Emphasize need for ability to exchange information in the
TEMP as part of Integrated testing■ Include requirement for common data standards/formats in all
SE and T&E Instructions and documentation■ DOT&E and OTAs encourage Test Planning Working Groups to
develop a common standard for data ■ Utilize professional organizations like ITEA and NDIA to
support the necessity of common data standards/formats
Recommendations for Integrating DT/OT
Page 19
© 2009 The MITRE Corporation. All rights reserved.
Agenda/Overview
■What/Why Integrated Testing?■When Should We Do It?■What Can/Should Testers Do?■Summary
20
© 2009 The MITRE Corporation. All rights reserved.
Integrated Testing Is Critical
■ Combines the engineering expertise of the developer and SE with the operational expertise of the user and operational tester
■ Ensures that requirements are written correctly■ Ensures all relevant testing is planned and budgeted■ Takes advantage of all available test data to inform
acquisition and operational decisions■ Accelerates deployment of mission capability to the
warfighter
21