IV&V TESTING STRATEGIES FOR INDEPENDENT VERIFICATION OF NASA MISSION SOFTWARE IMPLEMENTATION3rd Annual Workshop on Independent Validation and Verification

September 17, 2009

OBJECTIVES

Vision for an IV&V Test Environment Value of an IV&V Test Environment IV&V Test Environment Concept of Operations Characteristics of an IV&V test environment Software IV&V Testing Strategies Current status of the IV&V Test Environment

2

VISION: PROBLEM STATEMENT

The current method of analyzing source code in its static state doesn’t adequately evaluate the behavior of the code under operational and environmental conditions

System interactions, dependability, quality attributes, among other properties are hard to evaluate with static analysis

3

VISION: SCOPE

Develop, maintain and operate an adaptable test environment for the IV&V Program that enables the dynamic analysis of software behavior for multiple NASA missions

Compliment the inspection of the development test design and subsequent test results as well as manual and automatic source code analysis in its static state These methods have been effective at

uncovering evidence that the implementation answers the three questions

4

VISION: VALUE

Uncover issues that would not be found by static code analysis, manual requirements based code inspection, or test results inspection

Allow for the creation of goodness statements that will add more value to the project and provide a greater confidence in the assurance of the software

5

CONCEPT OF OPERATIONS

Intended to be common for for any NASA mission software IV&V project

For a given NASA mission, develop test plan using the System Reference Model (SRM)

For a given NASA mission flight software test article: Execute it Stimulate it according to IV&V test plan Observe its behavior

6

APPROACHES TO EXECUTING SUT

Mimic the intended target environment Acquire COTS emulators for the target processors Develop simulators Leverege simulators already created by the

developer Use the development test environment

Connect remotely Install our test environment at the development

test site Don’t execute

Analyze the developer’s test results7

APPROACHES TO STIMULATING SUT

Generate test scenarios from the system behavior models for a NASA mission found in theSRM Scope the type of scenarios based on the results

from validating the development test design Install and use developers simulation models

in our IV&V test environment Create our own simulation models to

stimulate the software Don’t stimulate

Developer incorporates IV&V test suite into their test plan and IV&V analyzes the test results 8

APPROACHES TO OBSERVING SUT

Manually inspect the test results Monitor the behavior of the SUT using formal

mechanisms such as state chart assertions created by the SRM and validation teams On-line during test Off-line after test, based on test results

9

TESTING STRATEGIES

A strategy is a plan of action designed to achieve a particular goal1

The expected goal of an analyst using the system: To determine whether or not the implemented

system behaves as we expect it to, doesn’t behave like we don’t what it to, and to ensure it behaves as it should under adverse conditions.

Possible strategies Improvement to the development test design Analyze the development test results Supplement the development test with our test

suite Test independent of the developer

10

1: From the Oxford English Dictionary

11

Develop Software Test SuiteAnalyze

Development Test Results

Configure IV&V Test Environment

Integrate IV&V Test Environment with Development

Facility

Execute Software Test Suite at Development Test

Facility

Execute Software Test Suite at IV&V

Facility

Analyze IV&V Test Results

Limitations do not exist in developer’s test designLimitations exist in developer’s test design

Development test facility is not available for IV&VDevelopment test facility is available for IV&V

TEST ENVIRONMENT CHARACTERISTICS Usable – common user interface across all

IV&V projects Adaptable – allow for the reconfiguration of

testing resources for a individual IV&V projects

Extensible – allow for the future integration of new simulation tools, simulation models, and development test articles

Accessible Interface with test monitors (e.g assertion state

chart repositories) Interface with development laboratories

Portable – for the possibility of testing at development site

12

STATUS Vision and Concept of Operations documents

are baselined Took inventory of available hardware and

software resources at the facility Created first draft of cost estimate and

acquiring vendor quotes Working on a proof of concept to explore the

process of dynamically testing flight software as a form of software implementation verification. The prototype will be used to improve our

Concept of Operations and validate our Vision. Using uncompiled mission code that was already

provided by developer13

CHALLENGES

Collecting all the information and materials needed to develop the test suite and configure the test environment in a timely manner. This may take multiple iterations

Configuring the test environment with the necessary simulators and emulators to successfully run and stimulate the software under test

14