Software Fault Injection

Kalynnda BerensScience Applications International

Corporation NASA Glenn Research Center

Software Fault Injection NASA Glenn Research Center

2

What is Software Fault Injection?

A testing technique that aids in understanding how software behaves when stressed in unusual ways.

A product-based assurance technique.

Variations in the technique allow it to be applied to many types of software and for different purposes.

Software Fault Injection NASA Glenn Research Center

3

How does SFI work?

Legal permutations or faults are input at interfaces (external and/or internal).

Outputs show whether the injected fault propagates through the software.

Requires instrumentation (software code) to observe the propagation process.

Software Fault Injection NASA Glenn Research Center

4

Uses for Software Fault Injection

Finding defects in software Robustness Testing COTS Validation/Determining failure

modes Safety Verification Security Assessment Software Testability Analysis

Software Fault Injection NASA Glenn Research Center

5

SFI Examples

Operating System Validation Ballista (CM) – Linux and VxWorks robustness WindowsNT

Network Security NCSA httpd server

Safety Advanced Automatic Train Control system Magneto Stereoaxis System

SFI can be used with or without source code

Software Fault Injection NASA Glenn Research Center

6

SFI without Source Code

Create software wrapper for COTS functions and other interfaces

“Trick” OS to call wrapper functions first Software under test usually run in debug

mode Wrapper can be used

Pass through for baselining response Call alternative function Call original function but change result

Software Fault Injection NASA Glenn Research Center

7

SFI wrapper operations

Application WrapperOS or

Hardware

Pass through wrapper:

Call alternative function:

Call original function but change result:

Application

Application

Wrapper

Wrapper OS orHardware

NewFunction

Software Fault Injection NASA Glenn Research Center

8

Center Initiative on SFI

Can SFI be used by an Independent V&V engineer?

Is SFI a useful and cost-effective technique for NASA? Are the errors and problems found of

sufficient severity or abundance? Are the costs of applying the technique

reasonable for the number/severity of errors found?

Is SFI a good tool for safer software?

Software Fault Injection NASA Glenn Research Center

9

Methodology

Determine scope Select projects Metrics Perform SFI on projects Create Test Plan (prototype due 1st

quarter, FY02)

Lessons Learned

Software Fault Injection NASA Glenn Research Center

10

Determine Scope Why narrow the scope?

SFI is a collection of related techniques Comparison across projects requires using one technique

for all

Why no source/interfaces technique chosen IV&V perspective (cost effective) “Outside” events or system limitations trigger many errors

Interfaces selected COTS software Hardware User input Communications medium

Software Fault Injection NASA Glenn Research Center

11

Project Selection

Potential Projects CM-2 Tempest Web Server (VxWorks and

Java) MDCA, FPP, SAMS, others

Selection Criteria Selection difficulties

Project support not free Contracted software not accessible

Final Choice

Software Fault Injection NASA Glenn Research Center

12

Metrics

Time spent per task Familiarization, researching errors,

instrumenting software, testing Subjective “effort” scale per task Software project metrics

SLOC, #classes/modules, complexity, interface information

Fault Injection metrics #faults, #failures, #faults no effect/correctly

handled

Software Fault Injection NASA Glenn Research Center

13

SFI Process

Obtain Tempest software (completed) Obtain access to VxWorks

(completed) ***Overcome compatibility problems Determine all interfaces to test Select errors to inject Create necessary wrappers for SFI Record test procedure and results

Software Fault Injection NASA Glenn Research Center

14

Tempest Interfaces

VxWorks OS Task creation and control functions C/C++ language functions File system functions Networking functions

Outside world Requests from external sources Standard HTML, built-in functions Tempest (VxWorks version) can execute OS

functions

Software Fault Injection NASA Glenn Research Center

15

Example Injection Errors

OS errors Memory allocation failures File errors (corrupted, not found) Single task abort, hang

External World errors Invalid request Too many requests Requests too frequent

Software Fault Injection NASA Glenn Research Center

16

Test Plan

How to perform software fault injection on “generic” software

Steps prior to actual testing Method of determining errors to

inject Procedure for performing the test Appendices of lessons learned,

example faults, other guidance

Software Fault Injection NASA Glenn Research Center

17

Difficulties Encountered

Tempest documentation limited VxWorks simulator does not

support networking Cost of hardware and full VxWorks

not within the budget Attempt to “fake” networking

unsuccessful

Software Fault Injection NASA Glenn Research Center

18

Status and Future Work

VxWorks incompatibilities not easily overcome

Shift to Java version of Tempest for now

Test VxWorks version of Tempest on actual hardware (if possible) or alternate operating system (Linux, uClinux, eCos)

If funding continues, test on actual flight experiment (CM-2).