Implementation and Test

Derived from Dr. Fawcett’s SlidesPhil Pratt-SzeligaCSE 784Fall 2009

Overview

Good Neighbor PolicyChanges to codeProving CorrectnessTestingDebugging

Good Neighbor Policy

When designing programs remember you do not have exclusive access to: Memory, Disk space, Screen formats I/O channel formats, peripherals public names, files, database locks operating system mutexes, semaphores, handles communication lines and environment variables

Release system resources as soon as you can. Don’t use more than you need

Assign memory and files dynamically when you need them and release them when done

If you change iostream formats (C++) save the original and restore to the original quickly.

Use namespaces to reduce the names in the public domain

Changes

If you modify functions in a server’s public interface You break the design of clients requiring redesign,

implementation, building and test If you modify the internal private implementation of a

server or add public functions You break the code of clients requiring recompilation

If you modify the private implementation of a class that clients use through an abstract interface You break the build of every client requiring relinking

If you modify the private implementation of a module packaged as a dynamic link library and exposing only public interfaces You break nothing, just copy the dll into the directory of any

component that needs the service

Proving Correctness

Definition – Correctness Ability of a program to perform exactly in accord to

its specifications If a component’s inputs satisfy a given set of

preconditions then a correct program guarantees its outputs satisfy its specification

To prove a function correct we verify two assertions The function calls only correct functions The current function meets its specifications when

preconditions are met

Inspections

Code Inspections To verify correctness of a component we need:

Specification of processing Required preconditions Statement of side-effects

Support for inspections Each module shall have a prologue that provides a

brief description of the module’s operations Each function in each module shall have a

prologue: Processing sections A list of inputs and preconditions A list of outputs and global data effected

Function Inspection Process

Enumerate all possible states of a function For each state determine if the function works

correctly If a state is found that the function is

confusingly complex, the function is not inspectable and the inspection is done

If a state is found that the function is incorrect, the inspection is done

If an invoked function is incorrect or not inspectable, so is the invoker

If all states pass then the function passes

White Box and Black Box Testing

White Box Testing Test implementation, showing that each block of

code operates as the designer intends Black Box Testing

Test requirements independent of implementation

White Box Tests:

Construction Tests

Unit Tests

Black Box Tests

Performance Tests

Validation Tests

Qualification Tests

Regression Tests

Integration Tests

White Box Testing

For each function in a component: Every program statement is executed at least once (100%

coverage) Each relational expression is tested for true and false values Every boundary condition is tested within and at the boundary State changes are predicted and verified All errors are predicted and error handling tested

Functions that are small, functionally cohesive and have narrow coupling are much easier to test

This is not exhaustive testing. In exhaustive testing every combination of paths within the function is exercised

White Box Tests

Construction Preliminary testing as component is being built Purpose is to prepare for unit testing

Unit Full white box testing as described in previous slide Purpose is to test that the module is correct

Integration Part white box and part black box Purpose is to ensure components operate together

Unit Test Procedure

•Test Name

•Author

•Preconditions

•Derived Requirements

•Test Procedure

•Expected Results

•Actual code

Black Box Tests

Integration Tests Part white box part black box Demonstrates that major blocks satisfy requirements

allocated to them (Helps debug qualification tests) Validation Tests

Does the system crash? Instrument the system to show that all inputs within

specifications produce outputs within specifications Inputs out of spec should produce error messages

Performance Tests Build test drivers and instrumentation software that exercises

timing key threads Analyze memory and disk usage

Black Box Tests

Qualification TestsStep by step demonstration that each A and

B spec requirement is metPass/fail criteria developed for each

requirement and reviewed with customerTest data notebook maintained throughout

test. Each test is not complete until the customer signs the notebook for that test

Test data notebook is supplied to the customer (one of a kind products)

Black Box Tests

Qualification Tests Inspection – Visually inspect documentation, source code.

“shall use the C# programming language”

Demonstration – Run software normally. “shall provide operator with a command menu for selections”

Analysis – Collect test data and analyze offline. “shall report 99.999% of targets”

Test – Instrument the software and provide special test inputs.

“shall initialize the system and enter operational mode within 30 seconds”

Qualification Test Procedures

Look at Qualification Test Procedure Handout

Black Box Tests (continued)

Regression TestsDoes it still work?Summary level tests to give quick

verification that a previously tested software product still works:

After minor code changesAfter changes in the platformAt the beginning of a new maintenance phase

Often used to substitute for re-qualification after changes are made to meet specifications during qualification

Bugs

You need to care about bugs for two reasons: In the short term, customers contact you for help, forcing you

to spend time and money on the current product instead of working on new ones

In the long term, customers start buying alternatives to your buggy product

What are bugs? Inconsistent user interface Poor performance Unmet specifications Crashes or data corruption

(Debugging Applications, John Robbins, Microsoft Press)

Bugs

Common sources of bugs:Short or impossible schedules“Code first, think later”Misunderstood requirementsEngineer ignorance or improper trainingLack of commitment to quality

Debugging Advice Debugging is hard, so learn from your mistakes Some language syntax is prone to errors, so minimize their use

Gotos Global variables Void * pointers Automatic Type Conversions

Explain your code to someone else Make the bug reproducible

Don’t start intensive debugging until you can reliably reproduce the problem Divide and conquer Display output to localize your search Write self-checking code Write to a log file Keep records

If the search is long, without records you will forget what you have already done

(from debugging book #1)

Debugging Books

The Practice of Programming, Kernighan & Pike, Addison-Wesley, 1999

Debugging Applications for Microsoft .Net and Windows, John Robbins, Microsoft Press, 2003

Unit Tests

John Robin’s advice for unit tests is as follows: Only the paranoid survive “I have many friends who are excellent engineers,

but when it comes to having them interface with my code, I verify their data down to the last bit”

“…I even have a healthy skepticism about myself” “Assertions, tracing and commenting are how I start

verifying my fellow developers who are calling my code”

“Unit testing is how I verify myself” Unit test drivers are scaffolds you use to call your

code outside the normal program environment

Unit Tests (continued)

Start writing unit tests as soon as you start writing code

Test each incremental change in isolation and spread test development over the whole code development phase

Think about how you are going to test your code before you write it

“While you are coding you should be running your unit tests all the time”

The key to effective unit tests is Code Coverage – the percent of lines you’ve executed during test

“The simple fact is that a line not been tested is a line waiting to crash”

End of presentation