Testing. Today’s Topics Why Testing? Basic Definitions Kinds of Testing Test-driven Development Code Reviews (not testing) 1

Testing

2

Today’s Topics

• Why Testing?• Basic Definitions• Kinds of Testing• Test-driven Development• Code Reviews (not testing)

3

Why Testing?

• Ideally, we’d prove that the code was correct using formal mathematics

• This is extremely difficult even for trivially small programs

• Not practical in most cases• Not even for mission critical or safety code

4

Why Testing?

• Near ideal: use symbolic or abstract model testing • Prove that model is correct• Automatically extract mathematical model from code• Prove properties of model for all possible cases

• In practice, can work well for simple properties• E.g., “System will not crash in this way”

• Doesn’t work well for complex systems / data structures

5

Why Testing?

• We can actually run code to see if it works• This is software testing

• Always necessary, even when you can prove correctness• Proof seldom tied to actual code

• Testing is NOT a lost resort; it is critical part of sw dev

6

Why Testing?

• Testing all too often an afterthought

• Test before faults become too expensive to fix or pressing deadlines

• The more code written before fault is detected, the more code needs to be fixed…

• Better to detect fault(s) on a small prototype or “finished” product?

7

Waterfall

Requirements analysis

Design

Implementation

Operation

Testing

Prototyping

8

Spiral

Draft a menu ofrequirements

EstablishrequirementsPlan

Analyze risk &prototype

Draft a menu ofarchitecture designs



Draft a menu ofprogram designs

Establisharchitecture

Establishprogram design

Implementation

TestingOperation

Plan

9

Agile

Customer provides “stories”(short requirement snippets)

System and acceptance tests

Do “spike” to evaluate & control risk

Prioritizestories and plan

Implement

OperationWrite/run/modify

unit tests

10

Testing Saves Lives and Money

• NIST Report: “The Economic Impact of Inadequate Infrastructure for Software Testing”

• Inadequate software testing costs US $22 - $59 billion annually

• Better approaches could cut this is half

11

Testing Saves Lives and Money

• Major failures: • Ariane 5 explosion• Mars Polar Lander• Intel Pentium FDIV bug• THERAC-25 radiation machine: killed 5

Ariane 5- exception-handling bug forced self destruct on maiden flight (64-bit to 16-bitconversion: about $370 mill lost)

12

Today’s Topics


13

Definitions: Testing

• Running a program to find faults (bugs)• … in code• … in specs• … in documentation• … in tests

14

Definitions: Test Case, Test Suite, Tester

• Test case: one execution of the program that may find a bug

• Test suite: a set of program executions, grouped together• Made of up test cases

• Tester: program that generates tests

15

Definitions: Coverage

• Measures or metrics• Abstraction of “what a test suite tests”

• Common measures:• Statement coverage• Decision coverage• Path coverage• Mutation coverage

16


• Statement coverage:• aka: line or basic block coverage• Which statements execute in test suite

• Decision coverage: • Which boolean expressions in control structures evaluate to true and

false during suite execution.

17


• Path coverage:• Which paths through a program’s control flow graph are taken in the test

suite

• Mutation coverage:• Ability to detect random variations in the code

18

Today’s Topics


19

Kinds of Testing

• White box• Black box• Unit • Integration• System • Acceptance

20

White Box Testing

• Opens up the box• aka glass box, clear box, structural testing

• Use source code to design test cases

21

Black Box Testing

• Testing does NOT look at source code or internal structures

• Sends program stream of inputs, observes outputs

• Abstracts away the internals• Useful perspective for system and integration testing

• Sometimes you don’t have access to the source code

22

Testing Stages

• Unit testing: first phase; done by module develops

• Integration testing: combines unit tested modules, tests interactions

• System testing: test whole program to make sure reqs met

• Acceptance testing: tests by users to see if system meets reqs

23

Unit Testing

• First phase; mostly done by module developers• Typically earliest type of testing• Unit could be as small as single function or method• Often relies on “stubs” to represent other modules or

incomplete code• Tools to support unit tests available for most popular

languages, e.g., Junit ( http://junit.org )

?

http://junit.org/

24

Integration Testing

• Combines unit-tested modules and tests interactions• Relies on completed units• After unit testing, before system testing• Test cases focus on interfaces between components and

assemblies of multiple components• Often more formal than unit testing• More detailed test plan(s)

??

?

25

System Testing

• Tests whole program; make sure reqs met• After integration testing• Focuses on “breaking the system”• Finds defects in completed product, not just component

interactions• Checks quality of requirements as well as system• Often includes stress testing: beyond bounds of well-defined

behavior

26

System Testing: Functional Testing

• Developer tests program from “user’s” perspective• Different mindset than unit testing

• Analogy: home inspection vs. walk-through • inspector => software engineer; looking for “hard facts”• buyer => system user; looking for functionality

27

Acceptance Testing

• Tests by user to see if system meets use requirements

• Form of black box testing

• Done by end-users to determine if system really meets needs

• May revise requirements

• May find bugs in the system

28

Today’s Topics


29

Test Drive Development

• Write test cases before code• Make sure code is tested as early as possible• Idea arises from Extreme Programming• Often used in agile development

• Write automated test cases first• Then write code to satisfy tests

30

Test Driven Development

• How to add a feature:

1. Add a test case that fails (but succeeds with new feature)2. Run all tests, make sure only new test fails 3. Write code to implement new feature4. Rerun all tests, make sure new test succeeds (no others fail)

31

Test Driven Development Cycle

32

Test Driven Development Benefits

• Results in lots of useful test cases• A very large regression set

• Forces attention to controllable and observable sys behavior • Only write code as needed to pass tests• May get good coverage of paths through program• Reduces temptation to tailor tests to idiosyncratic behaviors

• Testing is a first-class activity in this kind of development

33

Problems with Test Driven Development

• Needs institutional support• Difficult to integrate with waterfall process• Management may wonder why time is spent writing tests, not code

• Many test cases may create false confidence• If developers have written all tests, there may be blind spots due

34

Exhaustive vs. Representative Testing

• Can we test everything? (exhaustive)• Probably not! Permutations / combinations are exponential!

• Need to be thoughtful about creating representative tests• Create tests at boundary conditions• Want to “represent” the range of input values

35

Today’s Topics


36

Code Reviews

• Code Review ≠ Testing

• Method for finding bugs and determining code quality

• Code walkthrough: developer leads review team through code• Informal

• Code inspection: review team checks against concerns• Prepared offline (in many cases)• Led by a team moderator

37

Code Reviews

• Code inspections: found to be one of the most effective practices for finding faults• Some experiments show removal of 67-85% of defects via inspections

• Some consider XP’s pair programming as a kind of “code review” process

• Can review/walkthrough requirements and design docs, not just code

38

Next Steps

• Homework 5 due Friday!!

• Vision paper due 11/21!!

• Midterm due 11/25!!

Documents

Testing. Today’s Topics Why Testing? Basic Definitions Kinds of Testing Test-driven Development Code Reviews (not testing) 1