Upload
michelle-mason
View
22
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Software Testing. (Adapted from Chuck Cusack’s Notes) Based on material from Dick Hamlet and Joe Maybee, The Engineering of Software , Addison Wesley Longman, inc., 2001 Ian Sommerville, Software Engineering, 6 th Edition , Pearson Education Limited, 2001 - PowerPoint PPT Presentation
Citation preview
1
Software Testing
(Adapted from Chuck Cusack’s Notes)
Based on material from • Dick Hamlet and Joe Maybee, The Engineering of Software, Addison Wesley Longman, inc., 2001• Ian Sommerville, Software Engineering, 6th Edition, Pearson Education Limited, 2001 • Sebastian Elbaum, Software engineering lecture notes
2
The Software Life Cycle• Before we can discuss testing, we need to present the
context of testing—the software life cycle.• There are various forms of the software life cycle.
Requirements
Sy stem andsof tware design
Implementation andunit testing
Integration andsy stem testing
Operation andmaintenence
Requirements
Specif ication
Design
Coding
Testing
Software life cycle 1 Software life cycle 2
3
The Life Cycle and Testing• Although testing is not explicitly mentioned at
each stage of the lifecycle, every stage has some relationship to testing.
Requirements
Integration andsy stem testing
Operation andmaintenence
Requirements
Specif ication
Design
Coding
Testing
Software life cycle 1 Software life cycle 2
Test Plan
Implementation andunit testing
Sy stem andsof tware design
Test Plan
4
Software That Works• When developing software, it is important that
– it work according to the specification, and – it not contain any errors.
• Even perfect design and implementation rarely guarantees the first, and never the second.
• The solution is verification and validation.– Validation: Are we building the right product?– Verification: Are we building the product right?
• There are two techniques.– Software inspection (static)– Software testing (dynamic)
5
Software Inspection• Software inspection involves analyzing
– Requirements– Design diagrams– Source code
• As the name indicates, this is performed by looking at the various components (called a walkthrough), not by running any of the code.
• Because final code is not required, inspection can be performed at all stages of the life cycle.
• There is much more to inspection than this, but that is not the subject of these notes.
6
Software Testing• Software testing involves running the software
with test data to determine whether or not the software performs as required.
• There are a several important questions that need to be answered.– When do we develop test data?– What are good test data?– When should testing be done?– How should testing be done?– When should you stop?
• These notes attempt to answer these questions.
7
Why Test: The Optimist
• Optimist: The purpose of software testing is to verify that software meets the requirements.
• The optimistic view has several problems:– It is virtually impossible to prove that software
performs correctly for all possible inputs/situations.– It is in human nature to find what one is looking for.
Thus, we can too easily be convinced that the software is correct, when there are (sometimes very subtle) errors.
8
Why Test: The Pessimist• Pessimist: The purpose of software testing is to
find errors.• Although bleak, this is a much better definition.• There is still a slight problem. • What is an error?Errors:• Failure: When the software does something wrong (does
not do as specified).• Fault: The error (bug, defect) in the software (the code)
which causes a failure.
9
Why Test: The Realist
• Realist: The purpose of software testing is to find failures, so the faults causing the failures can be found and fixed.
• This view agrees with the pessimist, but is more precise.
• Put another way: “Try to break the system.”• This is a better view than the optimist, since:
– If we look for errors, we will probably find some.– When errors are found and fixed, we know that the
software is one step closer to being correct.
10
The “Not”s of Testing• Debugging is not the same as testing.
– Debug: Fix faults/remove failures.– Test: Find faults/discover failures.
• Testing cannot show the absence of errors, only show the presence of them.
• A system cannot (usually) be completely tested. • It cannot be assumed users will not make errors.
In other words, users will make errors.
11
Test Cases and Test Suite• A test case is a set of input data and the expected
output.• A test suite is a set of test cases.• The important questions are:
– What makes a good test case?– How do you determine the expected output?– When does a test suite contain enough test cases?
• We will attempt to answer these questions with a few suggestions, and several examples.
12
Choosing Test Cases• When choosing test cases, it is important to follow
these principles: – Enough test cases should be chosen so that it is fairly
certain that the software will work in all situations.– Both valid and invalid inputs should be tested.– Test cases on and near “boundaries” should be chosen.– The possible inputs should be divided into equivalence
partitions, with “just enough” test cases from each equivalence partition.
– If the system has “states,” sequences of test cases may be required.
13
Important Test Cases• Strings/arrays of various sizes, including 0 (empty), 1, 2,
and larger.• Strings/arrays that are too long.• Strings containing spaces.• Empty input files.• The first, last, and middle element of an array/string.• Index into array/string that is too small/large.• Loops that execute 0 times.• For ADTs (linked-lists, sets, stacks, and queues), empty,
partially full, and completely full (if applicable) data structures.
14
Equivalence Partitions• If a program breaks an array into parts, each part
of the array should be treated as an equivalence partition, and the test cases for each partition should include boundaries.
• If an input has restrictions, the partitions should correspond to the restrictions.
• For instance, if the input should be between 0 and 10, the partitions include numbers less than 0, between 0 and 10, and greater than 10.– Test cases should include –1, 0, 1, 5, 9, 10, and 11, at a
minimum.
15
Example: Binary Search• Recall the binary search algorithm:
int BinarySearch(int []A,int val, int left,int right) { if(left<right) { int middle=(left+right)/2; if(A[middle] == val ) return middle else if (A[middle] > val) return BinarySearch(A,val,left,middle-1); else return BinarySearch(A,val,middle+1,right); } else return -1;}
16
Partitions for Binary Search• The algorithm partitions the array into 3 parts:
• This suggests we break the test cases into 4 parts—One for each of the above parts, and one for a failed search.
• For each partition, include the boundaries and some case in the middle of each partition (x and y above).
• For a failed search, include values smaller and larger than all values, and one missing in the middle.
• We should also include an array of size 1.
lef t lef t+1 rightright-1mid mid+1mid-1x y
Part 1 Part 2 Part 3
17
A Test Suite for Binary Search• One test suite for binary search might be:
Input array Val return
10 10 1
10 3 -1
3, 5, 17, 23, 35, 57 29 -1
3, 5, 17, 23, 35, 57 1 -1
3, 5, 17, 23, 35, 57 60 -1
9, 11, 13, 15, 17, 19, 23, 34, 35 9 1
9, 11, 13, 15, 17, 19, 23, 34, 35 11 2
9, 11, 13, 15, 17, 19, 23, 34, 35 15 4
9, 11, 13, 15, 17, 19, 23, 34, 35 17 5
9, 11, 13, 15, 17, 19, 23, 34, 35 19 6
9, 11, 13, 15, 17, 19, 23, 34, 35 34 8
9, 11, 13, 15, 17, 19, 23, 34, 35 35 9
18
Example: Loan Calculator• Specification: The system should allow the user to
specify a principal amount, interest rate, and term, and compute the total amount of the loan.
• The first questions that should come to mind are:– Can interest rate, principal amount, and/or term be
zero? negative?– Should the interest rate be given as a percentage (5 for
5%) or decimal (.05 for 5%).• We will assume no, no, and decimal.• Given this, what test cases are appropriate?
19
Loan Calculator Test Cases• Test cases should include negative, zero, and positive
values for all of the inputs.• Does this require 27 test cases?• If we do not know how the system is to be implemented,
this may be the case, since it is possible that it handles one bad input, but not two.
• We will assume it is clear from the design and/or code that if any one or more of the inputs is not positive, the same code is executed.
20
Loan Calculator Test Suite• The following set of test
cases seems reasonable.Principal Interest Term Amount
-100 .05 10 Invalid
100 -.03 10 Invalid
100 .05 -10 Invalid
0 .05 10 Invalid
100 0 10 Invalid
100 .05 0 Invalid
100 .05 10 ?
100 .10 1 ?
10,000 .065 30 ?
250,000 .0825 15 ?
• I have not included the amount (the output), but it should certainly be included in a real test suite.
• Also, where amount says “Invalid”, the real result would depend on what the specification states.
21
Example: Stack ADT• The stack ADT (LIFO) is a data type that supports
the following operations – bool Push(item X) Place X on the top of the list.– item Pop( ) Remove and return the top element.– item Peek( ) Return the top element.
• When thinking of test cases, we should consider– an empty stack– a full stack, and– a partially full stack.
22
Testing A Stack• Since we cannot look inside the ADT, how do we
test the operations?• To test Push, we need to use either Pop or Peek.• If there is a problem, what caused it?• To further complicate things, is a newly created
empty stack the same as a stack that has had 3 elements pushed, and then popped?
23
A Stack Test Plan• We present a test suite,
assuming a stack that holds a maximum of 3 elements.
• The test cases in the following test plan should be tested in order on a single instance of a stack object.
Operation Result
Peek( ) null
Pop( ) null
Push(3) true
Peek( ) 3
Pop( ) 3
Pop( ) null
Push(1) true
Push(2) true
Push(3) true
Push(4) false
Pop( ) 3
Pop( ) 2
Pop( ) 1
Pop( ) null
Peek( ) null
24
Test Case Timeline• As the specification, design, and coding phases are
in progress, test cases may come to mind.• Things are easily forgotten, so it is important to
write these cases down, even though (in fact, especially since) testing may not occur for some time.
• When the time for testing arrives, insights have been gathered from a variety of people during the entire life cycle, making it likely that many of the problems have been anticipated.
25
Testing: The Boxes
• Black-box testing (functional, specification-based)– Based only on the specification and design process– E.g. stack ADT example
• White-box testing (structural, program-based, code-based, systematic, clear-box, glass-box, broken-box)– Based on the actual code– E.g. binary search example
26
Black-Box Testing• As noted before, black-box testing is based on the
specification, not the code.• If the specification is written properly, some test
cases should be fairly easy to generate.• It can be difficult to determine test cases that are
likely to fail.• It can also be difficult to determine equivalence
partitions.
27
White-Box Testing• When implementing “tricky” code, you can
develop test cases based on what might go wrong.• You can create test cases to be sure that every line
of code is tested.• Finding “boundaries” can be much easier when
you can look at the code.• Determining equivalence partitions can be easier.• If there are restrictions on the inputs, the code
might give some indication of which invalid inputs might be most problematic.
28
Which Box?• There is no clear answer to this question.• White-box testing is not always possible, since the
source code may not be available.• Code coverage (white-box) can be misleading,
since it does not guarantee that separate program units operate correctly together.
• White-box and black-box testing discover different types of faults. That is, they are complementary.
• Thus, picking test cases using a combination of white- and black-box testing will often result in the best test suite.
29
Testing: Stages
Testing can be separated into 5 stages:• Unit (function, component)• Module (ADT, class, group of functions)• Sub-system• System (The final product)• Acceptance (alpha)
In addition, beta testing may be performed.
30
Unit Testing• Units (functions, components, etc.) are tested in
isolation of any other code.• Unit testing can be helpful, since it gives some
confidence to the correctness of the “building blocks.”
• It is important to note that if all units are correct, it does not guarantee that the entire system will be correct, since the interactions between units might be incorrect.
31
Module Testing• As noted earlier, a module can be an abstract data type
(ADT), an object class, or a collection of related functions.• As with unit testing, module testing can be performed
independently of the other parts of the system.• For objects classes and ADTs, testing involves ensuring
the object/ADT performs all operations correctly, in all possible states of the object/ADT.
• Information-hiding can make testing ADTs and object classes difficult.– For instance to test push, one must use peek or pop. If there is an
error, which function caused it?
32
Sub-system Testing• Tests collections of modules that are integrated
into sub-systems.• The most common problems arise from interface
mismatch, including:– Interface misuse (syntactic): e.g. wrong parameter list
(wrong types, wrong order, wrong number).– Interface misunderstanding (semantic): e.g. passing an
unsorted array when a sorted array is expected.
33
System Testing• Similar to sub-system testing.• System (and sub-system) integration can be
performed using– Big-bang integration: Put it all together, and test it
(waiting for the “big bang”).– Incremental integration: Put it together in steps, testing
as each new sub-system or module is integrated.• These have obvious implications to system testing.• For incremental integration, there are two choices:
bottom-up and top-down.
34
Top-down Integration• With top-down integration, the higher-level
components are integrated and tested before the low-level components are integrated.
• This requires stub code to be written that either does nothing, or simulates the low-level modules.
• Examples:int square(int x) { // Just return a default value
return 1;} // Use a pre-defined sorting algorithm for nowvoid quickSort(int []array,int size) {
SomeLibrary.Sort(array,size);}
35
Bottom-up Integration• With bottom-up integration, low-level components
are tested before the high-level components are designed.
• This requires a test driver (or test harness) to be written that runs the low-level components.
• Example:int main() { for(int i=0;i<8;i++) { println(“The square of ” + i + “ is “ + square(i)); }}
36
Acceptance Testing• When the system has been integrated, it should be
tested with real data supplied by the actual user(s) of the system.
• This will increase the chance that the system will be error-free, since the actual data may include cases not considered by the software team.
• It will also test whether or not the requirements were correctly understood, and if the system will meet the needs it was designed to meet.
37
Testing Summary
• Testing good.• Not testing bad.• Planning for testing good.• Haphazard testing bad.