Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Software Testing 101or
Why Should We Bother With Software Testing?Crash Course Lecture
Marat Akhin
Saint-Petersburg State Polytechnical University
December 8th, 2009
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 1 / 51
Contents
1 Prelude
2 Basics
3 Design for testability
4 Coverage
5 Regression testing
6 Resume
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 2 / 51
What we WON'T cover today Heavy formal stu�
Heavy formal stu�
Input: T1,T2 � AST äëÿ ïðîãðàììû äî è ïîñëå ìîäèôèêàöèèOutput: M � ìíîæåñòâî ñâÿçàííûõ óçëîâ äëÿ T1 è T2
1: M← ∅2: Îòìåòèòü âñå óçëû T1 è T2 êàê ¾íåñâÿçàííûå¿
3: for x ∈ T1 do . Îáõîäèì T1 ñíèçó ââåðõ
4: if ∃y ∈ T2 : ¬isMatched(y) ∧match(x , y) = 1 then5: M←M ∪ (x , y)6: Îòìåòèòü x è y êàê ¾ñâÿçàííûå¿
7: end if
8: end for
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 3 / 51
What we WON'T cover today Details of testing techniques
Details of testing techniques
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 4 / 51
What we WON'T cover today Speci�c testing software
Speci�c testing software
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 5 / 51
What we WILL cover today Software testing 101
Software testing 101
Some basic things about software testing that you should know...
...if you want to be a good software testerShould you learn ¾Software testing 101¿ if you want to be a goodsoftware developer?YES
I hope you'll understand why in about one and a half hours =)
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 6 / 51
What we WILL cover today Software testing 101
Software testing 101
Some basic things about software testing that you should know......if you want to be a good software tester
Should you learn ¾Software testing 101¿ if you want to be a goodsoftware developer?YES
I hope you'll understand why in about one and a half hours =)
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 6 / 51
What we WILL cover today Software testing 101
Software testing 101
Some basic things about software testing that you should know......if you want to be a good software testerShould you learn ¾Software testing 101¿ if you want to be a goodsoftware developer?
YES
I hope you'll understand why in about one and a half hours =)
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 6 / 51
What we WILL cover today Software testing 101
Software testing 101
Some basic things about software testing that you should know......if you want to be a good software testerShould you learn ¾Software testing 101¿ if you want to be a goodsoftware developer?YES
I hope you'll understand why in about one and a half hours =)
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 6 / 51
What we WILL cover today Software testing 101
Software testing 101
Some basic things about software testing that you should know......if you want to be a good software testerShould you learn ¾Software testing 101¿ if you want to be a goodsoftware developer?YES
I hope you'll understand why in about one and a half hours =)
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 6 / 51
What we WILL cover today Why should we bother with software testing?
Why should we bother with software testing?
Short answer: because I tell you that you should do so!
Long answer: because experience of thousands and thousands ofdevelopers all over the world indicate that software testing really helpsin the development of quality software. Otherwise � why is it used inalmost every software development methodology?
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 7 / 51
What we WILL cover today Why should we bother with software testing?
Why should we bother with software testing?
Short answer: because I tell you that you should do so!Long answer: because experience of thousands and thousands ofdevelopers all over the world indicate that software testing really helpsin the development of quality software. Otherwise � why is it used inalmost every software development methodology?
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 7 / 51
Prelude What question does testing answer?
What question does testing answer?
Does this software work?
NO, testing cannot answer that
Does this software fail?
YES, testing can (and should) answer that
Testing = Destruction
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 8 / 51
Prelude What question does testing answer?
What question does testing answer?
Does this software work?
NO, testing cannot answer that
Does this software fail?
YES, testing can (and should) answer that
Testing = Destruction
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 8 / 51
Prelude What question does testing answer?
What question does testing answer?
Does this software work?
NO, testing cannot answer that
Does this software fail?
YES, testing can (and should) answer that
Testing = Destruction
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 8 / 51
Prelude What ¾testing¿ is and what isn't
What ¾testing¿ is and what isn't
It is:
Best friend of Verification and Validation
What is the di�erence between the two?
Verification checks whether the software is correct and rightValidation checks whether the software complies to a set ofrequirements
In other words:
We verify that ¾we did it right¿
We validate that ¾we did the right thing¿
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 9 / 51
Prelude What ¾testing¿ is and what isn't
What ¾testing¿ is and what isn't
It isn't:
Way to ensure 100% guarantee of any kind
The software will never fail with NPEThe database will never deadlockCalculations are always correctTime requirements are always ful�lled
We ¾can¿ give 100% guarantee but only in the most trivial cases �and testing itself is usually not needed when we talk about trivial stu�We'll cover why 100% guarantee is impossible a little bit later
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 10 / 51
Prelude What ¾testing¿ is and what isn't
What ¾testing¿ is and what isn't
It isn't:
Way to ensure 100% guarantee of any kind
The software will never fail with NPEThe database will never deadlockCalculations are always correctTime requirements are always ful�lled
We ¾can¿ give 100% guarantee but only in the most trivial cases �and testing itself is usually not needed when we talk about trivial stu�
We'll cover why 100% guarantee is impossible a little bit later
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 10 / 51
Prelude What ¾testing¿ is and what isn't
What ¾testing¿ is and what isn't
It isn't:
Way to ensure 100% guarantee of any kind
The software will never fail with NPEThe database will never deadlockCalculations are always correctTime requirements are always ful�lled
We ¾can¿ give 100% guarantee but only in the most trivial cases �and testing itself is usually not needed when we talk about trivial stu�We'll cover why 100% guarantee is impossible a little bit later
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 10 / 51
Prelude Why testing is so di�cult
Why testing is so di�cult
Theorem
Software testing is all about PAIN
Proof.
Software engineering is all about PAINSoftware testing ⊂ Software engineering
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 11 / 51
Prelude Why testing is so di�cult
Why testing is so di�cult
Software Method Pain
Build-and-�x It just doesn't scale up
Waterfall Model We can't predict everything at the start
Extreme Programming Writing good test cases is very di�cult
Rapid Prototyping Throwing away the prototype is hard
Uni�ed Process Following its many rules is a pain
Dan Berry: ¾The Inevitable Pain of Software Development¿
Fred Brooks: ¾No Silver Bullet¿
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 12 / 51
Prelude Why testing is so di�cult
Why testing is so di�cult
Brian Kernighan
¾Debugging is twice as hard as writing the code in the �rst place.Therefore, if you write the code as cleverly as possible, you are, byde�nition, not smart enough to debug it.¿
Massimo Arnoldi (feat. Kent Beck)
¾Unfortunately at least for me (and not only) testing goes against humannature. If you realize the pig in you, you will see that you program withouttests.¿
We have a very serious problem
Now we understand that software testing is really hard. But we have to doit to make quality software.
A very simple question � what do we do?
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 13 / 51
Prelude Why testing is so di�cult
Why testing is so di�cult
Brian Kernighan
¾Debugging is twice as hard as writing the code in the �rst place.Therefore, if you write the code as cleverly as possible, you are, byde�nition, not smart enough to debug it.¿
Massimo Arnoldi (feat. Kent Beck)
¾Unfortunately at least for me (and not only) testing goes against humannature. If you realize the pig in you, you will see that you program withouttests.¿
We have a very serious problem
Now we understand that software testing is really hard. But we have to doit to make quality software.
A very simple question � what do we do?
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 13 / 51
Prelude Why testing is so di�cult
Why testing is so di�cult
Brian Kernighan
¾Debugging is twice as hard as writing the code in the �rst place.Therefore, if you write the code as cleverly as possible, you are, byde�nition, not smart enough to debug it.¿
Massimo Arnoldi (feat. Kent Beck)
¾Unfortunately at least for me (and not only) testing goes against humannature. If you realize the pig in you, you will see that you program withouttests.¿
We have a very serious problem
Now we understand that software testing is really hard. But we have to doit to make quality software.
A very simple question � what do we do?
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 13 / 51
Prelude What do we do?
What do we do?
Stick our teethLearn how to minimize the pain of testing
Understand how the testing worksUnderstand how software testing can help software developmentUnderstand how software development can help software testing
Believe in software testing
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 14 / 51
Basics A very basic scheme of software testing
A very basic scheme of software testing
Run the softwareCheck the run results hoping to �nd errors
aka bugsaka faultsaka defectsaka �awsaka failures
Let's clear up the de�nition of ¾software error¿, shall we?
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 15 / 51
Basics Model of software error
Model of software error
FAILURE
FAULT
ERROR
Failure � observable from the outside
incorrect program behaviourFault � corrupted program state causedby an errorError � developer's mistake in aprogram
Let's check this model on an example
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 16 / 51
Basics Model of software error (example)
Model of software error (example)
Find the error in the following Java program
1 int sumCollection( f ina l Collection <Integer > c) {
2 int sum = 0;
3 for ( int i : c) {
4 sum += i;
5 }
6 return sum;
7 }
Possible over�ow at line 4
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 17 / 51
Basics Model of software error (example)
Model of software error (example)
Find the error in the following Java program
1 int sumCollection( f ina l Collection <Integer > c) {
2 int sum = 0;
3 for ( int i : c) {
4 sum += i;
5 }
6 return sum;
7 }
Possible over�ow at line 4
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 17 / 51
Basics Model of software error (example)
Model of software error (example)
c = {}
What happens?
1 int sumCollection( f ina l Collection <Integer > c) {
2 int sum = 0;
3 for ( int i : c) {
4 sum += i;
5 }
6 return sum;
7 }
No fault and no failure � program works correctly
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 18 / 51
Basics Model of software error (example)
Model of software error (example)
c = {}
What happens?
1 int sumCollection( f ina l Collection <Integer > c) {
2 int sum = 0;
3 for ( int i : c) {
4 sum += i;
5 }
6 return sum;
7 }
No fault and no failure � program works correctly
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 18 / 51
Basics Model of software error (example)
Model of software error (example)
c = {1, 2, 3, 5}
What happens?
1 int sumCollection( f ina l Collection <Integer > c) {
2 int sum = 0;
3 for ( int i : c) {
4 sum += i;
5 }
6 return sum;
7 }
No fault and no failure � program works correctly
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 19 / 51
Basics Model of software error (example)
Model of software error (example)
c = {1, 2, 3, 5}
What happens?
1 int sumCollection( f ina l Collection <Integer > c) {
2 int sum = 0;
3 for ( int i : c) {
4 sum += i;
5 }
6 return sum;
7 }
No fault and no failure � program works correctly
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 19 / 51
Basics Model of software error (example)
Model of software error (example)
c = {1, 2, 3, 5, +MAX_INT, -MAX_INT}
What happens?
1 int sumCollection( f ina l Collection <Integer > c) {
2 int sum = 0;
3 for ( int i : c) {
4 sum += i;
5 }
6 return sum;
7 }
Program goes through corrupted state (fault is present)But no failure visible from the outside
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 20 / 51
Basics Model of software error (example)
Model of software error (example)
c = {1, 2, 3, 5, +MAX_INT, -MAX_INT}
What happens?
1 int sumCollection( f ina l Collection <Integer > c) {
2 int sum = 0;
3 for ( int i : c) {
4 sum += i;
5 }
6 return sum;
7 }
Program goes through corrupted state (fault is present)But no failure visible from the outside
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 20 / 51
Basics Model of software error (example)
Model of software error (example)
c = {1, 2, 3, 5, +MAX_INT}
What happens?
1 int sumCollection( f ina l Collection <Integer > c) {
2 int sum = 0;
3 for ( int i : c) {
4 sum += i;
5 }
6 return sum;
7 }
Program goes through corrupted state (fault is present)And it fails with incorrect results
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 21 / 51
Basics Model of software error (example)
Model of software error (example)
c = {1, 2, 3, 5, +MAX_INT}
What happens?
1 int sumCollection( f ina l Collection <Integer > c) {
2 int sum = 0;
3 for ( int i : c) {
4 sum += i;
5 }
6 return sum;
7 }
Program goes through corrupted state (fault is present)And it fails with incorrect results
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 21 / 51
Basics How to �nd a software error
How to �nd a software error
Model can be represented as:
a mental representation of ¾how thissoftware should work¿requirements from software technicalspeci�cationdi�erent test casesset of correct software outputsanother (a priori correct)implementation of the samespeci�cation
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 22 / 51
Basics How to �nd a software error
How to �nd a software error
Testing run must ensure the following 3 properties to actually �nd an error:
Reachibility � test must execute (¾reach¿) the software errorCorruption � error must corrupt the program state to producesoftware faultPropagation � fault must persist through the program run and causea failure in program behaviour
Making these properties hold is a major pain in software testing
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 23 / 51
Basics How to �nd a software error
How to �nd a software error
To soothe this pain, we have to develop a program that is:
Controllable � we can easily make the program go where and how wewant it to goObservable � we can easily understand what the program is doing atany given time
If we don't think about that before we start developing � chances are we'llnever be able to make the program controllable and observable.
We need ¾Design for testability¿
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 24 / 51
Design for testability Test-driven development
Test-driven development
Most well known technique of ensuring ¾Design for testability¿Created by Kent Beck back in 1999Two simple principles:
We write test cases �rstThen we make them work by writing code
Forces the developer to create and maintain a good softwarespeci�cationMakes the software controllable and observable:
Each test case corresponds to a speci�c control �ow trace � thus givingus controllabilityEach test case checks a set of software outputs � thus giving usobservability
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 25 / 51
Design for testability Controllability
Controllability
Simulation and stubbing
We can easily control our own code � but we can't do that withthird-party code like OS routines, incomplete code, etc.We have to simulate or stub (S&S) third-party code to control whatit is doing
S&S may be faster than third-party codeS&S can easily produce hard-to-induce error
Downwards scalability
Many bugs live in ¾the dark corners¿ of software limitsTo make them appear we have to reach these limits � and it is easierto do so when the limits are nice and smallWe scale the limits down to reach them during testing
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 26 / 51
Design for testability Controllability
Controllability
Simulation and stubbing
We can easily control our own code � but we can't do that withthird-party code like OS routines, incomplete code, etc.We have to simulate or stub (S&S) third-party code to control whatit is doing
S&S may be faster than third-party codeS&S can easily produce hard-to-induce error
Downwards scalability
Many bugs live in ¾the dark corners¿ of software limitsTo make them appear we have to reach these limits � and it is easierto do so when the limits are nice and smallWe scale the limits down to reach them during testing
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 26 / 51
Design for testability Observability
Observability
Assertions
Improve observability by propagating some errors into failuresAnd we can observe the error directly � not just the results of it
Invariant checkers
Extension of assertions to do a full-scale program state checkingInspect program state at some points during execution to ensure statecorrectnessOf course we run invariant checkers only during testing � they are tooexpensive to run during actual program run
Logging
Very important in any serious project to trace what and how theprogram is doing at any given timeProjects program traces into developer-friendly form
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 27 / 51
Design for testability Observability
Observability
Assertions
Improve observability by propagating some errors into failuresAnd we can observe the error directly � not just the results of it
Invariant checkers
Extension of assertions to do a full-scale program state checkingInspect program state at some points during execution to ensure statecorrectnessOf course we run invariant checkers only during testing � they are tooexpensive to run during actual program run
Logging
Very important in any serious project to trace what and how theprogram is doing at any given timeProjects program traces into developer-friendly form
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 27 / 51
Design for testability Observability
Observability
Assertions
Improve observability by propagating some errors into failuresAnd we can observe the error directly � not just the results of it
Invariant checkers
Extension of assertions to do a full-scale program state checkingInspect program state at some points during execution to ensure statecorrectnessOf course we run invariant checkers only during testing � they are tooexpensive to run during actual program run
Logging
Very important in any serious project to trace what and how theprogram is doing at any given timeProjects program traces into developer-friendly form
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 27 / 51
Coverage How to measure testing?
How to measure testing?
OK � we have ¾design for testability¿And we have some kind of testing in our project
Question is:
How do we measure software testing?
We need a way to estimate testing ¾e�ciency¿ � yes, this is anothermajor pain in software testing
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 28 / 51
Coverage Four types of coverage
Four types of coverage
We use di�erent coverage criteria to measure testing e�ciencyThere are 4 basic types of coverage:
Graph based coverageLogic based coverageInput partition based coverageSyntax based coverage
Paul Ammann, Je� O�utt, ¾Introduction to Software Testing¿
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 29 / 51
Coverage Graph based coverage
Graph based coverage
Cover some elements of some graph corresponding to the programIn other words, ¾Find a graph and cover it!¿Most common type of coverage due to simplicity (we're not talkingabout computational simplicity here)Examples include:
Statement coverageBranch coveragePath coverageData �ow coverage
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 30 / 51
Coverage Graph based coverage
Statement coverage
Cover every node in program's CFG (control�ow graph)CFG node = basic block � part of code withone entry point, one exit point and no jumpinstructions within itThe simpliest of all statement coverage criteria
In our example
Single test run is enough to achieve 100% statementcoverage
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 31 / 51
Coverage Graph based coverage
Branch coverage
Cover every branch in program's CFG (control�ow graph)Stronger than statement coverage � we test allpossible control �ow jumps (but we do it¾context-free¿)
In our example
We need at least two test runs to achieve 100%branch coverage
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 32 / 51
Coverage Graph based coverage
Path coverage
Cover every possible path in program's CFG(control �ow graph)Much stronger than branch coverage � we testall possible control �ow jumps in all possiblecombinations (i.e., ¾context-sensitive¿)Unfortunately, covering all execution paths isNP-complete task
In our example
We need at least four test runs to achieve 100%path coverage
If we had even one back-edge in our CFG...
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 33 / 51
Coverage Logic based coverage
Logic based coverage
Every conditional statement gives us two possible branches to coverBut consider this one:
i f (p != q && (p == null || !p.equals(q))) {
...
}
There are four logical branches to cover � we cannot cover themsimply by covering all branches
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 34 / 51
Coverage Logic based coverage
Logic based coverage
Let's take a look at these logical branches
p != q && (p == null || !p.equals(q)) Result
T F T T
T T - T
T F F F
F - - F
Much stronger than path coverageEven more ¾context-sensitive¿ than path coverageEven more ¾NP-complete¿ than path coverage
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 35 / 51
Coverage Input partition based coverage
Input partition based coverage
In a way � evolution of logic based coverage criteriaWe have input data domain D
It can be partitioned into several equivalence classes Ei
The equivalence classes must satisfy two main properties:
Ei ∩ Ej = ∅,∀i , j : i 6= j (equivalence classes must be disjoint)⋃i Ei = D (equivalence classes must completely cover input data
domain)
Seems pretty simple, doesn't it?
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 36 / 51
Coverage Input partition based coverage
Input partition based coverage
Main problem with input partitioning is ambiguityConsider the following example:
Program that determines whether the �le is sorted in ascending ordescending orderThat's easy!
E1 � �le is sorted in ascending order
E2 � �le is sorted in descending order
E3 � �le is not sorted
What about a �le with only one record?That's correct � partitioning fails
Easier to use than logic based coverageLess complex than other coverage criteriasPartitioning can only be done manually in most cases � outweights�rst two advantages
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 37 / 51
Coverage Input partition based coverage
Input partition based coverage
Main problem with input partitioning is ambiguityConsider the following example:
Program that determines whether the �le is sorted in ascending ordescending orderThat's easy!
E1 � �le is sorted in ascending order
E2 � �le is sorted in descending order
E3 � �le is not sorted
What about a �le with only one record?That's correct � partitioning fails
Easier to use than logic based coverageLess complex than other coverage criteriasPartitioning can only be done manually in most cases � outweights�rst two advantages
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 37 / 51
Coverage Input partition based coverage
Input partition based coverage
Main problem with input partitioning is ambiguityConsider the following example:
Program that determines whether the �le is sorted in ascending ordescending orderThat's easy!
E1 � �le is sorted in ascending order
E2 � �le is sorted in descending order
E3 � �le is not sorted
What about a �le with only one record?That's correct � partitioning fails
Easier to use than logic based coverageLess complex than other coverage criteriasPartitioning can only be done manually in most cases � outweights�rst two advantages
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 37 / 51
Coverage Syntax based coverage
Syntax based coverage
A ¾very other¿ kind of coverage criteriaAs you all know � software is usually writen in a programming languageAny programming language has some syntax rules or grammarWe can use that fact to test our software � let's see how we can dothat
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 38 / 51
Coverage Syntax based coverage
Syntax based coverage
A program is a valid string in a given grammarWhat if we mutate this string into another one?
We can get an invalid string � we are not interested in themIf we get another valid string � it is a mutant of the original program
A mutant exhibits di�erent behaviour from the original programA ¾perfect¿ test should pass only on the original program � andshould fail on any possible mutant (kill a mutant)
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 39 / 51
Coverage Syntax based coverage
Syntax based coverage
That is our coverage criteriaI.e., how many mutants a test killsThis is the main idea of mutation testing
A very powerful technique of estimating testing e�ciencyVery di�cult to apply by handVery computation-intensive � NP-complete at the very least
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 40 / 51
Regression testing What kind of testing is most widely used?
What kind of testing is most widely used?
We have ¾design for testability¿We know how to estimate testing e�ciencyAnd we have some kind of testing in our project
What kind of testing it is?
Let's answer some questions to �nd the truth
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 41 / 51
Regression testing What kind of testing is most widely used?
What kind of testing is most widely used?
Does a software change big and fast during its development?
Usually � NOWe maintain, correct, adapt and modify our software most of the timeUnfortunately, these small changes can break our software big time
What do we test during development?
We test modi�ed software in order to ensure it isn't broken by ourmodi�cationsWait... That is regression testing!
Regression testing is the main testing activity in any software developmentproject � or, at the very least, it should be
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 42 / 51
Regression testing What kind of testing is most widely used?
What kind of testing is most widely used?
Does a software change big and fast during its development?
Usually � NOWe maintain, correct, adapt and modify our software most of the timeUnfortunately, these small changes can break our software big time
What do we test during development?
We test modi�ed software in order to ensure it isn't broken by ourmodi�cationsWait... That is regression testing!
Regression testing is the main testing activity in any software developmentproject � or, at the very least, it should be
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 42 / 51
Regression testing What kind of testing is most widely used?
What kind of testing is most widely used?
Does a software change big and fast during its development?
Usually � NOWe maintain, correct, adapt and modify our software most of the timeUnfortunately, these small changes can break our software big time
What do we test during development?
We test modi�ed software in order to ensure it isn't broken by ourmodi�cationsWait... That is regression testing!
Regression testing is the main testing activity in any software developmentproject � or, at the very least, it should be
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 42 / 51
Regression testing Regression testing
Regression testing
How does it work?
1 We modify our program P into P′
2 We select tests T′that need to be run on P
′
3 We add new tests T′′that test new functionality
4 We run all these tests5 We analyze testing results
Steps (2)�(5) each contain some nasty pain that we have to bear
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 43 / 51
Regression testing Regression test selection
Regression test selection
How can we select tests that need to be run after a small change?
Conservative approach � run all tests (full regression testing)Cheap approach � run tests whose test requirements are related tothe changesSmart approach � �nd parts of program that are a�ected by thechanges and run corresponding tests
Many regression test selection techniques have been developedBut almost none of them found it's way to wide-spread use
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 44 / 51
Regression testing Regression test selection
Regression test selection
Why is that so?
We need to ensure 4 basic properties of RTS:
Inclusiveness � ability to include tests that are ¾modi�cationrevealing¿. Unsafe RTS techniques have inclusiveness less than 100%Precision � ability to omit tests that are not ¾modi�cation revealing¿E�ciency � RTS is e�cient if it takes less time to determine whattests to omit than running these testsGenerality � ability to work in most practical situations
It is very di�cult to ensure all these properties at the same timeRemember ¾Good. Fast. Cheap. Pick any two¿?
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 45 / 51
Regression testing Maintaining regression test suite
Maintaining regression test suite
Regression test suite tends to grow in time and spaceWe need a way of maintaining test suiteHow to decide what tests to add and what to remove?
Add a new test for every part of new functionalityAdd a new test for every �xed bugRemove tests that are duplicates of the other testsRemove tests that have never failed
What do we do when a test fails?
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 46 / 51
Regression testing Maintaining regression test suite
Maintaining regression test suite
There are several possible reasons of a regression test failure
The software has a regression bug � Must �x and retestThe input test values are no longer valid � Must delete or update thetestThe expected output is no longer valid � Must update the test
Usually very hard to decide which situation we have
We always have to consider adding additional resources vs removingexcessive tests
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 47 / 51
Regression testing Running regression tests
Running regression tests
That's simple � take all the selected tests and run them, right?Not quite � what order do we run them in?Does it matter?
Yes, it does:
We want to get testing results ASAPThat is why we have to prioritize tests:
Di�erent heuristicsCoverage-based prioritizationHistory-based selection
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 48 / 51
Resume Summary
Summary
Now you have a very basic glimpse of:
What software testing isHow it interacts with software developmentWhat are the sources of software testing pain and how to overcomethem
There is one piece of advice left to share with you
¾Those who cannot remember the past are condemned to repeat it¿George Santayana
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 49 / 51
Resume Main rule of software testing
Main rule of software testing
It is very simple:
Either do software testing right by the book or don't do it at allYes, it is a ¾win or lose¿ situation
If you do software testing less than 100% � you're wearing rose-colouredglasses
You have a fake feeling of safetyYou think that you ensure good software quality
You imagine you're a very good developer
Sometimes it's better not to test software at all
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 50 / 51
Resume Thank you for your time and patience
Thank you for your time and patience
Marat Akhin (SPbSPU) Software Testing 101 08/12/2009 51 / 51