Upload
barnaby-barnett
View
214
Download
1
Tags:
Embed Size (px)
Citation preview
Testing
MSO 08/09, WP
Why testing ?
Obvious.
Still, how many of you actually test your assignment before submitting them?
How do you know if you have tested enough??
How much testing really cost?
Seemingly a lot, since there are plenty of software testing companies in NL; they seem to make good business!
2
Some numbers
US financial sector suffers 1.5 billion USD (2002) due to inadequate testing (US Nat. Inst. of Standards and Tech.)
Testing cost 30 – 70% of total development cost! (various studies)
Effort to find a bug (Ellims et all 2004, JTest)
at system testing 12 – 20 hrs (600 Eur per bug!)
at unit testing 0 – 3 hrs (90 Eur per bug)
3
Why is it so hard ?
Size
At the system level you can be dealing with millions LOCs!
OO adds a multitude of other complications.
Concurrency
4
Plan
Unit test, integration test, system test OO issues, concurrency issues Testing with JUnit Adequacy of your tests Assignment-7 Specification-based testing Regression testing
5
Dennis et al only discuss testing very globally
Taxonomy of testing
According to the "scope" : Unit testing Integration testing System testing
According to ... "other things" : Functional testing GUI testing Acceptance testing Performance testing Regression testing
6
Unit Testing
Testing each "program unit" in isolation.
Manageable size; finding bugs are easier at this level.
Widely supported by xUnit tools.
Automated tools e.g. T2.
We'll take a look at Junit.
7
Unit Testing
What is "unit" ? Options : method class package
Typically people take method as "unit"
But you should be aware that in OO methods can influence each other through their side effects on objects' states.
This implies that there are methods that should not be tested in isolation.
OO issues complicates testing
Intra-class side effect makes it difficult to test each method in isolation:
9
class Subscription { int price ; int n = 12 ;
public Subscription(p) { price = p }
public int pricePerMonth() { return price/n } public cancel() { price=0; n=0 }}
OO issues
We fix Subscription and find no more error. So, now we're sure that every client can safely call "pricePerMonth".
Watch out!! Inheritance adds another complication
10
class client {
m(Subscription s) { s.cancel() ; return s.pricePerMonth() }
class EvilSubscription extends Subscription {
public int pricePerMonth() { return 0/0 } ...
This implies that whenever you override a method, you need to test if you still respect the spec of the original method (Lipkov substitution).
Integration test
Test if a set of "units" (that logically belong together) functions expectedly.
But what is a "set of units" ? Class or package
Class level testing. Given a class C, check that
Lots of work to do by hand; use automated tool (T2).
11
Various sequences of calls to C's methods and updates to C's exposed fields do not violate C's class invariant and the specs of each method.
Integration test
Package level testing.
12
Subscription
price...
Customer
SubsctiptionPackage
classinv price 0Service
Potential problem: if there are fields in Subscription which are open to package-level access, but their values are constrained by a class invariant.
Implying you have to test that every method in the package repects the class invariant of every class in the package.
Concurrency Concurrent threads multiply the number of possible
scenarios to consider:
This implies you need lots of tests to cover all scenarios expensive!!
But even that is not possible without special tools: We can't control JVM to do a specific scheduling. This also implies that you can't replay an error in a
concurrent setup!
13
thread 1 : { price++ ; price++ }thread 2 : { price = 0 }
Will this satisfy : 0 price 2 ?
And multi-cores are coming! (shudder)
The famous V model of testing
14
Image is taken from Wikipedia.
V model of testing
15
This approach assumes a water-fall SDLC model.
SystemRequirement
SystemSpecifications
Package/classlevel
specifications
Implementation unit testing
integration testing
system testing
acceptancetesting
But more generally phases can run parallel.
System test System testing tests if a software as a whole functions as
expected.
Problem: at this level a software often has to be interacted through its (complex) GUI.
Testing through GUI is HARD to write and hard to automate!!
Alternative: split system testing to Functional system testing Separate GUI testing
16
Functional System Testing
You need a well defined interface (set of methods) through which we can directly access a software functionality (without going through the GUI).
We will test through this interface.
Ideally this interface is prepared all the way from your design phase.
17
Test Interface
GUI
Business Logic
Test Plan
For a large application you (obviously) need a test plan.
Because it involves lots of activities (and huge investment).
IEEE 829, test plan is a document:
1. describing scope, approach, resources, and schedules of testing activities.
2. identifying test items, features to test, testing tasks, who will do the tasks, risks, and contingency plans.
18
Test Plan
IEEE 829
c) Approach d) Test items e) Features to be tested; features not to be tested g) Pass/fail criteria h) Suspension and resumption criteria i) Test deliverables
Oh well, a test plan essentially: Defines the scope of the testing Explains how we do the testing Estimates the resources needed
19
(Unit) Testing with Junit
20
C Ctest
Junit
report
Java
public class Subscription {
private int price ; private int length ; ... public double pricePerMonth() { double r = (double) price / (double) length ; return r ; }}
Testing with Junit
21
C Ctest
Junit
report
Java
import org.junit.* ;import static org.junit.Assert.* ;
public class SubscriptionTest {
@Test public void test_if_pricePerMonth_returns_Euro() { Subscription S = new Subscription(200,2) ; assertTrue(S.pricePerMonth() == 1.0) ; }
@Test public void test_if_pricePerMonth_rounds_up() { Subscription S3 = new Subscription(200,3) ; assertTrue(S3.pricePerMonth() == (double) 0.67) ; }}
Read the Tutorial, linked from Opdracht-7.
JUnit with Eclipse
22
JUnit with Eclipse
23
Have we tested enough ?? In principle, no way to know that.
Still, we can use "coverage" as indication:
The precentage of "parts" of your code that are executed during your tests.
Keep in mind: 100% coverage does not imply absence of bugs!!
But low coverage is obviously not good.
24
Various concepts of "coverage"
Very abstract: Class coverage (have we tested all classes?) Method coverage (have we tested all methods?)
Code level: Line coverage Branch coverage Path coverage
25
Line, branch, and path coverage
Line coverage: percentage of lines executed by your tests. E.g. a test (1x) with
s.period = 0 s.price = 0
will give you 100% line coverage. Very weak. But in practice is used most
26
discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d }
Line, branch, and path coverage
Branch coverage: percentage of branches (of "if", "while" etc) executed by your tests.
E.g. the tests (2x)
s1.price=0 s1.period=0s2.price=2000 s2.period=2000
gives 100% branch coverage.27
discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d }
s.period > 12
s.price > 1000
But you still miss the "red" case!!
Line, branch, and path coverage
Path coverage: percentage of paths in the "control flow graph" that your tests pass through.
There are 4 paths. You will need 4 tests to cover them all. Much stronger. Complication with loop & recursion take e.g. "prime
paths".
28
discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d }
s.period > 12
s.price > 1000
Measuring Coverge in Eclipse
29
Read the Tutorial, linked from Opdracht-7.
Assignment 7 : Foo Subscription System
30
Foo architecture in more detail
31
Foo architecture in more detail
32
Discount Abstract class DiscountStrategy
description() calcDiscount(customer, activeDiscountStrategies)
Two concrete implementations: Discount_5top Discount_10pack
TASK 1: test these two classes (unit testing) Verify against the specifications (of the above two methods) 100% (emma) coverage!
33
Problem with traditional tests
We need concrete data as inputs and oracles need manual effort.
This is very fragile; if we change the functionality a bit, you'll have to re-calculate them.
34
@Test public void test_if_pricePerMonth_returns_Euro() { Subscription S = new Subscription(200,2) ; assertTrue(S.pricePerMonth() == 1.0) ; }
@Test public void test_if_pricePerMonth_rounds_up() { Subscription S3 = new Subscription(200,3) ; assertTrue(S3.pricePerMonth() == (double) 0.67) ; }
Alternative: specification-based testing We'll write specifications in-code (in Java). Alternative: use JML or OCL. You'll need additional tools. Your need to train your programmers in them.
TASK 2 : (optional, 1.5 pt)
Write specifications for ApplicationLogic, and do specification-based testing.
We'll limit to the methods addCustomer and removeCustomer of ApplicationLogic.
35
Architecture
36
ApplicationLogic
ApplicationLogic_SPEC
addCustomerremoveCustomerclassinv
ApplicationLogic_SPEC_test
app int addCustomer(String name, String email) {
... // pre-condition
int result = app.addCustomer(name,email) ;
.... // post-condition
return result
}
Example of in-code specification : spec-1
37
Suppose this is the spec of addCustomer:
"Name and email should not be null nor empty (pre-cond). If name does not occur in the system, then add the customer, return +1. Else don't add, return -1."
int addCustomer(String name, String email) {
assertTrue(name != null && email != null && ! name.equals("") && ... )
boolean occur = false ; for (String n : getCustNames()) if (n.equals(name)) { occur = true ; break }
int result = app.addCustomer(name,email) ;
if (occur) assertTrue(result == 1) else assertTrue(result == -1) ;
return result }
Example of in-code specification : spec-2
38
ApplicationLogic
ApplicationLogic_SPEC
addCustomerremoveCustomerclassinv
app
Suppose we want to impose this class invariant on the application logic:
"The value of the variable NextCustId should be non-negative."
boolean classinv() {
assertTrue(app.nextCustID 0) ;
return true ;
}
Examples of the tests
39
import org.junit.*;import static org.junit.Assert.*;
public class ApplicationLogic_test {
@Test public void test1() { ApplicationLogic.reset() ApplicationLogic_SPEC applogic = new ApplicationLogic_SPEC()
assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","[email protected]") assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","[email protected]") assertTrue (applogic.classinv()) }}
Advatanges of spec-based testing
No (fragile) oracle! But we still have to manually write the test sequences and inputs. Can be
automated! e.g. with T2 You must combine this with pair-programming. Disadvantage:
writing specs takes time Incure maintenace cost.
40
public void test1() { ... assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","[email protected]") assertTrue (applogic.classinv()) applogic.addCustomer("Pinky","[email protected]") assertTrue (applogic.classinv()) }
Regression test
The re-testing of a software that has been modified. Is a serious problem for large software may take days! Between versions, the changes are usually very limited (to
some modules). However, the impact may be global due to interactions
between modules. This forces you to test the entire software. That is, you need
a test set that will deliver full coverage over your entire software.
But if we just re-execute the whole set of tests collected so far, this will take too long!!
Challenge: how do we identify a good minimum set?
41