Embed Size (px)
Citation preview
Get Yourself Covered
– An Introduction to Test Coverage
Barry Savell
Associate Consultant
What is test coverage?
• A map of where we’ve been
– What has been tested
– And what hasn’t
• Dynamic testing
– Actually executing code
• Testing level
– System testing
• Realistically can’t cover everything
– Component testing
Why should you care about coverage?
• Risks – To safety
– To reputation
• Show S/W is fit for purpose
• Verify the components – Have the brakes been checked?
– What needs to be checked?
– What has been checked?
• All of the components – Even the simple parts can fail
– It only takes one O-ring …
Types of coverage
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
p1>0
And
p2>0
Result = p1
• Various measures
• Differing thoroughness
• Example
– Simple code
– Logic flow diagram
Types of coverage
• Function coverage
– Function called
– p1=0
– Statements may be missed
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
p1>0
And
p2>0
Result = p1
X
Types of coverage
• Function coverage
• Statement coverage
– Each statement executed
• p1=1, p2=1
– May miss null branches
– Commonest coverage
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
p1>0
And
p2>0
Result = p1
X
Types of coverage
• Function coverage
• Statement coverage
• Branch coverage
– Each branch executed
• p1=1, p2=1
– Including null branches
• p1=0, p2=1
– More test cases
– p2>0 is not tested
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
p1>0
And
p2>0
Result = p1
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
Types of coverage
• Function coverage
• Statement coverage
• Branch coverage
• Condition coverage
– Individual Boolean sub-clauses
– Must evaluate True and False
• p1=1, p2=0
• p1=0, p2=1
– Not all branches
p1>
0
Result = p1 p2
>0
X
Types of coverage
• Function coverage
• Statement coverage
• Branch coverage
• Condition Coverage
• Condition / Decision
coverage
– Condition and Branch
• p1=1, p2=0
• p1=0, p2=1
• p1=1, p2=1
Function Fred (p1, p2)
If p1 > 0 And p2 > 0 Then
Result = p1
Endif
Return
p1>
0
Result = p1 p2
>0
Modified Condition / Decision Coverage
• Condition / decision coverage
– All conditions tested
– All branches tested
– Still not a great test
• MC/DC
– Modified Condition / Decision Coverage
– Show each condition independently
affects outcome
If (a or b) and c then
branch 1
Else
branch 2
Endif
Condition/decision coverage tests
a=true, b=true, c=true - branch 1
a=false, b=false, c=false - branch 2
a, b and c are boolean conditions (eg p1 > 0)
Modified Condition / Decision Coverage
• Vary only condition a a=true, b=false, c=true -> branch 1
a=false, b=false, c=true - branch 2
• Vary only condition b a=false, b=false, c=true - branch 2
a=false, b=true, c=true - branch 1
• Vary only condition c a=false, b=true, c=true - branch 1
a=false, b=true, c=false - branch 2
• More tests – But some are duplicates
– Only four tests 1: a=true, b=false, c=true - branch 1
2: a=false, b=false, c=true - branch 2
3: a=false, b=true, c=true - branch 1
4: a=false, b=true, c=false - branch 2
If (a or b) and c then
branch 1
Else
branch 2
Endif
b
branch 1
c
branch 2
a
1,3 4
2
3,4
2,3,4
1
Modified Condition / Decision Coverage
• Condition / decision coverage
– All conditions tested
– All branches tested
– Still not a great test
• MC/DC
– Modified Condition / Decision Coverage
– Show each condition independently
affects outcome
– More tests
– Safety related / critical systems
• Multiple condition coverage
– All combinations tested
– Many tests
If (a or b) and c then
branch 1
Else
branch 2
Endif
Condition/decision
a=true, b=true, c=true - branch 1
a=false, b=false, c=false - branch 2
MC/DC tests
a=true, b=false, c=true - branch 1
a=false, b=false, c=true - branch 2
a=false, b=true, c=true - branch 1
a=false, b=true, c=false - branch 2
Multiple condition coverage tests
a=false, b=false, c=false - branch 2
a=false, b=false, c=true - branch 2
a=false, b=true, c=false - branch 2
a=false, b=true, c=true - branch 1
a=true, b=false, c=false - branch 2
a=true, b=false, c=true - branch 1
a=true, b=true, c=false - branch 2
a=true, b=true, c=true - branch 1
Other coverage types
• Loop coverage
• Entry / exit coverage
• Parameter value coverage
• State coverage (for FSMs)
• LCSAJ / JJ-path coverage
How is coverage measured
• Modify code
– To report execution during tests
• Instrument your own code
– Manual
– Conditional compilation
• Use a test tool
– Does the modification for you
• Re-run tests
– On unmodified code
gcov
Cantata
LDMS
Parasoft
VectorCAST
IBM RTRT
Squish (Froglogic)
Test Tools:
How much coverage do you need?
• Horses for courses
– Some things will need a lot
– Some not as much
• Perhaps
• Critical systems
– Safety related
– High integrity systems
• Business systems
– But are they really less critical?
Set coverage standards
• Set coverage targets – Appropriate for your project
– For component testing
– And system testing
• What is 100% – Which coverage?
– 80% is 20% not tested - which 20%?
• A good default – At component testing
• 100% MC/DC
– At system testing • Lower percentage
• Entry/Exit coverage
Preparation
• Design for testing
– Make your job easier
– Consider test issues in advance
• Design tests
– Plan tests
– To achieve coverage targets
How to get 100%
• Measure coverage
– Using automated tool
• Analyse results
• Add tests
– If needed
• Automatic test addition
– Available in some tools
100%
Get Yourself Covered - Summary
• Maps what has been tested
• Shows S/W fit for purpose
• Various types
– Modified Condition / Decision Coverage
• Involves instrumenting code
• Supported by tools
• Set coverage targets
– Appropriate to the system
Thanks
