What we will do / Structure
• Subjects:>Test Metric goals / stakeholders>Which metrics; some demo>Process around metrics>How to apply metrics / pittfalls>Wrap-up
• For each subject: intro, discussion• Discussion
>hand raise>hand raise with comments>open
Metric Goals / Stakeholders
• Hand raise only1. Who is using project metrics?2. Who is using test metrics?3. Who would like to use test metrics?
• Some opinions please!• Goals / Stakeholders for test
metrics? OPEN!• Your votes please
Which test metrics?
Test basis
Test object
Test Execution Defects
Repair
Production
Specifying test
cases / scripts Test cases/
scripts
Test Process
Size test basis
Size test object # defects in
test object# defects in
production
For each process:# hours effortlead time
# test cases
# = number of
# defects in test basis
# repair rounds
Deductible metrics
• Effort = Size /# hours• Productivity = # test cases / # hours• Efficiency =
# defects / (# hours or # test cases)> Specification> Test execution> Retest of repaired defects
• DDP Defect Detection Percentage (Europe)DRE Defect Removal Efficiency (USA)
• Defect injection rate for rework• Damage prevented?• More metrics:
> Availability of test environments> OPEN
Dutch test metrics experiences
• Dutch initiative to gather test metrics
• Parties involved>NESMA
Netherlands Software Metrics Association
>Testnet, including SogetiDutch Testing community
>LaQuSOLaboratory for Software QualityUniversities Eindhoven & Nijmegen
Some slides with data from the database will follow in the presentation, not in the hand-out, due to a lack of statistical evidence
Processes around metrics
• Collection in a project> Embedded in daily work> Weekly summarisation> Sanity checks> Cost: about 2% project budget
• Distribution• For a benchmark on the level of:
> Project releases> Organisation> Country> International: ISBSG
International Software Benchmarking Standards Group
To what extent?
• OPEN Discussion about>Accuracy of definitions>Number of types of defects>Is a batch test case the same as an
online test case?>Only testing of functionality or also
security, performance, usability>How to include regression testing?>Predictive value
average (mean), median, standard deviation, correlations with?Prediction model needed?
To apply test metrics
• Project>Estimating Testing New development>Estimating Testing Maintenance release>Controlling the test project
• Benchmark>Measure the effectiveness of the testing,
Improve inside / outside the test process?
• PITTFALLS?
Make a judgement based with metrics?
• Of person production/ quality of work>OPEN Discussion>Some Dutch effects
• Of Service Level Agreement>What to measure and how to judge
Wrap-up
• Business value out of metricsHand raise> Who would like to use test metrics?
• Changed opinions> Comments please (- + ,+ -)
• Reading: TMap Next> Chapter 13 Goal-Question-Metrics for test> Chapter 11 for data
• End of 2008: www.isbsg.orgParticipate? Leave your business card