Upload
tamah
View
22
Download
0
Embed Size (px)
DESCRIPTION
Iterative Development of a Domain-Specific Fault Classification An Industrial Case Study. Philip Preissing and Jan Schulte 2009-06-19. Table of Contents. Introduction RUAG Context Related Work & Quality Characteristics Development Process Case Study Discussion Summary. - PowerPoint PPT Presentation
Citation preview
Iterative Development of a Domain-Specific Fault Classification
An Industrial Case Study
Philip Preissing and Jan Schulte2009-06-19
WEMSE Workshop- 2009-06-19
2/16
Table of Contents
● Introduction▬ RUAG
▬ Context
● Related Work & Quality Characteristics
● Development Process
● Case Study
● Discussion
● Summary
RUAG Aerospace Sweden AB
● Headquarter in Göteborg▬ 360 employees
● Formerly SAAB Aerospace
● Highly reliable Satellite Equipment▬ Computers
▬ Antennas
▬ Microwave Systems
● Projects▬ Ariane 5
▬ Herschel/Planck
▬ Galileo
3/16
WEMSE Workshop- 2009-06-19
4/16
Problems at RUAG
● High dependability & reliability
● ECSS Standards
● Late faults several times more expensive
Software Testing accounts for 60% of Development Time
Master thesis to develop an optimization framework for the Verification & Validation Activities (VAs)
WEMSE Workshop- 2009-06-19
5/16
Goals
● Measure defects
Gain insight into VAs
Fault-slippage between VAs
Overlap in VAs
Fault classification(FC) to group similar types of defects
▬ Analyze problems in the process on a high level
▬ Simplify the measurement process
WEMSE Workshop- 2009-06-19
6/16
Example WEMSE Workshop- 2009-06-19
7/16
Related Work
● Schemes exist▬ Orthogonal Defect Classification (IBM)▬ Standard Classification for SW Anomalies (IEEE)▬ Origins, Types & Modes (HP)▬ FC in use at Ericsson
Classifications need to be adapted (Case studies)
● Development processes▬ Expert opinion▬ Commit comments
Development process needed
WEMSE Workshop- 2009-06-19
8/16
Quality characteristics
● Classes should…
… be at most 5-10
… describe the fault type
… be orthogonal
… be consistent
… be complete
… be applicable to every software artifact
WEMSE Workshop- 2009-06-19
9/16
Development Process
1. Initial Fault Classification
2. Iterative Refinement
1. Selection
2. Classification
3. Comparison & Discussion
3. Quality Review
WEMSE Workshop- 2009-06-19
10/16
Case Study – Initial Fault Classification
Class
Assignment
Understandability
Threading/ Concurrency
Function/Interface
Hardware Interface
Timing/ Performance
Algorithms
Robustness
WEMSE Workshop- 2009-06-19
● Select data source▬ Code Inspection sheets
● Classification of faults▬ Completely sure
▬ Uncertain
▬ Don‘t know
● Analysis
Several iterations
11/16
Case Study – Iterative Refinement
10
21
375
398
0
0
27
37
7
7
20
36
42
15
4
1
42
17
38
32
0 100 200 300 400 500
Assignment
Assignment
Understandability
Understandability
Threading/Concurrency
Threading/Concurrency
Algorithms
Algorithms
Function/Interface
Function/Interface
A1A2
A1A2
A1A2
A1A2
A1A2 Uncertain
Completely
WEMSE Workshop- 2009-06-19
12/16
Case Study – Quality Review
● Agreement Factor (Consistency)
▬ Initial: 0.30
▬ Final: 0.71
● Workshop
▬ Project Manager, Designer, Developer, Tester
▬ Only minor changes
< 0.45 Inadequate agreement
> 0.62 Good agreement
> 0.78 Excellent agreement
WEMSE Workshop- 2009-06-19
Advantages & Drawbacks
+ Analysis of all stages
+ Low developer involvement
- Participants have limited understanding
- Validation first performed at the end
13/16
WEMSE Workshop- 2009-06-19
14/16
Lessons learned
● A posteriori analysis difficult
● Documentation important
● Alternative data sources difficult
▬ Checkin comments
▬ Observation of developers
▬ Interviews
WEMSE Workshop- 2009-06-19
15/16
Summary
● Software Testing very expensive
● FC necessary to analyse & improve process
● Process to develop domain-specific FC
▬ Incorporating existing schemes
▬ Iterative refinement by analyzing all steps
▬ Quality review
● Validation in industrial case study at RUAG
WEMSE Workshop- 2009-06-19
16/16
Thank you very much!
Tack så mycket!Muchas gracias!
Vielen Dank!Muito obrigado!
Mille grazie!
Merci beaucoup !
WEMSE Workshop- 2009-06-19
Moltes gràcies!