Upload
rakesh-rana
View
69
Download
2
Embed Size (px)
Citation preview
Software Defect Prediction Techniques in the Automotive Domain: Evaluation, Selection and Adoption
Rakesh Rana
Slide | 01
Outline:
Part 1: Introduction
Part 2: The Main Problem
Part 3: Research Questions & Methodology
Part 4: Scientific Contributions
Part 5: Industrial Impact
Part 6: Conclusions
Agenda
Slide | 02
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Software (SW) in Automotive Domain
“It takes dozens of microprocessors running 100 million lines of code to get a premium car out of the driveway, and this software is only going to get more complex”
- IEEE spectrum
Driving Factors:
o Size & Complexity: >1 GB of software distributed over ~100 ECUs.
o Cost of SW & Electronics: for today’s premium carso ~40% of a vehicle’s development costs are
determined by electronics and software, and o 50 - 70% of the development costs for an ECU
(Electronic Control Unit) are related to software.
o Innovation & Trends: today, ~90% of all innovations are driven by electronics and software.
V&VFunctionalityMixed Blessing
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Slide | 03
S. Fürst, “Challenges in the design of automotive software,” in Proceedings of the Conference on Design, Automation and Test in Europe, 2010, pp. 256–258.R. N. Charette, “This car runs on code,” IEEE Spectrum, vol. 46, no. 3, p. 3, 2009.
Software Defect – Definition
Defect: An imperfection or deficiency in a work product where that work
product does not meet its requirements or specifications and needs to be
either repaired or replaced.
Error: A human action that produces an incorrect result.
Failure: (A) Termination of the ability of a product to perform a required
function or its inability to perform within previously specified limits Or
(B) An event in which a system or system component does not perform a
required function within specified limits.
Fault: A manifestation of an error in software.
Problem: (A) Difficulty or uncertainty experienced by one or more
persons, resulting from an unsatisfactory encounter with a system in use
or (B) a negative situation to overcome.
IEEE standard 1044, Classification for Software Anomalies
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Slide | 04
Software Defect – an example of defect description
Information type Attribute (example) Example defect description
Basic information Unique ID SWI-1234
Date & time stamp 12-Jul-201X, 16:33:04
Problem StatusOpen/ Resolved/ Closed/
CancelledOpen
Severity Major/ Minor/ Insignificant Minor
Problem TypeRequirement defect/ Design
defect/ Code defect Code defect
Title and Description
Title Misplaced white pixels on the home screen
Description
Precondition
System is up and running.
Action
Enter home screen in default mode (all tiles in standard size).
Note! This error is only applicable first time the user enters the home screen.
Once tiles in the home screen have been expanded, the misplaced pixels
disappear.
Observation
In the middle of the home screen there are some white and grey pixels in line
with the top of the phone tile (between the phone tile and the media tile).
Expected
There should not be any colour deviation of “stains” on the background screen.
Probability
100%
Additional Information Attachments Attached Log file
Comments
Update 201X-08-26:
When entering and exiting the Settings list, the misplaced pixels reappear, even
if tiles in the home screen have previously been expanded (and shrunk back to
default tile size again).
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Slide | 05
Applying SDP Techniques in the Automotive Domain
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Main Objective
1. Optimize allocation of test resources:
o Predicting the shape of defect inflow profile, and
o Predicting expected pre-release defect count for an on-going project.
Help experts within the automotive domain to:
2. Assess release readiness of software before release:
Slide | 06
Applying SDP Techniques in the Automotive Domain
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
The Main Problem
1. SW Development Process
o SW sourced through suppliers.
o Model-Based Development, and
2. Evaluation
Slide | 07
Research Questions
RQ1: Which defect prediction techniques are applicable at different stages of software life cycle in the automotive domain?
RQ2: How software reliability growth models can be used for defect predictions in the automotive domain?
RQ3: What are the consequences of mispredicting total number of defects and release readiness?
RQ4: How can correlation based models be used for defect prediction in the automotive domain?
RQ5: How to evaluate reliability characteristics of software at early stages of development using only behavioural models?
RQ6: What are important factors for industrial adoption of machine learning based defect prediction models?
To evaluate how software defect prediction techniques can be effectively applied over the software development life cycle within the automotive domain.
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Slide | 08
Des
crip
tive
An
alyt
ical
Fun
dam
enta
l
Ap
plie
d
Qu
anti
tati
ve
Qu
alit
ativ
e
Co
nce
ptu
al
Emp
iric
al
Des
crip
tive
-
Rev
iew
of
liter
atu
re
Des
crip
tive
oth
erEv
alu
ativ
e-
ded
uct
ive
Eval
uat
ive-
inte
rpre
tive
Form
ula
tive
-
mo
del
Form
ula
tive
-
fram
ewo
rkFo
rmu
lati
ve-
guid
elin
esC
on
cep
tual
anal
ysis
Lite
ratu
re
revi
ew/a
nal
y
Cas
e st
ud
y
Act
ion
rese
arch
Dat
a an
alys
is
Sim
ula
tio
n
Fiel
d s
tud
y
Paper 1
Paper 2
Paper 3
Paper 4
Paper 5
Paper 6
Paper 7
Paper 8
Paper 9
Paper 10
Paper
No
Research approachResearch type Research methods
Research Methodology
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Slide | 09
Research approach and methods are classification based on: R. L. Glass, I. Vessey, and V. Ramesh, “Research in software engineering: an analysis of the
literature,” Information and Software technology, vol. 44, no. 8, pp. 491–506, 2002.
Research type classification is based on: C. R. Kothari, Research Methodology: Methods and Techniques. New Age International, 2011.
RQ1: Which defect prediction techniques are applicable at different stages of
software life cycle in the automotive domain?
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Main contributions:
o Overview of automotive SW development life cycle for large platform projects,
o Seven different SDP techniques were mapped according to the development phase they can be applied,
o Mapped SDP techniques were also classified based on the purpose and application level, and
o Roadmap for increasing the efficiency of defect predictions by using field data.Production Software
Slide | 10
RQ2: How software reliability growth models can be used for defect
predictions in the automotive domain?
Evaluation:
o Two commonly used parameter estimation procedure (MLE, and NLR) were compared,
o Eight SRGMs were evaluated at function level (best: logistic Model),
o Eight SRGMs were further evaluated at Project/Release level using defect data from three different companies from embedded software domain (best: logistic & Gompertz Model),
o How to use historical projects data to make defect count prediction more accurate,
Selection:
o Proposed method for predicting shape of defect inflow profile, and
o The statistical distribution family of defect inflow data was also explored where beta distribution was shown to be prominent family among tested distributions.
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Slide | 11
Defect inflow intensity until half-way through the project Predicted shape of
defect inflow profileOverall trend Trend after reaching local maximum
Increasing Decreasing S-shape
Decreasing Decreasing Concave
Increasing Increasing Convex
ConcaveS-shape Convex
Empirical Evaluation at Project/Release levelIntroduction
The Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
RQ2: How software reliability growth models can be used for defect
predictions in the automotive domain?
Slide | 12
Predicted
shape of
defect inflow
profile
Recommended SRGMs
For testing
resource(s)
allocation
For release readiness
assessment using
current project data
S-shape Logistic Logistic
Concave Gompertz Gompertz
Convex Delayed-S Logistic
Recommendation on applying SRGMs for SDP in industry:
o Collect the right data from testing,
o Perform exploratory analysis of data,
o Define goals for applying SRGMs,
o Select appropriate model based on predicted shape/distribution, and
o Use historical information when available
Introduction
The Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
RQ2: How software reliability growth models can be used for defect
predictions in the automotive domain?
Slide | 13
Introduction
The Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
RQ3: What are the consequences of mispredicting total number of defects and
release readiness?
Consequences of mispredictions:
o Two axis of accuracy of prediction were explicitly identified –
1) the prediction of the asymptote or the total number of defects, and
2) when the total number of defects are discovered or the release readiness.
o Consequences of mispredictions the shape of defect inflow profile were examined, and
o The consequence model was validated at two companies which provided insights into which mispredictions are critical in industry and how these are managed.
Slide | 14
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Empirical Evaluation at Function (Feature) and ECU level
o Confirmed the Pareto principle of defect distribution (20-60 rule) for automotive software projects,
o Correlation between defects count in consecutive integration points - not found to be strong, and
o Evidence for strong correlation between number of defects found at IP4/IP5 with total pre-release defect count.
RQ4: How can correlation based models be used for defect prediction in the
automotive domain?
Slide | 15
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
RQ5: How to evaluate reliability characteristics of software at early stages of
development using only behavioural models?
Evaluation at Model/Code level
o Behavioural models in MBD offers significant opportunity to do reliability analysis early in the development process,
o Proposed a framework for early identification of design flaws and evaluating the efficacy of test suite to detect potential implementation defects that can potentially violate the safety goals, and
o The framework was validation to provide a proof-of-concept - 2 out of 24 mutants produced unexpected results and exposed the deficiency of the current test protocol.
Slide | 16
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
RQ6: What are important factors for industrial adoption of machine learning
based defect prediction
Adoption:
o Proposed a framework for adoption of machine learning based techniques in industry and its adaptation for software defect prediction,
o Identified nine main factors and various sub-attributes that affect the decisions of adoption of new techniques for SDP in the industry, and
o The framework was validation at two companies and specific challenges for adoption were identified.
Slide | 17
Software Reliability Growth Models (SRGMs)
Analogy Based Predictions
Correlation Based Models
Reliability analysis of Simulink Models
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Supporting experts with data based model
o Applying different SDP models based on testing data provides empirical evidence and predictions that support experts in making their own forecasts,
o Data based models brings more objectivity in the predictions thus helping experts to potentially avoid some biases.
Slide | 18
Func Increase Calib/Optimization
SRGMs
Correlation
Simulink Model Analysis
ML
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Efficient resource planning (cost/release timing)
o Automotive SW development is characterised by high time pressure/constraints, and
o Large part of verification & validation at vehicle level is done in late stages of development life cycle, thus:
o Reliability analysis early in development life-cycle can help in developing robust SW, and
o SDP about mid-way can provide useful estimates for efficient resource planning of testing resources.
Slide | 19
Introduction
The Main Problem
Research Questions & Methodology
Scientific Contributions
Industrial Impact
Conclusions
Conclusions
Slide | 20
Software Defect Prediction Techniques in the Automotive Domain:
Evaluation:
1. SRGMs To support test resource allocation decisions, For assessment of release readiness, and Using historical data to make better predictions.
2. Correlation based models For predicting defect count in next integration point, and For pre-release defect count predictions.
3. Behavioural models For early reliability analysis.
Selection:
1. How appropriate SRGMs could be selected by predicting the shape of defect inflow, and2. Using defect inflow distribution family to select appropriate SRGMs.
Adoption:
1. Explored factors that play important role for adoption (of ML based SDP techniques),2. Adoption framework is expected to help with wider adoption of such techniques.