Defect Prediction & Prevention In

Automotive Software Development

11-Dec-2013

Rakesh Rana

Agenda

• Introduction: Why do we need to focus on reliability?

• Research Question

• Structure of thesis

• Reliability growth models – theory

• Defect Prediction: Chapters 2 – 4

• Defect Prevention: Chapters 5 – 6

• Conclusions

• Future research directions

Road fatalities in the EU since 2001

• 2011 more than 30,000 people died on the roads of the EU (equivalent of a medium town)

• For every death on Europe's roads there are an estimated 4 permanently disabling injuries

such as damage to the brain or spinal cord, 8 serious injuries and 50 minor injuries.

Source: EU Commission, Mobility and Transport, Road Safety; http://ec.europa.eu/transport/road_safety/specialist/statistics/

Cars: Safety Goal

“Our aim for 2020 is that no one

should be killed or seriously

injured in a Volvo”

- Thomas Broberg, Volvo’s senior safety

adviser, 2009

Source: The Volvo S60 concept, The New York Times; http://wheels.blogs.nytimes.com/2009/10/14/volvo-sets-a-lofty-safety-goal/?_r=0

http://www.industryweek.com/product-development/volvo-eyes-no-death-goal-its-new-cars-2020

"The car of the future will be just like the farmer's horse. The

farmer can steer the horse and carriage but if he falls asleep

the horse can still (get) back home. And if the farmer tries to

steer the carriage against a tree or off a cliff, the horse will

refuse"

- Anders Eugensson, Volvo's head of government affairs, Dec 2012 to Wall Street Journal

Increasing role of Software in Cars

First automotive ECU -- single-function

controller -- 1977 -- GM.

By 1981, GM was using ~50000 lines of

code across its entire domestic passenger

car production.

Today premium-class automobiles contains

~100 m lines of software code running on

70-100 microprocessors based ECUs.

Source: http://www2.teknat.uu.se/forskning/program.php?vetenskapsid=1&hforskomr=6&id=39&lang=en

http://spectrum.ieee.org/green-tech/advanced-cars/this-car-runs-on-code

“It takes dozens of microprocessors

running 100 million lines of code to get

a premium car out of the driveway, and

this software is only going to get more

complex”

- Robert N. Charette, 2009 in IEEE Spectrum

Cars and Software: Challenges

Cost of software and electronics can reach ~35 to 40% of the cost of a car.

Complexity also brings with it reliability issues.

In 2005, Toyota voluntarily recalled 160 000 -- Prius hybrids -- software problem. Time

needed to repair the software ~90 minutes per vehicle!

May 2008, Chrysler recalled 24 535 -- Jeep Commanders -- problem in the automatic-

transmission software.

June 2008, Volkswagen recalled about 4000 Passats and about 2500 Tiguans --

problem in the engine-control-module software.

Nov 2008, GM recalled 12 662 -- Cadillac CTS -- software problem.

Problems with repair and warranty

More than 50% of the ECUs that mechanics replace in cars are technically error free:

They exhibit neither a hardware nor a software problem.

50% of warranty costs (IBM).

Source: http://spectrum.ieee.org/green-tech/advanced-cars/this-car-runs-on-code

With so much more software in cars and its increasing

complexity, how do we control the reliability issue?

Cars and Software: The Big Question

Reliability

Source: A. Avizienis, J. C. Laprie, and B. Randell, “Fundamental concepts of dependability,” Tech. Rep. Ser.-Univ. Newctle. Tyne Comput. Sci., 2001.

IEEE standard 1633, recommended practice on software

reliability definition for software reliability:

(A) The probability that software will not cause the failure of

a system for a specfied time under specified conditions.

(B) The ability of a program to perform a required function

under stated conditions for a stated period of time.

Reliability

Source: A. Avizienis, J. C. Laprie, and B. Randell, “Fundamental concepts of dependability,” Tech. Rep. Ser.-Univ. Newctle. Tyne Comput. Sci., 2001.

Dependability

Attributes

Safety Reliability

Reliability

Source: A. Avizienis, J. C. Laprie, and B. Randell, “Fundamental concepts of dependability,” Tech. Rep. Ser.-Univ. Newctle. Tyne Comput. Sci., 2001.

Threats

Reliability

Means

Faults

Errors

Failures

Fault Prevention

Fault Tolerance

Fault Removal

Fault Forecasting

Research Focus & Questions

(Addressed in chapters 2, 3 & 4) (Addressed in chapters 5 & 6)

Thesis

RG1. Evaluating the applicability of

Software Reliability Growth Models

(SRGMs) in the context of automotive

software development?

RG2. Propose and evaluate methods that

can potentially increase the reliability of

software in the automotive domain

RQ1. Do SRGMs fit

defect inflow data

from automotive

domain?

RQ2. Differences

between the widely

used parameter

estimation methods?

RQ3. Which

SRGMs have the

best long-term

predictive power?

RQ4. How to use

fault injection &

mutation testing

@models?

RQ3. How to test

models better in

simulating

environment?

Prediction Prevention

Chapters 2 – 4: Defect prediction

(IEEE standard 1044)

o defect: An imperfection or deficiency in a work product where that

work product does not meet its requirements or specifications and

needs to be either repaired or replaced.

Software reliability growth models: Theory

Source: Wood, Alan. "Software reliability growth models." Tandem Technical Report 96 (1996).

Concave Models

• Assumption

• Examples: Exponential Model,

Goel-Okumoto; Musa-Okumoto

S-Shaped Models

• Assumption

• Examples: InflectionS, DelayedS,

gompertz, logistic model.

Chapter 2: Evaluation of standard reliability growth models

in the context of automotive software systems*

*Proceedings of 14th Product-Focused Software Process Improvement (PROFES) 2013, Paphos, Cyprus

RQ1. Do SRGMs fit

defect inflow data from

automotive domain?

• Objective: Do widely used software reliability growth models fit to defect

inflow data from the automotive domain?

• Method: Analytical study, we evaluated eight commonly used software

reliability growth models on defect inflow data from the automotive

domain (large project on active safety function).

• Results: While three parameter models provide good fit to the defect

data, better results can be obtained by accounting for changes in the

testing effort over calendar time.

Chapter 2: Evaluation of standard reliability growth models

in the context of automotive software systems*

*Proceedings of 14th Product-Focused Software Process Improvement (PROFES) 2013, Paphos, Cyprus

RQ1. Do SRGMs fit

defect inflow data from

automotive domain?

Where ai is actual values, pi predicted values of total number

of defects for data set of size k and q is the number of

parameters of software reliability growth model equation.𝑴𝑺𝑬 =

𝟏𝒌(𝒂𝒊 − 𝒑𝒊)

𝒌 − 𝒑

Chapter 3: Comparing between Maximum Likelihood

Estimator and Non-Linear Regression estimation

procedures for Software Reliability Growth Modelling*

*Proceedings of 23nd International Workshop on Software Measurement, IWSM-Mensura 2013, Turkey.

RQ2. Differences between

the widely used parameter

estimation methods?

• Objective: To explore applicability and practical considerations for

applying two widely recommended and used parameter estimation

methods.

– Maximum likelihood estimation

– Non-linear regression estimation

• Method: Analytical study comparing the parameter estimates obtained

from these two methods for same data set and also comparing them to

results obtained via empirical equations and those reported in earlier

study.

• Results: While MLE is the recommended estimator with superior

statistical properties, its usability and applicability in all situations is

questionable. We further provide an improvised metric (BPRE) for

comparing the predictive accuracy.

BPRE, Balanced Predicted Relative Error

Metric Relative Error

(RE)

Predicted Relative Error

(PRE)

Balanced Predicted Relative

Error (BPRE)

Formula 𝑹𝑬 =𝑷 − 𝑨

𝑨𝑷𝑹𝑬 =

𝑷 − 𝑨

𝑷𝑩𝑷𝑹𝑬 =

𝑷 − 𝑨

𝜼𝑷 + 𝟏 − 𝜼 (𝟐𝑨 − 𝑷)Where η=1 if P > A, 0 otherwise

𝑹𝒂𝒏𝒈𝒆: (−∞,∞) [𝟎, 𝟏) [𝟎, 𝟏)

Over Prediction

+20% +16.67% +16.67%

Under Prediction

-20% -25.0% -16.67%

Chapter 3: Comparing between Maximum Likelihood

Estimator and Non-Linear Regression estimation

procedures for Software Reliability Growth Modelling*

*Proceedings of 23nd International Workshop on Software Measurement, IWSM-Mensura 2013, Turkey.

RQ2. Differences between

the widely used parameter

estimation methods?

𝑩𝑷𝑹𝑬 =𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 − 𝑨𝒄𝒕𝒖𝒂𝒍

𝜼 ∗ 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 + 𝟏 − 𝜼 ∗ (𝟐 ∗ 𝑨𝒄𝒕𝒖𝒂𝒍 − 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅); 𝒘𝒉𝒆𝒓𝒆 𝜼 =

𝟏 𝒊𝒇 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 > 𝑨𝒄𝒕𝒖𝒂𝒍𝟎 𝒊𝒇 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 < 𝑨𝒄𝒕𝒖𝒂𝒍

Chapter 4: Evaluating long-term predictive power of

standard reliability growth models on automotive systems*

*Proceedings of the 24th IEEE International Symposium on Software Reliability Engineering, Pasadena,2013

RQ3. Which SRGMs have the

best long-term predictive

power?

• Objective:

– Which SRGMs fit best to the defect data from automotive software projects?

– Which SRGMs have the best long-term predictive power?

– Which models growth rates are consistent between projects over time?

• Method: Analytical study comparing seven widely used SRGMs on full & partial

defect inflow data from four large automotive software projects.

Chapter 4: Evaluating long-term predictive power of

standard reliability growth models on automotive systems*

*Proceedings of the 24th IEEE International Symposium on Software Reliability Engineering, Pasadena,2013

RQ3. Which SRGMs have the

best long-term predictive

power?

• Results:

– Which SRGMs fit best to the defect data from automotive software projects?

– Which SRGMs have the best long term predictive power?

– Which models growth rates are consistent between projects over time?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Musa-Okumoto Goel-Okumoto Inflection-S Delayed-S Rayleigh Gompertz Logistic

BPRE for different models using growth rate from project A (100% data)

PRE-100%

PRE-90%

PRE-70%

PRE-50%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Musa-Okumoto Goel-Okumoto Inflection-S Delayed-S Rayleigh Gompertz Logistic

BPRE values (average across projects) for SRGMs using full & partial data

PRE-100%

PRE-90%

PRE-70%

PRE-50%

BPRE ~100% --

too bad prediction

BPRE ~0% -- predicted total

#Defects ~Actual total #Defects

Same SRGM, using full and

partial data (forecasting)

Research Focus & Questions

(Addressed in chapters 5 & 6)

Thesis

RG2. Propose and evaluate methods that

can potentially increase the reliability of

software in the automotive domain

Prevention

Models SRGMs

Models Functional Models

w.r.t. MBD

Prediction

Chapter 5: Increasing Efficiency of ISO-26262 Verification and

Validation by Combining Fault Injection and Mutation Testing

with Model Based Development*

*8th International Joint Conference on Software Technologies - ICSOFT-EA, Reykjavik, Iceland, July 2013

RQ4. How to use

fault injection &

mutation testing

@models?

• Objective:

– How models can be used more effectively for early verification and validation?

• Method: Descriptive-Qualitative case study based on empirical observations to

propose a framework which combines methods of fault injection and mutation

testing to be used at the model level that can be used for increasing the efficiency

of ISO-26262 compliance.

Chapter 5: Increasing Efficiency of ISO-26262 Verification and

Validation by Combining Fault Injection and Mutation Testing

with Model Based Development*

*8th International Joint Conference on Software Technologies - ICSOFT-EA, Reykjavik, Iceland, 2013

RQ4. How to use

fault injection &

mutation testing

@models?

a) Assign TSRs corresponding

to FSRs to Z-outputs

b) Inject Faults (simulating

common defects) to X-inputs

c) Identify critical fault scenarios;

Study fault propagation properties;

Build fault tolerance

e) Repeat Steps (b) & (c) to

test, correct & validate

the function for its “d”

dependencies

d) Cause Mutation to “n” blocks of

function & assess effectiveness of

given test suit using M.Testing

f) Examine Mutation not killed; Update

test cases or build new to detect such

failure scenarios/defects

Chapter 6: Improving Fault Injection in Automotive Model

Based Development using Fault Bypass Modelling*

*2nd Workshop on Software-Based Methods for Robust Embedded Systems, INFORMATIK, Germany,2 013

RQ3. How to test models better

in simulating environment?

Environment Model

SW system Model

Out_1

Output

Inp_2

Inp_1

Out_2

Natural/State

parameter(s)

• Objective:

– How simulations of functional models be used effectively for early verification

and validation?

• Method: Descriptive-Qualitative case study based on experiment, we propose

and provide proof-of-concept for “fault bypass modelling”, a simple yet effective

framework for correct analysis of simulation in closed loop mode.

Chapter 6: Improving Fault Injection in Automotive Model

Based Development using Fault Bypass Modelling*

*2nd Workshop on Software-Based Methods for Robust Embedded Systems, INFORMATIK, Germany,2 013

0 1 2 3 4 5 6 7 8 9 10 11 12 13 140

10

20

30

40

50

60

Time in sec

Sp

ee

d in

RP

M

Vehicle and wheel speed with fault injection (FBM)

Vehicle Speed

Wheel Speed

0 1 2 3 4 5 6 7 8 9 100

20

40

60

80

100

120

140

Time in sec

Sp

ee

d in

RP

M

Vehicle and wheel speed with fault injection

Vehicle Speed

Wheel Speed

RQ3. How to test models better

in simulating environment?

Vehicle Speed

Control Signal

Wheel Speed

Vehicle Speed

Wheel Speed

Relative Slip

ABS Model

Environment Model

Conclusions

RG1. Evaluating the applicability of software reliability growth models

in the context of automotive software development?

1. SRGMs are able to fit the defect inflow data from automotive domain.

2. MLE Vs. NLR

– MLE has superior statistical properties.

– MLE is not applicable in all cases.

– A new metric for predictive accuracy is introduced.

3. Logistic and Gompertz model provide best fit among widely used

SRGMs, these models also provide best long-term predictive power. The

results also indicates that information (in form of growth rates) can be used

to improve the predictive power of most SRGMs.

Conclusions

RG2. Propose and evaluate methods that can potentially increase

the reliability of software in the automotive domain.

4. A framework to combine fault injection and mutation testing approach

applied at behavioural models is introduced.

5. Fault Bypass Modelling is introduced that helps to develop robust

software, a proof-of-concept is provided.

Conclusions

Why predict and prevent software defects in automotive domain?

• Predicting defects inflow help us manage defects and testing resources

effectively.

• Effective defect management and defect prevention increases reliability

of software in cars and thus cars themselves. It also translates to lower

development costs and time to market.

• Consumers get cars that are cheaper, safe and reliable.

• Lower ownership costs and it allows adding more innovations to your

cars with the help of software.

Future Research Directions

• Exploratory analysis of defect inflow data distribution from

industrial software projects

• Evaluating SRGMs and their long-term predictive power for

embedded software projects from industry

• Explore Machine Learning to Predicts Defects and Analysing Risks

in Large Software Development Projects

• SRGMs based on functional/behavioural models of software

• Measuring impact of software quality and/or software reliability

assessment on software development projects

Thank You

Back Up Slides

Source: Healing with Art, community on Facebook, https://www.facebook.com/photo.php?fbid=10151903164088141&set=a.378605758140.163024.14524668140&type=1&theater