Upload
dwight-dixon
View
215
Download
0
Embed Size (px)
DESCRIPTION
CSE SW Measurement and Quality Engineering Copyright © , Dennis J. Frailey, All Rights Reserved CSE8314M14 version 3.09Slide 3 Requirements Volatility [NOT an IEEE Metric, but Similar] Goal: Determine the stability of the requirements, so you can decide: – How far you really are in your development, – How reliable your software is likely to be, and – What type of process to use for software development
Citation preview
version 3.09
Slide 1CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
SMU CSE 8314 / NTU SE 762-N
Software Measurement and Quality Engineering
Module 14Software Reliability
Models - Part 2
version 3.09
Slide 2CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Contents• Requirements Volatility• RADC Model• Summary
version 3.09
Slide 3CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Requirements Volatility[NOT an IEEE Metric, but Similar]
Goal: Determine the stability of the requirements, so you can decide: – How far you really are in your
development,– How reliable your software is likely to
be, and– What type of process to use for
software development
version 3.09
Slide 4CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Requirements VolatilityGeneral Rules of Thumb
• If requirements are stable, use “waterfall” or similar processes
• If requirements are unstable, use incremental or evolutionary development
version 3.09
Slide 5CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Requirements VolatilityPrimitive Data Collected
R = # of original requirements– In original specification, for example
C = # of changes to original requirements
A = # of additions to original requirements
D = # of deletions from original requirements
version 3.09
Slide 6CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Requirements VolatilityEquation
V = (C + A + D) / R
•Very large V means unstable requirements
•You measure periodically to see if things are stabilizing
version 3.09
Slide 7CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Typical Graph of Volatility
0
5
10
15
20
1 2 3 4 5 6 7 8 9 10
Volatility
version 3.09
Slide 8CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Requirements VolatilityUsage Notes - 1
• In a mature development effort for a production product, V should flatten out in the design phase, indicating stabilization of requirements
• If it continues to rise, it means you have an unstable development and should not be proceeding to later phases yet, unless this is a prototype effort
version 3.09
Slide 9CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Requirements VolatilityUsage Notes - 2
• If V is large, the implication is that the current software development effort is really a requirements definition effort, which suggests a prototype, incremental or evolutionary development approach– If intended to be final development, do
not go on to next step of process yet
version 3.09
Slide 10CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Requirements VolatilityVariation
T = Number of “TBD” (“to be determined”) requirements in original specification
• This gives you more insight on changes that MUST happen (TBDs)
• It also gives more insight on stability over time
V = (C + A + D + T) / R
version 3.09
Slide 11CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Typical Graphs of V
0
5
10
15
20
1 2 3 4 5 6 7 8 9 10
With TBDs Without TBDs
version 3.09
Slide 12CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
You Can Also Learn a Lot by Graphing the Individual Factors of the Equation
R = # of original requirements– In original specification, for example
C = # of changes to original requirementsA = # of additions to original requirementsD = # of deletions from original
requirementsT = # of TBDs
version 3.09
Slide 13CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Requirements Volatility Factors
Sample GraphRequirements Volatility
020406080
100120
1 2 3 4 5 6 7 8 9
Week
C A D T R V
R
VT
C AD
version 3.09
Slide 14CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Thresholds and Targets• The nature of the development
determines what thresholds should be established
• In a supposedly stable development, thresholds for stability should be very low - instability indicates development effort may be being wasted -- lots of rework ahead
Continued ...
version 3.09
Slide 15CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Thresholds and Targets• In a development that is expected
to be volatile, thresholds might be high and targets would be established to determine when stability has been achieved.
• Historical data is essential for establishing reliable thresholds and targets
version 3.09
Slide 16CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
RADC Measurements
Rome Air Development CenterUS Air Force
Rome Air Force Base
version 3.09
Slide 17CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
RADC Measurements• These are based on a large amount
of data collected from U.S.A.F. Projects:– 5 million lines of code– 59 projects– Dating back to 1976
• 24 reliability models were studied• Used as the basis for several
government standards
version 3.09
Slide 18CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Like IEEE, These Measurements Break the
Process into PhasesPredict
ReliabilityEstimate Reliability
Start Codin
g
Release Software
Requirements
Design
CodeTest
version 3.09
Slide 19CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Background of RADC MeasurementsAssumptions:
– # of faults is fixed at the start of formal test– # of faults correlates to # of failures (failures
are easier to measure and are the things the customer cares about)
Goals:– Get the number of faults as low as possible– Predict number of faults as early as possible– Use Predictions to Manage the Process
version 3.09
Slide 20CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Basic Approach to RADC Measurements for Reliability
(one variant)• Each factor that influences reliability is
expressed as a number N 0 < N < 1
N = reliability impact of the individual factorN near 0 means it lowers reliability N near 1 means higher reliability
• The product of all these factors is the net reliability
• Each factor may be defined as the product of other, more detailed factors
version 3.09
Slide 21CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
RADC Concept
R
F1 F3F2
F21 F22 F23
R = F1 * F2 * F3
F2 = F21 * F22 * F23
Assumptions: Factors are Bayesian, Independent, Homogeneous
version 3.09
Slide 22CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Use of RADC Formula - I• At start of project, you compute R and
use it as the “current reliability prediction”
• As you go through the project, you try to improve the factors represented by the Fis, thus improving the value of R
version 3.09
Slide 23CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Use of RADC Formula - II• e.g. if F3 represents programmer
capability and it has a value of 0.6, you could improve it to 0.7 or 0.8 by hiring more capable programmers or by training your staff in defect reduction techniques
• Eventually, you base your values on actual results rather than on predictions
version 3.09
Slide 24CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Reliability Expectation Improves Throughout the
Lifecycle
0
0.2
0.4
0.6
0.8
1
1.2
Requirements PreliminaryDesign
Detailed Design Code Test Release
Historical Actual
Estimates, based on Actual Code
(RE)
Predictions, based on Factors Known at This
Time
(RP)
Goal (based on Specific
System
version 3.09
Slide 25CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Note: The Whole Thing Can Also Be Done in Terms of Other Factors
• Mean time between failures• Probability of failure• Hazard function• Defects, etc.• Regardless of how it is expressed, the
idea is to:– Set goals based on system requirements– Determine the indicators for reliability– Improve early to achieve desired goals
version 3.09
Slide 26CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
What Factors does RADC Recommend?
• RADC has studied many software development efforts and has developed a recommended set of factors to use
version 3.09
Slide 27CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Predicted Reliability
RP = A * D * S
• A, D and S are factors known before you start developing the software
version 3.09
Slide 28CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Predicted ReliabilityFactors
A = Application type– Similar to Cocomo estimation model– Worse for embedded, real time, etc.
D = development environment– Tools, turnaround, etc– Personnel capability also included
S = software development methods & process– Factors included for each phase
version 3.09
Slide 29CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
S = Software Characteristics
S = S1 * S2
version 3.09
Slide 30CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
S = Software CharacteristicsS1 = Requirements & design methods &
process– Structured Analysis, OO, etc. score higher– Less Formal techniques score lower– Process Management is a Big Factor
S2 = Implementation methods & process– Language– Coding standards– etc.
version 3.09
Slide 31CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
S1 = Requirements and Design Methods
S1 = SA * ST * SQ
version 3.09
Slide 32CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
S1 = Requirements and Design Methods
SA = Anomaly management»Corrective action process»Risk tracking and contingency»etc.
ST = Traceability»Ability to trace design to requirements,
etc.SQ = Results of quality reviews
»Number of defects found
version 3.09
Slide 33CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
SQ = Results of Quality Reviews
Design
Design Inspection
Design Repair
Design Inspection
27 defects
SQ = .6 (too low)
3 defects
SQ = .9 (OK)
to Next Phase
Note: values of SQ shown are illustrations. Actual values depend on size of code, defect definitions.
version 3.09
Slide 34CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
General Algorithm
Do Somethin
g
SiToo Low?
Go On
No
YesRedo
version 3.09
Slide 35CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
S2 = Software Implementation
Methods and Process
S2 = SL * SS * SM * SU * SX * SR
version 3.09
Slide 36CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
S2 = Software Implementation
Methods and ProcessSL = Language type– Higher order languages are better– Ada better than C due to Discipline, etc.
SS = Program sizeSM = ModularitySU = Extent of reuseSX = McCabe complexity (of Design)SR = Review results (Defects Detected)
version 3.09
Slide 37CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Examples of Improving S2
• Reduce design complexity• Reduce number of defects allowed
before exiting a review or inspection• Use a better language• Reuse proven code• Make software more modular
version 3.09
Slide 38CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Estimation Measurements(Based on Actual Code)
F = Failure rate during testingT = Test environment (for Software)E = Operational environmentDuring software test:
RE = F * T
During system test:RE = F * E
version 3.09
Slide 39CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Rqmts & Design Coding andSoftware Test
Begins
System TestBegins
Release
Diagram of Estimation Measurements
Predictive Measurements
RE = F * T RE = F * E
version 3.09
Slide 40CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
T = Test Environment
TE = Test Effort -- amount of work spent testing
TM = Test Methods -- sophistication, etc.
TC = Test Coverage -- percent of paths tested, etc.
T = TE * TM * TC
version 3.09
Slide 41CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
E = Operating EnvironmentEW = Workload
EV = Input Variability
RE = F * E = F * EW * EV
version 3.09
Slide 42CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Summary of RADC Measurement Usage - At Start
of Project1) Establish a reliability goal based on the
objectives of the product2) Using organization’s history, characterize
the process in terms of the correlation between expected defect or reliability level and the various RADC parameters
3) Use this information to predict the reliability or defect level and use that information to affect the planning process– e.g. use to decide what language, CASE tools,
etc.
version 3.09
Slide 43CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Summary of RADC Measurement Usage - During
Project Execution4) Track actuals and compare with
historical data and plans5) Adjust behavior if actuals are
inadequate to meet goals6) Record actuals for future use
version 3.09
Slide 44CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
Summary• Requirements volatility is easy to
measure early in the project and it can give you a useful prediction of reliability and stability
• RADC and IEEE reliability models use different techniques for different phases of development
• RADC uses facts about the development process to predict reliability
version 3.09
Slide 45CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
References• Bowen, C., et al., Methodology for Software
Reliability Prediction, RADC-TR-87-171, Vol I & II, Rome Air Development Center, 1987.
• Lyu, Michael R., Handbook of Software Reliability Engineering, IEEE, 1996, Catalog # RS00030. ISBN 0-07-039400-8.
• Musa, John, Software Reliability Engineering: More Reliable Software, Faster Development and Testing, McGraw Hill. ISBN: 0-07-913271-5.
• Xie, M. Software Reliability Modeling, World Scientific, London, 1991. ISBN 981-02-0640-2.
version 3.09
Slide 46CSE 8314 - SW Measurement and Quality EngineeringCopyright © 1995-2003, Dennis J. Frailey, All Rights Reserved CSE8314M14
END OFMODULE 14