Upload
nur-islam
View
377
Download
4
Embed Size (px)
DESCRIPTION
Citation preview
SOFTWARE RELIABILITY AND QUALITY
By Nur ISLAM
AIM
Categorising and specifying the reliability of software systems
Discussing various issues associated with Software Quality Assurance (SQA)
SOFTWARE RELIABILITY
WHAT IS SOFTWARE RELIABILITY?
Probability of failure-free operation for a specified time in a specified environment for a given purpose
This means quite different things depending on the system and the users of that system
Informally, reliability is a measure of how well system users think it provides the services they require
Contd…
Cannot be defined objectively Reliability measurements which are quoted out
of context are not meaningful Requires operational profile for its
definition The operational profile defines the expected
pattern of software usage Must consider fault consequences
Not all faults are equally serious. System is perceived as more unreliable if there are more serious faults
SOFTWARE RELIABILITY
What does matter most, assessing the core of the program (running 90% of the time) or non-core section (running 10% of the time) Use tools called program profilers (in Unix it is
called prof) Removing defects/bugs does not indicate
its effectiveness in increasing the reliability of the product Removing defects from non-core section does
not have the same effect as removing the ones in core section.
SOFTWARE RELIABILITY
The perceived software reliability is an observant dependent If you do not face the problem, you do
not report it Different users have different views of
the systems and thus different quality and reliability assessments
The software reliability keeps changing as the defects are detected and fixed
HARDWARE Vs SOFTWARE RELIABILITY
An important characteristic feature that sets hardware and software reliability issues apart is the difference between their failure patterns
Hardware components fail due to very different reasons as compared to software components.
Hardware components fail mostly due to wear and tear, whereas software components fail due to bugs
Reliability Metrics
Hardware metrics not really suitable for software as they are based on component failures and the need to repair or replace a component once it has failed. The design is assumed to be correct
Software failures are always design failures. Often the system continues to be available in spite of the fact that a failure has occurred
Reliability Metrics
Probability of failure on demand This is a measure of the likelihood that the system
will fail when a service request is made POFOD = 0.001 means 1 out of 1000 service
requests result in failure Relevant for safety-critical or non-stop systems
Rate of fault occurrence (ROCOF) Frequency of occurrence of unexpected behaviour ROCOF of 0.02 means 2 failures are likely in each
100 operational time units Relevant for operating systems, transaction
processing systems
Reliability Metrics
Mean time to failure Measure of the time between observed failures MTTF of 500 means that the time between failures is 500
time units Relevant for systems with long transactions e.g. CAD
systems Availability
Measure of how likely the system is available for use. Takes repair/restart time into account
Availability of 0.998 means software is available for 998 out of 1000 time units
Relevant for continuously running systems e.g. telephone switching systems
Failure Consequences
Reliability does not take consequences into account
Transient faults have no real consequences but other faults might cause data loss or corruption
May be worthwhile to identify different classes of failure, and use different metrics for each
Failure Consequences
When specifying reliability both the number of failures and the consequences of each matter
Failures with serious consequences are more damaging than those where repair and recovery is straightforward
In some cases, different reliability specifications may be defined for different failure types
Failure Classification
Transient - only occurs with certain inputs
Permanent - occurs on all inputsRecoverable - system can recover
without operator helpUnrecoverable - operator has to
help Non-corrupting - failure does not
corrupt system state or data Corrupting - system state or data
are altered
Reliability Growth Modelling
Growth model is a mathematical model of the system reliability change as it is tested and faults are removed
Used as a means of reliability prediction by extrapolating from current data
Depends on the use of statistical testing to measure the reliability of a system version
Reliability Growth ModellingStep Function Model
The simplest reliability growth model:▪ a step function model
The basic assumption:▪ reliability increases by a constant amount
each time an error is detected and repaired. Assumes:▪ all errors contribute equally to reliability
growth▪ highly unrealistic:▪ we already know that different errors contribute
differently to reliability growth.
Reliability Growth Modelling
Jelinski and Moranda Model Realizes each time an error is
repaired reliability does not increase by a constant amount.
Reliability improvement due to fixing of an error is assumed to be proportional to the number of errors present in the system at that time.
Reliability Growth ModellingLittlewood and Verall’s Model
Assumes different fault have different sizes, thereby contributing unequally to failures.
Allows for negative reliability growth Large sized faults tends to be detected
and fixed earlier As number of errors is driven down with
the progress in test, so is the average error size, causing a law of diminishing return in debugging
Reliability Growth Modelling
Applicability of models: There is no universally applicable
reliability growth model. Reliability growth is not independent of
application. Fit observed data to several growth
models.▪ Take the one that best fits the data.
Statistical Testing
The objective is to determine reliability rather than discover errors.
Uses data different from defect testing.
Statistical Testing
Different users have different operational profile: i.e. they use the system in different ways formally, operational profile:▪ probability distribution of input
Divide the input data into a number of input classes: e.g. create, edit, print, file operations, etc.
Assign a probability value to each input class: a probability for an input value from that class
to be selected.
Statistical Testing
Determine the operational profile of the software: This can be determined by analyzing the usage
pattern. Manually select or automatically generate a
set of test data: corresponding to the operational profile.
Apply test cases to the program: record execution time between each failure it may not be appropriate to use raw execution time
After a statistically significant number of failures have been observed: reliability can be computed
Statistical Testing
Relies on using large test data set. Assumes that only a small percentage of
test inputs: likely to cause system failure.
It is straight forward to generate tests corresponding to the most common inputs: but a statistically significant percentage of
unlikely inputs should also be included. Creating these may be difficult:
especially if test generators are used.
Statistical Testing
Pros and cons -Say by yourself
SOFTWARE QUALITY
What is Software Quality?
Software quality is: - The degree to which a
system, component, or process meets specified requirements.
- The degree to which a system, component, or process meets customer or user needs or expectations.
Software Quality Criteria
Correctness Efficiency Flexibility Robustness Interoperability Maintainability Performance Portability Reliability Reusability Testability Usability Availability Understandability
Software Quality Management System
A quality management system is a principal methodology used by organizations to ensure that the products they develop have the desired quality
A quality system is the responsibility of the system as a whole, and the full support of the top management is a must
A good quality system must be well documented
Software Quality Management System
The quality system activities encompass the following: Auditing of projects Review of the quality system Development of standards, procedures
and guidelines… etc Production of reports for the top
management summarizing the effectiveness of the quality system in the organization
Evolution of Quality Systems Pre WWII, the usual way to produce
quality products is to inspect the final product for defective units
Then Quality Control (QC) principle was found: focuses not only on detecting the defective products and eliminating them, but also on determining the causes behind the defects (and fixing them), so that the product rejection rate can be reduced
Evolution of Quality Systems The Quality Assurance (QA) :the basic
premise of modern quality assurance is that if an organization's processes are good and are followed rigorously, then the products are bound to be of good quality
Total Quality Management (TQM) advocates that the process followed by an organization must continuously be improved through process management
Evolution of Quality Systems
TQM requires continuous improvement (more than just documenting the process and optimizing them through redesign)
Over the last six decades the quality management has shifted from inspection to Total quality management, and the quality assurance has shifted from product to process assurance
Evolution of Quality Systems
Quality Assurance Method
Quality Paradigm
Inspection
Quality Control (QC)
Quality Assurance (QA)
Total Quality Management (TQM)
Product Assurance
Process Assurance
Product Metrics versus Process Metrics
Product metrics help measure the characteristics of a product being developed, whereas process metric help measure how a process is performing
Product metrics: LOC ,FP, PM, time to develop the product, the complexity of the system … etc
Process metrics: review effectiveness, inspection efficiency … etc
THANK YOU