INFO 636 Software Engineering Process I Prof. Glenn Booker

www.ischool.drexel.edu

INFO 636 Software Engineering Process I

Prof. Glenn Booker

Week 9 – Quality Management

1INFO636 Week 9

www.ischool.drexel.eduINFO636 Week 9 2

Software Quality Management

• The PSP focuses on managing defects to produce a quality product

• This, in turn, helps improve productivity– Productivity still falls as the size of a system

increases, due to its increasing complexity

• To improve further, must improve your process quality


What is Quality?

• Crosby considered software quality to be “conformance to requirements”

• Meeting user needs is often key to determine quality

• We can distinguish between the quality of the product created, and the quality of the process used to create it


Product Quality

• A software product must provide the functions defined in its requirements, and must be a functioning application– Beyond that, how sensitive the user is to other

aspects of usability, reliability, compatibility, etc. varies wildly

– Poor quality products tend to have a lot of focus on testing to find defects


Process Quality

• Process quality focuses on meeting the developer’s needs

• A quality process supports creating good products– Process consistency and repeatability are

generally critical– Efficiency and usability are less obvious

objectives


Cost of Quality

• Quality is a big cost issue for software• Each defect has costs for

– detecting it– finding the source– determining a solution– fixing the problem (req’ts, design, and/or code)– reviewing the fix– testing the fix to see if it works– testing the fix to check for regression problems– updating documentation to reflect the fix


Cost of Quality

• The relative cost of finding and correcting a defect increases through the life cycle– During requirements or design phases,

a defect has a relative cost of about 1– Coding and testing phases, cost 10-40– After release to the customer, cost 100-1000


Cost of Quality

• Many of you also noted that the time to fix defects found in testing or later is much higher than those found earlier (see pp. 276-7)

• PSP experience has shown that reviews are twice as efficient at finding defects as testing


Testing Schedules

• Many organizations focus on getting into system testing quickly, as a sign of progress

• However, more time in requirements and design would produce higher quality code which needs less testing time


Cost of Defect Removal

• Typical developers inject (create) about 100 defects per thousand LOC– About half are found by compiler– The rest need to be found by inspection or

testing

• Using the PSP, about 70% of defects entering inspection are found


Cost of Defect Removal

• Inspection and review average about ½ hour per defect, whereas testing averages 8-10 hours per defect – so the cost benefit of inspection becomes clear


An Example

• Consider a 50,000 LOC project– About 5000 defects are initially created, but

2500 are found by the compiler (and are presumably fixed immediately)

– Without reviews or inspections, those 2500 defects would take at least 2500*8 = 20,000 hours to find and fix using testing alone


An Example

– Using inspections, we would expect to find 1750 of those defects, and fix them in ½ hour each for 875 hours

– The remaining 750 defects would be found by testing, and fixed in 750*8=6000 hours

• Total defect removal time with inspection and test is 6,875 hours, versus 20,000 hours for testing alone– That saves $1.3M at $100/hr


Cost of (Low) Quality

• Poor quality can be measured to determine its impact on the project– Failure cost = the cost of finding and repairing

failures– Appraisal cost = the cost of conducting

reviews (called), and testing which finds no defects

– Prevention cost = the cost of preventing the failures from occurring


Cost of Quality (COQ)

• For the PSP we simplify these to– Failure cost = time for compile and test– Appraisal cost = time for reviews and

inspections– Prevention costs are low, since we have few

projects to examine for defect trends

• Based on this, define COQ measures


COQ Measures

– Failure COQ = 100*(compile time + test time)/(total devel time)

– Appraisal COQ = 100*(design review time + code review time)/(total devel time)

– Total COQ = Failure COQ + Appraisal COQ– Appraisal cost as % of total quality =

100*(appraisal COQ)/(total COQ)– A/FR ratio = Appraisal COQ/(Failure COQ)


COQ Measures

• The purpose of these measures is to help look for trends in quality over several projects– Hope to see higher ‘appraisal cost as % of

total quality’ over time– This should lead to lower defect rates for later

projects


Quality Strategy

• Important to keep product and process measures separate – both are important

• Avoid productivity measures by themselves – they can help with planning, and be compared to other measures, but don’t use them as a sole objective


Process Yield

• Total process yield is helpful – the percent of defects removed before test

• You can also evaluate yield for every life cycle phase, but only after hidden defects have been later discovered

• The life cycle is a filtering process– Each phase has some kind of step to help find

and remove defects to keep from feeding them into the next phase


No One Answer

• The PSP is a personal process for many reasons – one is that there is no one best way to find and remove defects– Need to look for the best ways for you to

improve your work– Then reevaluate your strategy periodically


Benchmarking

• It can be helpful to compare your product and process measures to other people’s, or to industry standards– For example, plot the A/FR ratio over time

• It should increase (p. 288)

– Or look at A/FR versus the yield– Or Test defects per kLOC versus A/FR


Benchmarking

• Productivity can be compared to other measures, such as yield, to look for trends

• Yield acts like the filter part of the life cycle process– Each step’s yield reduces the number of

defects to be found later, so a small improvement in yield can be significant


Injection and Removal

• The key to understanding your software development process is to balance the injection of new defects with activities to detect and remove them

• To achieve greater control, need to examine the factors which cause defects to occur


Defect Causes

• Education – didn’t know how to do it• Communication – weren’t informed about

something• Oversight – forgot to do something• Transcription – knew what to do, but made

a mistake• Process – the process misled your actions


Yield Management

• While the overall process yield doesn’t care if you have 80% review yield, and 40% test yield, or 40% review and 80% test, the costs and A/FR ratio of those situations are quite different

• This is why so much emphasis is placed on the quality of early reviews (see recommendations on pp. 298-300)


Defect Prevention

• Finding and fixing defects is purely defensive

• To be more proactive, need to prevent defects from being created

• Analyze the types of defects which occur frequently in your experience

• Determine the causes of common defect types (see slide 24 again)


Defect Prevention

• Then look for ways to prevent those defects from being made again, e.g.– Improve review checklists– Get additional instruction – Conduct more detailed design– Get independent reviews

• And change your processes accordingly


Defect Prevention

• The critical follow-up is to re-examine your defects later, and see if your changes helped or not

• Incidentally, defect prevention and continuous process improvement are the highest levels of process maturity– CMM and CMMI level 5 activities

Documents

INFO 636 Software Engineering Process I Prof. Glenn Booker