5 Quality

SoberIT Software Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Errata!   “Estimation of Software” has a bug!

  “The multiplier for high complexity and external output type is higher than the multiplier for high complexity and external input type.”

  Instead, you should look up the values from the table for external inquiry type, as with other user types

  The reason:  Some descriptions of Albrecht function point method have

instructed to do as described in the material, but this suggestion has been changed later.



Lecture on February 16th   Student presentations!   Prepare a presentation of 15-20 minutes on topic of your interest

(related to software project management), and get 5 lecture summary points

  And/or   Attend the session, participate in discussions and share your own

experiences, and earn max 5 lecture summary points by writing a summary of the presentations & discussion

  If you want to give a presentation, please send email to [email protected]



T-76.5612 Software Project Management

5: Quality in Software Project Management

Tuomas Niinimäki (many slides by Juha Itkonen & Mika Mäntylä)

Helsinki University of Technology


HELSINKI UNIVERSITY OF TECHNOLOGY 4

A Plan



The Reality



What quality practices can do for my project?   Provide headlights (Information on quality)

  Where are we?   … and what lies ahead?

  Enable changing our plan   We can get there, but later than planned   We can get almost there and stick to our

schedule   But if we run full steam ahead according

to our plan we will definitely crash



Contents   Setting quality goals   From goals to quality practices   Quality information in project planning and tracking



Quality Goals



Strive for good quality

Bad quality leads to rework, higher costs, delays, and

problems in operation

9



What is quality (in software)?



ISO 9126 Quality attributes   Functionality

  Suitability   Accurateness   Interoperability   Security

  Reliability   Maturity   Fault tolerance   Recoverability

  Usability   Understandability   Learnability   Operability   Attractiveness

  Efficiency   Time behavior   Resource behavior

  Maintainability   Analyzability   Changeability   Stability   Testability

  Portability   Adaptability   Installability   Conformance   Replaceability

+ suggested metrics for each quality attribute



PRODUCT PEOPLE PROCESS

QUALITY



PRODUCT PEOPLE PROCESS

  Product is error-free

  Product meets the specifications

  Product has certain measurable attribute   e.g. durability,

weight, SMS support

  The expectations of the customer or user have been met

  Someone gets value from the result   What is value?

Differs depending on who you ask: developer, user, tech. support

  Sustainability

  Conformity / commitment

  Control and management

  Expectations, estimates, forecasts

  Reproducibility

  Improvement   Learning from

mistakes and successes

QUALITY



Quality goals



Setting the target - Quality goals   Generic quality characteristics are not enough

  Customer’s requirements and needs   Product characteristics   Application domain   Development technologies and environment   Project characteristics

  Specific quality goals are needed   Specific goals for this project   Unambiguous   Connected to a concrete project, product, and problems

we need to define precisely what qualities we require of a system



From a general to a specific quality goal: An Example

  Adaptability   Attributes of software that bear on the opportunity for its

adaptation to different specified environments without applying other actions or means than those provided for this purpose for the software considered.

  Adaptability   The MobWebClient software must work with all advertised

compatible browser and e-mail environments, with or without java, without specific configuration.



Quality goals   Quality goals point out the important quality characteristics for

  the project,   the product, and   the whole organization

  Quality goals   show where to focus effort and resources   help selecting suitable practices and methods   steer every activity of the project into the right direction

  Quality goals can reflect different qualities for different parts of the system



Focusing the QA efforts – Quality Risks   Risk – a potential threat to projects objectives

  With uncertain probability   Quality goals tell us what are the important qualities to achieve

in this project   Quality risk analysis focuses our QA efforts to the most

important things   E.g. if functional correctness is one of our quality goals:

What are the most important functions that we should test thoroughly vs. all the functions we could test?



What we should test vs. what we could test?

Reduced value due to defects

Cost of finding defects



Example – Risk analysis matrix Characteristic Impact Probability Exposure

Usability 3 4 12

Functional correctness

2 2 4

Conformance to standard X

1 3 3



Example – Statistical Risk Analysis Matrix

Interest Calculation

Close Account

Create Account

Dev Cust. Avg New Func.

Design Qual. Size

Com- plexity

Weigh. Sum

Risk Exposure

5 5 1 3

3 3 3

1 3 2

2 1 1,5

2 3 3 3 37

2 2 2 3 31

3 3 2 3 41

111

62

61,5

Impact * Probability = Re

Modified slide, originally from Ståle Amland

Weight

The numbers and calculations are not important – Idea is to get the features into priority order



Prioritising risks (I x P = E)   I = Impact, P = Probability, E = Exposure

  What does the exposure number mean?   Scale of a risk   Used to rank risks

  high I x low P = low I x high P   Risks with very severe impacts can have average exposure   Usually impact is easier to estimate than probability   Can lead to low exposures for high impact risks



Prioritising risks   Goal is to prioritize

  Which risks deserve most attention

  The discussion about the severities and probabilities is probably more valuable than the exact ranking

  In QA, prioritization means distribution of efforts, not order of actions   Less risky application areas cannot be ignored   Requirement priority is not test priority



Quality Practices



Quality practices   Many practices strive for better quality

  Testing   Reviews, inspections   Conventions, guidelines   Documenting   Design and development methods & techniques   Project management practices



Quality practices   Plan which practices are used to achieve and measure each of

the quality goals   Applying existing practices already in use   Learning best practices from other projects and organizations   Improving and combining existing practices   Inventing new practices

  Quality goals give guidance for performing the practices



Mapping the goals and practices

Practices Quality goals Quality threat



Usability Performance

DB changes break existing client sw

One developer has no experience in J2EE

Functional testing 3 1 GUI prototyping 2 Code reviews 3

3 = good practice 2 = helps 1 = might help



Mapping the goals and practices

Practices Quality goals Quality threat



Usability Performance

DB changes break existing client sw

One developer has no experience in J2EE

Functional testing 3 1 GUI prototyping 2 Code reviews 3 1 2 Regression testing client sw

1 3

  Documents how quality goals are to be achieved   Focuses activities into right issues   Reveals and communicates

  explicitly identified threats   quality goals with weak practices



How to measure the quality goals   Some quality characteristics can be measured

quantitatively = Hard metrics

 e.g. features passing tests, performance

  Some can be assessed subjectively = Subjective evaluation

 e.g. usability, conformance to a standard

  Some cannot be measured at all (in a practical situation) = Indirect metrics

 e.g. maintainability, reusability



Evaluating and tracking quality - Hard metrics

  Hard metrics are concrete and clear   If the goal can be quantified, do so

  What measures are used?   Who, how and when?   How the measures are used?

  Numbers are easy to present and understand   Easy to track and observe changes and trends   Easy to misinterpret

  E.g. what can we infer based on the number of found defects alone?  Historical data helps in detecting trends  Characteristics of defects (severity, technical type)



Evaluating and tracking quality - Subjective evaluation

  Subjective evaluation as a measure   Define how the goal is assessed

  What are the metrics   Who, how and when   How the results are used

  Reviews and assessments   Subjective opinion

  Qualitative data  E.g., subjective assessment of quality of a video stream

  Tracking not as straightforward -> must quantify  E.g., compare to reference video streams of different

quality



Evaluating and tracking quality - Indirect metrics   Define how to achieve, evaluate and track these goals too!   Indirect assumptions

  Maintainability  E.g., following a coding standard leads to better code

quality, understandability and source code documentation   Reusability

 E.g., writing and executing unit tests for all code leads to better design, robust code, and better maintainability

 E.g., product line architecture leads to re-usable software   Hard metrics for following these practices

  Maintainability  E.g., coding standards violations found in code reviews

  Reusability  E.g., number / amount of unit tests, unit test peer

reviews  E.g., amount of project specific glue code



Goal – Question – Metric (GQM)   Goals define the purpose and objective for measurement

  Purpose   Quality issue   Object   Viewpoint

  Questions characterize and refine the measurement goals   The goal can be achieved when answers to the questions are

available   Metrics are identified for each question

  Metrics provide the answers to the questions



Project Planning



Planning the project with quality goals and practices

  Describe quality goals and major threats   Make sure that each quality goal is meaningful and described in

the context of this project   Make sure that all required quality practices are planned

  Effort estimates   Schedule   Resourcing

  Plan how to track quality practices and measure achieved quality  What measures are used?  Who, how and when?  How the measures are collected and used?

  Remember: Quality assurance takes effort and calendar time



Testing and project planning   Test releases

  Make clear what and when is released for testing   Plan what is the required quality level of test releases – and how to

ensure / measure it   Building and delivering the test release takes time

  Do not forget that fixing takes time and effort   You don’t know how many bugs will be found   You must plan how many bugs will be found   Allocate time and resources to test, fix and re-test (verify the fixes)

  More bugs means more fixing and slower testing   Test schedules are relative

  Development will be late   Code will be buggy   Plan how to handle slipping dates and underestimated workload



Iterative Testing and Deployment

  Testing, stabilizing, and deployment takes time   Testing requires many iterations

  Found defects has to be fixed   Fixes has to be re-tested   Fixing causes more defects…

  If you do more during development the risk is smaller during testing and deployment   If all testing is left at the end, the testing phase is hard to

estimate and plan   Waterfall process

t

Release

Iteration Heartbeat Testing and deployment iterations

Release to customer

Release to testing



Project Tracking and Reporting



Making quality visible in project tracking

  Keep quality goals visible in project meetings   Track the planned metrics   Publish the metrics   React to deviations

  Connect quality metrics in progress tracking   What does it mean to get a task done?   Reviews, unit tests, functional tests, regression tests, demo, …



Making quality visible in project tracking

  Find out the reasons behind the metrics   small number of found bugs due to ignored testing tasks   large number of found bugs due to changed specifications -> broken

test cases

Number of defects

High QA quality Low QA quality

A B

C D



Test reporting in project management   As a project manager you need test reports

  Not as bug descriptions, not as plain numbers of test cases and bugs

  Test reports describe what has been tested and assess the current level of achieved quality based on that testing

  Test reports should provide information to assess how well the stated goals have been achieved

  Important content in test reports   Assessment of the achieved quality

 What was tested, how thoroughly, what was coverage  What was found, bugs by severity, issues, metrics

  Progress according to the plan



Reporting with weak metrics: Testing Dashboard

  If you cannot have direct quality measures, use indirect   Use of certain quality assurance practices gives indirectly hints of

the likely quality of the final system   Qualitative assessment and the reasoning behind is much better

than plain numbers

Modified from: Kaner et al. 2002. Lessons Learned in Software testing.



Test reporting for steering group   It depends! (Usually higher level of abstraction, focus on specific

issues)

0 5

10 15 20 25 30 35

1 3 5 7 9 11 13 15 17 19 21 23

Designed

Implemented

Tested

Needs rework

Finished

0 5

10 15 20 25 30 35

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Remaining

Planned



Quality information • Review results

• Test results, bug counts

• Development progress

• Metrics of different qualities (performance, load, …)

Managing project • To re-scope an iteration

• To focus resources into right areas

• To make sure achieving good enough quality

• To find out we are making the right product

we need to forecast the likely quality of the final system when it is under development



Quality Assurance Through Time Horizons

  Heartbeat time horizon   Quality practices are part of each

development task

  Iteration time horizon   Tasks to assure the quality of an

increment

  Release time horizon   Tasks to assure the quality of the

whole release

t

Release

Iteration Heartbeat

Blue = QA practices



Quality information and time horizons Release time horizon

  Not connected to any iterations or milestones – hard to track and manage

  Typical approach to manage   Informal reviews   System testing   Acceptance testing   Testing non-functional quality characteristics

  Performance   Usability   Load and stress testing   …

i i i i i i i Release project

Time



Quality information and time horizons Iteration time horizon

  Quality assurance goals for each iteration   Activities that provide information by the end of (each) iteration

  Not necessary connected to individual development tasks

  Information available at the end of each iteration, at latest   Review results   Unit test and integration test results   Function test and system test results   Quality metrics (e.g. performance, bug counts, test reports)

i i i i

i

i i i i i i i i i

Release project

Iteration

Time



Quality information and time horizons Heartbeat time horizon

  Quality assurance activities as part of each development task   Information created early and continuously

  Development progress, bug counts, performed tests, …   Code review results   Unit tests and results   Automatic builds and smoke tests   Function test and integration test results

i i i i i i

i

i i i i i i i i i i i i

Release project

Iteration

Heartbeat

Time



Quality information and time horizons

i i i i i i

i i i i

i i i i i i i

i i i i i i i i i

i i i i i i i i i i i i

Release project

Iteration

Heartbeat

Time

Heartbeat quality practices •  Information for steering the iteration •  Tracking progress

Iteration quality practices •  Information for steering the current and forthcoming iterations •  Evaluating achieved quality

Release quality practices •  Independent activities •  Tracking and planning iterations



Gather information on short time horizons   Progress against iteration (quality) goals must be tracked in

heartbeat rhythm   Plan how you get the quality information before it’s too late

  Find actively ways of digging quality information out as early and often as possible   Talk to testers and developers

  The best way is to get something completed and tested in short iterations   The risk of poor quality is not delayed to the end of the project

  Shorter feedback loop -> easier and faster to fix the problems   Pay attention to heartbeat practices   Assure good enough quality in each iteration

  Do not push risks like a growing snowball in front of you



Who is responsible for quality?   Make sure that quality is everybody’s responsibility

  and it shows in the project plan   It is good to have a project “QA manager” role

  Responsibility to actively track quality in the heartbeat rhythm and help project manager to make QA activities to happen

  Reports project’s progress from the quality point of view in the project meetings   Tracking quality goals in iteration time horizon

  Tries to be less eager to sacrifice quality for new fancy features   Keep separate testers tightly in the project

  In the heartbeat rhythm  Project meetings  Reviews  Collaborating with developers



Summary   Define concrete and meaningful quality goals   Identify practices and activities that are applied to achieve the

goals and track progress   Plan activities and tracking

  quality assurance activities in the schedule and effort estimations   how to react to deviations

  Remember the different time horizons   When is the information needed in order to be useful?   Plan activities to provide the information promptly

  Identify also the major quality threats   and plan how the threats are handled in the project and when   Do not leave big threats to be revealed at the end of the project



Thank you.

Questions?

Tuomas Niinimäki

[email protected]