IQUAL IQUAL 1 of 60 Software Quality Engineering in international standardization and practice Witold Suryn Ph.D. Eng. École de technologie supérieure

IQUALIQUAL

1 of 60

Software Quality Engineering in Software Quality Engineering in international standardization and international standardization and

practicepractice

Witold Suryn Ph.D. Eng.

École de technologie supérieure

Montréal, Canada

Higher School of Economics

Moscow

22nd of May 2007

IQUALIQUAL

2 of 60

AgendaAgenda

A few words about Presenter

What is Software Quality Engineering

Software Quality Engineering in ISO

Software Quality Engineering in practice

IQUALIQUAL

3 of 60

A few words about PresenterA few words about Presenter

Professor of Software and IT Engineering Dept., École de technologie

supérieure, Montréal, Canada

Director of GELOG : IQUAL Research Group at ÉTS

Software Quality expert at ISO/IEC JTC1 SC7 WG6

Editor of ISO/IEC 25001, 250023, Co-editor of ISO/IEC 9126-4 and

25000.

International Secretary of ISO/IEC JTC1 Subcommittee SC7 – System

and Software Engineering

IQUALIQUAL

4 of 60

AgendaAgenda





IQUALIQUAL

5 of 60

What is Software Quality Engineering (1)What is Software Quality Engineering (1)

Is Software Quality Engineering

just another term for

Software Quality Assurance?

IQUALIQUAL

6 of 60


What is Software

+Strategy

could be

IQUALIQUAL

7 of 60

What is Software Quality Engineering (3)What is Software Quality Engineering (3)The definition

1. The application of a continuous, systematic, disciplined, quantifiable approach to the development and maintenance of quality throughout the whole life cycle of software products and systems; that is, the application of quality engineering to software,

2. The study of approaches as in (1).

©Witold Suryn 2003-2007

IQUALIQUAL

8 of 60


Cooperation

SQA SpecialistChief Developer/Teste

r

Quality Engineer

Project Manager Budget Controller

IQUALIQUAL

9 of 60


Cooperation, but why?

Functionalities Quality

Customer

Software

DevelopmentProcess

Dev/Test

Processes SQA

SQA Spec

To-dosSQE

SQ Eng

Mngr

Fin Contr

IQUALIQUAL

10 of 60


The presented diagram introduces the subject of

SQE processes and procedures

that will be discussed in the last part of this presentation

IQUALIQUAL

11 of 60

AgendaAgenda





IQUALIQUAL

12 of 60

Who works on Software Engineering inWho works on Software Engineering in

ISO (1)ISO (1)ISO/IEC JTC 1 SC7

System Software Documentation

WG2

WG6Process

Assessment

WG10Tools And

Environment

WG4

Life Cycle Management

WG7

System Assurance

WG9

Business Planning Group

SWG 1

Software EngineeringBody of Knowledge

WG20ODP and Modeling

Languages

WG19

Secretariat

Asset Management

WG21

Architecture Management

WG12Functional Size Measurement

Vocabulary

WG22

SWG 5

Source: Prof. M. Azuma

Software Product Quality Measurement and

Evaluation

System Quality Management

WG23

SLC Profiles and Guidelines for VSE

IT ServiceManagement

WG25

WG24 Architecture

WG42

CIF

JWG

IQUALIQUAL

13 of 60


ISO (2)ISO (2)Structure

P-Members (Participating Members)33 National Bodies with the right to voteAmong others UK, Germany, Canada, Japan, USA, France.

O-Members (Observing Members)20 National Bodies with the right to commentAmong others Russian Federation, Austria, Poland.

LiaisonsCategory A – 9 (IEEE, INCOSE, OMG and others)Category B – 1 (CEN)Category C – 12 (COSMIC, UKSMA, OFPUG and others)ISO Internal - 16

IQUALIQUAL

14 of 60


ISO (3)ISO (3) How to know about us?

http://www.jtc1-sc7.org/

IQUALIQUAL

15 of 60

Who works on Software Quality Engineering in ISOWho works on Software Quality Engineering in ISO

Working Group 6 (WG6)Convener Prof. Motoei Azuma (Japan)Most Active Members:

Japan,USA,UK,Canada,France,Denmark,Brazil,Czech Republic,Australia.

IQUALIQUAL

16 of 60

Software Quality Engineering in ISO (1)Software Quality Engineering in ISO (1)

ISO/IEC JTC1 SC7 WG6 best known international standards

ISO/IEC 9126-1 : 2000 - Software engineering - Product quality - Part 1: Quality model (IS)ISO/IEC 9126-2 : 2003 - Software engineering - Product quality – Part 2: External metrics (TR)ISO/IEC 9126-3 : 2003 - Software engineering - Product quality – Part 3: Internal metrics (TR)ISO/IEC 9126-4 : 2004 - Software engineering - Product quality – Part 4: Quality in use metrics (TR)ISO/IEC 14598 (1 to 6) : 1999-2001 - Information technology - Software product evaluation

IQUALIQUAL

17 of 60


In 2001 WG6 launched international survey on the applicability of ISO/IEC 9126. The results were the following:

ISO/IEC 9126 is widely known

but is not used Ease of use (not for novices, oh, no!)

Completeness (ex: missing quality requirements specifications);

Consistency with other ISO standards published in parallel (ex: ISO/IEC 15939 measurement model);

Scope of applicabilityUser guidance as part of the series of standards;

Addition of execution workflow recommendations.

IQUALIQUAL

18 of 60


WG6 has responded to this feedback by designing the second generation of quality standards, referred to as SQuaRE, Software Product Quality Requirements and Evaluation or ISO/IEC 25000. The following strategy was adopted to develop this second generation of quality standards:

merger of ISO/IEC 9126 and 14598 into one, harmonized structure,introduction of a new organization of the standards,introduction of a new general reference model,introduction of detailed guides,introduction of a standards on Measurement Elements, introduction of a standard on Quality Requirements, introduction of guidance on the practical use of the series with examples,coordination and harmonization of the measure model with ISO/IEC 15939.

IQUALIQUAL

19 of 60


ISO/IEC 25000 SQuaRE or Software Product Quality Requirements and Evaluation

IQUALIQUAL

20 of 60


ISO/IEC 25000 SQuaRE

Reference model

IQUALIQUAL

21 of 60


ISO/IEC 25000 SQuaRE

quality model

IQUALIQUAL

22 of 60


ISO/IEC 25000 SQuaRE quality measuresOver 200 measures

Under scrutiny:Quality in use – corrections published in 2005 by M-A.Côté and W.Suryn,

Internal quality - corrections published in 2006 by W.Suryn and B.Gil

External quality – in publication in 2007 by W.Suryn, M.Ouelett, C.Nelson and B.Salmi

IQUALIQUAL

23 of 60


ISO/IEC 25000 SQuaRE – publication stage:Published:

25000 – Guide to SQuaRE25001 – Planning and management25051 - Requirements for quality of Commercial Off-The-Self (COTS) software product and instructions for testing

FDIS/DTR (last stage before publication)25020 - Measurement reference model and guide 25021 – Measurement Elements25030 – Quality requirements

CD (internal working/balloting stage)25010 – Quality model25012 – Data quality25022/23/24 – measures25040 - evaluation

IQUALIQUAL

24 of 60

Software Quality Engineering in ISO (9)Software Quality Engineering in ISO (9)How much of Engineering is in ISO/IEC 25000 SQuaRE?


IQUALIQUAL

25 of 60

Software Quality Engineering in ISO (Software Quality Engineering in ISO (1010))ISO/IEC 19759 SWEBOK – Software Engineering Body of

Knowledge

Editors:

Partners:

www.swebok.org

IQUALIQUAL

26 of 60

Software Quality Engineering in ISO (Software Quality Engineering in ISO (111)1)

ISO/IEC 19759 SWEBOK – Software Engineering Body of Knowledge

1998 1999 2000 2001 2002 2003

Straw ManVersion

Stone Man Version

Iron ManVersion

(Sub-phase 1)

Iron Man Version(Sub-phase 2)

IQUALIQUAL

27 of 60

Software Quality Engineering in ISO (Software Quality Engineering in ISO (112)2)ISO/IEC 19759 SWEBOK – Structure – Part 1

IQUALIQUAL

28 of 60

Software Quality Engineering in ISO (Software Quality Engineering in ISO (113)3)ISO/IEC 19759 SWEBOK – Structure – Part 2

IQUALIQUAL

29 of 60

Software Quality Engineering in ISO (Software Quality Engineering in ISO (114)4)ISO/IEC 19759 SWEBOK – Structure – KA 10 – Software Quality

IQUALIQUAL

30 of 60


How much of Engineering

is inSWEBOK (1)

Published inSuryn W., et al “Software Quality Engineering – where to find it in Software Engineering Body of Knowledge (SWEBOK)”. Proceedings of 14th International Software Quality Management & INSPIRE Conference 2006.

IQUALIQUAL

31 of 60


How much of Engineering

is inSWEBOK (2)

Published inSuryn W., et al “Software Quality Engineering – where to find it in Software Engineering Body of Knowledge (SWEBOK)”. Proceedings of 14th International Software Quality Management & INSPIRE Conference 2006.

Conclusions:•Very process oriented (QA notion)•Main focus on quality measurement and evaluation•Quality requirements are absent•No engineering notions at all

The recommendations will be passed to SWEBOK Editors directlyand through ISO channels

IQUALIQUAL

32 of 60

AgendaAgenda





IQUALIQUAL

33 of 60

Software Quality Engineering in practice (1)Software Quality Engineering in practice (1)

Engineering issues to consider

Quality model

Quality requirements

Quality design

Quality measurement and evaluation

Management issues to consider

Process model

Conflict management

Financial and maturity constraints

IQUALIQUAL

34 of 60


Quality model (1)

McCall B.Boehm Dromey

Good models, but:• Present personal views of the authors• No community consensus• Have no measures associated with them• Mostly useless in practice (*)• As such, almost unknown in industry

(*) - Côté M_A., Suryn W., Martin R., Laporte C.Y., “Evolving a Corporate Software Quality Assessment Exercise: A Migration Path to ISO/IEC 9126”. Software Quality Professional, VOL. 6, NO. 3/© 2004, ASQ

IQUALIQUAL

35 of 60


Quality model (2)

ISO/IEC 9126

• Presents best-practice international view• Community consensus• Over 200 measures associated with them• The only useful in practice• The only known in industry, but• Not free from serious critics (*)

(*) - Suryn W., Abran A., “ISO/IEC SQuaRE. The 2nd generation of standard for quality of software product”. Proceedings of 7th IASTED International Conference on Software Engineering and Applications, SEA 2003

IQUALIQUAL

36 of 60


Quality requirements (1)

External QualityRequirement

External QualityRequirement

Quality in Use Requirements

Quality in Use Requirements

QualityIn Use

QualityIn Use

InternalQuality

InternalQuality

Internal QualityRequirement

Internal QualityRequirement

ExternalQuality

ExternalQuality

Requirements Products

Validation

Validation

Validation

Feed Back to the Next Cycle (Needs)

Quality requirements:

• Quality in use – non-technical, end-user perspective

• External quality – mostly technical, applied to running software/system

• Internal quality – technical, applied to software in development

• Operational quality- applied to product deployed massively

IQUALIQUAL

37 of 60



Legend:Red solid arrows ask the question HOWBoxes ask the question WHAT (choices)Blue dashed arrows ask about TRACING

7

6 4

3

2

1

Stakeholder Requirements1,..,S

Quality Requirements1,..,Q

Operational QualityRequirements

1,..,O

Quality in UseRequirements

1,..,U

External QualityRequirements

1,…,E

Internal QualityRequirements

1,..,I

5


IQUALIQUAL

38 of 60



Requirements:• Functional• Non-functional

Solution Design

ComponentsDesign

Construction (coding)

Unit & IntegrationTests

System Tests(ALL)

StakeholderRequirements

Quality Requirements:•In Use Characterstics•External Characteristcs•Internal Characteristcs•Operational

Quality for Solution Design•QiU attributes•External Sub-charact.

Quality for Component Design•External Sub-charact.•Internal Sub-charact.

Quality for Component Construction

•Internal Sub-charact.

External and Internal MeasuresQuality in use &External Measures

Influence/Modify through Engineering Decisions

The consolidated traceability model CTM is © Witold Suryn


Solution Design

ComponentsDesign



System Tests(ALL)








Influence/Modify through Engineering Decisions


Solution Design

ComponentsDesign



System Tests(ALL)


Solution Design

ComponentsDesign



System Tests(ALL)














Influence/Modify through Engineering DecisionsInfluence/Modify through Engineering Decisions

The consolidated traceability model CTM is © Witold Suryn

IQUALIQUAL

39 of 60

Software Quality Engineering in practice (6)Software Quality Engineering in practice (6)Quality requirements (4): How to get them?

An experiment with master students

IQUALIQUAL

40 of 60


Quality design (1)Quality design translates quality requirements into their technical representations (or in simpler form: executable to-dos) applicable in next phases,The results of the quality design process also have to be:

applicable immediately in the software/system design phase, no matter the development model chosen,further refined and complemented to descend to the level required in a construction phase

IQUALIQUAL

41 of 60


Quality design (2)

Quality Requirements Analysis and Definition

Identification of quality attributes applicable on the

system level.

For example in ISO 9126 from quality in use and external quality, or operational quality attributes from TL9000.

Specification of technical elements related to identified

quality attributes

Communication with

development team

Quality Requirements Analysis and Definition

Identification of quality attributes applicable on the

system level.

For example in ISO 9126 from quality in use and external quality, or operational quality attributes from TL9000.

Specification of technical elements related to identified

quality attributes

Communication with

development team

IQUALIQUAL

42 of 60


Quality design (3)Bob: from my quality requirements definition phase I have identified the following, system-level quality attributes: system availability has to be minimum 98%, the mean recovery time should not exceed 1 min/failure and the customer requests that the relative user efficiency be at least 0.9.

Frank: Well, it sounds Ok, but what I am supposed to do with it?

Bob: these attributes have the following impact on your design: availability of 98% means that the system must be available 59 minutes out of every operational hour at any time, so your design should strive to exhibit high reliability. One of the options you may take into account is using stable technologies and enhanced application of reuse

recovery time below 1 min/failure may request applying super efficient recovery algorithms together with extra fast hardware

user efficiency of 0.9 and above translates into such a functional, user interface and on-line help design that there will be virtually no difference in operation between and average user and an expert.

IQUALIQUAL

43 of 60


Quality design (4): personalized quality model

measures measures measures measures measures measures

We rarely engineer into our product everything from this model, because:

1.For a given product not everything is:

• equally important,

• necessary, or

• Feasible,

2.The budget is always limited (*)(*) will be discussed in Managerial Issues part

partialpartial partial partial

IQUALIQUAL

44 of 60


Quality design (5): personalized design map

Quality Requirements

System quality technical elementsSystem design

Functional and non-Functional Requirements

System quality design

Program designProgram quality design

Program quality technical elements

Personalized Quality Model

System quality

attributes

System quality

measures

System quality target values

Program quality

attributes

Program quality

measures

Program quality target values

Quality Requirements

System quality technical elementsSystem design

Functional and non-Functional Requirements

System quality design

Program designProgram quality design

Program quality technical elements

Personalized Quality Model

System quality

attributes

System quality

measures

System quality target values

Program quality

attributes

Program quality

measures

Program quality target values


IQUALIQUAL

45 of 60

Software Quality Engineering in practice (12)Software Quality Engineering in practice (12)Quality measurement and evaluation (1): managerial support

Establish and sustain

Measurement Commitment

Plan the Measurement

Process

Perform the Measurement

Process

Evaluate Measurement

Commitment

PlanningInformation

InfomationProducts &

PerformanceMesures

Core Measurement Process

Measurement Experience Base

Technical andManagement

ProcessesInformation Needs Information Products

Improvement ActionsScope of ISO/IEC 15939

Infomation Products &Evaluation Results

Legend

Activity Data Flow Data Store




Process


Process


Commitment

PlanningInformation


PerformanceMesures










Process


Process


Commitment

PlanningInformation


PerformanceMesures







Legend

Activity Data Flow Data Store

Legend

Activity Data Flow Data Store©ISO, Geneva, Switzerland

IQUALIQUAL

46 of 60


Quality measurement and

evaluation (2):

generic measurement

model

©ISO, Geneva, Switzerland

IQUALIQUAL

47 of 60


Quality measurement and evaluation (3):

quality measures vs. quality model

Software ProductQuality

QualityCharacteristics

QualitySub-Characteristics

Quality Measures

Quality MeasureElements

MeasurementFunction

indicate

indicate

are applied to

generates

composed of

composed of

Software ProductQuality

QualityCharacteristics

QualitySub-Characteristics

Quality Measures

Quality MeasureElements

MeasurementFunction

indicate

indicate

are applied to

generates

composed of

composed of


IQUALIQUAL

48 of 60

Software Quality Engineering in practice (15)Software Quality Engineering in practice (15)Quality measurement and evaluation (4): evaluation process


IQUALIQUAL

49 of 60

Software Quality Engineering in practice (16)Software Quality Engineering in practice (16)Quality measurement and evaluation (5): evaluation of data

Unacceptable

Minimally acceptable

Target range

Exceeds requirements

satisfactory

unsatisfactory

measured value

measurement scale rating levels

worst case

current level

planned level


IQUALIQUAL

50 of 60


Software Quality Implementation Process Model SQIM (1)

In “Fundamental Principles of Software Engineering – a Journey” P.Bourque et al propose the following principle:

Manage quality throughout the life cycle as formally as possible

SQIM applies this principle in following ways:engineering of quality into a software product should be conducted throughout the whole life cycle of software,quality engineering is similar to a development process, so it should follow similar rules and applie similar structures,To be generic, SQIM adheres to the SLC generic model published in ISO/IEC 15288 - Information Technology - Life Cycle Management - System Life Cycle ProcessesThe quality model that SQIM adheres to is the one which is widely accepted and recognized, the quality model from ISO/IEC 9126.


IQUALIQUAL

51 of 60

Software Quality Software Quality Engineering in Engineering in

practice (17)practice (17)Software Quality

Implementation

Process Model

SQIM (2)©Witold Suryn 2004-2007

IQUALIQUAL

52 of 60


Software Quality Implementation: A Quality Engineer


QualityEngineer

Technical User Non-technical User

Technical Analyst Non-technical Analyst

Developer/TesterBusiness Case

Controller

customer

supplier

QualityEngineer

Technical User Non-technical User

Technical Analyst Non-technical Analyst

Developer/TesterBusiness Case

Controller

customer

supplier

IQUALIQUAL

53 of 60


Software Quality Implementation: Conflict management


DevelopmentProcess

DevelopmentChange ControlProcess

QualityChange ControlProcess

Quality Engineering

Process

Change request

Accepted ?

Accepted ?

External ChangeRequest

Internal ChangeRequest

Change request

Y

Quality?

Y

N

Process

Process

Y

Engineer

Change request


Engineer

DevelopmentProcess

DevelopmentProcess





Quality Engineering

Process

Quality Engineering

Process

Change requestChange request

Accepted ?

Accepted ?







Change request

Y

Change requestChange request

Y

Quality?

Y

Quality?

Y

N

Process

N

Process

Process

Y

Process

Y

EngineerEngineer

Change request


Change request


EngineerEngineer

IQUALIQUAL

54 of 60

Software Quality Engineering in practice (20)Software Quality Engineering in practice (20)Software Quality Implementation:

Financial and maturity constraints (1)

A few basic facts of engineering quality into software:Everything in software engineering boils down to user’s satisfaction,Satisfaction is conditional to the overall behavior of the system, with software product in the first place,The behavior of any software product is perceived through features and quality,Features and quality of software product are expressed through requirements,Any behavior-related requirement for software product may only be realized through code


IQUALIQUAL

55 of 60


Financial and maturity constraints (2)When engineering quality into software, one could take into consideration the following HARD fact:

In most development projects functionality and quality are natural enemies.


IQUALIQUAL

56 of 60




CostCost

FeaturesFeatures Operation Operation

Quality Quality CostCostCostCost

FeaturesFeatures Operation Operation FeaturesFeaturesFeaturesFeatures Operation Operation Operation Operation

Quality Quality Quality Quality

IQUALIQUAL

57 of 60




• Economic perspective

Cost = a0f0 + a∑∑ features + b∑∑ quality aspectsquality aspects

ororC = C = a0f0 + + afaf + bq+ bq

Where: Where: a, b a, b –– proportions of investmentproportions of investmentaa00 -- initial investment levelinitial investment levelff00 -- initial set of featuresinitial set of features

IQUALIQUAL

58 of 60



Maturity evaluation: CMMI, SPICE, ISO9000, but is REAL QUALITY there?

Or perhaps quality is a real an indicator of maturity?

Young, immature companies can afford developing just a set of attractive functionalities

Mature organizations can develop quality tooso the level of quality observed in a software product is an indicator of the level of maturity of its developer

The evaluation of maturity of a software development organization can lead to conclusions that may not entirely reflect the reality.

All best processes will not replace the tangible indicators of the real maturity: functionalities and quality of the product.

Since functionalities are always in a product and quality only sometimes, quality is a more restrictive indicator.

IQUALIQUAL

59 of 60

If you are interested in more informationIf you are interested in more information

About ISO/IEC JTC1 SC7, go to:

http://www.jtc1-sc7.org/

About Software Quality Engineering, go to:

http://profs.logti.etsmtl.ca/wsuryn/research/

Suryn W., Coallier F., Hvanberg T.E.

Software Quality Engineering – A Practitioner’s Approach

Targeted publication date 2008

IQUALIQUAL

60 of 60

Thank you

Questions and Answers

ВОПРОСЫ И ОТВЕТЫ

Documents

IQUAL IQUAL 1 of 60 Software Quality Engineering in international standardization and practice Witold Suryn Ph.D. Eng. École de technologie supérieure