read.pudn.comread.pudn.com/downloads454/doc/1911476/ISO_IEC 9126-2.pdf · ISO/IEC TR 9126-2:2003(E) © ISO/IEC 2003 – All rights reserved iii Contents Page Foreword

Reference numberISO/IEC TR 9126-2:2003(E)

© ISO/IEC 2003

TECHNICAL REPORT

ISO/IECTR

9126-2

First edition2003-07-01

Software engineering — Product quality — Part 2: External metrics

Génie du logiciel — Qualité des produits —

Partie 2: Métrologie externe

ISO/IEC TR 9126-2:2003(E)

PDF disclaimer This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat accepts no liability in this area.

Adobe is a trademark of Adobe Systems Incorporated.

Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.

© ISO/IEC 2003 All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO's member body in the country of the requester.

ISO copyright office Case postale 56 • CH-1211 Geneva 20 Tel. + 41 22 749 01 11 Fax + 41 22 749 09 47 E-mail [email protected] Web www.iso.org

Published in Switzerland

ii © ISO/IEC 2003 – All rights reserved

ISO/IEC TR 9126-2:2003(E)

© ISO/IEC 2003 – All rights reserved iii

Contents Page

Foreword............................................................................................................................................................ vi

Introduction ...................................................................................................................................................... vii

1 Scope........................................................................................................................................................... 1

2 Conformance .............................................................................................................................................. 2

3 Normative references................................................................................................................................. 2

4 Terms and definitions ................................................................................................................................ 2

5 Abbreviated terms...................................................................................................................................... 2

6 Use of software quality metrics ................................................................................................................ 3

7 How to read and use the metrics tables .................................................................................................. 4

8 Metrics tables.............................................................................................................................................. 4 8.1 Functionality metrics............................................................................................................................... 5 8.1.1 Suitability metrics ........................................................................................................................ 5 8.1.2 Accuracy metrics ......................................................................................................................... 5 8.1.3 Interoperability metrics................................................................................................................ 5 8.1.4 Security metrics............................................................................................................................ 5 8.1.5 Functionality compliance metrics .............................................................................................. 6 8.2 Reliability metrics .................................................................................................................................. 14 8.2.1 Maturity metrics.......................................................................................................................... 14 8.2.2 Fault tolerance metrics .............................................................................................................. 14 8.2.3 Recoverability metrics ............................................................................................................... 14 8.2.4 Reliability compliance metrics.................................................................................................. 14 8.3 Usability metrics .................................................................................................................................... 25 8.3.1 Understandability metrics ......................................................................................................... 25 8.3.2 Learnability metrics ................................................................................................................... 25 8.3.3 Operability metrics..................................................................................................................... 26 8.3.4 Attractiveness metrics............................................................................................................... 26 8.3.5 Usability compliance metrics.................................................................................................... 26 8.4 Efficiency metrics .................................................................................................................................. 41 8.4.1 Time behaviour metrics ............................................................................................................. 41 8.4.2 Resource utilization metrics ..................................................................................................... 41 8.4.3 Efficiency compliance metrics.................................................................................................. 41 8.5 Maintainability metrics .......................................................................................................................... 52 8.5.1 Analysability metrics ................................................................................................................. 52 8.5.2 Changeability metrics ................................................................................................................ 52 8.5.3 Stability metrics.......................................................................................................................... 52 8.5.4 Testability metrics...................................................................................................................... 52 8.5.5 Maintainability compliance metrics.......................................................................................... 52 8.6 Portability metrics.................................................................................................................................. 60 8.6.1 Adaptability metrics ................................................................................................................... 60

ISO/IEC TR 9126-2:2003(E)

iv © ISO/IEC 2003 – All rights reserved

8.6.2 Installability metrics ...................................................................................................................60 8.6.3 Co-existence metrics..................................................................................................................60 8.6.4 Replaceability metrics................................................................................................................60 8.6.5 Portability compliance metrics..................................................................................................60

Annex A (informative) Considerations When Using Metrics........................................................................68 A.1 Interpretation of measures....................................................................................................................68 A.1.1 Potential differences between test and operational contexts of use....................................68 A.1.2 Issues affecting validity of results............................................................................................69 A.1.3 Balance of measurement resources .................................................................................................69 A.1.4 Correctness of specification..............................................................................................................69 A.2 Validation of Metrics ..............................................................................................................................69 A.2.1 Desirable Properties for Metrics ...............................................................................................69 A.2.2 Demonstrating the Validity of Metrics ......................................................................................70 A.3 Use of metrics for estimation (judgement) and prediction (forecast) ..............................................71 A.3.1 Quality characteristics prediction by current data .................................................................71 A.3.2 Current quality characteristics estimation on current facts ..................................................71 A.4 Detecting deviations and anomalies in quality problem prone components ..................................72 A.5 Displaying measurement results..........................................................................................................72

Annex B (informative) Use of Quality in Use, External & Internal Metrics (Framework Example) ...........73 B.1 Introduction ............................................................................................................................................73 B.2 Overview of Development and Quality Process .................................................................................73 B.3 Quality Approach Steps ........................................................................................................................74 B.3.1 General.........................................................................................................................................74 B.3.2 Step #1 Quality requirements identification ............................................................................74 B.3.3 Step #2 Specification of the evaluation....................................................................................75 B.3.4 Step #3 Design of the evaluation ..............................................................................................77 B.3.5 Step #4 Execution of the evaluation .........................................................................................77 B.3.6 Step #5 Feedback to the organization ......................................................................................77

Annex C (informative) Detailed explanation of metric scale types and measurement types ...................78 C.1 Metric Scale Types.................................................................................................................................78 C.2 Measurement Types...............................................................................................................................79 C.2.1 Size Measure Type......................................................................................................................79 C.2.2 Time measure type .....................................................................................................................82 C.2.2.0 General.........................................................................................................................................82 C.2.3 Count measure type ...................................................................................................................83

Annex D (informative) Term(s).........................................................................................................................85 D.1 Definitions...............................................................................................................................................85 D.1.1 Quality..........................................................................................................................................85 D.1.2 Software and user.......................................................................................................................85 D.1.3 Measurement...............................................................................................................................85

Table 8.1.1 Suitability metrics ...........................................................................................................................7 Table 8.1.2 Accuracy metrics ............................................................................................................................9 Table 8.1.3 Interoperability metrics ................................................................................................................10 Table 8.1.4 Security metrics ............................................................................................................................11

ISO/IEC TR 9126-2:2003(E)

© ISO/IEC 2003 – All rights reserved v

Table 8.1.5 Functionality compliance metrics............................................................................................... 13 Table 8.2.1 Maturity metrics ............................................................................................................................ 15 Table 8.2.2 Fault tolerance metrics ................................................................................................................ 19 Table 8.2.3 Recoverability metrics ................................................................................................................. 21 Table 8.2.4 Reliability compliance metrics .................................................................................................... 24 Table 8.3.1 Understandability metrics ........................................................................................................... 27 Table 8.3.2 Learnability metrics...................................................................................................................... 30 Table 8.3.3 Operability metrics a) Conforms with operational user expectations .................................. 32 Table 8.3.3 Operability metrics b) Controllable ......................................................................................... 33 Table 8.3.3 Operability metrics c) Suitable for the task operation........................................................... 34 Table 8.3.3 Operability metrics d) Self descriptive (Guiding)................................................................... 35 Table 8.3.3 Operability metrics e) Operational error tolerant (Human error free) .................................. 36 Table 8.3.3 Operability metrics f) Suitable for individualisation ............................................................. 37 Table 8.3.4 Attractiveness metrics ................................................................................................................. 39 Table 8.3.5 Usability compliance metrics ...................................................................................................... 40 Table 8.4.1 Time behaviour metrics a) Response time .............................................................................. 42 Table 8.4.1 Time behaviour metrics b) Throughput .................................................................................. 44 Table 8.4.1 Time behaviour metrics c) Turnaround time .......................................................................... 45 Table 8.4.2 Resource utilisation metrics a) I/O devices resource utilisation.......................................... 47 Table 8.4.2 Resource utilisation metrics b) Memory resource utilisation............................................... 48 Table 8.4.2 Resource utilisation metrics c) Transmission resource utilisation ..................................... 49 Table 8.4.3 Efficiency compliance metrics .................................................................................................... 51 Table 8.5.1 Analysability metrics.................................................................................................................... 53 Table 8.5.2 Changeability metrics .................................................................................................................. 55 Table 8.5.3 Stability metrics ............................................................................................................................ 57 Table 8.5.4 Testability metrics ........................................................................................................................ 58 Table 8.5.5 Maintainability compliance metrics............................................................................................ 59 Table 8.6.1 Adaptability metrics ..................................................................................................................... 61 Table 8.6.2 Installability metrics ..................................................................................................................... 63 Table 8.6.3 Co-existence metrics ................................................................................................................... 65 Table 8.6.4 Replaceability metrics.................................................................................................................. 66 Table 8.6.5 Portability compliance metrics ................................................................................................... 67 Table B.1 Quality Measurement Model ......................................................................................................... 73 Table B.2 User Needs Characteristics & Weights ........................................................................................ 74 Table B.3 Quality measurement tables ......................................................................................................... 75 Table B.4 Measurement plan........................................................................................................................... 77

ISO/IEC TR 9126-2:2003(E)

vi © ISO/IEC 2003 – All rights reserved

Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.

International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.

The main task of the joint technical committee is to prepare International Standards. Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote.

In exceptional circumstances, the joint technical committee may propose the publication of a Technical Report of one of the following types:

— type 1, when the required support cannot be obtained for the publication of an International Standard, despite repeated efforts;

— type 2, when the subject is still under technical development or where for any other reason there is the future but not immediate possibility of an agreement on an International Standard;

— type 3, when the joint technical committee has collected data of a different kind from that which is normally published as an International Standard (“state of the art”, for example).

Technical Reports of types 1 and 2 are subject to review within three years of publication, to decide whether they can be transformed into International Standards. Technical Reports of type 3 do not necessarily have to be reviewed until the data they provide are considered to be no longer valid or useful.

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.

ISO/IEC TR 9126-2:2003, which is a Technical Report of type 2, was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology, Subcommittee SC 7, Software and system engineering.

This document is being issued in the Technical Report (type 2) series of publications (according to the Procedures for the technical work of ISO/IEC JTC 1) as a “prospective standard for provisional application” in the field of external metrics for quantitatively measuring external software because there is an urgent need for guidance on how standards in this field should be used to meet an identified need.

This document is not to be regarded as an “International Standard”. It is proposed for provisional application so that information and experience of its use in practice may be gathered. Comments on the content of this document should be sent to the ISO Central Secretariat.

A review of this Technical Report (type 2) will be carried out not later than three years after its publication with the options of: extension for another three years; conversion into an International Standard; or withdrawal.

ISO/IEC 9126 consists of the following parts, under the general title Software engineering — Product quality :

Part 1: Quality model

Part 2: External metrics

Part 3: Internal metrics

Part 4: Quality in use metrics

ISO/IEC TR 9126-2:2003(E)

© ISO/IEC 2003 – All rights reserved vii

Introduction

This Technical Report provides external metrics for measuring attributes of six external quality characteristics defined in ISO/IEC 9126-1. The metrics listed in this Technical Report are not intended to be an exhaustive set. Developers, evaluators, quality managers and acquirers may select metrics from this Technical Report for defining requirements, evaluating software products, measuring quality aspects and other purposes. They may also modify the metrics or use metrics which are not included here. This Technical Report is applicable to any kind of software product, although each of the metrics is not always applicable to every kind of software product.

ISO/IEC 9126-1 defines terms for the software quality characteristics and how these characteristics are decomposed into subcharacteristics. ISO/IEC 9126-1, however, does not describe how any of these subcharacteristics could be measured. ISO/IEC TR 9126-2 defines external metrics, ISO/IEC TR 9126-3 defines internal metrics and ISO/IEC 9126-4 defines quality in use metrics, for measurement of the characteristics or the subcharacteristics. Internal metrics measure the software itself, external metrics measure the behaviour of the computer-based system that includes the software, and quality in use metrics measure the effects of using the software in a specific context of use.

This Technical Report is intended to be used together with ISO/IEC 9126-1. It is strongly recommended to read ISO/IEC 14598-1 and ISO/IEC 9126-1, prior to using this Technical Report, particularly if the reader is not familiar with the use of software metrics for product specification and evaluation.

Clauses 1 to 7 and Annexes A to D are common to ISO/IEC TR 9126-2, ISO/IEC TR 9126-3, and ISO/IEC 9126-4.

TECHNICAL REPORT ISO/IEC TR 9126-2:2003(E)

© ISO/IEC 2003 – All rights reserved 1

Software engineering — Product quality —

Part 2: External metrics

1 Scope

This Technical Report defines external metrics for quantitatively measuring external software quality in terms of characteristics and subcharacteristics defined in ISO/IEC 9126-1, and is intended to be used together with ISO/IEC 9126-1.

This Technical Report contains:

I. an explanation of how to apply software quality metrics

II. a basic set of metrics for each subcharacteristic

III. an example of how to apply metrics during the software product life cycle

This Technical Report does not assign ranges of values of these metrics to rated levels or to grades of compliance, because these values are defined for each software product or a part of the software product, by its nature, depending on such factors as category of the software, integrity level and users' needs. Some attributes may have a desirable range of values, which does not depend on specific user needs but depends on generic factors; for example, human cognitive factors.

This Technical Report can be applied to any kind of software for any application. Users of this Technical Report can select or modify and apply metrics and measures from this Technical Report or may define application-specific metrics for their individual application domain. For example, the specific measurement of quality characteristics such as safety or security may be found in International Standards or Technical Reports provided by IEC 65 and ISO/IEC JTC 1/SC 27.

Intended users of this Technical Report include:

— Acquirer (an individual or organization that acquires or procures a system, software product or software service from a supplier);

— Evaluator (an individual or organization that performs an evaluation. An evaluator may, for example, be a testing laboratory, the quality department of a software development organization, a government organization or a user);

— Developer (an individual or organization that performs development activities, including requirements analysis, design, and testing through acceptance during the software life cycle process);

— Maintainer (an individual or organization that performs maintenance activities);

— Supplier (an individual or organization that enters into a contract with the acquirer for the supply of a system, software product or software service under the terms of the contract) when validating software quality at qualification test;

— User (an individual or organization that uses the software product to perform a specific function) when evaluating quality of software product at acceptance test;

— Quality manager (an individual or organization that performs a systematic examination of the software product or software services) when evaluating software quality as part of quality assurance and quality control.

ISO/IEC TR 9126-2:2003(E)

2 © ISO/IEC 2003 – All rights reserved

2 Conformance

There are no conformance requirements in this Technical Report.

NOTE General conformance requirements for metrics are in ISO/IEC 9126-1 Quality model.

3 Normative references

ISO/IEC 9126-1:2001, Software engineering — Product quality — Part 1: Quality model

ISO/IEC TR 9126-31), Software engineering — Product quality — Part 3: Internal metrics

ISO/IEC 9126-41), Software engineering — Product quality — Part 4: Quality in use metrics

ISO/IEC 14598-1:1999, Information technology — Software product evaluation — Part 1: General overview

ISO/IEC 14598-2:2000, Software engineering — Product evaluation — Part 2: Planning and management

ISO/IEC 14598-3:2000, Software engineering — Product evaluation — Part 3: Process for developers

ISO/IEC 14598-4:1999, Software engineering — Product evaluation — Part 4: Process for acquirers

ISO/IEC 14598-5:1998, Information technology — Software product evaluation — Part 5: Process for evaluators

ISO/IEC 14598-6:2001, Software engineering — Product evaluation — Part 6: Documentation of evaluation modules

ISO/IEC 12207:1995, Information technology — Software life cycle processes

ISO/IEC 14143-1:1998, Information technology — Software measurement — Functional size measurement — Part 1: Definition of concepts

ISO/IEC 2382-20:1990, Information technology — Vocabulary — Part 20: System development

ISO 9241-10:1996, Ergonomic requirements for office work with visual display terminals (VDTs) — Part 10: Dialogue principles

4 Terms and definitions

For the purposes of this document, the terms and definitions given in ISO/IEC 14598-1:1999 and ISO/IEC 9126-1:2001 apply. They are also listed in Annex D.

5 Abbreviated terms

The following abbreviations are used in this Technical Report:

SQA — Software Quality Assurance (Group)

SLCP — Software Life Cycle Processes

1) To be published.

ISO/IEC TR 9126-2:2003(E)


6 Use of software quality metrics

These Technical Reports (ISO/IEC TR 9126-2 External metrics, ISO/IEC TR 9126-3 Internal metrics and ISO/IEC 9126-4 Quality in use metrics) provide a suggested set of software quality metrics (external, internal and quality in use metrics) to be used with the ISO/IEC 9126-1 Quality model. The user of these Technical Reports may modify the metrics defined, and/or may also use metrics not listed. When using a modified or a new metric not identified in these Technical Reports, the user should specify how the metrics relate to the ISO/IEC 9126-1 quality model or any other substitute quality model that is being used.

The user of these Technical Reports should select the quality characteristics and subcharacteristics to be evaluated, from ISO/IEC 9126-1; identify the appropriate direct and indirect measures, identify the relevant metrics and then interpret the measurement result in an objective manner. The user of these Technical Reports also may select product quality evaluation processes during the software life cycle from the ISO/IEC 14598 series of standards. These give methods for measurement, assessment and evaluation of software product quality. They are intended for use by developers, acquirers and independent evaluators, particularly those responsible for software product evaluation (see Figure 1).

software product effect of softwareproduct

quality in usemetrics

quality inuse

internalquality

internal metrics external metrics

externalquality

contexts ofusedepends on

influences influences

depends on

Figure 1 – Relationship between types of metrics

The internal metrics may be applied to a non-executable software product during its development stages (such as request for proposal, requirements definition, design specification or source code). Internal metrics provide the users with the ability to measure the quality of the intermediate deliverables and thereby predict the quality of the final product. This allows the user to identify quality issues and initiate corrective action as early as possible in the development life cycle.

The external metrics may be used to measure the quality of the software product by measuring the behaviour of the system of which it is a part. The external metrics can only be used during the testing stages of the life cycle process and during any operational stages. The measurement is performed when executing the software product in the system environment in which it is intended to operate.

The quality in use metrics measure whether a product meets the needs of specified users to achieve specified goals with effectiveness, productivity, safety and satisfaction in a specified context of use. This can be only achieved in a realistic system environment.

User quality needs can be specified as quality requirements by quality in use metrics, by external metrics, and sometimes by internal metrics. These requirements specified by metrics should be used as criteria when a product is evaluated.

It is recommended to use internal metrics having a relationship as strong as possible with the target external metrics so that they can be used to predict the values of external metrics. However, it is often difficult to design a rigorous theoretical model that provides a strong relationship between internal metrics and external metrics. Therefore, a hypothetical model that may contain ambiguity may be designed and the extent of the relationship may be modelled statistically during the use of metrics.

Recommendations and requirements related to validity and reliability are given in ISO/IEC 9126-1, Clause A.4. Additional detailed considerations when using metrics are given in Annex A of this Technical Report.

ISO/IEC TR 9126-2:2003(E)


7 How to read and use the metrics tables

The metrics listed in Clause 8 are categorized by the characteristics and subcharacteristics in ISO/IEC 9126-1. The following information is given for each metric in the table:

a) Metric name: Corresponding metrics in the internal metrics table and external metrics table have similar names.

b) Purpose of the metric: This is expressed as the question to be answered by the application of the metric.

c) Method of application: Provides an outline of the application.

d) Measurement, formula and data element computations: Provides the measurement formula and explains the meanings of the used data elements.

NOTE In some situations more than one formula is proposed for a metric.

e) Interpretation of measured value: Provides the range and preferred values.

f) Metric scale type: Type of scale used by the metric. Scale types used are; Nominal scale, Ordinal scale, Interval scale, Ratio scale and Absolute scale.

NOTE A more detailed explanation is given in Annex C.

g) Measure type: Types used are; Size type (e.g. Function size, Source size), Time type (e.g. Elapsed time, User time), Count type (e.g. Number of changes, Number of failures).

NOTE A more detailed explanation is given in Annex C.

h) Input to measurement: Source of data used in the measurement.

i) ISO/IEC 12207 SLCP Reference: Identifies software life cycle process(es) where the metric is applicable.

j) Target audience: Identifies the user(s) of the measurement results.

8 Metrics tables

The metrics listed in this clause are not intended to be an exhaustive set and may not have been validated. They are listed by software quality characteristics and subcharacteristics, in the order introduced in ISO/IEC 9126-1.

Metrics, which may be applicable, are not limited to these listed here. Additional specific metrics for particular purposes are provided in other related documents, such as functional size measurement or precise time efficiency measurement.

NOTE 1 It is recommended to refer a specific metric or measurement form from specific standards, technical reports or guidelines. Functional size measurement is defined in ISO/IEC 14143. An example of precise time efficiency measurement can be referred from ISO/IEC 14756.

Metrics should be validated before application in a specific environment (see Annex A).

NOTE 2 This list of metrics is not finalized, and may be revised in future versions of this Technical Report. Readers of this Technical Report are invited to provide feedback.

ISO/IEC TR 9126-2:2003(E)


8.1 Functionality metrics

An external functionality metric should be able to measure an attribute such as the functional behaviour of a system containing the software. The behaviour of the system may be observed from the following perspectives:

a) Differences between the actual executed results and the quality requirements specification;

NOTE 1 The quality requirements specification for functionality is usually described as the functional requirements specification.

b) FunctionaI inadequacy detected during real user operation which is not stated but is implied as a requirement in the specification.

NOTE 2 When implied operations or functions are detected, they should be reviewed, approved and stated in the specifications. Their extent to be fulfilled should be agreed.

8.1.1 Suitability metrics

An external suitability metric should be able to measure an attribute such as the occurrence of an unsatisfying function or the occurrence of an unsatisfying operation during testing and user operation of the system.

An unsatisfying function or operation may be:

a) Functions and operations that do not perform as specified in user manuals or requirement specification.

b) Functions and operations that do not provide a reasonable and acceptable outcome to achieve the intended specific objective of the user task.

8.1.2 Accuracy metrics

An external accuracy metric should be able to measure an attribute such as the frequency of users encountering the occurrence of inaccurate matters which includes:

a) Incorrect or imprecise result caused by inadequate data; for example, data with too few significant digits for accurate calculation;

b) Inconsistency between actual operation procedures and described ones in the operation manual;

c) Differences between the actual and reasonable expected results of tasks performed during operation.

8.1.3 Interoperability metrics

An external interoperability metric should be able to measure an attribute such as the number of functions or occurrences of less communicativeness involving data and commands, which are transferred easily between the software product and other systems, other software products, or equipment which are connected.

8.1.4 Security metrics

An external security metric should be able to measure an attribute such as the number of functions with, or occurrences of security problems, which are:

a) Failing to prevent leak of secure output information or data;

b) Failing to prevent loss of important data;

c) Failing to defend against illegal access or illegal operation.

ISO/IEC TR 9126-2:2003(E)


NOTE 1 It is recommended that penetration tests be performed to simulate attack, because such a security attack does not normally occur in the usual testing. Real security metrics may only be taken in “real life system environment”, that is “quality in use”.

NOTE 2 Security protection requirements vary widely from the case of a stand-alone-system to the case of a system connected to the Internet. The determination of the required functionality and the assurance of their effectiveness have been addressed extensively in related standards. The user of this standard should determine security functions using appropriate methods and standards in those cases where the impact of any damage caused is important or critical. In the other case the user may limit his scope to generally accepted “Information Technology (IT)” protection measures such as virus protection backup methods and access control.

8.1.5 Functionality compliance metrics

An external functionality compliance metric should be able to measure an attribute such as the number of functions with, or occurrences of compliance problems, which are the software product failing to adhere to standards, conventions, contracts or other regulatory requirements.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

1.1

Suita

bilit

y m

etric

s

Exte

rnal

sui

tabi

lity

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SL

CP

Ref

eren

ce

Targ

et

audi

ence

Func

tiona

l ad

equa

cy

How

ade

quat

e ar

e th

e ev

alua

ted

func

tions

?

Num

ber o

f fun

ctio

ns th

at

are

suita

ble

for p

erfo

rmin

g th

e sp

ecifi

ed ta

sks

com

parin

g to

the

num

ber

of fu

nctio

n ev

alua

ted.

X=1-

A/B

A=

Num

ber o

f fun

ctio

ns in

whi

ch p

robl

ems

are

dete

cted

in e

valu

atio

n

B= N

umbe

r of f

unct

ions

eva

luat

ed

0 <=

X <

= 1

Th

e cl

oser

to

1.0,

the

mor

e ad

equa

te.

Abso

lute

X= C

ount

/C

ount

A=

Cou

ntB=

Cou

nt

Req

uire

men

t sp

ecifi

catio

n (R

eq. S

pec.

) Ev

alua

tion

repo

rt

6.5

Valid

atio

n,

6.3

Qua

lity

Assu

ranc

e,5.

3 Q

ualif

icat

ion

test

ing

Dev

elop

er,

SQA

Func

tiona

l im

plem

enta

tion

com

plet

enes

s

How

com

plet

e is

the

impl

emen

tatio

n ac

cord

ing

to

requ

irem

ent

spec

ifica

tions

?

Do

func

tiona

l tes

ts (b

lack

bo

x te

st) o

f the

sys

tem

ac

cord

ing

to th

e re

quire

men

t spe

cific

atio

ns.

Cou

nt th

e nu

mbe

r of

mis

sing

func

tions

det

ecte

d in

eva

luat

ion

and

com

pare

w

ith th

e nu

mbe

r of f

unct

ion

desc

ribed

in th

e re

quire

men

t spe

cific

atio

ns.

X =

1 - A

/ B

A =

Num

ber o

f mis

sing

func

tions

det

ecte

d in

ev

alua

tion

B =

Num

ber o

f fun

ctio

ns d

escr

ibed

in

requ

irem

ent s

peci

ficat

ions

0<=X

<=1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Req

. spe

c.

Eval

uatio

n re

port

6.5

Valid

atio

n,

6.3

Qua

lity

Assu

ranc

e,5.

3 Q

ualif

icat

ion

test

ing

Dev

elop

er,

SQA

FOO

TNO

TES

1 In

put t

o th

e m

easu

rem

ent p

roce

ss is

the

upda

ted

requ

irem

ent s

peci

ficat

ion.

Any

cha

nges

iden

tifie

d du

ring

life

cycl

e m

ust b

e ap

plie

d to

the

requ

irem

ent s

peci

ficat

ions

bef

ore

usin

g in

m

easu

rem

ent p

roce

ss.

2 Th

is m

etric

is s

ugge

sted

as

expe

rimen

tal u

se.

NO

TE

Any

mis

sing

func

tion

cann

ot b

e ex

amin

ed b

y te

stin

g be

caus

e it

is n

ot im

plem

ente

d. F

or d

etec

ting

mis

sing

func

tions

, it i

s su

gges

ted

that

eac

h fu

nctio

n st

ated

in a

re

quire

men

t spe

cific

atio

n be

test

ed o

ne b

y on

e du

ring

func

tiona

l tes

ting.

Suc

h re

sults

bec

ome

inpu

t to

“Fun

ctio

nal i

mpl

emen

tatio

n co

mpl

eten

ess”

met

ric. F

or d

etec

ting

func

tions

whi

ch a

re im

plem

ente

d bu

t ina

dequ

ate,

it is

sug

gest

ed th

at e

ach

func

tion

be te

sted

for

mul

tiple

spe

cifie

d ta

sks.

Suc

h re

sults

bec

ome

inpu

t to

the

“Fun

ctio

nal

adeq

uacy

” met

ric. T

here

fore

, use

rs o

f met

rics

are

sugg

este

d to

use

bot

h th

ese

met

rics

durin

g fu

nctio

nal t

estin

g.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

1.1

(con

tinue

d)

Exte

rnal

sui

tabi

lity

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SL

CP

Ref

eren

ce

Targ

et

audi

ence

Func

tiona

l im

plem

enta

tion

cove

rage

How

cor

rect

is th

e fu

nctio

nal

impl

emen

tatio

n?

Do

func

tiona

l tes

ts (b

lack

bo

x te

st) o

f the

sys

tem

ac

cord

ing

to th

e re

quire

men

t spe

cific

atio

ns.

Cou

nt th

e nu

mbe

r of

inco

rrect

ly im

plem

ente

d or

m

issi

ng fu

nctio

ns d

etec

ted

in e

valu

atio

n an

d co

mpa

re

with

the

tota

l num

ber o

f fu

nctio

ns d

escr

ibed

in th

e re

quire

men

t spe

cific

atio

ns

Cou

nt th

e nu

mbe

r of

func

tions

that

are

com

plet

e ve

rsus

the

ones

that

are

no

t.

X=1-

A /

B

A= N

umbe

r of i

ncor

rect

ly im

plem

ente

d or

m

issi

ng fu

nctio

ns d

etec

ted

in e

valu

atio

n B=

Num

ber o

f fun

ctio

ns d

escr

ibed

in

requ

irem

ent s

peci

ficat

ions

0<=X

<=1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Req

. spe

c.

Eval

uatio

n re

port

6.5

Valid

atio

n,

6.3

Qua

lity

Assu

ranc

e,5.

3 Q

ualif

icat

ion

test

ing

Dev

elop

er,

SQA

FOO

TNO

TES

1 In

put t

o th

e m

easu

rem

ent p

roce

ss is

the

upda

ted

requ

irem

ent s

peci

ficat

ion.

Any

cha

nges

iden

tifie

d du

ring

life

cycl

e m

ust b

e ap

plie

d to

the

requ

irem

ent s

peci

ficat

ions

bef

ore

usin

g in

m

easu

rem

ent p

roce

ss.

2 Th

is m

easu

re re

pres

ents

a b

inar

y ga

te c

heck

ing

of d

eter

min

ing

the

pres

ence

of a

feat

ure.

Func

tiona

l sp

ecifi

catio

n st

abili

ty

(vol

atili

ty)

How

sta

ble

is th

e

func

tiona

l sp

ecifi

catio

n af

ter

ente

ring

oper

atio

n?

Cou

nt th

e nu

mbe

r of

func

tions

des

crib

ed i

n fu

nctio

nal s

peci

ficat

ions

th

at h

ad to

be

chan

ged

afte

r the

sys

tem

is p

ut in

to

oper

atio

n an

d co

mpa

re

with

the

tota

l num

ber o

f fu

nctio

ns d

escr

ibed

in th

e re

quire

men

t spe

cific

atio

ns.

X =

1- A

/ B

A=

Num

ber o

f fun

ctio

ns c

hang

ed a

fter

ente

ring

oper

atio

n st

artin

g fro

m e

nter

ing

oper

atio

n B=

Num

ber o

f fun

ctio

ns d

escr

ibed

in

requ

irem

ent s

peci

ficat

ions

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X= C

ount

/ C

ount

Req

. spe

c.

Eval

uatio

n re

port

6.8

Prob

lem

R

esol

utio

n5.

4 O

pera

tion

Mai

ntai

ner

SQA

FOO

TNO

TE

This

met

ric is

sug

gest

ed a

s ex

perim

enta

l use

.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

1.2

Accu

racy

met

rics

Exte

rnal

acc

urac

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Accu

racy

to

expe

ctat

ion

Are

diffe

renc

es

betw

een

the

actu

al

and

reas

onab

le

expe

cted

re

sults

acc

epta

ble?

Do

inpu

t .vs

. out

put t

est

case

s an

d co

mpa

re th

e ou

tput

to re

ason

able

ex

pect

ed re

sults

. C

ount

the

num

ber o

f cas

es

enco

unte

red

by th

e us

ers

with

an

unac

cept

able

di

ffere

nce

from

reas

onab

le

expe

cted

resu

lts.

X=A

/ T

A= N

umbe

r of c

ases

enc

ount

ered

by

the

user

s w

ith a

diff

eren

ce a

gain

st to

reas

onab

le

expe

cted

resu

lts b

eyon

d al

low

able

T=

Ope

ratio

n tim

e

0<=X

Th

e cl

oser

to

0 is

the

bette

r.

Rat

io

A= C

ount

T= T

ime

X= C

ount

/ Ti

me

Req

. spe

c.

Use

r op

erat

ion

man

ual

Hea

ring

to

user

s Te

st re

port

6.5

Valid

atio

n 6.

3 Q

ualit

y As

sura

nce

Dev

elop

er

Use

r

FOO

TNO

TE

Rea

sona

ble

expe

cted

resu

lts m

ight

be

iden

tifie

d in

a re

quire

men

t spe

cific

atio

n, a

use

r ope

ratio

n m

anua

l, or

use

rs’ e

xpec

tatio

ns.

Com

puta

tiona

l Ac

cura

cy

How

ofte

n do

the

end

user

s en

coun

ter

inac

cura

te re

sults

?

Rec

ord

the

num

ber o

f in

accu

rate

com

puta

tions

ba

sed

on s

peci

ficat

ions

.

X=A

/ T

A= N

umbe

r of i

nacc

urat

e co

mpu

tatio

ns

enco

unte

red

by u

sers

T=

Ope

ratio

n tim

e

0<=X

Th

e cl

oser

to

0 is

the

bette

r.

Rat

io

A= C

ount

T= T

ime

X= C

ount

/ Ti

me

Req

. spe

c.

Test

repo

rt 6.

5 Va

lidat

ion

6.3

Qua

lity

Assu

ranc

e

Dev

elop

er

Use

r

Prec

isio

n H

ow o

ften

do th

e en

d us

ers

enco

unte

r re

sults

with

in

adeq

uate

pr

ecis

ion

?

Rec

ord

the

num

ber o

f re

sults

with

inad

equa

te

prec

isio

n.

X=A

/ T

A= N

umbe

r of

resu

lts e

ncou

nter

ed b

y th

e us

ers

with

leve

l of p

reci

sion

diff

eren

t fro

m

requ

ired

T= O

pera

tion

time

0<=X

Th

e cl

oser

to

0 is

the

bette

r.

Rat

io

A= C

ount

T= T

ime

X= C

ount

/ Ti

me

Req

. spe

c.

Test

repo

rt

6.5

Valid

atio

n 6.

3 Q

ualit

y As

sura

nce

Dev

elop

er

Use

r

NO

TE

D

ata

elem

ents

for

com

puta

tion

of e

xter

nal

met

rics

are

desi

gned

to

use

exte

rnal

ly a

cces

sibl

e in

form

atio

n, b

ecau

se i

t is

hel

pful

for

end

use

rs,

oper

ator

s,

mai

ntai

ners

or

acqu

irers

to u

se e

xter

nal m

etric

s. T

here

fore

, the

tim

e ba

sis

met

ric o

ften

appe

ars

in e

xter

nal m

etric

s an

d is

diff

eren

t fro

m in

tern

al o

nes.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

1.3

Inte

rope

rabi

lity

met

rics

Exte

rnal

inte

rope

rabi

lity

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Dat

a ex

chan

geab

ility

(Dat

a fo

rmat

ba

sed)

How

cor

rect

ly h

ave

the

exch

ange

in

terfa

ce fu

nctio

ns fo

r sp

ecifi

ed d

ata

trans

fer b

een

impl

emen

ted?

Test

eac

h do

wns

tream

in

terfa

ce fu

nctio

n ou

tput

re

cord

form

at o

f the

sy

stem

acc

ordi

ng to

the

data

fiel

ds s

peci

ficat

ions

. C

ount

the

num

ber o

f dat

a fo

rmat

s th

at a

re a

ppro

ved

to b

e ex

chan

ged

with

oth

er

softw

are

or s

yste

m d

urin

g te

stin

g on

dat

a ex

chan

ges

in c

ompa

ring

with

the

tota

l nu

mbe

r.

X= A

/ B

A= N

umbe

r of d

ata

form

ats

whi

ch a

re

appr

oved

to b

e ex

chan

ged

succ

essf

ully

with

ot

her s

oftw

are

or s

yste

m d

urin

g te

stin

g on

da

ta e

xcha

nges

B=

Tot

al n

umbe

r of d

ata

form

ats

to b

e ex

chan

ged

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Req

. spe

c.

(Use

r man

ual)

Test

repo

rt

6.5

Valid

atio

n D

evel

oper

FOO

TNO

TE

It is

reco

mm

ende

d to

test

spe

cifie

d da

ta tr

ansa

ctio

n.

Dat

a ex

chan

geab

ility

(Use

r’s s

ucce

ss

atte

mpt

bas

ed)

How

ofte

n do

es th

e en

d us

er fa

il to

ex

chan

ge

data

bet

wee

n ta

rget

so

ftwar

e an

d ot

her

softw

are?

H

ow o

ften

are

the

data

tran

sfer

s be

twee

n ta

rget

so

ftwar

e an

d ot

her

softw

are

succ

essf

ul?

Can

use

r usu

ally

su

ccee

d in

ex

chan

ging

dat

a?

Cou

nt th

e nu

mbe

r of c

ases

th

at in

terfa

ce fu

nctio

ns

wer

e us

ed a

nd fa

iled.

a) X

= 1

- A

/ B

A= N

umbe

r of c

ases

in w

hich

use

r fai

led

to

exch

ange

dat

a w

ith o

ther

sof

twar

e or

sy

stem

s

B= N

umbe

r of c

ases

in w

hich

use

r at

tem

pted

to e

xcha

nge

data

b)

Y=

A / T

T=

Per

iod

of o

pera

tion

time

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

0<=Y

Th

e cl

oser

to

0, is

the

bette

r.

a)

Abso

lute

b)

Rat

io

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Y= C

ount

/ Ti

me

T= T

ime

Req

. spe

c.

(Use

r man

ual)

Test

repo

rt

5.4

Ope

ratio

n M

aint

aine

r

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

1.4

Secu

rity

met

rics

Exte

rnal

sec

urity

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Acce

ss

audi

tabi

lity

How

com

plet

e is

the

audi

t tra

il co

ncer

ning

th

e us

er a

cces

s to

th

e sy

stem

and

dat

a? Ev

alua

te th

e am

ount

of

acce

sses

that

the

syst

em

reco

rded

in th

e ac

cess

hi

stor

y da

taba

se.

X= A

/ B

A= N

umbe

r of “

user

acc

esse

s to

the

syst

em

and

data

” rec

orde

d in

the

acce

ss h

isto

ry

data

base

B=

Num

ber o

f “ u

ser a

cces

ses

to th

e sy

stem

an

d da

ta” d

one

durin

g ev

alua

tion

0<=X

<=1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Test

spe

c.

Test

repo

rt

6.5

Valid

atio

n

Dev

elop

er

FOO

TNO

TES

1 Ac

cess

es to

dat

a m

ay b

e m

easu

red

only

with

test

ing

activ

ities

. 2

This

met

ric is

sug

gest

ed a

s an

exp

erim

enta

l use

. 3

It is

reco

mm

ende

d th

at p

enet

ratio

n te

sts

be p

erfo

rmed

to s

imul

ate

atta

cks,

bec

ause

suc

h se

curit

y at

tack

s do

not

nor

mal

ly o

ccur

in th

e us

ual t

estin

g. R

eal s

ecur

ity m

etric

s m

ay o

nly

be

take

n in

"rea

l life

sys

tem

env

ironm

ent",

that

is "q

ualit

y in

use

". 4

“Use

r acc

ess

to th

e sy

stem

and

dat

a” re

cord

may

incl

ude

“viru

s de

tect

ion

reco

rd“ f

or v

irus

prot

ectio

n. T

he a

im o

f the

con

cept

of c

ompu

ter v

irus

prot

ectio

n is

to c

reat

e su

itabl

e sa

fegu

ards

w

ith w

hich

the

occu

rrenc

e of

com

pute

r viru

ses

in s

yste

ms

can

be p

reve

nted

or d

etec

ted

as e

arly

as

poss

ible

.

Acce

ss

cont

rolla

bilit

y H

ow c

ontro

llabl

e is

ac

cess

to th

e

syst

em?

Cou

nt n

umbe

r of d

etec

ted

illega

l ope

ratio

ns w

ith

com

parin

g to

num

ber o

f ille

gal o

pera

tions

as

in th

e sp

ecifi

catio

n.

X= A

/ B

A= N

umbe

r of d

etec

ted

diffe

rent

type

s of

ille

gal o

pera

tions

B=

Num

ber o

f typ

es o

f ille

gal o

pera

tions

as

in

the

spec

ifica

tion

0<=X

<=1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Test

spe

c.

Test

repo

rt O

pera

tion

repo

rt

6.5

Valid

atio

n 6.

3 Q

ualit

y As

sura

nce

Dev

elop

er

FOO

TNO

TES

1 If

it is

nec

essa

ry to

com

plem

ent d

etec

tion

of u

nexp

ecte

d ille

gal o

pera

tions

add

ition

al in

tens

ive

abno

rmal

ope

ratio

n te

stin

g sh

ould

be

cond

ucte

d.

2 It

is re

com

men

ded

that

pen

etra

tion

test

s be

per

form

ed to

sim

ulat

e at

tack

, bec

ause

suc

h se

curit

y at

tack

s do

not

nor

mal

ly o

ccur

in th

e us

ual t

estin

g. R

eal s

ecur

ity m

etric

s m

ay o

nly

be

take

n in

"rea

l life

sys

tem

env

ironm

ent",

that

is "q

ualit

y in

use

". 3

Func

tions

pre

vent

una

utho

rized

per

sons

from

cre

atin

g, d

elet

ing

or m

odify

ing

prog

ram

s or

info

rmat

ion.

The

refo

re, i

t is

sugg

este

d to

incl

ude

such

ille

gal o

pera

tion

type

s in

test

cas

es.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

1.4

(con

tinue

d)

Exte

rnal

sec

urity

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Dat

a co

rrup

tion

prev

entio

n W

hat i

s th

e fre

quen

cy

of d

ata

corru

ptio

n ev

ents

?

Cou

nt th

e oc

curre

nces

of

maj

or a

nd m

inor

dat

a co

rrupt

ion

even

ts.

a) X

= 1

– A

/ N

A= N

umbe

r of t

imes

that

a m

ajor

dat

a co

rrupt

ion

even

t occ

urre

d N

= N

umbe

r of t

est c

ases

trie

d to

cau

se d

ata

corru

ptio

n ev

ent

b)

Y= 1

- B /

N

B= N

umbe

r of t

imes

that

a m

inor

dat

a co

rrupt

ion

even

t occ

urre

d c)

Z=

A /

T or

B /

T T=

per

iod

of o

pera

tion

time

(dur

ing

oper

atio

n te

stin

g)

0<=X

<= 1

Th

e clo

ser t

o 1.

0 is

the

bette

r. 0<

=Y<=

1

The

close

r to

1.0

is th

e be

tter.

0<=Z

Th

e clo

ser t

o 0,

is

the

bette

r.

a)

Abso

lute

b)

Abso

lute

c)

Rat

io

A= C

ount

B= C

ount

N

= C

ount

X=

Cou

nt/

Cou

nt

Y= C

ount

/C

ount

T=

Tim

e Z=

Cou

nt/

Tim

e

Test

spe

c.

Test

repo

rt O

pera

tion

repo

rt

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.4

Ope

ratio

n

Mai

ntai

ner

Dev

elop

er

FOO

TNO

TES

1 In

tens

ive

abno

rmal

ope

ratio

n te

stin

g is

nee

ded

to o

btai

n m

inor

and

maj

or d

ata

corru

ptio

n ev

ents

. 2

It is

reco

mm

ende

d to

gra

de th

e im

pact

of d

ata

corru

ptio

n ev

ents

suc

h as

the

follo

win

g ex

ampl

es:

Maj

or (f

atal

) dat

a co

rrupt

ion

even

t: - r

epro

duct

ion

and

reco

very

impo

ssib

le;

- sec

ond

affe

ctio

n di

strib

utio

n to

o w

ide;

- i

mpo

rtanc

e of

dat

a its

elf.

Min

or d

ata

corru

ptio

n ev

ent:

- rep

rodu

ctio

n or

reco

very

pos

sibl

e an

d - n

o se

cond

affe

ctio

n di

strib

utio

n;

- im

porta

nce

of d

ata

itsel

f. 3

Dat

a el

emen

ts fo

r com

puta

tion

of e

xter

nal m

etric

s ar

e de

sign

ed to

use

ext

erna

lly a

cces

sibl

e in

form

atio

n, b

ecau

se it

is h

elpf

ul fo

r end

use

rs, o

pera

tors

, mai

ntai

ners

or a

cqui

rers

to u

se

exte

rnal

met

rics.

The

refo

re, c

ount

ing

even

ts a

nd ti

mes

use

d he

re a

re d

iffer

ent f

rom

cor

resp

ondi

ng in

tern

al m

etric

. 4

It is

reco

mm

ende

d th

at p

enet

ratio

n te

sts

be p

erfo

rmed

to s

imul

ate

atta

ck, b

ecau

se s

uch

secu

rity

atta

cks

do n

ot n

orm

ally

occ

ur in

the

usua

l tes

ting.

R

eal s

ecur

ity m

etric

s m

ay o

nly

be ta

ken

in "r

eal l

ife s

yste

m e

nviro

nmen

t", th

at is

"qua

lity

in u

se"

5 Th

is m

etric

is s

ugge

sted

as

an e

xper

imen

tal u

se.

6 D

ata

back

up is

one

of t

he e

ffect

ive

way

s to

pre

vent

dat

a co

rrupt

ion.

The

cre

atio

n of

bac

k up

ens

ures

that

nec

essa

ry d

ata

can

be re

stor

ed q

uick

ly in

the

even

t tha

t par

ts o

f the

ope

rativ

e da

ta a

re lo

st. H

owev

er, d

ata

back

up

is re

gard

ed a

s a

part

of th

e co

mpo

sitio

n of

the

relia

bilit

y m

etric

s in

this

repo

rt.

7 It

is s

ugge

sted

that

this

met

ric b

e us

ed e

xper

imen

tally

.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

1.5

Func

tiona

lity

com

plia

nce

met

rics

Exte

rnal

func

tiona

lity

com

plia

nce

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Func

tiona

l co

mpl

ianc

e H

ow c

ompl

iant

is th

e fu

nctio

nalit

y of

the

prod

uct t

o ap

plic

able

re

gula

tions

, st

anda

rds

and

conv

entio

ns?

Cou

nt th

e nu

mbe

r of i

tem

s re

quiri

ng c

ompl

ianc

e th

at

have

bee

n m

et a

nd

com

pare

with

the

num

ber

of it

ems

requ

iring

co

mpl

ianc

e in

the

spec

ifica

tion.

D

esig

n te

st c

ases

in

acco

rdan

ce w

ith

com

plia

nce

item

s.

Con

duct

func

tiona

l tes

ting

for t

hese

test

cas

es.

Cou

nt th

e nu

mbe

r of

com

plia

nce

item

s th

at

have

bee

n sa

tisfie

d.

X =

1 -

A /

B A=

Num

ber o

f fun

ctio

nalit

y co

mpl

ianc

e ite

ms

spec

ified

that

hav

e no

t bee

n im

plem

ente

d du

ring

test

ing

B= T

otal

num

ber o

f fun

ctio

nalit

y co

mpl

ianc

e ite

ms

spec

ified

0<=

X <=

1 Th

e clo

ser t

o 1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B= C

ount

X= C

ount

/C

ount

Prod

uct

desc

riptio

n (U

ser m

anua

l or

Sp

ecifi

catio

n)

of c

ompl

ianc

e an

d re

late

d st

anda

rds,

co

nven

tions

or

regu

latio

ns Te

st

spec

ifica

tion

and

repo

rt

5.3

Qua

lific

atio

n te

stin

g 6.

5 Va

lidat

ion

Supp

lier

Use

r

FOO

TNO

TES

1 It

may

be

usef

ul to

col

lect

sev

eral

mea

sure

d va

lues

alo

ng ti

me,

to a

naly

se th

e tre

nd o

f inc

reas

ingl

y sa

tisfie

d co

mpl

ianc

e ite

ms

and

to d

eter

min

e w

heth

er th

ey a

re fu

lly s

atis

fied

or n

ot.

2 It

is s

ugge

sted

to c

ount

num

ber o

f fai

lure

s, b

ecau

se p

robl

em d

etec

tion

is a

n ob

ject

ive

of e

ffect

ive

test

ing

and

also

sui

tabl

e fo

r cou

ntin

g an

d re

cord

ing.

Inte

rfac

e

stan

dard

co

mpl

ianc

e

How

com

plia

nt a

re

the

inte

rface

s to

ap

plic

able

re

gula

tions

, st

anda

rds

and

conv

entio

ns?

Cou

nt th

e nu

mbe

r of

inte

rface

s th

at m

eet

requ

ired

com

plia

nce

and

com

pare

with

the

num

ber

of in

terfa

ces

requ

iring

co

mpl

ianc

e as

in th

e sp

ecifi

catio

ns.

X= A

/ B

A= N

umbe

r of c

orre

ctly

impl

emen

ted

inte

rface

s as

spe

cifie

d B=

Tot

al n

umbe

r of i

nter

face

s re

quiri

ng

com

plia

nce

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Prod

uct

desc

riptio

n of

co

mpl

ianc

e an

d re

late

d st

anda

rds,

co

nven

tions

or

regu

latio

ns Te

st

spec

ifica

tion

and

repo

rt

6.5

Valid

atio

n D

evel

oper

FOO

TNO

TE

All s

peci

fied

attri

bute

s of

a s

tand

ard

mus

t be

test

ed.

ISO/IEC TR 9126-2:2003(E)


8.2 Reliability metrics

An external reliability metric should be able to measure attributes related to the behaviours of the system of which the software is a part during execution testing to indicate the extent of reliability of the software in that system during operation. Systems and software are not distinguished from each other in most cases.

8.2.1 Maturity metrics

An external maturity metric should be able to measure such attributes as the software freedom of failures caused by faults existing in the software itself.

8.2.2 Fault tolerance metrics

An external fault tolerance metric should be related to the software capability of maintaining a specified performance level in cases of operation faults or infringement of its specified interface.

8.2.3 Recoverability metrics

An external recoverability metric should be able to measure such attributes as the software with system being able to re-establish its adequate level of performance and recover the data directly affected in the case of a failure.

8.2.4 Reliability compliance metrics

An external reliability compliance metric should be able to measure an attribute such as the number of functions with, or occurrences of compliance problems, in which the software product fails to adhere to standards, conventions or regulations relating to reliability.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

2.1

Mat

urity

met

rics

Exte

rnal

mat

urity

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Estim

ated

la

tent

faul

t de

nsity

How

man

y pr

oble

ms

still

exis

t tha

t may

em

erge

as

futu

re

faul

ts?

Cou

nt th

e nu

mbe

r of f

aults

de

tect

ed d

urin

g a

defin

ed

trial

per

iod

and

pred

ict

pote

ntia

l num

ber o

f fut

ure

faul

ts u

sing

a re

liabi

lity

grow

th e

stim

atio

n m

odel

.

X= {A

BS( A

1 -

A2 )

} / B

(X

: Est

imat

ed re

sidu

ary

late

nt fa

ult d

ensi

ty)

ABS(

)= A

bsol

ute

Valu

e

A1 =

tota

l num

ber o

f pre

dict

ed la

tent

faul

ts in

a

softw

are

prod

uct

A2 =

tota

l num

ber o

f act

ually

det

ecte

d fa

ults

B=

pro

duct

siz

e

0<=X

It

depe

nds

on

stag

e of

te

stin

g.

At th

e la

ter

stag

es,

smal

ler i

s be

tter.

Abso

lute

A1=

Cou

nt

A2=

Cou

nt

B=

Siz

e X=

Cou

nt/

Siz

e

Test

repo

rt O

pera

tion

repo

rt Pr

oble

m

repo

rt

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n 6.

5 Va

lidat

ion

6.

3 Q

ualit

y As

sura

nce

Dev

elop

er

Test

er

SQA

Use

r

FOO

TNO

TES

1 W

hen

tota

l num

ber o

f act

ually

det

ecte

d fa

ults

bec

omes

larg

er th

an to

tal n

umbe

r of p

redi

cted

late

nt fa

ults

, it i

s re

com

men

ded

to p

redi

ct a

gain

and

est

imat

e m

ore

larg

er n

umbe

r. Es

timat

ed la

rger

num

bers

are

inte

nded

to p

redi

ct re

ason

able

late

nt fa

ilure

s, b

ut n

ot to

mak

e th

e pr

oduc

t loo

k be

tter.

2 It

is re

com

men

ded

to u

se s

ever

al re

liabi

lity

grow

th e

stim

atio

n m

odel

s an

d ch

oose

the

mos

t su

itabl

e on

e an

d re

peat

pre

dict

ion

with

mon

itorin

g de

tect

ed fa

ults

. 3

It m

ay b

e he

lpfu

l to

pred

ict u

pper

and

low

er n

umbe

r of l

aten

t fau

lts.

4 It

is n

eces

sary

to c

onve

rt th

is v

alue

(X) t

o th

e <0

,1>

inte

rval

if m

akin

g su

mm

aris

atio

n of

cha

ract

eris

tics.

Failu

re d

ensi

ty

agai

nst t

est

case

s

How

man

y fa

ilure

s w

ere

dete

cted

dur

ing

defin

ed tr

ial p

erio

d?

Cou

nt th

e nu

mbe

r of

dete

cted

failu

res

and

perfo

rmed

test

cas

es.

X= A

1 /

A2

A1 =

num

ber o

f det

ecte

d fa

ilure

s A2

= n

umbe

r of p

erfo

rmed

test

cas

es

0<=X

It

depe

nds

on

stag

e of

te

stin

g.

At th

e la

ter

stag

es,

smal

ler i

s be

tter.

Abso

lute

A1

= C

ount

A2

= C

ount

B

= S

ize

X=

Cou

nt/

Cou

nt

Test

repo

rt O

pera

tion

repo

rt Pr

oble

m

repo

rt

5.3

Inte

grat

ion

5.3

Q

ualif

icat

ion

test

ing

5.4

Ope

ratio

n 6.

3 Q

ualit

y As

sura

nce

Dev

elop

er

Test

er

SQA

FOO

TNO

TES

1 Th

e la

rger

is th

e be

tter,

in e

arly

sta

ge o

f tes

ting.

On

the

cont

rary

, the

sm

alle

r is

the

bette

r, in

late

r sta

ge o

f tes

ting

or o

pera

tion.

It is

reco

mm

ende

d to

mon

itor t

he tr

end

of th

is m

easu

re

alon

g w

ith th

e tim

e.

2 Th

is m

etric

dep

ends

on

adeq

uacy

of t

est c

ases

so

high

ly th

at th

ey s

houl

d be

des

igne

d to

incl

ude

appr

opria

te c

ases

: e.g

., no

rmal

, exc

eptio

nal a

nd a

bnor

mal

cas

es.

3 It

is n

eces

sary

to c

onve

rt th

is v

alue

(X) t

o th

e <0

,1>

inte

rval

if m

akin

g su

mm

aris

atio

n of

cha

ract

eris

tics.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

2.1

(con

tinue

d)

Exte

rnal

mat

urity

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Failu

re

reso

lutio

n H

ow m

any

failu

re

cond

ition

s ar

e re

solv

ed?

Cou

nt th

e nu

mbe

r of

failu

res

that

did

not

re

occu

r dur

ing

defin

ed tr

ial

perio

d un

der t

he s

imila

r co

nditi

ons.

M

aint

ain

a pr

oble

m

reso

lutio

n re

port

desc

ribin

g st

atus

of a

ll th

e fa

ilure

s.

X=

A1 /

A2

A1

= n

umbe

r of r

esol

ved

failu

res

A2 =

tota

l num

ber o

f act

ually

det

ecte

d fa

ilure

s

0<=X

<= 1

Th

e cl

oser

to

1.0

is b

ette

r as

mor

e fa

ilure

s ar

e re

solv

ed.

a)

Abso

lute

A1=

Cou

nt

A2=

Cou

nt

A3 =

C

ount

X=

Cou

nt/

Cou

nt

Test

repo

rt O

pera

tion

(test

) re

port

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n

Use

r SQ

A M

aint

aine

r

FOO

TNO

TES

1 It

is re

com

men

ded

to m

onito

r the

tren

d w

hen

usin

g th

is m

easu

re.

2 To

tal n

umbe

r of p

redi

cted

late

nt fa

ilure

s m

ight

be

estim

ated

usi

ng re

liabi

lity

grow

th m

odel

s ad

just

ed w

ith a

ctua

l his

toric

al d

ata

rela

ting

to s

imila

r sof

twar

e pr

oduc

t. In

suc

h a

case

, the

nu

mbe

r of a

ctua

l and

pre

dict

ed fa

ilure

s ca

n be

com

para

ble

and

the

num

ber o

f res

idua

l unr

esol

ved

failu

res

can

be m

easu

rabl

e.

Faul

t den

sity

H

ow m

any

faul

ts

wer

e de

tect

ed d

urin

g de

fined

tria

l per

iod?

Cou

nt th

e nu

mbe

r of

dete

cted

fau

lts a

nd

com

pute

den

sity

.

X= A

/ B

A

= nu

mbe

r of d

etec

ted

faul

ts

B =

prod

uct s

ize

0<=X

It

depe

nds

on

stag

e of

te

stin

g.

At th

e la

ter

stag

es,

smal

ler i

s be

tter.

Abso

lute

A=

C

ount

B

= S

ize

X=

Cou

nt/

Siz

e

Test

repo

rt O

pera

tion

repo

rt Pr

oble

m

repo

rt

5.3

Inte

grat

ion

5.3

Q

ualif

icat

ion

test

ing

5.4

Ope

ratio

n 6.

3 Q

ualit

y As

sura

nce

Dev

elop

er

Test

er

SQA

FOO

TNO

TES

1 Th

e la

rger

is th

e be

tter,

in e

arly

sta

ge o

f tes

ting.

On

the

cont

rary

, the

sm

alle

r is

the

bette

r, in

late

r sta

ge o

f tes

ting

or o

pera

tion.

It is

reco

mm

ende

d to

mon

itor t

he tr

end

of th

is m

easu

re

alon

g w

ith th

e tim

e.

2 Th

e nu

mbe

r of d

etec

ted

faul

ts d

ivid

ed b

y th

e nu

mbe

r of t

est c

ases

indi

cate

s ef

fect

iven

ess

of te

st c

ases

. 3

It is

nec

essa

ry to

con

vert

this

val

ue (X

) to

the

<0,1

> in

terv

al if

mak

ing

sum

mar

isat

ion

of c

hara

cter

istic

s.

4 W

hen

coun

ting

faul

ts, p

ay a

ttent

ion

to th

e fo

llow

ings

: - P

ossi

bilit

y of

dup

licat

ion,

bec

ause

mul

tiple

repo

rts m

ay c

onta

in th

e sa

me

faul

ts a

s ot

her r

epor

t; - P

ossi

bilit

y of

oth

ers

than

faul

ts, b

ecau

se u

sers

or t

este

rs m

ay n

ot fi

gure

out

whe

ther

thei

r pro

blem

s ar

e op

erat

ion

erro

r, en

viro

nmen

tal e

rror o

r sof

twar

e fa

ilure

.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

2.1

(con

tinue

d)

Exte

rnal

mat

urity

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Faul

t rem

oval

H

ow m

any

faul

ts

have

bee

n co

rrect

ed?

Cou

nt th

e nu

mbe

r of f

aults

re

mov

ed d

urin

g te

stin

g an

d co

mpa

re w

ith th

e to

tal

num

ber o

f fau

lts d

etec

ted

and

tota

l num

ber o

f fau

lts

pred

icte

d.

a) X

= A

1 /

A2

A1

= n

umbe

r of c

orre

cted

faul

ts

A2 =

tota

l num

ber o

f act

ually

det

ecte

d fa

ults

b)

Y=

A1

/ A

3

A3 =

tota

l num

ber o

f pre

dict

ed la

tent

faul

ts in

th

e so

ftwar

e pr

oduc

t

0<=X

<= 1

Th

e cl

oser

to

1.0

is b

ette

r as

few

er

faul

ts re

mai

n.

0<=Y

Th

e cl

oser

to

1.0

is b

ette

r as

few

er

faul

ts r

emai

n.

a)

Abso

lute

b)

Abso

lute

A1=

Cou

nt

A2=

Cou

nt

A3=

Cou

nt

X= C

ount

/ C

ount

Y=

Cou

nt/

Cou

nt

Test

repo

rt O

rgan

izat

ion

data

base

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

6.5

Valid

atio

n 6.

3 Q

ualit

y As

sura

nce

Dev

elop

er

SQA

Mai

ntai

ner

FOO

TNO

TES

1 It

is re

com

men

ded

to m

onito

r the

tren

d du

ring

a de

fined

per

iod

of ti

me.

2

Tota

l num

ber o

f pre

dict

ed la

tent

faul

ts m

ay b

e es

timat

ed u

sing

relia

bilit

y gr

owth

mod

els

adju

sted

with

act

ual h

isto

rical

dat

a re

latin

g to

sim

ilar s

oftw

are

prod

uct.

3 It

is re

com

men

ded

to m

onito

r the

est

imat

ed fa

ults

reso

lutio

n ra

tio Y

, so

that

if Y

> 1

, inv

estig

ate

the

reas

on w

heth

er it

is b

ecau

se m

ore

faul

ts h

ave

been

det

ecte

d ea

rly o

r bec

ause

so

ftwar

e pr

oduc

t con

tain

s an

unu

sual

num

ber o

f fau

lts.

Oth

erw

ise,

whe

n Y

< 1,

inve

stig

ate

whe

ther

it is

bec

ause

ther

e ar

e le

ss th

an th

e us

ual n

umbe

r of d

efec

ts in

the

softw

are

prod

ucts

or b

ecau

se th

e te

stin

g w

as n

ot a

dequ

ate

to d

etec

t all

poss

ible

faul

ts.

4 It

is n

eces

sary

to c

onve

rt th

is v

alue

(Y) t

o th

e <0

,1>

inte

rval

if m

akin

g su

mm

aris

atio

n of

cha

ract

eris

tics.

5

Whe

n co

untin

g fa

ults

, pay

atte

ntio

n to

the

poss

ibilit

y of

dup

licat

ion,

bec

ause

mul

tiple

repo

rts m

ay c

onta

in th

e sa

me

faul

ts a

s ot

her r

epor

t.

Mea

n tim

e be

twee

n fa

ilure

s (M

TBF)

How

freq

uent

ly d

oes

the

softw

are

fail

in

oper

atio

n?

Cou

nt th

e nu

mbe

r of

failu

res

occu

rred

durin

g a

defin

ed p

erio

d of

ope

ratio

n an

d co

mpu

te th

e av

erag

e in

terv

al b

etw

een

the

failu

res.

a) X

= T

1 /

A

b)

Y =

T2

/ A

T1 =

ope

ratio

n tim

e

T2 =

sum

of t

ime

inte

rval

s be

twee

n co

nsec

utiv

e fa

ilure

occ

urre

nces

A

= to

tal n

umbe

r of a

ctua

lly d

etec

ted

failu

res

(Fai

lure

s oc

curre

d du

ring

obse

rved

op

erat

ion

time)

0<X,

Y Th

e lo

nger

is

the

bette

r as

long

er ti

me

can

be

expe

cted

be

twee

n fa

ilure

s.

a)

Rat

io

b)

Rat

io

A =

Cou

nt

T1 =

T

ime

T2 =

T

ime

X =T

ime

/ C

ount

Y

=Tim

e/ C

ount

Test

repo

rt O

pera

tion

(test

) rep

ort

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

test

ing

5.4

Ope

ratio

n

Mai

ntai

ner

Use

r

FOO

TNO

TES

1 Th

e fo

llow

ing

inve

stig

atio

n m

ay b

e he

lpfu

l: - d

istri

butio

n of

tim

e in

terv

al b

etw

een

failu

re o

ccur

renc

es;

- cha

nges

of m

ean

time

alon

g w

ith in

terv

al o

pera

tion

time

perio

d;

- dis

tribu

tion

indi

catin

g w

hich

func

tion

has

frequ

ent f

ailu

re o

ccur

renc

es a

nd o

pera

tion

beca

use

of fu

nctio

n an

d us

e de

pend

ency

. 2

Failu

re ra

te o

r haz

ard

rate

cal

cula

tion

may

be

alte

rnat

ivel

y us

ed.

3 It

is n

eces

sary

to c

onve

rt th

is v

alue

(X,Y

) to

the

<0,1

> in

terv

al if

mak

ing

sum

mar

isat

ion

of c

hara

cter

istic

s.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

2.1

(con

tinue

d)

Exte

rnal

mat

urity

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Test

cov

erag

e(S

peci

fied

oper

atio

n sc

enar

io te

stin

g co

vera

ge)

How

muc

h of

re

quire

d te

st c

ases

ha

ve b

een

exec

uted

du

ring

test

ing?

Cou

nt th

e nu

mbe

r of t

est

case

s pe

rform

ed d

urin

g te

stin

g an

d co

mpa

re th

e nu

mbe

r of t

est c

ases

re

quire

d to

obt

ain

adeq

uate

test

cov

erag

e.

X= A

/ B

A= N

umbe

r of a

ctua

lly p

erfo

rmed

test

cas

es

repr

esen

ting

oper

atio

n sc

enar

io d

urin

g te

stin

g B=

Num

ber o

f tes

t cas

es to

be

perfo

rmed

to

cove

r req

uire

men

ts

0<=X

<=1

The

clos

er t

o 1.

0 is

the

bette

r tes

t co

vera

ge.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Req

. spe

c.,

Tes

t spe

c. o

r U

ser m

anua

lTe

st re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g

6.5

Valid

atio

n 6.

3 Q

ualit

y As

sura

nce

Dev

elop

er

Test

er

SQA

FOO

TNO

TE

Test

cas

es m

ay b

e no

rmal

ised

by

softw

are

size

, tha

t is:

test

den

sity

cov

erag

e Y=

A /

C, w

here

C=

Size

of p

rodu

ct to

be

test

ed.

The

larg

er Y

is th

e be

tter.

Size

may

be

func

tiona

l siz

e th

at u

ser c

an m

easu

re.

Test

mat

urity

Is

the

prod

uct w

ell

test

ed?

CO

MM

ENT(

S) T

his

is to

pre

dict

the

succ

ess

rate

the

prod

uct w

ill ac

hiev

e in

futu

re te

stin

g.

Cou

nt th

e nu

mbe

r of

pass

ed te

st c

ases

whi

ch

have

bee

n ac

tual

ly

exec

uted

and

com

pare

it

to th

e to

tal n

umbe

r of t

est

case

s to

be

perfo

rmed

as

per r

equi

rem

ents

.

X= A

/ B

A= N

umbe

r of p

asse

d te

st c

ases

dur

ing

test

ing

or o

pera

tion

B= N

umbe

r of t

est c

ases

to b

e pe

rform

ed to

co

ver r

equi

rem

ents

0<=X

<=1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Req

. spe

c.,

Tes

t spe

c.,

or U

ser

man

ual

Test

repo

rt O

pera

tion

repo

rt

5.3

Qua

lific

atio

n te

stin

g 6.

3 Q

ualit

y As

sura

nce

Dev

elop

er

Test

er

SQA

FOO

TNO

TES

1 It

is re

com

men

ded

to p

erfo

rm s

tress

test

ing

usin

g liv

e hi

stor

ical

dat

a es

peci

ally

from

pea

k pe

riods

. It

is a

lso

reco

mm

ende

d to

ens

ure

that

the

follo

win

g te

st ty

pes

are

exec

uted

and

pas

sed

succ

essf

ully

: - U

ser o

pera

tion

scen

ario

; - P

eak

stre

ss;

- Ove

rload

ed d

ata

inpu

t. 2

Pass

ed te

st c

ases

may

be

norm

alis

ed b

y so

ftwar

e si

ze, t

hat i

s:

pass

ed te

st c

ase

dens

ity Y

= A

/ C, w

here

C

= Si

ze o

f pro

duct

to b

e te

sted

. Th

e la

rger

Y is

bet

ter.

Size

may

be

func

tiona

l siz

e th

at u

ser c

an m

easu

re.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

2.2

Faul

t tol

eran

ce m

etric

s

Exte

rnal

faul

t tol

eran

ce m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Bre

akdo

wn

avoi

danc

e H

ow o

ften

the

softw

are

prod

uct

caus

es th

e br

eak

dow

n of

the

tota

l pr

oduc

tion

envi

ronm

ent?

Cou

nt th

e nu

mbe

r of

brea

kdow

ns o

ccur

renc

e w

ith re

spec

t to

num

ber o

f fa

ilure

s.

If it

is u

nder

ope

ratio

n,

anal

yse

log

of u

ser

oper

atio

n hi

stor

y.

X= 1

- A

/ B

A= N

umbe

r of b

reak

dow

ns

B= N

umbe

r of f

ailu

res

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A =C

ount

B

=Cou

nt

X =C

ount

/ C

ount

Test

repo

rt O

pera

tion

repo

rt

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n

Use

r M

aint

aine

r

FOO

TNO

TES

1 Th

e br

eakd

own

mea

ns th

e ex

ecut

ion

of a

ny u

ser t

asks

is s

uspe

nded

unt

il sy

stem

is re

star

ted,

or i

ts c

ontro

l is

lost

unt

il sy

stem

is fo

rced

to b

e sh

ut d

own.

2

Whe

n no

ne o

r fe

w fa

ilure

s ar

e ob

serv

ed, t

ime

betw

een

brea

kdow

ns m

ay b

e m

ore

suita

ble.

Failu

re

avoi

danc

e H

ow m

any

faul

t pa

ttern

s w

ere

brou

ght

unde

r con

trol t

o av

oid

criti

cal a

nd s

erio

us

failu

res?

Cou

nt th

e nu

mbe

r of

avoi

ded

faul

t pat

tern

s an

d co

mpa

re it

to th

e nu

mbe

r of

faul

t pat

tern

s to

be

cons

ider

ed

X=A

/ B

A= N

umbe

r of a

void

ed c

ritic

al a

nd s

erio

us

failu

re o

ccur

renc

es a

gain

st te

st c

ases

of

faul

t pat

tern

B=

Num

ber o

f exe

cute

d te

st c

ases

of f

ault

patte

rn (a

lmos

t cau

sing

failu

re) d

urin

g te

stin

g

0<=X

<= 1

Th

e cl

oser

to

1.0

is b

ette

r, as

the

user

ca

n m

ore

ofte

n av

oid

criti

cal o

r se

rious

fa

ilure

.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Test

repo

rt O

pera

tion

repo

rt

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n 6.

5 Va

lidat

ion

Use

r M

aint

aine

r

FOO

TNO

TES

1 It

is re

com

men

ded

to c

ateg

oris

e fa

ilure

avo

idan

ce le

vels

whi

ch is

the

exte

nt o

f miti

gatin

g im

pact

of f

aults

, for

exa

mpl

e:

-Crit

ical

: ent

ire s

yste

m s

tops

/ or

ser

ious

dat

abas

e de

stru

ctio

n;

-Ser

ious

: im

porta

nt fu

nctio

ns b

ecom

e in

oper

able

and

no

alte

rnat

ive

way

of o

pera

ting

(wor

karo

und)

; -A

vera

ge: m

ost f

unct

ions

are

stil

l ava

ilabl

e, b

ut li

mite

d pe

rform

ance

occ

urs

with

lim

ited

or a

ltern

ate

oper

atio

n (w

orka

roun

d);

-Sm

all:

a fe

w fu

nctio

ns e

xper

ienc

e lim

ited

perfo

rman

ce w

ith li

mite

d o

pera

tion;

-N

one:

impa

ct d

oes

not r

each

end

use

r. 2

Failu

re a

void

ance

leve

ls m

ay b

e ba

sed

on a

risk

mat

rix c

ompo

sed

by s

ever

ity o

f con

sequ

ence

and

freq

uenc

y of

occ

urre

nce

prov

ided

by

ISO

/IEC

150

26 S

yste

m a

nd s

oftw

are

inte

grity

. 3

Faul

t pat

tern

exa

mpl

es

- out

of r

ange

dat

a - d

eadl

ock

Faul

t tre

e an

alys

is te

chni

que

may

be

used

to d

etec

t fau

lt pa

ttern

s.

4 Te

st c

ases

can

incl

ude

the

hum

an in

corre

ct o

pera

tion.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

2.2

(con

tinue

d)

Exte

rnal

faul

t tol

eran

ce m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Inco

rrec

t op

erat

ion

avoi

danc

e

How

man

y fu

nctio

ns

are

impl

emen

ted

with

in

corre

ct o

pera

tions

av

oida

nce

capa

bilit

y? C

ount

the

num

ber o

f tes

t ca

ses

of in

corre

ct

oper

atio

ns w

hich

wer

e av

oide

d to

cau

se c

ritic

al

and

serio

us fa

ilure

s an

d co

mpa

re it

to th

e nu

mbe

r of

exe

cute

d te

st c

ases

of

inco

rrect

ope

ratio

n pa

ttern

s to

be

cons

ider

ed.

X=A

/ B

A= N

umbe

r of a

void

ed c

ritica

l and

ser

ious

fa

ilure

s oc

curre

nces

B=

Num

ber o

f exe

cute

d te

st c

ases

of i

ncor

rect

op

erat

ion

patte

rns

(alm

ost c

ausin

g fa

ilure

) du

ring

test

ing

0<=X

<= 1

Th

e cl

oser

to

1.0

is b

ette

r, as

mor

e in

corre

ct u

ser

oper

atio

n is

av

oide

d.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Test

repo

rt O

pera

tion

repo

rt

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n

Use

r M

aint

aine

r

FOO

TNO

TES

1 Al

so d

ata

dam

age

in a

dditi

on to

sys

tem

failu

re.

2 In

corre

ct o

pera

tion

patte

rns

- In

corre

ct d

ata

type

s as

par

amet

ers

- In

corre

ct s

eque

nce

of d

ata

inpu

t -

Inco

rrect

seq

uenc

e of

ope

ratio

n.

3 Fa

ult t

ree

anal

ysis

tech

niqu

e m

ay b

e us

ed to

det

ect i

ncor

rect

ope

ratio

n pa

ttern

s.

4 Th

is m

etric

may

be

used

exp

erim

enta

lly.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

2.3

Rec

over

abili

ty m

etric

s

Exte

rnal

reco

vera

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Avai

labi

lity

How

ava

ilabl

e is

the

syst

em fo

r use

dur

ing

the

spec

ified

per

iod

of ti

me?

Test

sys

tem

in a

pr

oduc

tion

like

envi

ronm

ent f

or a

spe

cifie

d pe

riod

of ti

me

perfo

rmin

g al

l use

r ope

ratio

ns.

Mea

sure

the

repa

ir tim

e pe

riod

each

tim

e th

e sy

stem

was

una

vaila

ble

durin

g th

e tri

al.

Com

pute

mea

n tim

e to

re

pair.

a)

X= {

To /

(To

+ Tr

) }

b)

Y= A

1 /

A2

To

= o

pera

tion

time

Tr =

tim

e to

repa

ir A1

= to

tal a

vaila

ble

case

s of

use

r’s

succ

essf

ul s

oftw

are

use

whe

n us

er a

ttem

pt

to u

se

A2=

tota

l num

ber o

f cas

es o

f use

r’s a

ttem

pt

to u

se th

e so

ftwar

e du

ring

obse

rvat

ion

time.

Th

is is

from

the

user

cal

labl

e fu

nctio

n op

erat

ion

view

.

0<=X

<=1

The

larg

er

and

clos

er to

1.

0 is

bet

ter,

as th

e us

er

can

use

the

softw

are

for

mor

e tim

e.

0<=Y

<=1

The

larg

er

and

clos

er to

1.

0 is

the

bette

r.

(a),(

b)

Abso

lute

To =

Tim

eTr

= T

ime

X= T

ime/

Ti

me

A1=

Cou

ntA2

= C

ount

Y= C

ount

/C

ount

Test

repo

rt O

pera

tion

repo

rt

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n

Use

r M

aint

aine

r

FOO

TNO

TE

It is

reco

mm

ende

d th

at th

is m

etric

incl

udes

onl

y th

e au

tom

atic

reco

very

pro

vide

d by

the

softw

are

and

excl

udes

the

mai

nten

ance

wor

k of

hum

an.

Mea

n do

wn

time

Wha

t is

the

aver

age

time

the

syst

em s

tays

un

avai

labl

e w

hen

a fa

ilure

occ

urs

befo

re

grad

ual s

tart

up?

Mea

sure

the

dow

n tim

e ea

ch ti

me

the

syst

em is

un

avai

labl

e du

ring

a sp

ecifi

ed tr

ial p

erio

d an

d co

mpu

te th

e m

ean

time.

X= T

/ N

T=

Tot

al d

own

time

N

= N

umbe

r of o

bser

ved

brea

kdow

ns

The

wor

st c

ase

or d

istri

butio

n of

dow

n tim

e sh

ould

be

mea

sure

d.

0<X

The

smal

ler i

s th

e be

tter,

syst

em w

ill be

do

wn

for

shor

ter t

ime.

Rat

io

T= T

ime

N=

Cou

ntX=

Tim

e/C

ount

Test

repo

rt O

pera

tion

repo

rt

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n 6.

5 Va

lidat

ion

Use

r M

aint

aine

r

FOO

TNO

TES

1 It

is re

com

men

ded

that

this

reco

vera

bilit

y m

etric

incl

udes

onl

y th

e au

tom

atic

reco

very

pro

vide

d by

the

softw

are

and

excl

udes

the

mai

nten

ance

wor

k of

hum

an.

2 It

is n

eces

sary

to c

onve

rt th

is v

alue

(X) t

o th

e <0

,1>

inte

rval

if m

akin

g su

mm

aris

atio

n of

cha

ract

eris

tics.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

2.3

(con

tinue

d)

Exte

rnal

reco

vera

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Mea

n re

cove

ry

time

W

hat i

s th

e av

erag

e tim

e th

e sy

stem

take

s to

com

plet

e re

cove

ry

from

initi

al p

artia

l re

cove

ry?

Mea

sure

the

full

reco

very

tim

es fo

r eac

h of

the

time

the

syst

em w

as b

roug

ht

dow

n du

ring

the

spec

ified

tri

al p

erio

d an

d co

mpu

te

the

mea

n tim

e.

X= S

um(T

) / B

T=

Tim

e to

reco

very

dow

ned

softw

are

syst

em a

t eac

h op

portu

nity

N

= N

umbe

r of c

ases

whi

ch o

bser

ved

softw

are

syst

em e

nter

ed in

to re

cove

ry

0<X

The

smal

ler i

s th

e be

tter.

Rat

io

T= T

ime

N=

Cou

ntX=

Tim

e/C

ount

Test

repo

rt O

pera

tion

repo

rt

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n 6.

5 Va

lidat

ion

Use

r M

aint

aine

r

FOO

TNO

TES

1 It

is re

com

men

ded

to m

easu

re th

e m

axim

um ti

me

of th

e w

orst

cas

e or

dis

tribu

tion

of re

cove

ry ti

me

for m

any

case

s.

2 It

is re

com

men

ded

that

this

reco

vera

bilit

y m

etric

incl

udes

onl

y th

e au

tom

atic

reco

very

pro

vide

d by

the

softw

are

and

excl

udes

the

mai

nten

ance

wor

k of

hum

an.

3 It

is re

com

men

ded

to d

istin

guis

h th

e gr

ades

of r

ecov

ery

diffi

culty

, for

exa

mpl

e, re

cove

ry o

f des

troye

d da

taba

se is

mor

e di

fficu

lt th

an re

cove

ry o

f des

troye

d tra

nsac

tion.

4

It is

nec

essa

ry to

con

vert

this

val

ue (X

) to

the

<0,1

> in

terv

al if

mak

ing

sum

mar

isat

ion

of c

hara

cter

istic

s.

Res

tart

abili

ty

How

ofte

n th

e sy

stem

ca

n re

star

t pro

vidi

ng

serv

ice

to u

sers

w

ithin

a re

quire

d tim

e?

Cou

nt th

e nu

mbe

r of t

imes

th

e sy

stem

rest

arts

and

pr

ovid

es s

ervi

ce to

use

rs

with

in a

targ

et re

quire

d tim

e an

d co

mpa

re it

to th

e to

tal n

umbe

r of r

esta

rts,

whe

n th

e sy

stem

was

br

ough

t dow

n du

ring

the

spec

ified

tria

l per

iod.

X =

A / B

A=

Num

ber o

f res

tarts

whi

ch m

et to

requ

ired

time

durin

g te

stin

g or

use

r ope

ratio

n su

ppor

t B=

Tot

al n

umbe

r of r

esta

rts d

urin

g te

stin

g or

us

er o

pera

tion

supp

ort

0<=X

<=1

The

larg

er

and

clos

er to

1.

0 is

bet

ter,

as th

e us

er

can

rest

art

easi

ly.

Abso

lute

A =C

ount

B

=Cou

nt

X =C

ount

/ C

ount

Test

repo

rt O

pera

tion

repo

rt

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n 6.

5 Va

lidat

ion

Use

r M

aint

aine

r

FOO

TNO

TES

1 It

is re

com

men

ded

to e

stim

ate

diffe

rent

tim

e to

rest

art t

o co

rresp

ond

to th

e se

verit

y le

vel o

f ino

pera

bilit

y, s

uch

as d

ata

base

des

truct

ion,

lost

mul

ti tra

nsac

tion,

lost

sin

gle

trans

actio

n, o

r te

mpo

rary

dat

a de

stru

ctio

n.

2 It

is re

com

men

ded

that

this

reco

vera

bilit

y m

etric

incl

udes

onl

y th

e au

tom

atic

reco

very

pro

vide

d by

the

softw

are

and

excl

udes

the

mai

nten

ance

wor

k of

hum

an.

Res

tora

bilit

y H

ow c

apab

le is

the

prod

uct i

n re

stor

ing

itsel

f afte

r abn

orm

al

even

t or a

t req

uest

?

Cou

nt th

e nu

mbe

r of

succ

essf

ul re

stor

atio

ns

and

com

pare

it to

the

num

ber o

f tes

ted

rest

orat

ion

requ

ired

in th

e sp

ecifi

catio

ns.

Res

tora

tion

requ

irem

ent

exam

ples

: da

taba

se c

heck

poin

t, tra

nsac

tion

chec

kpoi

nt,

redo

func

tion,

und

o fu

nctio

n et

c.

X= A

/ B

A= N

umbe

r of r

esto

ratio

n ca

ses

succ

essf

ully

do

ne

B= N

umbe

r of r

esto

ratio

n ca

ses

test

ed a

s pe

r req

uire

men

ts

0<=X

<=1

The

larg

er

and

clos

er to

1.

0 is

bet

ter,

as h

e pr

oduc

t is

mor

e ca

pabl

e to

re

stor

e in

de

fined

ca

ses.

Abso

lute

A= C

ount

B= C

ount

X= C

ount

/ C

ount

Req

. spe

c.,

Test

spe

c. o

r U

ser m

anua

l Te

st re

port

Ope

ratio

n re

port

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n 6.

5 Va

lidat

ion

Use

r M

aint

aine

r

FOO

TNO

TE

It is

reco

mm

ende

d th

at th

is m

etric

incl

udes

onl

y th

e au

tom

atic

reco

very

pro

vide

d by

the

softw

are

and

excl

udes

the

mai

nten

ance

wor

k of

hum

an.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

2.3

(con

tinue

d)

Exte

rnal

reco

vera

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Res

tore

ef

fect

iven

ess

How

effe

ctiv

e is

the

rest

orat

ion

capa

bilit

y? C

ount

the

num

ber o

f te

sted

rest

orat

ion

mee

ting

targ

et re

stor

atio

n tim

e an

d co

mpa

re it

to th

e nu

mbe

r of

rest

orat

ions

requ

ired

with

spe

cifie

d ta

rget

tim

e.

X= A

/ B

A= N

umbe

r of c

ases

suc

cess

fully

rest

ored

m

eetin

g th

e ta

rget

rest

ore

time

B= N

umbe

r of c

ases

per

form

ed

0<=X

<=1

The

larg

er

and

clos

er to

1.

0 is

the

bette

r, as

the

rest

orat

ion

proc

ess

in

prod

uct i

s m

ore

effe

ctiv

e.

Abso

lute

A= C

ount

B= C

ount

X= C

ount

/ C

ount

Test

repo

rt O

pera

tion

repo

rt

5.3

Inte

grat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n 6.

5 Va

lidat

ion

Use

r M

aint

aine

r

FOO

TNO

TE

It is

reco

mm

ende

d th

at th

is m

etric

incl

udes

onl

y th

e au

tom

atic

reco

very

pro

vide

d by

the

softw

are

and

excl

udes

the

mai

nten

ance

wor

k of

hum

an.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

2.4

Rel

iabi

lity

com

plia

nce

met

rics

Exte

rnal

relia

bilit

y co

mpl

ianc

e m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Rel

iabi

lity

com

plia

nce

How

com

plia

nt is

the

relia

bilit

y of

the

prod

uct t

o ap

plic

able

re

gula

tions

, st

anda

rds

and

conv

entio

ns.

Cou

nt th

e nu

mbe

r of i

tem

s re

quiri

ng c

ompl

ianc

e th

at

have

bee

n m

et a

nd

com

pare

with

the

num

ber

of it

ems

requ

iring

co

mpl

ianc

e as

in th

e sp

ecifi

catio

n.

X =

1 -

A / B

A=

Num

ber o

f rel

iabi

lity

com

plia

nce

item

s sp

ecifi

ed th

at h

ave

not b

een

impl

emen

ted

durin

g te

stin

g B=

Tot

al n

umbe

r of r

elia

bilit

y co

mpl

ianc

e ite

ms

spec

ified

0<=

X <=

1 Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X= C

ount

/C

ount

Prod

uct

desc

riptio

n (U

ser

man

ual o

r Sp

ecifi

catio

n)

of c

ompl

ianc

e an

d re

late

d st

anda

rds,

co

nven

tions

or

regu

latio

ns Te

st

spec

ifica

tion

and

repo

rt

5.3

Qua

lific

atio

n te

stin

g 6.

5 Va

lidat

ion

Supp

lier

Use

r

FOO

TNO

TE

It m

ay b

e us

eful

to c

olle

ct s

ever

al m

easu

red

valu

es a

long

tim

e, to

ana

lyse

the

trend

of i

ncre

asin

gly

satis

fied

com

plia

nce

item

s an

d to

det

erm

ine

whe

ther

they

are

fully

sat

isfie

d or

not

.

ISO/IEC TR 9126-2:2003(E)


8.3 Usability metrics

Usability metrics measure the extent to which the software can be understood, learned, operated, attractive and compliant with usability regulations and guidelines.

Many external usability metrics are tested by users attempting to use a function. The results will be influenced by the capabilities of the users and the host system characteristics. This does not invalidate the measurements, since the evaluated software is run under explicitly specified conditions by a sample of users who are representative of an identified user group. (For general-purpose products, representatives of a range of user groups may be used.) For reliable results a sample of at least eight users is necessary, although useful information can be obtained from smaller groups. Users should carry out the test without any hints or external assistance.

Metrics for understandability, learnability and operability have two types of method of application: user test or test of the product in use.

NOTE 1 User test

Users attempting to use a function test many external metrics. These measures can vary widely among different individuals. A sample of users who are representative of an identified user group should carry out the test without any hints or external assistance. (For general-purpose products, representatives of a range of user groups may be used.) For reliable results a sample of at least eight users is necessary, although useful information can be obtained from smaller groups.

It should be possible for the measures to be used to establish acceptance criteria or to make comparisons between products. This means that the measures should be counting items of known value. Results should report the mean value and the standard error of the mean.

Many of these metrics can be tested with early prototypes of the software. Which metrics are to be applied will depend on the relative importance of different aspects of usability, and the extent of subsequent quality in use testing.

NOTE 2 Test of the product in use

Rather than test specific functions, some external metrics observe the use of a function during more general use of the product to achieve a typical task as part of a test of the quality in use (ISO/IEC 9126-4). This has the advantage that fewer tests are required. The disadvantage is that some functions may only rarely be used during normal use.

It should be possible for the measures to be used to establish acceptance criteria or to make comparisons between products. This means that the measures should be counting items of known value. Results should report the mean value and the standard error of the mean.

8.3.1 Understandability metrics

Users should be able to select a software product, which is suitable for their intended use. An external understandability metric should be able to assess whether new users can understand:

• whether the software is suitable

• how it can be used for particular tasks.

8.3.2 Learnability metrics

An external learnability metric should be able to assess how long users take to learn how to use particular functions, and the effectiveness of help systems and documentation.

Learnability is strongly related to understandability, and understandability measurements can be indicators of the learnability potential of the software.

ISO/IEC TR 9126-2:2003(E)


8.3.3 Operability metrics

An external operability metric should be able to assess whether users can operate and control the software. Operability metrics can be categorized by the dialogue principles in ISO 9241-10:

• suitability of the software for the task

• self-descriptiveness of the software

• controllability of the software

• conformity of the software with user expectations

• error tolerance of the software

• suitability of the software for individualization

The choice of functions to test will be influenced by the expected frequency of use of functions, the criticality of the functions, and any anticipated usability problems.

8.3.4 Attractiveness metrics

An external attractiveness metric should be able to assess the appearance of the software, and will be influenced by factors such as screen design and colour. This is particularly important for consumer products.

8.3.5 Usability compliance metrics

An external usability compliance metric should be able to assess adherence to standards, conventions, style guides or regulations relating to usability.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.1

Und

erst

anda

bilit

y m

etric

s

Exte

rnal

und

erst

anda

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Com

plet

enes

s of

des

crip

tion

Wha

t pro

porti

on o

f fu

nctio

ns (o

r typ

es o

f fu

nctio

ns) i

s un

ders

tood

afte

r re

adin

g th

e pr

oduc

t de

scrip

tion?

Con

duct

use

r tes

t and

in

terv

iew

use

r with

qu

estio

nnai

res

or o

bser

ve

user

beh

avio

ur.

Cou

nt th

e nu

mbe

r of

func

tions

whi

ch a

re

adeq

uate

ly u

nder

stoo

d an

d co

mpa

re w

ith th

e to

tal

num

ber o

f fun

ctio

ns in

the

prod

uct.

X =

A / B

A

= N

umbe

r of f

unct

ions

(or t

ypes

of

func

tions

) und

erst

ood

B =

Tota

l num

ber o

f fun

ctio

ns (o

r typ

es o

f fu

nctio

ns)

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Use

r man

ual

Ope

ratio

n

(test

) rep

ort

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n

Use

r M

aint

aine

r

FOO

TNO

TE

This

indi

cate

s w

heth

er p

oten

tial u

sers

und

erst

and

the

capa

bilit

y of

the

prod

uct a

fter r

eadi

ng th

e pr

oduc

t des

crip

tion.

D

emon

stra

tion

acce

ssib

ility

W

hat p

ropo

rtion

of

the

dem

onst

ratio

ns/

tuto

rials

can

the

user

ac

cess

?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

. C

ount

the

num

ber o

f fu

nctio

ns th

at a

re

adeq

uate

ly d

emon

stra

ble

and

com

pare

with

the

tota

l nu

mbe

r of f

unct

ions

re

quiri

ng d

emon

stra

tion

capa

bilit

y.

X =

A / B

A=

Num

ber o

f dem

onst

ratio

ns /

tuto

rials

that

th

e us

er s

ucce

ssfu

lly a

cces

ses

B=

Num

ber o

f dem

onst

ratio

ns /

tuto

rials

av

aila

ble

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Use

r man

ual

Ope

ratio

n

(test

) rep

ort

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n

Use

r M

aint

aine

r

FOO

TNO

TE

This

indi

cate

s w

heth

er u

sers

can

find

the

dem

onst

ratio

ns a

nd/o

r tut

oria

ls.

Dem

onst

ratio

n ac

cess

ibili

ty in

us

e

Wha

t pro

porti

on o

f th

e de

mon

stra

tions

/ tu

toria

ls c

an th

e us

er

acce

ss w

hene

ver

user

act

ually

nee

ds to

do

dur

ing

oper

atio

n?

Obs

erve

the

beha

viou

r of

the

user

who

is tr

ying

to

see

dem

onst

ratio

n/tu

toria

l. O

bser

vatio

n m

ay e

mpl

oy

hum

an c

ogni

tive

actio

n m

onito

ring

appr

oach

with

vi

deo

cam

era.

X =

A / B

A=

Num

ber o

f cas

es in

whi

ch u

ser

succ

essf

ully

see

s de

mon

stra

tion

whe

n us

er

atte

mpt

s to

see

dem

onst

ratio

n

B= N

umbe

r of c

ases

in w

hich

use

r atte

mpt

s to

see

dem

onst

ratio

n du

ring

obse

rvat

ion

perio

d

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Use

r man

ual

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd (v

ideo

ta

pe a

nd

actio

n re

cord

)

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n

Use

r M

aint

aine

r

FOO

TNO

TE

This

indi

cate

s w

heth

er u

sers

can

find

the

dem

onst

ratio

ns a

nd/o

r tut

oria

ls w

hile

usi

ng th

e pr

oduc

t.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.1

(con

tinue

d)

Exte

rnal

und

erst

anda

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Dem

onst

ratio

n ef

fect

iven

ess

Wha

t pro

porti

on o

f fu

nctio

ns c

an th

e us

er o

pera

te

succ

essf

ully

afte

r a

dem

onst

ratio

n or

tu

toria

l?

Obs

erve

the

beha

viou

r of

the

user

who

is tr

ying

to

see

dem

onst

ratio

n/tu

toria

l. O

bser

vatio

n m

ay e

mpl

oy

hum

an c

ogni

tive

actio

n m

onito

ring

appr

oach

with

vi

deo

cam

era.

X =

A / B

A=

Num

ber o

f fun

ctio

ns o

pera

ted

succ

essf

ully

B=

Num

ber o

f dem

onst

ratio

ns/tu

toria

ls

acce

ssed

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Use

r man

ual

Ope

ratio

n

(test

) rep

ort

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n

Use

r M

aint

aine

r

FOO

TNO

TE

This

indi

cate

s w

heth

er u

sers

can

ope

rate

func

tions

suc

cess

fully

afte

r an

onlin

e de

mon

stra

tion

or tu

toria

l. Ev

iden

t fu

nctio

ns

Wha

t pro

porti

on o

f fu

nctio

ns (o

r typ

es o

f fu

nctio

n) c

an b

e id

entif

ied

by th

e us

er

base

d up

on s

tart

up

cond

ition

s?

Con

duct

use

r tes

t and

in

terv

iew

use

r with

qu

estio

nnai

res

or o

bser

ve

user

beh

avio

ur.

Cou

nt th

e nu

mbe

r of

func

tions

that

are

evi

dent

to

the

user

and

com

pare

w

ith th

e to

tal n

umbe

r of

func

tions

.

X =

A / B

A

= N

umbe

r of

func

tions

(or t

ypes

of

func

tions

) ide

ntifi

ed b

y th

e us

er

B =

Tota

l num

ber o

f act

ual f

unct

ions

(or

type

s of

func

tions

)

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Use

r man

ual

Ope

ratio

n

(test

) rep

ort

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n

Use

r M

aint

aine

r

FOO

TNO

TE

This

indi

cate

s w

heth

er u

sers

are

abl

e to

loca

te fu

nctio

ns b

y ex

plor

ing

the

inte

rface

(e.g

. by

insp

ectin

g th

e m

enus

). Fu

nctio

n un

ders

tand

-ab

ility

Wha

t pro

porti

on o

f th

e pr

oduc

t fun

ctio

ns

will

the

user

be

able

to

und

erst

and

corre

ctly

?

Con

duct

use

r tes

t and

in

terv

iew

use

r with

qu

estio

nnai

res.

C

ount

the

num

ber o

f use

r in

terfa

ce fu

nctio

ns w

here

pu

rpos

es a

re e

asily

un

ders

tood

by

the

user

an

d co

mpa

re w

ith th

e nu

mbe

r of f

unct

ions

av

aila

ble

for u

ser.

X= A

/ B

A= N

umbe

r of i

nter

face

func

tions

who

se

purp

ose

is c

orre

ctly

des

crib

ed b

y th

e us

er

B= N

umbe

r of f

unct

ions

ava

ilabl

e fro

m th

e in

terfa

ce

0 <=

X <

= 1

The

clos

er to

1.

0, th

e be

tter.

Abso

lute

A=

Cou

nt

B= C

ount

X= C

ount

/C

ount

Use

r man

ual

Ope

ratio

n

(test

) rep

ort

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n

Use

r M

aint

aine

r

FOO

TNO

TE

This

indi

cate

s w

heth

er u

sers

are

abl

e to

und

erst

and

func

tions

by

expl

orin

g th

e in

terfa

ce (e

.g. b

y in

spec

ting

the

men

us).

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.1

(con

tinue

d)

Exte

rnal

und

erst

anda

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Und

erst

anda

ble

inpu

t and

ou

tput

Can

use

rs

unde

rsta

nd w

hat i

s re

quire

d as

inpu

t dat

a an

d w

hat i

s pr

ovid

ed

as o

utpu

t by

softw

are

syst

em?

Con

duct

use

r tes

t and

in

terv

iew

use

r with

qu

estio

nnai

res

or o

bser

ve

user

beh

avio

ur.

Cou

nt th

e nu

mbe

r of i

nput

an

d ou

tput

dat

a ite

ms

unde

rsto

od b

y th

e us

er

and

com

pare

with

the

tota

l nu

mbe

r of t

hem

ava

ilabl

e fo

r use

r.

X= A

/ B

A= N

umbe

r of i

nput

and

out

put d

ata

item

s w

hich

use

r suc

cess

fully

und

erst

ands

B=

Num

ber o

f inp

ut a

nd o

utpu

t dat

a ite

ms

av

aila

ble

from

the

inte

rface

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Use

r man

ual

Ope

ratio

n

(test

) rep

ort

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r M

aint

aine

r

FOO

TNO

TE

This

indi

cate

s w

heth

er u

sers

can

und

erst

and

the

form

at in

whi

ch d

ata

shou

ld b

e in

put a

nd c

orre

ctly

iden

tify

the

mea

ning

of o

utpu

t dat

a.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.2

Lear

nabi

lity

met

rics

Exte

rnal

lear

nabi

lity

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Ease

of

func

tion

lear

ning

How

long

doe

s th

e us

er ta

ke to

lear

n to

us

e a

func

tion?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

.

T= M

ean

time

take

n to

lear

n to

use

a

func

tion

corre

ctly

0<T

The

shor

ter i

s th

e be

tter.

Rat

io

T= T

ime

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r M

aint

aine

r

FOO

TNO

TE

This

met

ric is

gen

eral

ly u

sed

as o

ne o

f exp

erie

nced

and

just

ified

. Ea

se o

f le

arni

ng to

pe

rfor

m a

task

in

use

How

long

doe

s th

e us

er ta

ke to

lear

n ho

w to

per

form

the

spec

ified

task

ef

ficie

ntly

?

Obs

erve

use

r beh

avio

ur

from

whe

n th

ey s

tart

to

lear

n un

til th

ey b

egin

to

oper

ate

effic

ient

ly.

T= S

um o

f use

r ope

ratio

n tim

e un

til u

ser

achi

eved

to p

erfo

rm th

e sp

ecifi

ed ta

sk w

ithin

a

shor

t tim

e

0<T

The

shor

ter i

s th

e be

tter.

Rat

io

T= T

ime

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r M

aint

aine

r

FOO

TNO

TES

1 It

is re

com

men

ded

to d

eter

min

e an

exp

ecte

d us

er’s

ope

ratin

g tim

e as

a s

hort

time.

Suc

h us

er’s

ope

ratin

g tim

e m

ay b

e th

e th

resh

old,

for e

xam

ple,

whi

ch is

70%

of t

ime

at th

e fir

st u

se a

s th

e fa

ir pr

opor

tion.

2

Effo

rt m

ay a

ltern

ativ

ely

repr

esen

t tim

e by

per

son-

hour

uni

t.

Effe

ctiv

enes

s of

the

user

do

cum

enta

tion

and/

or h

elp

syst

em

Wha

t pro

porti

on o

f ta

sks

can

be

com

plet

ed c

orre

ctly

af

ter u

sing

the

user

do

cum

enta

tion

and/

or

help

sys

tem

?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

. C

ount

the

num

ber o

f tas

ks

succ

essf

ully

com

plet

ed

afte

r acc

essi

ng o

nlin

e he

lp

and/

or d

ocum

enta

tion

and

com

pare

with

the

tota

l nu

mbe

r of t

asks

test

ed.

X= A

/ B

A= N

umbe

r of t

asks

suc

cess

fully

com

plet

ed

afte

r acc

essi

ng o

nlin

e he

lp a

nd/o

r do

cum

enta

tion

B =

Tota

l of n

umbe

r of t

asks

test

ed

0<=X

<=1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TE

Thre

e m

etric

s ar

e po

ssib

le: c

ompl

eten

ess

of th

e do

cum

enta

tion,

com

plet

enes

s of

the

help

faci

lity,

or c

ompl

eten

ess

of th

e he

lp a

nd d

ocum

enta

tion

used

in c

ombi

natio

n.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.2

(con

tinue

d)

Exte

rnal

lear

nabi

lity

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Effe

ctiv

enes

s of

use

r do

cum

enta

tion

and/

or h

elp

syst

ems

in u

se

Wha

t pr

opor

tion

of

func

tions

can

be

used

co

rrect

ly a

fter r

eadi

ng

the

docu

men

tatio

n or

us

ing

help

sys

tem

s?

Obs

erve

use

r beh

avio

ur.

Cou

nt th

e nu

mbe

r of

func

tions

use

d co

rrect

ly

afte

r rea

ding

the

docu

men

tatio

n or

usi

ng

help

sys

tem

san

d co

mpa

re

with

the

tota

l num

ber o

f fu

nctio

ns.

X =

A / B

A

= N

umbe

r of f

unct

ions

that

can

be

used

B

= To

tal o

f num

ber o

f fun

ctio

ns p

rovi

ded

0<=X

<=1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Use

r man

ual

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TE

This

met

ric is

gen

eral

ly u

sed

as o

ne o

f exp

erie

nced

and

just

ified

met

rics

rath

er th

an th

e ot

hers

. H

elp

acce

ssib

ility

W

hat p

ropo

rtion

of

the

help

topi

cs c

an

the

user

loca

te?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

. C

ount

the

num

ber o

f tas

ks

for w

hich

cor

rect

onl

ine

help

is lo

cate

d an

d co

mpa

re w

ith th

e to

tal

num

ber o

f tas

ks te

sted

.

X =

A / B

A

= N

umbe

r of t

asks

for w

hich

cor

rect

onl

ine

help

is lo

cate

d

B =

Tota

l of n

umbe

r of t

asks

test

ed

0<=X

<=1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A=

Cou

ntB=

Cou

nt

X= C

ount

/C

ount

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

Hel

p fr

eque

ncy

How

freq

uent

ly d

oes

a us

er h

ave

to a

cces

s he

lp to

lear

n op

erat

ion

to c

ompl

ete

his/

her w

ork

task

?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

. C

ount

the

num

ber o

f cas

es

that

a u

ser a

cces

ses

help

to

com

plet

e hi

s/he

r tas

k.

X =

A A

= N

umbe

r of a

cces

ses

to h

elp

until

a u

ser

com

plet

es h

is/h

er ta

sk.

0<=

X

The

clos

er to

0

is th

e be

tter.

Abso

lute

X=

Cou

nt

A =C

ount

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.3

Ope

rabi

lity

met

rics

a) C

onfo

rms

with

ope

ratio

nal u

ser e

xpec

tatio

ns

Exte

rnal

Ope

rabi

lity

met

rics

a) C

onfo

rms

with

ope

ratio

nal u

ser e

xpec

tatio

ns

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

a) X

= 1

- A

/ B

A= N

umbe

r of m

essa

ges

or fu

nctio

ns w

hich

us

er fo

und

unac

cept

ably

inco

nsis

tent

with

th

e us

er’s

exp

ecta

tion

B=

Num

ber o

f mes

sage

s or

func

tions

0<=X

<=1

The

clos

er to

1.

0 is

the

bette

r.

a)

Abso

lute

A= C

ount

B= C

ount

X= C

ount

/C

ount

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

Ope

ratio

nal

cons

iste

ncy

in

use

How

con

sist

ent a

re

the

com

pone

nt o

f the

us

er in

terfa

ce?

Obs

erve

the

beha

viou

r of

the

user

and

ask

the

opin

ion.

b) Y

= N

/ U

OT

N=

Num

ber o

f ope

ratio

ns w

hich

use

r fou

nd

unac

cept

ably

inco

nsis

tent

with

the

user

’s

expe

ctat

ion

U

OT=

use

r ope

ratin

g tim

e (d

urin

g ob

serv

atio

n pe

riod)

0<=Y

Th

e sm

alle

r an

d cl

oser

to

0.0

is th

e be

tter.

b)

Rat

io

UO

T=

Tim

e N

= C

ount

Y= C

ount

/Ti

me

FOO

TNO

TES

1 U

ser’s

exp

erie

nce

of o

pera

tion

is u

sual

ly h

elpf

ul to

reco

gnis

e se

vera

l ope

ratio

n pa

ttern

s, w

hich

der

ive

user

’s e

xpec

tatio

n.

2 Bo

th o

f “in

put p

redi

ctab

ility”

and

“out

put p

redi

ctab

ility”

are

effe

ctiv

e fo

r ope

ratio

nal c

onsi

sten

cy.

3 Th

is m

etric

may

be

used

to m

easu

re “E

asy

to d

eriv

e op

erat

ion”

and

“Sm

ooth

Com

mun

icat

ion”

.

ISO/IEC TR 9126-2:2003(E)


Tab

le 8

.3.3

Ope

rabi

lity

met

rics

b)

Con

trol

labl

e

Exte

rnal

Ope

rabi

lity

met

rics

b) C

ontr

olla

ble

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Erro

r cor

rect

ion

Can

use

r eas

ily

corre

ct e

rror o

n ta

sks?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

.

T= T

c -

Ts

Tc =

Tim

e of

com

plet

ing

corre

ctio

n of

sp

ecifi

ed ty

pe e

rrors

of p

erfo

rmed

task

Ts

= T

ime

of s

tarti

ng c

orre

ctio

n of

spe

cifie

d ty

pe e

rrors

of p

erfo

rmed

task

0<T

The

shor

ter i

s th

e be

tter.

Rat

io

Ts, T

c=

Tim

e T=

Tim

e

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TE

Use

r of t

his

met

ric is

sug

gest

ed to

spe

cify

type

s of

erro

rs fo

r tes

t cas

es b

y co

nsid

erin

g, fo

r exa

mpl

e, s

ever

ity (d

ispl

ayin

g er

ror o

r des

troyi

ng d

ata)

, typ

e of

inpu

t/out

put e

rror (

inpu

t tex

t erro

r, ou

tput

dat

a er

ror t

o da

taba

se o

r gra

phic

al e

rror o

n di

spla

y) o

r typ

e of

erro

r ope

ratio

nal s

ituat

ion

(inte

ract

ive

use

or e

mer

gent

ope

ratio

n).

Erro

r cor

rect

ion

in u

se

Can

use

r eas

ily

reco

ver h

is/h

er e

rror

or re

try ta

sks?

Obs

erve

the

beha

viou

r of

the

user

who

is o

pera

ting

softw

are.

a)

X= A

/ U

OT

A= n

umbe

r of t

imes

that

the

user

suc

ceed

s to

can

cel t

heir

erro

r ope

ratio

n U

OT=

use

r ope

ratin

g tim

e du

ring

obse

rvat

ion

perio

d C

OM

MEN

T(S)

W

hen

func

tion

is te

sted

one

by

one

, the

ratio

can

be

also

cal

cula

ted,

that

is

the

ratio

of n

umbe

r of f

unct

ions

whi

ch u

ser

succ

eeds

to c

ance

l his

/her

ope

ratio

n to

all

func

tions

.

0<=X

Th

e hi

gher

is

the

bette

r.

Rat

io

A= C

ount

U

OT

=

Tim

e X

= C

ount

/ T

ime

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

C

an u

ser e

asily

re

cove

r his

/her

inpu

t? O

bser

ve th

e be

havi

our o

f th

e us

er w

ho is

ope

ratin

g so

ftwar

e.

b)

X =

A / B

A=

Num

ber o

f scr

eens

or f

orm

s w

here

the

inpu

t dat

a w

ere

succ

essf

ully

mod

ified

or

chan

ged

befo

re b

eing

ela

bora

ted

B =

Num

ber o

f scr

eens

or f

orm

s w

here

use

r tri

ed to

mod

ify o

r to

chan

ge th

e in

put d

ata

durin

g ob

serv

ed u

ser o

pera

ting

time

0<=X

<=1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

, B=

Cou

ntX=

Cou

nt/

Cou

nt

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.3

Ope

rabi

lity

met

rics

c)

Sui

tabl

e fo

r the

task

ope

ratio

n

Exte

rnal

Ope

rabi

lity

met

rics

c) S

uita

ble

for t

he ta

sk o

pera

tion

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Def

ault

valu

e av

aila

bilit

y in

us

e

Can

use

r eas

ily

sele

ct p

aram

eter

va

lues

for h

is/h

er

conv

enie

nt

oper

atio

n?

Obs

erve

the

beha

viou

r of

the

user

who

is o

pera

ting

softw

are.

C

ount

how

man

y tim

es

user

atte

mpt

s to

est

ablis

h or

to s

elec

t par

amet

er

valu

es a

nd fa

ils, (

beca

use

user

can

not

use

def

ault

valu

es p

rovi

ded

by th

e so

ftwar

e).

X =

1 - A

/ B

A= T

he n

umbe

r of t

imes

that

the

user

fail

to

esta

blis

h or

to s

elec

t par

amet

er v

alue

s in

a

shor

t per

iod

(bec

ause

use

r can

not

use

de

faul

t val

ues

prov

ided

by

the

softw

are)

B=

Tot

al n

umbe

r of t

imes

that

the

user

at

tem

pt to

est

ablis

h or

to s

elec

t par

amet

er

valu

es

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B=

Cou

nt

X= C

ount

/C

ount

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TES

1 It

is re

com

men

ded

to o

bser

ve a

nd re

cord

ope

rato

r’s b

ehav

iour

and

dec

ide

how

long

per

iod

is a

llow

able

to s

elec

t par

amet

er v

alue

s as

“sho

rt pe

riod”

. 2

Whe

n pa

ram

eter

set

ting

func

tion

is te

sted

by

each

func

tion,

the

ratio

of a

llow

able

func

tion

can

be a

lso

calc

ulat

ed.

3 It

is re

com

men

ded

to c

ondu

ct fu

nctio

nal t

est t

hat c

over

s pa

ram

eter

-set

ting

func

tions

.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.3

Ope

rabi

lity

met

rics

d)

Sel

f des

crip

tive

(Gui

ding

)

Exte

rnal

Ope

rabi

lity

met

rics

d) S

elf d

escr

iptiv

e (G

uidi

ng)

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Mes

sage

un

ders

tand

-ab

ility

in u

se

Can

use

r eas

ily

unde

rsta

nd

mes

sage

s fro

m

softw

are

syst

em?

Is th

ere

any

m

essa

ge w

hich

ca

used

the

user

a

dela

y in

un

ders

tand

ing

befo

re

star

ting

the

next

ac

tion?

C

an u

ser e

asily

m

emor

ise

impo

rtant

m

essa

ge?

Obs

erve

use

r beh

avio

ur

who

is o

pera

ting

softw

are.

X

= A

/ UO

T A

= nu

mbe

r of t

imes

that

the

user

pau

ses

for

a lo

ng p

erio

d or

suc

cess

ivel

y an

d re

peat

edly

fa

ils a

t the

sam

e op

erat

ion,

bec

ause

of t

he

lack

of m

essa

ge c

ompr

ehen

sion

. U

OT

= us

er o

pera

ting

time

(obs

erva

tion

perio

d)

0<=X

Th

e sm

alle

r an

d cl

oser

to

0.0

is th

e be

tter.

Rat

io

A =C

ount

U

OT

=

Tim

e X

= C

ount

/ T

ime

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TES

1 Th

e ex

tent

of e

ase

of m

essa

ge c

ompr

ehen

sion

is re

pres

ente

d by

how

long

that

mes

sage

cau

sed

dela

y in

use

r un

ders

tand

ing

befo

re s

tarti

ng th

e ne

xt a

ctio

n.

Ther

efor

e, it

is re

com

men

ded

to o

bser

ve a

nd re

cord

ope

rato

r’s b

ehav

iour

and

dec

ide

wha

t len

gth

of p

ause

is c

onsi

dere

d a

“long

per

iod”

. 2

It is

reco

mm

ende

d to

inve

stig

ate

the

follo

win

g as

pos

sibl

e ca

uses

of t

he p

robl

ems

of u

ser’s

mes

sage

com

preh

ensi

on.

a)At

tent

iven

ess

: Atte

ntiv

enes

s im

plie

s th

at u

ser s

ucce

ssfu

lly re

cogn

ises

impo

rtant

mes

sage

s pr

esen

ting

info

rmat

ion

such

as

guid

ance

on

next

use

r act

ion,

nam

e of

dat

a ite

ms

to b

e lo

oked

at,

and

war

ning

of c

aref

ul o

pera

tion.

- D

oes

user

eve

r fai

l to

wat

ch w

hen

enco

unte

ring

impo

rtant

mes

sage

s?

- Can

use

r avo

id m

ista

kes

in o

pera

tion,

bec

ause

of

reco

gnis

ing

impo

rtant

mes

sage

s?

b) M

emor

abilit

y: M

emor

abilit

y im

plie

s th

at u

ser r

emem

ber i

mpo

rtant

mes

sage

s pr

esen

ting

info

rmat

ion

such

as

guid

ance

on

the

next

use

r act

ion,

nam

e of

dat

a ite

ms

to b

e lo

oked

at,

and

war

ning

of c

aref

ul o

pera

tion.

- C

an u

ser e

asily

rem

embe

r im

porta

nt m

essa

ges?

- I

s re

mem

berin

g im

porta

nt m

essa

ges

help

ful t

o th

e us

er?

- Is

it re

quire

d fo

r the

use

r to

rem

embe

r onl

y a

few

impo

rtant

mes

sage

s an

d no

t so

muc

h?

3 W

hen

mes

sage

s ar

e te

sted

one

by

one,

the

ratio

of c

ompr

ehen

ded

mes

sage

s to

the

tota

l can

be

also

cal

cula

ted.

4

Whe

n se

vera

l use

rs a

re o

bser

ved

who

are

par

ticip

ants

of o

pera

tiona

l tes

ting,

the

ratio

of u

sers

who

com

preh

ende

d m

essa

ges

to a

ll us

ers

can

be c

alcu

late

d.

Self-

expl

anat

ory

erro

r mes

sage

s

In w

hat p

ropo

rtion

of

erro

r con

ditio

ns d

oes

the

user

pro

pose

the

corre

ct re

cove

ry

actio

n?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

. X=

A /

B A

=Num

ber o

f erro

r con

ditio

ns fo

r whi

ch th

e us

er p

ropo

ses

the

corre

ct re

cove

ry a

ctio

n B

=Num

ber o

f erro

r con

ditio

ns te

sted

0 <=

X <

= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

X =C

ount

/ C

ount

A

=Cou

nt

B =C

ount

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TE

This

met

ric is

gen

eral

ly u

sed

as o

ne o

f exp

erie

nced

and

just

ified

.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.3

Ope

rabi

lity

met

rics

e)

Ope

ratio

nal e

rror

tole

rant

(Hum

an e

rror

free

)

Exte

rnal

ope

rabi

lity

met

rics

e) O

pera

tiona

l err

or to

lera

nt (H

uman

err

or fr

ee)

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Ope

ratio

nal

erro

r re

cove

rabi

lity

in u

se

Can

use

r eas

ily

reco

ver h

is/h

er w

orse

si

tuat

ion?

Obs

erve

the

beha

viou

r of

the

user

who

is o

pera

ting

softw

are.

X =

1 -

A /

B

A= N

umbe

r of u

nsuc

cess

fully

reco

vere

d si

tuat

ion

(afte

r a u

ser e

rror o

r cha

nge)

in

whi

ch u

ser w

as n

ot in

form

ed a

bout

a ri

sk b

y th

e sy

stem

B=

Num

ber o

f use

r erro

rs o

r cha

nges

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

, B=

Cou

ntX=

Cou

nt/

Cou

nt

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TE

The

form

ula

abov

e is

repr

esen

tativ

e of

the

wor

st c

ase.

Use

r of t

his

met

ric m

ay ta

ke a

ccou

nt o

f the

com

bina

tion

of 1

) the

num

ber o

f erro

rs w

here

the

user

is /

is n

ot w

arne

d by

the

softw

are

syst

em a

nd 2

) the

num

ber o

f occ

asio

ns w

here

the

user

suc

cess

fully

/ un

succ

essf

ully

reco

vers

the

situ

atio

n.

Tim

e be

twee

n hu

man

err

or

oper

atio

ns in

us

e

Can

use

r ope

rate

the

softw

are

long

eno

ugh

with

out h

uman

erro

r? O

bser

ve th

e be

havi

our o

f th

e us

er w

ho is

ope

ratin

g so

ftwar

e.

X =

T / N

(at t

ime

t dur

ing

[ t-T

, t] )

T

= op

erat

ion

time

perio

d du

ring

obse

rvat

ion

( or T

he s

um o

f ope

ratin

g tim

e be

twee

n us

er’s

hum

an e

rror o

pera

tions

) N

= nu

mbe

r of o

ccur

renc

es o

f use

r’s h

uman

er

ror o

pera

tion

0<X

The

high

er is

th

e be

tter.

Rat

io

T =

Tim

e N

= C

ount

X =

Ti

me

/ C

ount

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TES

1 H

uman

erro

r ope

ratio

n m

ay b

e de

tect

ed b

y co

untin

g be

low

use

r’s b

ehav

iour

: a)

Sim

ple

hum

an e

rror (

Slip

s): T

he n

umbe

r of t

imes

that

the

user

just

sim

ply

mak

es e

rrors

to in

put o

pera

tion;

b)

Inte

ntio

nal e

rror (

Mis

take

s): T

he n

umbe

r of t

imes

that

the

user

repe

ats

fail

an e

rror a

t the

sam

e op

erat

ion

with

mis

unde

rsta

ndin

g du

ring

obse

rvat

ion

perio

d;

c) O

pera

tion

hesi

tatio

n pa

use:

The

num

ber o

f tim

es th

at th

e us

er p

ause

s fo

r a lo

ng p

erio

d w

ith h

esita

tion

durin

g ob

serv

atio

n pe

riod.

U

ser o

f thi

s m

etric

is s

ugge

sted

to m

easu

re s

epar

atel

y fo

r eac

h ty

pe li

sted

abo

ve.

2 It

seem

s th

at a

n op

erat

ion

paus

e im

plie

s a

user

’s h

esita

tion

oper

atio

n.

It de

pend

s on

the

func

tion,

ope

ratio

n pr

oced

ure,

app

licat

ion

dom

ain,

and

use

r whe

ther

it is

con

side

red

a lo

ng p

erio

d or

not

for t

he u

ser t

o pa

use

the

oper

atio

n. T

here

fore

, the

eva

luat

or is

re

ques

ted

to ta

ke th

em in

to a

ccou

nt a

nd d

eter

min

e th

e re

ason

able

thre

shol

d tim

e. F

or a

n in

tera

ctiv

e op

erat

ion,

a "l

ong

perio

d" th

resh

old

rang

e of

1m

in. t

o 3

min

.

How

freq

uent

ly d

oes

the

user

suc

cess

fully

co

rrect

inpu

t erro

rs?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

. a)

X=

A /

B

A= N

umbe

r of i

nput

erro

rs w

hich

the

user

su

cces

sful

ly c

orre

cts

B= N

umbe

r of a

ttem

pts

to c

orre

ct in

put

erro

rs

0<=X

<=1

The

clos

er to

1.

0 is

the

bette

r.

a)

Abso

lute

A= C

ount

B=

Cou

ntX=

Cou

nt/

Cou

nt

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

Und

oabi

lity

(Use

r err

or

corr

ectio

n)

How

freq

uent

ly d

oes

the

user

cor

rect

ly

undo

erro

rs?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

. b)

Y=

A /

B

A= N

umbe

r of e

rror c

ondi

tions

whi

ch th

e us

er s

ucce

ssfu

lly c

orre

cts

B= T

otal

num

ber o

f erro

r con

ditio

ns te

sted

0 <=

Y <

= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

b)

Abso

lute

A= C

ount

B=

Cou

nt

Y= C

ount

/ C

ount

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TE

This

met

ric is

gen

eral

ly u

sed

as o

ne o

f exp

erie

nced

and

just

ified

.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.3

Ope

rabi

lity

met

rics

f)

Sui

tabl

e fo

r ind

ivid

ualis

atio

n

Exte

rnal

ope

rabi

lity

met

rics

f) S

uita

ble

for i

ndiv

idua

lisat

ion

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Cus

tom

isab

ility

Can

use

r eas

ily

cust

omis

e op

erat

ion

proc

edur

es fo

r his

/her

co

nven

ienc

e?

Can

a u

ser,

who

in

stru

cts

end

user

s,

easi

ly s

et c

usto

mis

ed

oper

atio

n pr

oced

ure

tem

plat

es fo

r pr

even

ting

thei

r er

rors

? W

hat p

ropo

rtion

of

func

tions

can

be

cust

omis

ed?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

. X=

A /

B

A= N

umbe

r of f

unct

ions

suc

cess

fully

cu

stom

ised

B=

Num

ber o

f atte

mpt

s to

cus

tom

ise

0 <=

X <

= 1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B=

Cou

nt

X= C

ount

/ C

ount

Use

r man

ual

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TES

1 R

atio

of u

ser’s

failu

res

to c

usto

mis

e m

ay b

e m

easu

red.

Y

= 1

- (C

/ D

) C

= N

umbe

r of c

ases

in w

hich

a u

ser f

ails

to c

usto

mis

e op

erat

ion

D =

Tot

al n

umbe

r of c

ases

in w

hich

a u

ser a

ttem

pted

to c

usto

mis

e op

erat

ion

for h

is/h

er c

onve

nien

ce.

0<=Y

<= 1

, The

clo

ser t

o 1.

0 is

the

bette

r. 2

It is

reco

mm

ende

d to

rega

rd th

e fo

llow

ing

as v

aria

tions

of c

usto

mis

ing

oper

atio

ns:

- cho

se a

ltern

ativ

e op

erat

ion,

suc

h as

usi

ng m

enu

sele

ctio

n in

stea

d of

com

man

d in

put;

- com

bine

d us

er’s

ope

ratio

n pr

oced

ure,

suc

h as

reco

rdin

g an

d ed

iting

ope

ratio

n pr

oced

ures

; - s

et c

onst

rain

ed te

mpl

ate

oper

atio

n, s

uch

as p

rogr

amm

ing

proc

edur

es o

r mak

ing

a te

mpl

ate

for i

nput

gui

danc

e.

3 Th

is m

etric

is g

ener

ally

use

d as

one

of e

xper

ienc

ed a

nd ju

stifi

ed.

Ope

ratio

n pr

oced

ure

redu

ctio

n

Can

use

r eas

ily

redu

ce o

pera

tion

proc

edur

es fo

r his

/her

co

nven

ienc

e?

Cou

nt u

ser’s

stro

kes

for

spec

ified

ope

ratio

n an

d co

mpa

re th

em b

etw

een

befo

re a

nd a

fter

cust

omis

ing

oper

atio

n.

X =

1 -

A / B

A

= N

umbe

r of r

educ

ed o

pera

tion

proc

edur

es a

fter c

usto

mis

ing

oper

atio

n B

= N

umbe

r of o

pera

tion

proc

edur

es b

efor

e cu

stom

isin

g op

erat

ion

0<=X

< 1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B=

Cou

nt

X= C

ount

/C

ount

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TES

1 It

is re

com

men

ded

to ta

ke s

ampl

es fo

r eac

h di

ffere

nt u

ser t

ask

and

to d

istin

guis

h be

twee

n an

ope

rato

r who

is a

ski

lled

user

or a

beg

inne

r.

2 N

umbe

r of o

pera

tion

proc

edur

es m

ay b

e re

pres

ente

d by

cou

ntin

g op

erat

ion

stro

kes

such

as

clic

k, d

rug,

key

touc

h, s

cree

n to

uch,

etc

. 3

This

incl

udes

key

boar

d sh

ortc

uts.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.3

f) (

cont

inue

d)

Exte

rnal

ope

rabi

lity

met

rics

f) S

uita

ble

for i

ndiv

idua

lisat

ion

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Phys

ical

ac

cess

ibili

ty

Wha

t pro

porti

on o

f fu

nctio

ns c

an b

e ac

cess

ed b

y us

ers

with

phy

sica

l ha

ndic

aps?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

. X=

A /

B

A= N

umbe

r of f

unct

ions

suc

cess

fully

ac

cess

ed

B= N

umbe

r of f

unct

ions

0 <=

X <

= 1

The

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B=

Cou

nt

X= C

ount

/C

ount

Ope

ratio

n

(test

) rep

ort

Use

r m

onito

ring

reco

rd

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TE

Exam

ples

of p

hysi

cal i

nacc

essi

bilit

y ar

e in

abilit

y to

use

a m

ouse

and

blin

dnes

s.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.4

Attr

activ

enes

s m

etric

s

Exte

rnal

attr

activ

enes

s m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Attr

activ

e in

tera

ctio

n

How

attr

activ

e is

the

inte

rface

to th

e us

er?

Que

stio

nnai

re to

use

rs.

Que

stio

nnai

re to

ass

ess

the

attra

ctiv

enes

s of

the

inte

rface

to u

sers

, afte

r exp

erie

nce

of

usag

e

Dep

end

on it

s qu

estio

nnai

re

scor

ing

met

hod.

Abso

lute

Cou

nt

Que

stio

nnai

re

resu

lt 6.

5 Va

lidat

ion

5.3

Qua

lific

atio

n te

stin

g

5.4

Ope

ratio

n

Use

r H

uman

in

terfa

ce

desi

gner

Inte

rfac

e ap

pear

ance

cu

stom

isab

ility

Wha

t pro

porti

on o

f in

terfa

ce e

lem

ents

ca

n be

cus

tom

ised

in

appe

aran

ce to

the

user

’s s

atis

fact

ion?

Con

duct

use

r tes

t and

ob

serv

e us

er b

ehav

iour

. X=

A /

B A=

Num

ber o

f int

erfa

ce e

lem

ents

cu

stom

ised

in a

ppea

ranc

e to

use

r’s

satis

fact

ion

B= N

umbe

r of i

nter

face

ele

men

ts th

at th

e us

er w

ishe

s to

cus

tom

ise

0 <=

X <

= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Use

rs’

requ

ests

O

pera

tion

(te

st) r

epor

t

6.5

Valid

atio

n 5.

3 Q

ualif

icat

ion

test

ing

5.

4 O

pera

tion

Use

r H

uman

in

terfa

ce

desi

gner

FOO

TNO

TE

This

met

ric is

gen

eral

ly u

sed

as o

ne o

f exp

erie

nced

and

just

ified

.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

3.5

Usa

bilit

y co

mpl

ianc

e m

etric

s

Exte

rnal

usa

bilit

y co

mpl

ianc

e m

etric

s M

etric

nam

e Pu

rpos

e M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Usa

bilit

y co

mpl

ianc

e

How

com

plet

ely

does

th

e so

ftwar

e ad

here

to

the

stan

dard

s,

conv

entio

ns, s

tyle

gu

ides

or r

egul

atio

ns

rela

ting

to u

sabi

lity?

Spec

ify re

quire

d co

mpl

ianc

e ite

ms

base

d on

sta

ndar

ds, c

onve

ntio

ns,

styl

e gu

ides

or r

egul

atio

ns

rela

ting

to u

sabi

lity.

D

esig

n te

st c

ases

in

acco

rdan

ce w

ith

com

plia

nce

item

s.

Con

duct

func

tiona

l tes

ting

for t

hese

test

cas

es.

X =

1 -

A / B

A=

Num

ber o

f usa

bilit

y co

mpl

ianc

e ite

ms

spec

ified

that

hav

e no

t bee

n im

plem

ente

d du

ring

test

ing

B= T

otal

num

ber o

f usa

bilit

y co

mpl

ianc

e ite

ms

spec

ified

0<=

X <=

1 Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X= C

ount

/C

ount

Prod

uct

desc

riptio

n (U

ser m

anua

l or

Sp

ecifi

catio

n)

of c

ompl

ianc

e an

d re

late

d st

anda

rds,

co

nven

tions

, st

yle

guid

es

or re

gula

tions

Test

sp

ecifi

catio

n an

d re

port

5.3

Qua

lific

atio

n te

stin

g 6.

5 Va

lidat

ion

Supp

lier

Use

r

FOO

TNO

TE

It m

ay b

e us

eful

to c

olle

ct s

ever

al m

easu

red

valu

es a

long

tim

e, to

ana

lyse

the

trend

of i

ncre

asin

gly

satis

fied

com

plia

nce

item

s an

d to

det

erm

ine

whe

ther

they

are

fully

sat

isfie

d or

not

.

ISO/IEC TR 9126-2:2003(E)


8.4 Efficiency metrics

An external efficiency metric should be able to measure such attributes as the time consumption and resource utilisation behaviour of computer system including software during testing or operations.

It is recommended that the maximal and distribution time are investigated for many cases of testing or operations, because the measure is affected strongly and fluctuates depending on the conditions of use, such as load of processing data, frequency of use, number of connecting sites and so on. Therefore, efficiency metrics may include the ratio of measured actual value with error fluctuation to the designed value with allowed error fluctuation range, required by specification.

It is recommended to list and to investigate the role played by factors such as “CPU” and memory used by other software, network traffic, and scheduled background processes. Possible fluctuations and valid ranges for measured values should be established and compared to requirement specifications.

It is recommended that a task be identified and defined to be suitable for software application: for example, a transaction as a task for business application: a switching or data packet sending as a task for communication application; an event control as a task for control application; and an output of data produced by user callable function for common user application.

NOTE 1 Response time: Time needed to get the result from pressing a transmission key. This means that response time includes processing time and transmission time. Response time is applicable only for an interactive system. There is no significant difference when it is a standalone system. However, in the case of Internet system or other real time system, sometimes transmission time is much longer.

NOTE 2 Processing time: The elapsed time in a computer between receiving a message and sending the result. Sometimes it includes operating overhead time, other times it only means time used for an application program.

NOTE 3 Turn around time: Time needed to get the result from a request. In many cases one turn around time includes many responses. For example, in a case of banking cash dispenser, turn around time is a time from pressing initial key until you get money, meanwhile you must select type of transaction and wait for a message, input password and wait for the next message etc.

8.4.1 Time behaviour metrics

An external time behaviour metric should be able to measure such attributes as the time behaviour of computer system including software during testing or operations.

8.4.2 Resource utilization metrics

An external resource utilization metric should be able to measure such attributes as the utilized resources behaviour of computer system including software during testing or operating.

8.4.3 Efficiency compliance metrics

An external efficiency compliance metric should be able to measure an attribute such as the number of functions with, or occurrences of compliance problems, which is the software product failing to adhere to standards, conventions or regulations relating to efficiency.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

4.1

Tim

e be

havi

our m

etric

s a

) Res

pons

e tim

e

Exte

rnal

tim

e be

havi

our m

etric

s

a) R

espo

nse

time

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Res

pons

e tim

e W

hat i

s th

e tim

e ta

ken

to c

ompl

ete

a sp

ecifi

ed ta

sk?

H

ow lo

ng d

oes

it ta

ke

befo

re th

e sy

stem

re

spon

se to

a

spec

ified

ope

ratio

n?

Star

t a s

peci

fied

task

. M

easu

re th

e tim

e it

take

s fo

r the

sam

ple

to c

ompl

ete

its o

pera

tion.

Ke

ep a

reco

rd o

f eac

h at

tem

pt.

T =

( tim

e of

gai

ning

the

resu

lt)

- (

time

of c

omm

and

entry

fini

shed

)

0 <

T Th

e so

oner

is

the

bette

r.

Rat

io

T= T

ime

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Sys

./Sw

. In

tegr

atio

n5.

3 Q

ualif

icat

ion

test

ing

5.4

Ope

ratio

n 5.

5 M

aint

e-na

nce

Use

r D

evel

oper

M

aint

aine

r SQ

A

FOO

TNO

TE

It is

reco

mm

ende

d to

take

acc

ount

of t

ime

band

wid

th a

nd to

use

sta

tistic

al a

naly

sis

with

mea

sure

s fo

r a lo

t of t

asks

(sam

ple

shot

s) a

nd n

ot fo

r onl

y on

e ta

sk.

Res

pons

e tim

e (M

ean

time

to

resp

onse

)

Wha

t is

the

aver

age

wai

t tim

e th

e us

er

expe

rienc

es a

fter

issu

ing

a re

ques

t unt

il th

e re

ques

t is

com

plet

ed w

ithin

a

spec

ified

sys

tem

load

in

term

s of

con

curre

nt

task

s an

d sy

stem

ut

ilisat

ion?

Exec

ute

a nu

mbe

r of

scen

ario

s of

con

curre

nt

task

s.

Mea

sure

the

time

it ta

kes

to c

ompl

ete

the

sele

cted

op

erat

ion(

s).

Keep

a re

cord

of e

ach

atte

mpt

and

com

pute

the

mea

n tim

e fo

r eac

h sc

enar

io.

X =

Tmea

n / T

Xmea

n Tm

ean

= ∑

(Ti)

/ N,

(for

i=1

to N

) TX

mea

n =

requ

ired

mea

n re

spon

se ti

me

Ti=

resp

onse

tim

e fo

r i-th

eva

luat

ion

(sho

t) N

= nu

mbe

r of e

valu

atio

ns (s

ampl

ed s

hots

)

0 <=

X

The

near

er to

1.

0 an

d le

ss

than

1.0

is th

e be

tter.

Abso

lute

Tmea

n=

Tim

e TX

mea

n=

Tim

e Ti

= Ti

me

N=

Cou

nt

X= T

ime/

Ti

me

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Sys.

/Sw

. In

tegr

atio

n5.

3 Q

ualif

icat

ion

test

ing

5.4

Ope

ratio

n 5.

5 M

aint

e-na

nce

Use

r D

evel

oper

M

aint

aine

r SQ

A

FOO

TNO

TE

Req

uire

d m

ean

resp

onse

tim

e ca

n be

der

ived

from

spe

cific

atio

n of

requ

ired

real

-tim

e pr

oces

sing

, use

r exp

ecta

tion

of b

usin

ess

need

s or

obs

erva

tion

of u

ser r

eact

ion.

A u

ser c

ogni

tive

of th

e as

pect

s of

hum

an e

rgon

omic

s m

ight

be

cons

ider

ed.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

4.1

a) (

cont

inue

d)

Exte

rnal

tim

e be

havi

our m

etric

s

a) R

espo

nse

time

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Res

pons

e tim

e (W

orst

cas

e re

spon

se ti

me

ratio

)

Wha

t is

the

abso

lute

lim

it on

tim

e re

quire

d in

fulfi

lling

a fu

nctio

n?

In th

e w

orst

cas

e, c

an

user

stil

l get

resp

onse

w

ithin

the

spec

ified

tim

e lim

it?

In th

e w

orst

cas

e, c

an

user

stil

l get

repl

y fro

m th

e so

ftwar

e w

ithin

a ti

me

shor

t en

ough

to b

e to

lera

ble

for u

ser?

Cal

ibra

te th

e te

st.

Emul

ate

a co

nditio

n wh

ereb

y th

e sy

stem

reac

hes

a m

axim

um lo

ad s

ituat

ion.

R

un a

pplic

atio

n an

d m

onito

r re

sult(

s).

X= T

max

/ R

max

Tm

ax=

MA

X(Ti

) (fo

r i=

1 to

N)

Rm

ax =

requ

ired

max

imum

resp

onse

tim

e M

AX(T

i)= m

axim

um re

spon

se ti

me

amon

g ev

alua

tions

N

= nu

mbe

r of e

valu

atio

ns (s

ampl

ed s

hots

) Ti

= re

spon

se ti

me

for i

-th e

valu

atio

n (s

hot)

CO

MM

ENT(

S) D

istri

butio

n m

ay b

e ca

lcul

ated

as

illus

trate

d be

low

. St

atis

tical

max

imal

ratio

Y=

Tdev

/ R

max

T

dev

= Tm

ean

+ K

( D

EV )

Tde

v is

tim

e de

viat

ed fr

om m

ean

time

to th

e pa

rticu

lar t

ime:

e.g

. 2 o

r 3 ti

mes

of s

tand

ard

devi

atio

n.

K: c

oeffi

cien

t (2

or 3

) D

EV=S

QR

T{ ∑

( (Ti

-Tm

ean)

**2)

/ (N

-1)}

(for

i=1

to N

) Tm

ean

= ∑

(Ti)

/ N,

(for

i=1

to N

) TX

mea

n =

requ

ired

mea

n re

spon

se ti

me

0 <

X Th

e ne

arer

to

1 an

d le

ss

than

1 is

the

bette

r.

Abso

lute

Tmax

= Ti

me

Rm

ax=

Tim

e Ti

= Ti

me

N=

Cou

nt

X= T

ime/

Ti

me

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Sys.

/Sw

. In

tegr

atio

n5.

3 Q

ualif

icat

ion

test

ing

5.4

Ope

ratio

n 5.

5 M

aint

e-na

nce

Use

r D

evel

oper

M

aint

aine

r SQ

A

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

4.1

Tim

e be

havi

our m

etric

s

b) T

hrou

ghpu

t

Exte

rnal

tim

e be

havi

our m

etric

s

b) T

hrou

ghpu

t M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Thro

ughp

ut

How

man

y ta

sks

can

be s

ucce

ssfu

lly

perfo

rmed

ove

r a

give

n pe

riod

of ti

me?

Cal

ibra

te e

ach

task

ac

cord

ing

to th

e in

tend

ed

prio

rity

give

n.

Sta

rt se

vera

l job

task

s.

Mea

sure

the

time

it ta

kes

for t

he m

easu

red

task

to

com

plet

e its

ope

ratio

n.

Kee

p a

reco

rd o

f eac

h at

tem

pt.

X =

A /

T A

= nu

mbe

r of c

ompl

eted

task

s T

= ob

serv

atio

n tim

e pe

riod

0 <

X Th

e la

rger

is

the

bette

r.

Rat

io

A= C

ount

T=

Tim

e X=

C

ount

/ Ti

me

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Sys.

/Sw

. In

tegr

atio

n5.

3 Q

ualif

icat

ion

test

ing

5.4

Ope

ratio

n 5.

5 M

aint

e-na

nce

Use

r D

evel

oper

M

aint

aine

r SQ

A

Thro

ughp

ut

(Mea

n am

ount

of

thro

ughp

ut)

Wha

t is

the

aver

age

num

ber o

f con

curre

nt

task

s th

e sy

stem

can

ha

ndle

ove

r a s

et u

nit

of ti

me?

Cal

ibra

te e

ach

task

ac

cord

ing

to in

tend

ed

prio

rity.

E

xecu

te a

num

ber o

f co

ncur

rent

task

s.

Mea

sure

the

time

it ta

kes

to c

ompl

ete

the

sele

cted

ta

sk in

the

give

n tra

ffic.

K

eep

a re

cord

of e

ach

atte

mpt

.

X =

Xmea

n / R

mea

n Xm

ean

= ∑

(Xi)/

N

Rm

ean

= re

quire

d m

ean

thro

ughp

ut

Xi =

Ai /

Ti

Ai =

num

ber o

f con

curre

nt ta

sks

obse

rved

ov

er s

et p

erio

d of

tim

e fo

r i-th

eva

luat

ion

Ti =

set

per

iod

of ti

me

for i

-th e

valu

atio

n N

= n

umbe

r of e

valu

atio

ns

0 <

X Th

e la

rger

is

the

bette

r.

Abso

lute

Xmea

n=

Cou

nt

Rm

ean=

C

ount

Ai

= C

ount

Ti

= Ti

me

Xi=

Cou

nt/

Tim

e N

= C

ount

X

= C

ount

/C

ount

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.4

Ope

ratio

n 5.

5 M

aint

e-na

nce

Use

r D

evel

oper

M

aint

aine

r SQ

A

Thro

ughp

ut

(Wor

st c

ase

thro

ughp

ut

ratio

)

Wha

t is

the

abso

lute

lim

it on

the

syst

em in

te

rms

of th

e nu

mbe

r an

d ha

ndlin

g of

co

ncur

rent

task

s as

th

roug

hput

?

Cal

ibra

te th

e te

st.

Emul

ate

the

cond

ition

w

here

by th

e sy

stem

re

ache

s a

situ

atio

n of

m

axim

um lo

ad. R

un jo

b ta

sks

conc

urre

ntly

and

m

onito

r res

ult(s

).

X =

Xmax

/ R

max

Xm

ax =

MA

X(Xi

) (fo

r i =

1 to

N)

Rm

ax =

requ

ired

max

imum

thro

ughp

ut.

MAX

(Xi)

= m

axim

um n

umbe

r of j

ob ta

sks

amon

g e

valu

atio

ns

Xi =

Ai /

Ti

Ai =

num

ber o

f con

curre

nt ta

sks

obse

rved

ov

er s

et p

erio

d of

tim

e fo

r i-th

eva

luat

ion

Ti =

set

per

iod

of ti

me

for i

-th e

valu

atio

n N

= nu

mbe

r of e

valu

atio

ns

0 <

X

The

larg

er is

th

e be

tter.

Abso

lute

Xmax

= C

ount

R

max

= C

ount

Ai

= C

ount

Ti

= Ti

me

Xi=

Cou

nt/

Tim

e N

= C

ount

Xd

ev=

Cou

nt

X =

Cou

nt/

Cou

nt

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.4

Ope

ratio

n 5.

5 M

aint

e-na

nce

Use

r D

evel

oper

M

aint

aine

r SQ

A

FOO

TNO

TE

1 D

istri

butio

n m

ay b

e ca

lcul

ated

as

illust

rate

d be

low

. St

atis

tical

max

imal

ratio

Y=

Xdev

/ X

max

X

dev

= Xm

ean

+ K

( D

EV )

Xde

v is

tim

e de

viat

ed fr

om m

ean

time

to th

e pa

rticu

lar t

ime:

e.g

. 2 o

r 3 ti

mes

of s

tand

ard

devi

atio

n.

K: c

oeffi

cien

t (2

or 3

) D

EV=S

QR

T{ ∑

( (Xi

-Xm

ean)

**2)

/ (N

-1)}

(for i

=1 to

N)

Xmea

n =

∑(X

i)/N

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

4.1

Tim

e be

havi

our m

etric

s

c) T

urna

roun

d tim

e

Exte

rnal

tim

e be

havi

our m

etric

s

c) T

urna

roun

d tim

e M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Turn

arou

nd

time

Wha

t is

the

wai

t tim

e th

e us

er e

xper

ienc

es

afte

r iss

uing

an

inst

ruct

ion

to s

tart

a gr

oup

of re

late

d ta

sks

and

thei

r com

plet

ion?

Cal

ibra

te th

e te

st

acco

rdin

gly.

St

art t

he jo

b ta

sk. M

easu

re

the

time

it ta

kes

for t

he jo

b ta

sk to

com

plet

e its

op

erat

ion.

Ke

ep a

reco

rd o

f eac

h at

tem

pt.

T =

Tim

e be

twee

n us

er’s

fini

shin

g ge

tting

ou

tput

resu

lts a

nd u

ser’s

fini

shin

g re

ques

t

0 <

T Th

e sh

orte

r th

e be

tter.

Rat

io

T= T

ime

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Sys.

/Sw

. In

tegr

atio

n5.

3 Q

ualif

icat

ion

test

ing

5.4

Ope

ratio

n 5.

5 M

aint

e-na

nce

Use

r D

evel

oper

M

aint

aine

r SQ

A

FOO

TNO

TE

It is

reco

mm

ende

d to

take

acc

ount

of t

ime

band

wid

th a

nd to

use

sta

tistic

al a

naly

sis

with

mea

sure

s fo

r man

y ta

sks

(sam

ple

shot

s), n

ot o

nly

one

task

(sho

t).

Turn

arou

nd

time

(Mea

n tim

e fo

r tur

naro

und)

Wha

t is

the

aver

age

wai

t tim

e th

e us

er

expe

rienc

es a

fter

issu

ing

an in

stru

ctio

n to

sta

rt a

grou

p of

re

late

d ta

sks

and

thei

r com

plet

ion

with

in a

spe

cifie

d sy

stem

load

in te

rms

of c

oncu

rrent

task

s an

d sy

stem

ut

ilisat

ion?

Cal

ibra

te th

e te

st.

Em

ulat

e a

cond

ition

w

here

a lo

ad is

pla

ced

on

the

syst

em b

y ex

ecut

ing

a nu

mbe

r of c

oncu

rrent

ta

sks

(sam

pled

sho

ts).

M

easu

re th

e tim

e it

take

s to

com

plet

e th

e se

lect

ed

job

task

in

the

give

n tra

ffic.

K

eep

a re

cord

of e

ach

atte

mpt

.

X =

Tmea

n/TX

mea

n Tm

ean

= ∑

(Ti)/

N, (

for

i=1

to N

) TX

mea

n =

requ

ired

mea

n tu

rnar

ound

tim

e Ti

= tu

rnar

ound

tim

e fo

r i-th

eva

luat

ion

(sho

t) N

= n

umbe

r of e

valu

atio

ns (s

ampl

ed s

hots

)

0 <

X Th

e sh

orte

r is

the

bette

r.

Abso

lute

Tmea

n=

Tim

e TX

mea

n=

Tim

e Ti

= Ti

me

N=

Cou

nt

X= T

ime/

Ti

me

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Sys.

/Sw

. In

tegr

atio

n5.

3 Q

ualif

icat

ion

test

ing

5.4

Ope

ratio

n 5.

5 M

aint

e-na

nce

Use

r D

evel

oper

M

aint

aine

r SQ

A

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

4.1

c) (

cont

inue

d)

Exte

rnal

tim

e be

havi

our m

etric

s

c) T

urna

roun

d tim

e M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Turn

arou

nd

time

(Wor

st

case

tu

rnar

ound

tim

e ra

tio)

Wha

t is

the

abso

lute

lim

it on

tim

e re

quire

d in

fulfi

lling

a jo

b ta

sk?

In th

e w

orst

cas

e,

how

long

doe

s it

take

fo

r sof

twar

e sy

stem

to

per

form

spe

cifie

d ta

sks?

Cal

ibra

te th

e te

st.

Emul

ate

a co

nditi

on w

here

by

the

syst

em re

ache

s m

axim

um lo

ad in

term

s of

ta

sks

perfo

rmed

. Run

the

sele

cted

job

task

and

m

onito

r res

ult(s

).

X= T

max

/ R

max

Tm

ax=

MA

X(Ti

) (fo

r i=

1 to

N)

Rm

ax =

requ

ired

max

imum

turn

arou

nd ti

me

MAX

(Ti)=

max

imum

turn

arou

nd ti

me

amon

g ev

alua

tions

N

= nu

mbe

r of e

valu

atio

ns (s

ampl

ed s

hots

) Ti

= tu

rnar

ound

tim

e fo

r i-th

eva

luat

ion

(sho

t)

0 <

X

The

near

er to

1.

0 an

d le

ss

than

1.0

is th

e be

tter.

Abso

lute

X= T

ime/

Ti

me

Tmax

=

Tim

e

Rm

ax=

Tim

e Ti

= Ti

me

N=

Cou

nt

Tdev

=

Tim

e

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.4

Ope

ratio

n 5.

5 M

aint

e-na

nce

Use

r D

evel

oper

M

aint

aine

r SQ

A

FOO

TNO

TE

Dis

tribu

tion

may

be

calc

ulat

ed a

s illu

stra

ted

belo

w.

Stat

istic

al m

axim

al ra

tio Y

= Td

ev /

Rm

ax

Tde

v =

Tmea

n +

K (

DEV

) T

dev

is ti

me

devi

ated

from

mea

n tim

e to

the

parti

cula

r tim

e: e

.g. 2

or 3

tim

es o

f sta

ndar

d de

viat

ion.

K:

coe

ffici

ent (

2 or

3)

DEV

=SQ

RT{

∑( (

Ti-T

mea

n) **

2) /

(N-1

)} (fo

r i=1

to N

) Tm

ean

= ∑

(Ti)

/ N,

(for

i=1

to N

) TX

mea

n =

requ

ired

mea

n tu

rnar

ound

tim

e

Wai

ting

time

Wha

t pro

porti

on o

f th

e tim

e do

use

rs

spen

d w

aitin

g fo

r the

sy

stem

to re

spon

d?

Exec

ute

a nu

mbe

r of

scen

ario

s of

con

curre

nt

task

s.

Mea

sure

the

time

it ta

kes

to c

ompl

ete

the

sele

cted

op

erat

ion(

s).

Keep

a re

cord

of e

ach

atte

mpt

and

com

pute

the

mea

n tim

e fo

r eac

h sc

enar

io.

X =

Ta

/ Tb

Ta =

tota

l tim

e sp

ent w

aitin

g

Tb =

task

tim

e

0<=

X Th

e sm

alle

r th

e be

tter.

Abso

lute

Ta=

Tim

e Tb

= Ti

me

X= T

ime/

Tim

e

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Sys.

/Sw

. In

tegr

atio

n5.

3 Q

ualif

icat

ion

test

ing

5.4

Ope

ratio

n 5.

5 M

aint

e-na

nce

Use

r D

evel

oper

M

aint

aine

r SQ

A

FOO

TNO

TE

If th

e ta

sks

can

be p

artia

lly c

ompl

eted

, the

Tas

k ef

ficie

ncy

met

ric s

houl

d be

use

d w

hen

mak

ing

com

paris

ons.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

4.2

Res

ourc

e ut

iliza

tion

met

rics

a)

I/O

dev

ices

reso

urce

util

izat

ion

Exte

rnal

reso

urce

util

isat

ion

met

rics

a)

I/O

dev

ices

reso

urce

util

isat

ion

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

I/O d

evic

es

utili

satio

n

Is th

e I/O

dev

ice

utilis

atio

n to

o hi

gh,

caus

ing

inef

ficie

ncie

s?

Exec

ute

conc

urre

ntly

a

larg

e nu

mbe

r of t

asks

, re

cord

I/O

dev

ice

utilis

atio

n, a

nd c

ompa

re

with

the

desi

gn o

bjec

tives

.

X =

A / B

A

= tim

e of

I/O

dev

ices

occ

upie

d B

= sp

ecifi

ed ti

me

whi

ch is

des

igne

d to

oc

cupy

I/O

dev

ices

0 <=

X <

= 1

The

less

than

an

d ne

arer

to

the

1.0

is th

e be

tter.

Abso

lute

A= T

ime

B= T

ime

X= T

ime/

Tim

e

Test

ing

repo

rt O

pera

tion

repo

rt

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

SQA

I/O l

oadi

ng

limits

W

hat i

s th

e ab

solu

te

limit

on I/

O u

tilis

atio

n in

fulfi

lling

a fu

nctio

n? C

alib

rate

the

test

co

nditi

on. E

mul

ate

a co

nditi

on w

here

by th

e sy

stem

reac

hes

a si

tuat

ion

of m

axim

um lo

ad. R

un

appl

icat

ion

and

mon

itor

resu

lt(s)

.

X =

Amax

/ R

max

Am

ax =

MAX

(Ai),

(fo

r i =

1 to

N)

Rm

ax =

requ

ired

max

imum

I/O

mes

sage

s M

AX(A

i) =

Max

imum

num

ber o

f I/O

m

essa

ges

from

1st

to i-

th e

valu

atio

n N

= nu

mbe

r of e

valu

atio

ns.

0<=

X

The

smal

ler i

s th

e be

tter.

Abso

lute

Amax

=

Cou

nt

Rm

ax =

C

ount

Ai

= C

ount

N=

Cou

nt

X =

Cou

nt/

Cou

nt

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r D

evel

oper

M

aint

aine

r SQ

A

I/O re

late

d er

rors

H

ow o

ften

does

the

user

enc

ount

er

prob

lem

s in

I/O

de

vice

rela

ted

oper

atio

ns?

Cal

ibra

te th

e te

st

cond

ition

s. E

mul

ate

a co

nditi

on w

here

by th

e sy

stem

reac

hes

a si

tuat

ion

of m

axim

um I/

O lo

ad. R

un

the

appl

icat

ion

and

reco

rd

num

ber o

f erro

rs d

ue to

I/O

failu

re a

nd w

arni

ngs.

X =

A / T

A

= nu

mbe

r of w

arni

ng m

essa

ges

or s

yste

m

failu

res

T =

Use

r ope

ratin

g tim

e du

ring

user

ob

serv

atio

n

0 <=

X

The

smal

ler i

s th

e be

tter.

Rat

io

A =

Cou

nt

T =

Tim

e X

= C

ount

/ Ti

me

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r M

aint

aine

r SQ

A

Mea

n I/

O

fulfi

lmen

t rat

io

Wha

t is

the

aver

age

num

ber o

f I/O

rela

ted

erro

r mes

sage

s an

d fa

ilure

s ov

er a

sp

ecifi

ed le

ngth

of

time

and

spec

ified

ut

ilisat

ion?

Cal

ibra

te th

e te

st

cond

ition

. Em

ulat

e a

cond

ition

whe

reby

the

syst

em re

ache

s a

situ

atio

n of

max

imum

load

. Run

the

appl

icat

ion

and

reco

rd

num

ber o

f erro

rs d

ue to

I/O

failu

re a

nd w

arni

ngs.

X =

Amea

n / R

mea

n Am

ean

= ∑

(Ai)/

N

Rm

ean

= re

quire

d m

ean

num

ber o

f I/O

m

essa

ges

Ai =

num

ber o

f I/O

erro

r mes

sage

s fo

r i-th

ev

alua

tion

N =

num

ber o

f eva

luat

ions

0<=

X

The

smal

ler i

s th

e be

tter.

Abso

lute

Amea

n =

Cou

nt

Rm

ean

= C

ount

Ai

= C

ount

N=

Cou

nt

X =

Cou

nt/

Cou

nt

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r D

evel

oper

M

aint

aine

r SQ

A

Use

r wai

ting

time

of I/

O

devi

ces

utili

satio

n

Wha

t is

the

impa

ct o

f I/O

dev

ice

utilis

atio

n on

the

user

wai

t tim

es?

Exec

ute

conc

urre

ntly

a

larg

e am

ount

of t

asks

and

m

easu

re th

e us

er w

ait

times

as

a re

sult

of I/

O

devi

ce o

pera

tion.

T =

Tim

e sp

ent t

o w

ait f

or fi

nish

of I

/O

devi

ces

oper

atio

n

0 <

T Th

e sh

orte

r is

the

bette

r.

Rat

io

T= T

ime

Te

stin

g re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r D

evel

oper

M

aint

aine

r SQ

A FO

OTN

OTE

It

is re

com

men

ded

that

the

max

imal

and

dis

tribu

ted

time

are

to b

e in

vest

igat

ed fo

r sev

eral

cas

es o

f tes

ting

or o

pera

ting,

bec

ause

the

mea

sure

s ar

e te

nd to

be

fluct

uate

d by

con

ditio

n of

use

.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

4.2

Res

ourc

e ut

iliza

tion

met

rics

b)

Mem

ory

reso

urce

util

izat

ion

Exte

rnal

reso

urce

util

isat

ion

met

rics

b)

Mem

ory

reso

urce

util

isat

ion

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Max

imum

m

emor

y ut

ilisa

tion

Wha

t is

the

abso

lute

lim

it on

mem

ory

requ

ired

in fu

lfillin

g a

func

tion?

Cal

ibra

te th

e te

st

cond

ition

. Em

ulat

e a

cond

ition

whe

reby

the

syst

em re

ache

s a

situ

atio

n of

max

imum

load

. Run

ap

plic

atio

n an

d m

onito

r re

sult(

s).

X =

Amax

/ R

max

Am

ax =

MAX

(Ai),

(fo

r i =

1 to

N)

Rm

ax =

requ

ired

max

imum

mem

ory

rela

ted

erro

r mes

sage

s M

AX(A

i) =

Max

imum

num

ber o

f mem

ory

rela

ted

erro

r mes

sage

s fro

m 1

st to

i-th

ev

alua

tion

N=

num

ber o

f eva

luat

ions

0<=

X

The

smal

ler i

s th

e be

tter.

Abso

lute

Amax

= C

ount

R

max

= C

ount

Ai

= C

ount

N

= C

ount

X

= C

ount

/ C

ount

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r D

evel

oper

M

aint

aine

r SQ

A

Mea

n oc

curr

ence

of

mem

ory

err

or

Wha

t is

the

aver

age

num

ber o

f mem

ory

rela

ted

erro

r m

essa

ges

and

failu

res

over

a

spec

ified

leng

th o

f tim

e an

d a

spec

ified

lo

ad o

n th

e sy

stem

?

Cal

ibra

te th

e te

st

cond

ition

. Em

ulat

e a

cond

ition

whe

reby

the

syst

em re

ache

s a

situ

atio

n of

max

imum

load

. Run

the

appl

icat

ion

and

reco

rd

num

ber o

f er

rors

due

to

mem

ory

failu

re a

nd

war

ning

s.

X =

Amea

n / R

mea

n Am

ean

= ∑

(Ai)/

N

Rm

ean

= re

quire

d m

ean

num

ber o

f mem

ory

rela

ted

erro

r mes

sage

s Ai

= n

umbe

r of m

emor

y re

late

d er

ror

mes

sage

s fo

r i-th

eva

luat

ion

N =

num

ber o

f eva

luat

ions

0<=

X

The

smal

ler i

s th

e be

tter.

Abso

lute

Amea

n=

Cou

nt

Rm

ean=

C

ount

Ai

= C

ount

N

= C

ount

X

= C

ount

/ C

ount

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r D

evel

oper

M

aint

aine

r SQ

A

Rat

io o

f m

emor

y er

ror/t

ime

How

man

y m

emor

y er

rors

wer

e ex

perie

nced

ove

r a

set p

erio

d of

tim

e an

d sp

ecifi

ed re

sour

ce

utilis

atio

n?

Cal

ibra

te th

e te

st

cond

ition

s.

Em

ulat

e a

cond

ition

w

here

by th

e sy

stem

re

ache

s a

situ

atio

n of

m

axim

um lo

ad.

Run

the

appl

icat

ion

and

reco

rd n

umbe

r of

erro

rs

due

to m

emor

y fa

ilure

and

w

arni

ngs.

X =

A / T

A =

num

ber o

f war

ning

mes

sage

s or

sys

tem

fa

ilure

s

T =

Use

r ope

ratin

g tim

e du

ring

user

ob

serv

atio

n

0 <=

X

The

smal

ler i

s th

e be

tter.

Rat

io

A =

Cou

nt

T =

Tim

e

X =

Cou

nt/

Tim

e

Test

ing

repo

rt

Ope

ratio

n re

port

show

ing

elap

se ti

me

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r M

aint

aine

r SQ

A

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

4.2

Res

ourc

e ut

iliza

tion

met

rics

c)

Tra

nsm

issi

on re

sour

ce u

tiliz

atio

n

Exte

rnal

reso

urce

util

isat

ion

met

rics

c)

Tra

nsm

issi

on re

sour

ce u

tilis

atio

n M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Max

imum

tr

ansm

issi

on

utili

satio

n

Wha

t is

the

abso

lute

lim

it of

tran

smis

sion

s re

quire

d to

fulfi

l a

func

tion?

Eval

uate

wha

t is

requ

ired

for t

he s

yste

m to

reac

h a

situ

atio

n of

max

imum

load

. Em

ulat

e th

is c

ondi

tion.

R

un a

pplic

atio

n an

d m

onito

r res

ult(s

).

X =

Amax

/ R

max

Am

ax =

MAX

(Ai),

(fo

r i =

1 to

N)

Rm

ax =

requ

ired

max

imum

num

ber o

f tra

nsm

issi

on re

late

d er

ror m

essa

ges

and

failu

res

MAX

(Ai)

= M

axim

um n

umbe

r of t

rans

mis

sion

re

late

d er

ror m

essa

ges

and

failu

res

from

1st

to

i-th

eva

luat

ion

N=

num

ber o

f eva

luat

ions

0<=

X

The

smal

ler i

s th

e be

tter.

Abso

lute

Amax

=

Cou

nt

Rm

ax =

C

ount

Ai

= C

ount

N=

Cou

nt

X =

Cou

nt/

Cou

nt

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r D

evel

oper

M

aint

aine

r SQ

A

Med

ia d

evic

e ut

ilisa

tion

bala

ncin

g

Wha

t is

the

degr

ee o

f sy

nchr

onis

atio

n be

twee

n di

ffere

nt

med

ia o

ver a

set

pe

riod

of ti

me?

Cal

ibra

te th

e te

st

cond

ition

s. E

mul

ate

a co

nditi

on w

here

by th

e sy

stem

reac

hes

a si

tuat

ion

of m

axim

um tr

ansm

issi

on

load

. Run

the

appl

icat

ion

and

reco

rd th

e de

lay

in th

e pr

oces

sing

of d

iffer

ent

med

ia ty

pes.

X =

Sync

Tim

e/T

Sync

Tim

e =

Tim

e de

vote

d to

a c

ontin

uous

re

sour

ce

T =

requ

ired

time

perio

d du

ring

whi

ch

diss

imila

r med

ia a

re e

xpec

ted

to fi

nish

thei

r ta

sks

with

syn

chro

nisa

tion

The

smal

ler i

s th

e be

tter.

Rat

io

Sync

Tim

e =

Tim

e T

= Ti

me

X =

Tim

e/Ti

me

Test

ing

repo

rtO

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r M

aint

aine

r SQ

A

Mea

n

occu

rren

ce o

f tr

ansm

issi

on

erro

r

Wha

t is

the

aver

age

num

ber o

f tra

nsm

issi

on-re

late

d er

ror m

essa

ges

and

failu

res

over

a

spec

ified

leng

th o

f tim

e an

d sp

ecifi

ed

utilis

atio

n?

Cal

ibra

te th

e te

st

cond

ition

. Em

ulat

e a

cond

ition

whe

reby

the

syst

em re

ache

s a

situ

atio

n of

max

imum

load

. Run

the

appl

icat

ion

and

reco

rd

num

ber o

f erro

rs d

ue to

tra

nsm

issi

on fa

ilure

and

w

arni

ngs.

X =

Amea

n / R

mea

n Am

ean

= ∑

(Ai)/

N

Rm

ean

= re

quire

d m

ean

num

ber o

f tra

nsm

issi

on re

late

d er

ror m

essa

ges

and

failu

res

Ai =

Num

ber o

f tra

nsm

issi

on re

late

d er

ror

mes

sage

s an

d fa

ilure

s fo

r i-th

eva

luat

ion

N =

num

ber o

f eva

luat

ions

0<=

X

The

smal

ler i

s th

e be

tter.

Abso

lute

Amea

n=

Cou

nt

Rm

ean=

C

ount

Ai

= C

ount

N

= C

ount

X

= C

ount

/ C

ount

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r D

evel

oper

M

aint

aine

r SQ

A

Mea

n of

tr

ansm

issi

on

erro

r per

tim

e

How

man

y tra

nsm

issi

on-re

late

d er

ror m

essa

ges

wer

e ex

perie

nced

ove

r a

set p

erio

d of

tim

e an

d sp

ecifi

ed re

sour

ce

utilis

atio

n?

Cal

ibra

te th

e te

st

cond

ition

s. E

mul

ate

a co

nditi

on w

here

by th

e sy

stem

reac

hes

a si

tuat

ion

of m

axim

um tr

ansm

issi

on

load

. Run

the

appl

icat

ion

and

reco

rd n

umbe

r of

erro

rs d

ue to

tran

smis

sion

fa

ilure

and

war

ning

s.

X =

A / T

A

= nu

mbe

r of w

arni

ng m

essa

ges

or s

yste

m

failu

res

T =

Use

r ope

ratin

g tim

e du

ring

user

ob

serv

atio

n

0 <=

X

The

smal

ler i

s th

e be

tter.

Rat

io

A =

Cou

nt

T =

Tim

e X

= C

ount

/ Ti

me

Test

ing

repo

rt O

pera

tion

repo

rt sh

owin

g el

apse

tim

e

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r M

aint

aine

r SQ

A

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

4.2

c) (

cont

inue

d)

Exte

rnal

reso

urce

util

isat

ion

met

rics

c)

Tra

nsm

issi

on re

sour

ce u

tilis

atio

n M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Tran

smis

sion

ca

paci

ty

utili

satio

n

Is s

oftw

are

syst

em

capa

ble

of p

erfo

rmin

g ta

sks

with

in e

xpec

ted

trans

mis

sion

ca

paci

ty?

Exec

ute

conc

urre

ntly

sp

ecifi

ed ta

sks

with

m

ultip

le u

sers

, obs

erve

tra

nsm

issi

on c

apac

ity a

nd

com

pare

spe

cifie

d on

e.

X =

A / B

A

= tra

nsm

issi

on c

apac

ity

B =

spec

ified

tran

smis

sion

cap

acity

whi

ch is

de

sign

ed t

o be

use

d by

the

sof

twar

e du

ring

exec

utio

n

0 <=

X <

= 1

The

less

than

an

d ne

arer

to

the

1.0

is th

e be

tter.

Abso

lute

A= S

ize

B= S

ize

X= S

ize

/Si

ze

Test

ing

repo

rt O

pera

tion

repo

rt

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

SQA

FOO

TNO

TE

It is

reco

mm

ende

d to

mea

sure

dyn

amic

ally

pea

ked

valu

e w

ith m

ultip

le u

sers

.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

4.3

Effic

ienc

y co

mpl

ianc

e m

etric

s

Effic

ienc

y co

mpl

ianc

e m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Effic

ienc

y C

ompl

ianc

e H

ow c

ompl

iant

is th

e ef

ficie

ncy

of th

e pr

oduc

t to

appl

icab

le

regu

latio

ns,

stan

dard

s an

d co

nven

tions

?

Cou

nt th

e nu

mbe

r of i

tem

s re

quiri

ng c

ompl

ianc

e th

at

have

bee

n m

et a

nd

com

pare

with

the

num

ber

of it

ems

requ

iring

co

mpl

ianc

e in

the

spec

ifica

tion.

X =

1 -

A / B

(X

: Rat

io o

f sat

isfie

d co

mpl

ianc

e ite

ms

rela

ting

to e

ffici

ency

) A=

Num

ber o

f effi

cien

cy c

ompl

ianc

e ite

ms

spec

ified

that

hav

e no

t bee

n im

plem

ente

d du

ring

test

ing

B= T

otal

num

ber o

f effi

cien

cy c

ompl

ianc

e ite

ms

spec

ified

0<=

X <=

1 Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X= C

ount

/C

ount

Prod

uct

desc

riptio

n (U

ser m

anua

l or

Sp

ecifi

catio

n)

of c

ompl

ianc

e an

d re

late

d st

anda

rds,

co

nven

tions

or

regu

latio

ns Te

st

spec

ifica

tion

and

repo

rt

5.3

Qua

lific

atio

n te

stin

g 6.

5 Va

lidat

ion

Supp

lier

Use

r

FOO

TNO

TE

It m

ay b

e us

eful

to c

olle

ct s

ever

al m

easu

red

valu

es a

long

tim

e, to

ana

lyse

the

trend

of i

ncre

asin

g sa

tisfie

d co

mpl

ianc

e ite

ms

and

to d

eter

min

e w

heth

er th

ey a

re fu

lly s

atis

fied

or n

ot.

ISO/IEC TR 9126-2:2003(E)


8.5 Maintainability metrics

An external maintainability metric should be able to measure such attributes as the behaviour of the maintainer, user, or system including the software, when the software is maintained or modified during testing or maintenance.

8.5.1 Analysability metrics

An external analysability metric should be able to measure such attributes as the maintainer’s or user’s effort or spent of resources when trying to diagnose deficiencies or causes of failures, or for identifying parts to be modified.

8.5.2 Changeability metrics

An external changeability metric should be able to measure such attributes as the maintainer’s or user’s effort by measuring the behaviour of the maintainer, user or system including the software when trying to implement a specified modification.

8.5.3 Stability metrics

An external stability metric should be able to measure attributes related to unexpected behaviour of the system including the software when the software is tested or operated after modification.

8.5.4 Testability metrics

An external testability metric should be able to measure such attributes as the maintainer’s or user’s effort by measuring the behaviour of the maintainer, user or system including software when trying to test the modified or non-modified software.

8.5.5 Maintainability compliance metrics

An external maintainability compliance metric should be able to measure an attribute such as the number of functions or occurrences of compliance problems, where the software product fails to adhere to required standards, conventions or regulations relating to maintainability.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

5.1

Anal

ysab

ility

met

rics

Exte

rnal

ana

lysa

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Audi

t tra

il ca

pabi

lity

Can

use

r ide

ntify

sp

ecifi

c op

erat

ion

whi

ch c

ause

d fa

ilure

? C

an m

aint

aine

r eas

ily

find

spe

cific

op

erat

ion

whi

ch

caus

ed fa

ilure

?

Obs

erve

beh

avio

ur o

f use

r or

mai

ntai

ner w

ho is

tryi

ng

to re

solv

e fa

ilure

s.

X= A

/ B

A= N

umbe

r of d

ata

actu

ally

reco

rded

dur

ing

oper

atio

n B=

Num

ber o

f dat

a pl

anne

d to

be

reco

rded

en

ough

to m

onito

r sta

tus

of s

oftw

are

durin

g op

erat

ion

0<=X

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

Dia

gnos

tic

func

tion

supp

ort

How

cap

able

are

the

diag

nost

ic fu

nctio

ns

in s

uppo

rting

cau

sal

anal

ysis

? C

an u

ser i

dent

ify th

e sp

ecifi

c op

erat

ion

whi

ch c

ause

d fa

ilure

? (U

ser m

ay b

e ab

le to

av

oid

fallin

g in

to th

e sa

me

failu

re

occu

rrenc

e ag

ain

with

al

tern

ativ

e op

erat

ion.

) C

an m

aint

aine

r eas

ily

find

caus

e of

failu

re?

Obs

erve

beh

avio

ur o

f use

r or

mai

ntai

ner w

ho is

tryi

ng

to re

solv

e fa

ilure

s us

ing

diag

nost

ics

func

tions

.

X= A

/ B

A=

Num

ber o

f fai

lure

s w

hich

mai

ntai

ner c

an

diag

nose

(usi

ng th

e di

agno

stic

s fu

nctio

n) to

un

ders

tand

the

caus

e-ef

fect

rela

tions

hip

B= T

otal

num

ber o

f reg

iste

red

failu

res

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

Failu

re a

naly

sis

capa

bilit

y C

an u

ser i

dent

ify

spec

ific

oper

atio

n w

hich

cau

sed

failu

re?

Can

mai

ntai

ner e

asily

fin

d ca

use

of fa

ilure

?

Obs

erve

beh

avio

ur o

f use

r or

mai

ntai

ner w

ho is

tryi

ng

to re

solv

e fa

ilure

s.

X=1-

A /

B

A= N

umbe

r of f

ailu

res

of w

hich

cau

ses

are

still

not f

ound

B=

Tot

al n

umbe

r of r

egis

tere

d fa

ilure

s

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r D

evel

oper

M

aint

aine

r O

pera

tor

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

5.1

(con

tinue

d)

Exte

rnal

ana

lysa

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Failu

re a

naly

sis

effic

ienc

y

Can

use

r effi

cien

tly

anal

yse

caus

e of

fa

ilure

?

(Use

r som

etim

es

perfo

rms

mai

nten

ance

by

setti

ng p

aram

eter

s.)

Can

mai

ntai

ner

easi

ly fi

nd c

ause

of

failu

re?

Obs

erve

beh

avio

ur o

f use

r or

mai

ntai

ner w

ho is

tryi

ng

to re

solv

e fa

ilure

s.

X= S

um(T

) / N

0<

=X

The

shor

ter i

s th

e be

tter.

Rat

io

T= T

ime

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TES

1 It

is re

com

men

ded

to m

easu

re m

axim

al ti

me

of th

e w

orst

cas

e an

d tim

e du

ratio

n (b

andw

idth

) to

repr

esen

t dev

iatio

n.

2 It

is re

com

men

ded

to e

xclu

de n

umbe

r of f

ailu

res

of w

hich

cau

ses

are

not y

et fo

und

whe

n m

easu

rem

ent i

s do

ne. H

owev

er, t

he ra

tio o

f suc

h ob

scur

e fa

ilure

s sh

ould

be

also

mea

sure

d an

d pr

esen

ted

toge

ther

. 3

From

the

indi

vidu

al u

ser’s

poi

nt o

f vie

w, t

ime

is o

f con

cern

, whi

le e

ffort

may

als

o be

of c

once

rn fr

om th

e m

aint

aine

r’s p

oint

of v

iew

. The

refo

re, p

erso

n-ho

urs

may

be

used

inst

ead

of ti

me.

Stat

us

mon

itorin

g ca

pabi

lity

Can

use

r ide

ntify

sp

ecifi

c op

erat

ion

w

hich

cau

sed

failu

re

by g

ettin

g m

onito

red

data

dur

ing

oper

atio

n?

Can

mai

ntai

ner e

asily

fin

d ca

use

of fa

ilure

by

get

ting

mon

itore

d da

ta d

urin

g op

erat

ion?

Obs

erve

beh

avio

ur o

f use

r or

mai

ntai

ner w

ho is

tryi

ng

to g

et m

onito

red

data

re

cord

ing

stat

us o

f so

ftwar

e du

ring

oper

atio

n.

X= 1

- A

/ B

A= N

umbe

r of c

ases

whi

ch m

aint

aine

r (or

us

er) f

aile

d to

get

mon

itor d

ata

B=

Num

ber o

f cas

es w

hich

mai

ntai

ner

(or

user

) atte

mpt

ed to

get

mon

itor d

ata

reco

rdin

g st

atus

of s

oftw

are

durin

g op

erat

ion

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r D

evel

oper

M

aint

aine

r O

pera

tor

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

5.2

Cha

ngea

bilit

y m

etric

s

Exte

rnal

cha

ngea

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Cha

nge

cycl

e ef

ficie

ncy

Can

the

user

's

prob

lem

be

solv

ed to

hi

s sa

tisfa

ctio

n w

ithin

an

acc

epta

ble

time

scal

e?

Mon

itor i

nter

actio

n be

twee

n us

er a

nd s

uppl

ier.

Rec

ord

the

time

take

n fro

m th

e in

itial

use

r's

requ

est t

o th

e re

solu

tion

of

prob

lem

.

Aver

age

Tim

e : T

av =

Sum

(Tu)

/ N

Tu

= Tr

c - T

sn

Tsn=

Tim

e at

whi

ch u

ser f

inis

hed

to s

end

requ

est f

or m

aint

enan

ce to

sup

plie

r with

pr

oble

m re

port

Trc=

Tim

e at

whi

ch u

ser r

ecei

ved

the

revi

sed

vers

ion

rele

ase

(or s

tatu

s re

port)

N

= N

umbe

r of r

evis

ed v

ersi

ons

0<Ta

v Th

e sh

orte

r is

the

bette

r.,

exce

pt o

f th

e nu

mbe

r of

revi

sed

vers

ions

was

la

rge.

Rat

io

Tu=

Tim

e Tr

c,

Tsn

=

Tim

e N

= C

ount

Ta

v= T

ime

Prob

lem

re

solu

tion

re

port

Mai

nten

ance

re

port

O

pera

tion

repo

rt

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Use

r M

aint

aine

r O

pera

tor

Cha

nge

impl

emen

tatio

n el

apse

tim

e

Can

the

mai

ntai

ner

easi

ly c

hang

e th

e so

ftwar

e to

reso

lve

the

failu

re p

robl

em?

Obs

erve

the

beha

viou

r of

the

user

and

mai

ntai

ner

whi

le tr

ying

to c

hang

e th

e so

ftwar

e.

Oth

erw

ise,

inve

stig

ate

prob

lem

reso

lutio

n re

port

or m

aint

enan

ce re

port.

Aver

age

Tim

e : T

av =

Sum

(Tm

) / N

Tm

=Tou

t - T

in

Tout

= Ti

me

at w

hich

the

caus

es o

f fai

lure

ar

e re

mov

ed w

ith c

hang

ing

the

softw

are

(or

stat

us is

repo

rted

back

to u

ser)

Tin=

Tim

e at

whi

ch th

e ca

uses

of f

ailu

res

are

foun

d ou

t N

= N

umbe

r of r

egis

tere

d an

d re

mov

ed

failu

res

0<Ta

v Th

e sh

orte

r is

the

bette

r, ex

cept

of t

he

num

ber o

f fa

ilure

s w

as

larg

e.

Rat

io

Tm=

Tim

e Ti

n,

Tout

=

Tim

e Ta

v= T

ime

Prob

lem

re

solu

tion

re

port

Mai

nten

ance

re

port

O

pera

tion

repo

rt

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TES

1 It

is re

com

men

ded

to m

easu

re m

axim

al ti

me

of th

e w

orst

cas

e an

d tim

e ba

ndw

idth

to re

pres

ent d

evia

tion.

2

It is

reco

mm

ende

d to

exc

lude

the

failu

res

for w

hich

cau

ses

have

not

yet

bee

n fo

und

whe

n th

e m

easu

rem

ent i

s do

ne. H

owev

er, t

he ra

tio o

f suc

h ob

scur

e fa

ilure

s sh

ould

be

also

mea

sure

d an

d pr

esen

ted

toge

ther

. 3

From

the

indi

vidu

al u

ser’s

poi

nt o

f vie

w, t

ime

is o

f con

cern

, whi

le e

ffort

may

als

o be

of c

once

rn fr

om th

e m

aint

aine

r’s p

oint

of v

iew

. The

refo

re, p

erso

n-ho

urs

may

be

used

inst

ead

of ti

me.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

5.2

(con

tinue

d)

Exte

rnal

cha

ngea

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Mod

ifica

tion

com

plex

ity

Can

the

mai

ntai

ner

easi

ly c

hang

e th

e so

ftwar

e to

reso

lve

prob

lem

?

Obs

erve

beh

avio

ur o

f m

aint

aine

r who

is tr

ying

to

chan

ge th

e so

ftwar

e.

Oth

erw

ise,

inve

stig

ate

prob

lem

reso

lutio

n re

port

or m

aint

enan

ce re

port

and

prod

uct d

escr

iptio

n.

T =

Sum

(A /

B) /

N

A= W

ork

time

spen

t to

chan

ge

B= S

ize

of s

oftw

are

chan

ge

N=

Num

ber o

f cha

nges

0<T

The

shor

ter i

s th

e be

tter o

r th

e re

quire

d nu

mbe

r of

chan

ges

wer

e ex

cess

ive.

Rat

io

A= T

ime

B= S

ize

N=

Cou

nt

T= T

ime

Prob

lem

re

solu

tion

re

port

Mai

nten

ance

re

port

O

pera

tion

repo

rt

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TE

A si

ze o

f sof

twar

e ch

ange

may

be

chan

ged

exec

utab

le s

tate

men

ts o

f pro

gram

cod

e, n

umbe

r of c

hang

ed it

ems

of re

quire

men

ts s

peci

ficat

ion,

or c

hang

ed p

ages

of d

ocum

ent e

tc.

Para

met

eris

ed

mod

ifiab

ility

C

an th

e us

er o

r the

m

aint

aine

r eas

ily

chan

ge p

aram

eter

to

chan

ge s

oftw

are

and

reso

lve

prob

lem

s?

Obs

erve

beh

avio

ur o

f the

us

er o

r the

mai

ntai

ner

whi

le tr

ying

to c

hang

e th

e so

ftwar

e.

Oth

erw

ise,

inve

stig

ate

prob

lem

reso

lutio

n re

port

or m

aint

enan

ce re

port.

X=1-

A /

B

A= N

umbe

r of c

ases

whi

ch m

aint

aine

r fai

ls

to c

hang

e so

ftwar

e by

usi

ng p

aram

eter

B=

Num

ber o

f cas

es w

hich

mai

ntai

ner

atte

mpt

s to

cha

nge

softw

are

by u

sing

pa

ram

eter

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Prob

lem

re

solu

tion

re

port

M

aint

enan

ce

repo

rt

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r U

ser

Softw

are

chan

ge c

ontr

ol

capa

bilit

y

Can

the

user

eas

ily

iden

tify

revi

sed

vers

ions

?

Can

the

mai

ntai

ner

easi

ly c

hang

e th

e so

ftwar

e to

reso

lve

prob

lem

s?

Obs

erve

the

beha

viou

r of

user

or m

aint

aine

r whi

le

tryin

g to

cha

nge

the

softw

are.

O

ther

wis

e, in

vest

igat

e pr

oble

m re

solu

tion

repo

rt or

mai

nten

ance

repo

rt.

X= A

/ B

A= N

umbe

r of c

hang

e lo

g da

ta a

ctua

lly

reco

rded

B=

Num

ber o

f cha

nge

log

data

pla

nned

to

be re

cord

ed e

noug

h to

trac

e so

ftwar

e ch

ange

s

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter o

r the

cl

oser

to0

the

few

er

chan

ges

have

ta

ken

plac

e.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Use

r man

ual

or

spec

ifica

tion

Prob

lem

re

solu

tion

re

port

Mai

nten

ance

re

port

O

pera

tion

repo

rt

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

5.3

Stab

ility

met

rics

Exte

rnal

sta

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Cha

nge

su

cces

s ra

tio

Can

use

r ope

rate

so

ftwar

e sy

stem

w

ithou

t fai

lure

s af

ter

mai

nten

ance

? C

an m

aint

aine

r eas

ily

miti

gate

failu

res

caus

ed b

y m

aint

enan

ce s

ide

effe

cts?

Obs

erve

beh

avio

ur o

f use

r or

mai

ntai

ner w

ho is

op

erat

ing

softw

are

syst

em

afte

r mai

nten

ance

. C

ount

failu

res

whi

ch u

ser

or m

aint

aine

r enc

ount

ered

du

ring

oper

atin

g so

ftwar

e be

fore

and

afte

r m

aint

enan

ce.

Oth

erw

ise,

inve

stig

ate

prob

lem

reso

lutio

n re

port,

op

erat

ion

repo

rt or

m

aint

enan

ce re

port.

X= N

a / T

a Y

= { (

Na

/ Ta)

/ (N

b / T

b) }

Na

= N

umbe

r of c

ases

whi

ch u

ser

enco

unte

rs fa

ilure

s du

ring

oper

atio

n af

ter

softw

are

was

cha

nged

N

b =

Num

ber o

f cas

es w

hich

use

r en

coun

ters

failu

res

durin

g op

erat

ion

befo

re

softw

are

is c

hang

ed

Ta =

Ope

ratio

n tim

e du

ring

spec

ified

ob

serv

atio

n pe

riod

afte

r sof

twar

e is

cha

nged

Tb

= O

pera

tion

time

durin

g sp

ecifi

ed

obse

rvat

ion

perio

d be

fore

sof

twar

e is

ch

ange

d

0<=X

,Y

The

smal

ler

and

clos

er to

0

is th

e be

tter.

Rat

io

Na,

Nb=

C

ount

Ta

,Tb=

Ti

me

X= C

ount

/Ti

me

Y=[(C

ount

/Ti

me)

/ (C

ount

/ Ti

me)

]

Prob

lem

re

solu

tion

re

port

Mai

nten

ance

re

port

O

pera

tion

repo

rt

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TES

1 X

and

Y im

ply

“ fre

quen

cy o

f enc

ount

erin

g fa

ilure

s af

ter c

hang

e” a

nd “f

luct

uate

d fre

quen

cy o

f enc

ount

erin

g fa

ilure

s be

fore

/afte

r cha

nge”

. 2

Use

r may

nee

d sp

ecifi

ed p

erio

d to

det

erm

ine

side

effe

cts

of s

oftw

are

chan

ges,

whe

n th

e re

visi

on-u

p of

sof

twar

e is

intro

duce

d fo

r res

olvi

ng p

robl

ems.

3

It is

reco

mm

end

to c

ompa

re th

is fr

eque

ncy

befo

re a

nd a

fter c

hang

e.

4 If

chan

ged

func

tion

is id

entif

ied,

it is

reco

mm

ende

d to

det

erm

ine

whe

ther

enc

ount

ered

failu

res

are

dete

cted

in th

e ch

ange

d fu

nctio

n its

elf o

r in

the

othe

r one

s. T

he e

xten

t of i

mpa

cts

may

be

rate

d fo

r eac

h fa

ilure

.

Mod

ifica

tion

impa

ct

loca

lisat

ion

(Em

ergi

ng

failu

re a

fter

chan

ge)

Can

use

r ope

rate

so

ftwar

e sy

stem

w

ithou

t fai

lure

s af

ter

mai

nten

ance

? C

an m

aint

aine

r eas

ily

miti

gate

failu

res

caus

ed b

y m

aint

enan

ce s

ide

effe

cts?

Cou

nt fa

ilure

s oc

curre

nces

af

ter c

hang

e, w

hich

are

m

utua

lly c

hain

ing

and

affe

cted

by

chan

ge.

X= A

/ N

A=

Num

ber o

f fai

lure

s em

erge

d af

ter f

ailu

re

is re

solv

ed b

y ch

ange

dur

ing

spec

ified

pe

riod

N=

Num

ber o

f res

olve

d fa

ilure

s

0<=X

Th

e sm

alle

r an

d cl

oser

to

0 is

the

bette

r.

Abso

lute

A= C

ount

N

= C

ount

X=

Cou

nt/

Cou

nt

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TE

X im

plie

s “c

hain

ing

failu

re e

mer

ging

per

reso

lved

failu

re”.

It is

reco

mm

end

to g

ive

prec

ise

mea

sure

by

chec

king

whe

ther

cau

se o

f cur

rent

failu

re is

attr

ibut

ed to

cha

nge

for p

revi

ous

failu

re

reso

lutio

n, a

s po

ssib

le.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

5.4

Test

abili

ty m

etric

s

Exte

rnal

test

abili

ty m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Avai

labi

lity

of

built

-in te

st

func

tion

Can

use

r and

m

aint

aine

r eas

ily

perfo

rm o

pera

tiona

l te

stin

g w

ithou

t ad

ditio

nal t

est f

acilit

y pr

epar

atio

n?

Obs

erve

beh

avio

ur o

f use

r or

mai

ntai

ner w

ho is

te

stin

g so

ftwar

e sy

stem

af

ter m

aint

enan

ce.

X= A

/ B

A= N

umbe

r of c

ases

in w

hich

mai

ntai

ner c

an

use

suita

bly

built

-in te

st fu

nctio

n B=

Num

ber o

f cas

es o

f tes

t opp

ortu

nitie

s

0 <=

X <

=1

The

larg

er

and

the

clos

er

to 1

.0 is

the

bette

r.

Abso

lute

A= C

ount

B=

Cou

nt

X= C

ount

/C

ount

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TE

Exam

ples

of b

uilt-

in te

st fu

nctio

ns in

clud

e si

mul

atio

n fu

nctio

n, p

re-c

heck

func

tion

for r

eady

to u

se, e

tc.

Re-

test

ef

ficie

ncy

Can

use

r and

m

aint

aine

r eas

ily

perfo

rm o

pera

tiona

l te

stin

g an

d de

term

ine

whe

ther

the

softw

are

is re

ady

for o

pera

tion

or n

ot?

Obs

erve

beh

avio

ur o

f use

r or

mai

ntai

ner w

ho is

te

stin

g so

ftwar

e sy

stem

af

ter m

aint

enan

ce.

X= S

um(T

) / N

T=

Tim

e sp

ent t

o te

st to

mak

e su

re w

heth

er

repo

rted

failu

re w

as re

solv

ed o

r not

N

= N

umbe

r of r

esol

ved

failu

res

0<X

The

smal

ler i

s th

e be

tter.

Rat

io

T= T

ime

N=

Cou

ntX=

Tim

e /C

ount

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TE

X im

plie

s “a

vera

ge ti

me

(effo

rt) to

test

afte

r fai

lure

reso

lutio

n”. I

f fai

lure

s ar

e no

t res

olve

d or

fixe

d, e

xclu

de th

em a

nd s

epar

atel

y m

easu

re ra

tio o

f suc

h fa

ilure

s.

Test

re

star

tabi

lity

Can

use

r and

m

aint

aine

r eas

ily

perfo

rm o

pera

tiona

l te

stin

g w

ith c

heck

po

ints

afte

r m

aint

enan

ce?

Obs

erve

beh

avio

ur o

f use

r or

mai

ntai

ner w

ho is

te

stin

g so

ftwar

e sy

stem

af

ter m

aint

enan

ce.

X =

A / B

A

= N

umbe

r of c

ases

in w

hich

mai

ntai

ner

can

paus

e an

d re

star

t exe

cutin

g te

st ru

n at

de

sire

d po

ints

to c

heck

ste

p by

ste

p B=

Num

ber o

f cas

es o

f pau

se o

f exe

cutin

g te

st ru

n

0 <=

X <

=1

The

larg

er

and

the

clos

er

to 1

.0 is

the

bette

r.

Abso

lute

A= C

ount

B=

Cou

nt

X= C

ount

/ C

ount

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

5.5

Mai

ntai

nabi

lity

com

plia

nce

met

rics

Exte

rnal

mai

ntai

nabi

lity

com

plia

nce

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Mai

ntai

nabi

lity

com

plia

nce

How

com

plia

nt is

the

mai

ntai

nabi

lity

of th

e pr

oduc

t to

appl

icab

le

regu

latio

ns,

stan

dard

s an

d co

nven

tions

?

Cou

nt th

e nu

mbe

r of i

tem

s re

quiri

ng c

ompl

ianc

e th

at

have

bee

n m

et a

nd

com

pare

with

the

num

ber

of it

ems

requ

iring

co

mpl

ianc

e in

the

spec

ifica

tion.

X =

1- A

/ B

A=

Num

ber o

f mai

ntai

nabi

lity

com

plia

nce

item

s sp

ecifi

ed th

at h

ave

not b

een

impl

emen

ted

durin

g te

stin

g B=

Tot

al n

umbe

r of m

aint

aina

bilit

y co

mpl

ianc

e ite

ms

spec

ified

0<=

X <=

1 Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X= C

ount

/C

ount

Prod

uct

desc

riptio

n (U

ser m

anua

l or

Sp

ecifi

catio

n)

of c

ompl

ianc

e an

d re

late

d st

anda

rds,

co

nven

tions

or

regu

latio

ns Te

st

spec

ifica

tion

and

repo

rt

5.3

Qua

lific

atio

n te

stin

g 6.

5 Va

lidat

ion

Supp

lier

Use

r

FOO

TNO

TE

It m

ay b

e us

eful

to c

olle

ct s

ever

al m

easu

red

valu

es a

long

tim

e, to

ana

lyse

the

trend

of i

ncre

asin

g sa

tisfie

d co

mpl

ianc

e ite

ms

and

to d

eter

min

e w

heth

er th

ey a

re fu

lly s

atis

fied.

ISO/IEC TR 9126-2:2003(E)


8.6 Portability metrics

An external portability metric should be able to measure such attributes as the behaviour of the operator or system during the porting activity.

8.6.1 Adaptability metrics

An external adaptability metric should be able to measure such attributes as the behaviour of the system or the user who is trying to adapt software to different specified environments. When a user has to apply an adaptation procedure other than previously provided by software for a specific adaptation need, user’s effort required for adapting should be measured.

8.6.2 Installability metrics

An external installability metric should be able to measure such attributes as the behaviour of the system or the user who is trying to install the software in a user specific environment.

8.6.3 Co-existence metrics

An external co-existence metric should be able to measure such attributes as the behaviour of the system or the user who is trying to use the software with other independent software in a common environment sharing common resources.

8.6.4 Replaceability metrics

An external replaceability metric should be able to measure such attributes as the behaviour of the system or the user who is trying to use the software in place of other specified software in the environment of that software.

8.6.5 Portability compliance metrics

An external portability compliance metric should be able to measure such attributes as the number of functions with, or occurrences of compliance problems, where the software product fails to adhere to required standards, conventions or regulations relating to portability.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

6.1

Adap

tabi

lity

met

rics

Exte

rnal

ada

ptab

ility

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Adap

tabi

lity

of

data

str

uctu

res

Can

use

r or

mai

ntai

ner e

asily

ad

apt s

oftw

are

to

data

set

s in

new

en

viro

nmen

t?

Obs

erve

use

r’s o

r m

aint

aine

r’s b

ehav

iour

w

hen

user

is tr

ying

to

adap

t sof

twar

e to

op

erat

ion

envi

ronm

ent.

X =

A /

B

A =

The

num

ber o

f dat

a w

hich

are

ope

rabl

e an

d bu

t are

not

obs

erve

d du

e to

inco

mpl

ete

oper

atio

ns c

ause

d by

ada

ptat

ion

limita

tions

B=

The

num

ber o

f dat

a w

hich

are

exp

ecte

d to

be

oper

able

in th

e en

viro

nmen

t to

whi

ch

the

softw

are

is a

dapt

ed

0<=X

<=1

The

larg

er

and

clos

er to

1.

0 is

the

bette

r.

Abso

lute

A= C

ount

B= C

ount

X= C

ount

/C

ount

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TE

Thes

e da

ta m

ainl

y in

clud

e ty

pes

of d

ata

such

as

data

file

s, d

ata

tupl

es o

r dat

abas

es to

be

adap

ted

to d

iffer

ent d

ata

volu

mes

, dat

a ite

ms

or d

ata

stru

ctur

es. A

and

B o

f the

form

ula

are

nece

ssar

y to

cou

nt th

e sa

me

type

s of

dat

a. S

uch

an a

dapt

atio

n m

ay b

e re

quire

d w

hen,

for e

xam

ple,

the

busi

ness

sco

pe is

ext

ende

d.

Har

dwar

e en

viro

nmen

tal

adap

tabi

lity

(ada

ptab

ility

to

hard

war

e de

vice

s an

d ne

twor

k fa

cilit

ies)

Can

use

r or

mai

ntai

ner e

asily

ad

apt s

oftw

are

to

envi

ronm

ent?

Is

sof

twar

e sy

stem

ca

pabl

e en

ough

to

adap

t its

elf t

o op

erat

ion

envi

ronm

ent?

Obs

erve

use

r’s o

r m

aint

aine

r’s b

ehav

iour

w

hen

user

is tr

ying

to

adap

t sof

twar

e to

op

erat

ion

envi

ronm

ent.

X= 1

- A

/ B

A= N

umbe

r of o

pera

tiona

l fun

ctio

ns o

f whi

ch

task

s w

ere

not c

ompl

eted

or n

ot e

noug

h re

sulte

d to

mee

t ade

quat

e le

vels

dur

ing

com

bine

d op

erat

ing

test

ing

with

en

viro

nmen

tal

hard

war

e B=

Tot

al n

umbe

r of f

unct

ions

whi

ch w

ere

test

ed

0<=X

<=1

The

larg

er is

th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TE

It is

reco

mm

ende

d to

con

duct

ove

rload

com

bina

tion

test

ing

with

har

dwar

e en

viro

nmen

tal c

onfig

urat

ions

whi

ch m

ay p

ossi

bly

be o

pera

tiona

lly c

ombi

ned

in a

var

iety

of u

ser o

pera

tiona

l en

viro

nmen

ts.

Org

anis

atio

nal

envi

ronm

ent

adap

tabi

lity

(Org

anis

atio

n ad

apta

bilit

y to

in

fras

truc

ture

of

org

anis

atio

n)

Can

use

r or

mai

ntai

ner e

asily

ad

apt s

oftw

are

to

envi

ronm

ent?

Is

sof

twar

e sy

stem

ca

pabl

e en

ough

to

adap

t its

elf t

o th

e op

erat

iona

l en

viro

nmen

t?

Obs

erve

use

r’s o

r m

aint

aine

r’s b

ehav

iour

w

hen

user

is tr

ying

to

adap

t sof

twar

e to

op

erat

ion

envi

ronm

ent.

X= 1

- A

/ B

A=

Num

ber o

f ope

rate

d fu

nctio

ns in

whi

ch

the

task

s w

ere

not c

ompl

eted

or n

ot e

noug

h re

sulte

d to

mee

t ade

quat

e le

vels

dur

ing

oper

atio

nal t

estin

g w

ith u

ser’s

bus

ines

s en

viro

nmen

t B=

Tot

al n

umbe

r of f

unct

ions

whi

ch w

ere

test

ed

0<=X

<=1

The

larg

er is

th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TES

1 It

is re

com

men

ded

to c

ondu

ct te

stin

g w

hich

take

s ac

coun

t of t

he v

arie

ties

of c

ombi

natio

ns o

f inf

rast

ruct

ure

com

pone

nts

of p

ossi

ble

user

’s b

usin

ess

envi

ronm

ents

. 2

“Org

anis

atio

nal e

nviro

nmen

t ada

ptab

ility”

is c

once

rned

with

the

envi

ronm

ent o

f the

bus

ines

s op

erat

ion

of th

e us

er’s

org

anis

atio

n. “S

yste

m s

oftw

are

envi

ronm

enta

l ada

ptab

ility”

is

conc

erne

d w

ith th

e en

viro

nmen

t of t

he te

chni

cal o

pera

tion

of s

yste

ms.

The

refo

re,

ther

e is

a c

lear

dis

tinct

ion.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

6.1

(con

tinue

d)

Exte

rnal

ada

ptab

ility

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Port

ing

user

fr

iend

lines

s C

an u

ser o

r m

aint

aine

r eas

ily

adap

t sof

twar

e to

en

viro

nmen

t?

Obs

erve

use

r’s o

r m

aint

aine

r’s b

ehav

iour

w

hen

user

is tr

ying

to

adap

t sof

twar

e to

op

erat

iona

l env

ironm

ent.

T= S

um o

f use

r ope

ratin

g tim

e sp

ent t

o co

mpl

ete

adap

tatio

n of

the

softw

are

to u

ser’s

en

viro

nmen

t, w

hen

user

atte

mpt

to in

stal

l or

chan

ge s

etup

0<T

The

shor

ter i

s th

e be

tter.

Rat

io

T=Ti

me

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TE

T im

plie

s “u

ser e

ffort

requ

ired

to a

dapt

to u

ser’s

env

ironm

ent”.

Per

son-

hour

may

be

used

inst

ead

of ti

me.

Syst

em

softw

are

envi

ronm

enta

l ad

apta

bilit

y (a

dapt

abilit

y to

O

S, n

etw

ork

softw

are

and

co-

oper

ated

ap

plic

atio

n so

ftwar

e)

Can

use

r or

mai

ntai

ner e

asily

ad

apt s

oftw

are

to

envi

ronm

ent?

Is

sof

twar

e sy

stem

ca

pabl

e en

ough

to

adap

t its

elf t

o op

erat

ion

envi

ronm

ent?

Obs

erve

use

r’s o

r m

aint

aine

r’s b

ehav

iour

w

hen

user

is tr

ying

to

adap

t sof

twar

e to

op

erat

ion

envi

ronm

ent.

X= 1

- A

/ B

A=

Num

ber o

f ope

ratio

nal f

unct

ions

of w

hich

ta

sks

wer

e no

t com

plet

ed o

r wer

e no

t en

ough

resu

lted

to m

eet a

dequ

ate

leve

l du

ring

com

bine

d op

erat

ing

test

ing

with

op

erat

ing

syst

em s

oftw

are

or c

oncu

rrent

ap

plic

atio

n so

ftwar

e

B= T

otal

num

ber o

f fun

ctio

ns w

hich

wer

e te

sted

0<=X

<=1

The

larg

er is

th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TES

1 It

is re

com

men

ded

to c

ondu

ct o

verlo

ad c

ombi

natio

n te

stin

g w

ith o

pera

ting

syst

em s

oftw

ares

or c

oncu

rrent

app

licat

ion

softw

ares

whi

ch a

re p

ossi

bly

com

bine

d op

erat

ed in

a v

arie

ty o

f use

r op

erat

iona

l env

ironm

ents

. 2

“Org

anis

atio

nal e

nviro

nmen

t ada

ptab

ility”

is c

once

rned

with

the

envi

ronm

ent f

or b

usin

ess

oper

atio

n of

use

r’s o

rgan

isat

ion.

“Sys

tem

sof

twar

e en

viro

nmen

tal a

dapt

abilit

y” is

con

cern

ed w

ith

the

envi

ronm

ent f

or te

chni

cal o

pera

tions

on

syst

ems.

The

refo

re, t

here

is a

cle

ar d

istin

ctio

n.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

6.2

Inst

alla

bilit

y m

etric

s

Exte

rnal

inst

alla

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Ease

of

inst

alla

tion

Can

use

r or

mai

ntai

ner e

asily

in

stal

l sof

twar

e to

op

erat

ion

envi

ronm

ent?

Obs

erve

use

r’s o

r m

aint

aine

r’s b

ehav

iour

w

hen

user

is tr

ying

to

inst

all s

oftw

are

to

oper

atio

n en

viro

nmen

t.

X =

A / B

A

= N

umbe

r of c

ases

whi

ch a

use

r su

ccee

ded

to in

cha

ngin

g th

e in

stal

l op

erat

ion

for h

is/h

er c

onve

nien

ce

B =

Tota

l num

ber o

f cas

es w

hich

a u

ser

atte

mpt

ed to

cha

nge

the

inst

all o

pera

tion

for

his/

her c

onve

nien

ce

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B=

Cou

nt

X= C

ount

/C

ount

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TES

1 Th

is m

etric

is s

ugge

sted

as

expe

rimen

tal u

se.

2 W

hen

time

basi

s m

etric

is re

quire

d, s

pent

tim

e fo

r ins

talla

tion

may

be

mea

sura

ble.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

6.2

(con

tinue

d)

Exte

rnal

inst

alla

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Ease

of S

etup

R

etry

C

an u

ser o

r m

aint

aine

r eas

ily re

-try

set

-up

inst

alla

tion

of s

oftw

are?

Obs

erve

use

r’s o

r m

aint

aine

r’s b

ehav

iour

w

hen

user

is tr

ying

to re

-try

set

-up

inst

alla

tion

of

softw

are.

X =

1 -

A / B

A

= N

umbe

r of c

ases

in w

hich

use

r fai

ls in

re

-tryi

ng s

et-u

p du

ring

set-u

p op

erat

ion

B =

Tota

l num

ber o

f cas

es in

whi

ch u

ser

atte

mpt

to re

-try

setu

p du

ring

set-u

p op

erat

ion

0<=X

<= 1

Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B=

Cou

nt

X= C

ount

/C

ount

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TE

This

met

ric is

sug

gest

ed a

s ex

perim

enta

l use

.

FOO

TNO

TES

The

follo

win

g co

mpl

emen

tary

met

rics

may

be

used

. 1

Effo

rtles

s in

stal

latio

n U

ser’s

man

ual a

ctio

ns fo

r ins

talla

tion

X =

A A=

The

num

ber o

f use

r’s m

anua

l act

ions

nee

ded

for i

nsta

llatio

n 0<

X he

sm

alle

r is

the

bette

r. 2

Inst

alla

tion

ease

In

stal

latio

n su

ppor

ting

leve

l X =

A A

is ra

ted

with

, for

exa

mpl

e:

- Onl

y ex

ecut

ing

inst

alla

tion

prog

ram

whe

re n

othi

ng m

ore

is n

eede

d (e

xcel

lent

); - I

nstru

ctio

nal g

uide

for i

nsta

llatio

n (g

ood)

; - S

ourc

e co

de o

f pro

gram

nee

ds m

odifi

catio

n fo

r ins

talla

tion

(poo

r).

X= D

irect

Inte

rpre

tatio

n of

mea

sure

d va

lue

3 O

pera

tiona

l ins

talla

tion

effo

rt re

duct

ion

Use

r Ins

tall

Ope

ratio

n Pr

oced

ure

Red

uctio

n R

atio

X =

1-

A / B

A

= N

umbe

r of i

nsta

ll op

erat

ion

proc

edur

es w

hich

a u

ser h

ad to

do

afte

r pro

cedu

re re

duct

ion

B =

Num

ber o

f ins

tall

oper

atio

n pr

oced

ures

nor

mal

ly

0<=

X <=

1 Th

e cl

oser

to 1

.0 is

the

bette

r. 4

Ease

of u

ser’s

man

ual i

nsta

ll op

erat

ion

Easi

ness

leve

l of u

ser’s

man

ual i

nsta

ll op

erat

ion

X =

Scor

e of

eas

ines

s le

vel o

f use

r’s m

anua

l ope

ratio

n Ex

ampl

es o

f eas

ines

s le

vel a

re fo

llow

ing:

[v

ery

easy

] req

uirin

g on

ly u

ser’s

sta

rting

of i

nsta

ll or

set

-up

func

tions

and

then

obs

ervi

ng in

stal

latio

n;

[eas

y] re

quiri

ng o

nly

user

’s a

nsw

erin

g of

que

stio

n fro

m in

stal

l or s

et-u

p fu

nctio

ns;

[not

eas

y] re

quiri

ng u

ser’s

look

ing

up p

aram

eter

s fro

m ta

bles

or f

illing

-in b

oxes

; [c

ompl

icat

ed] r

equi

ring

user

’s s

earc

hing

par

amet

er fi

les,

look

ing

up p

aram

eter

s fro

m fi

les

to b

e ch

ange

d an

d w

ritin

g th

em.

X= D

irect

Inte

rpre

tatio

n of

mea

sure

d va

lue

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

6.3

Co-

exis

tenc

e m

etric

s

Exte

rnal

co-

exis

tenc

e m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Avai

labl

e co

-ex

iste

nce

How

ofte

n us

er

enco

unte

rs a

ny

cons

train

ts o

r un

expe

cted

failu

res

whe

n op

erat

ing

conc

urre

ntly

with

ot

her s

oftw

are?

Use

eva

luat

ed s

oftw

are

conc

urre

ntly

with

oth

er

softw

are

whi

ch u

ser o

ften

uses

.

X =

A / T

A

= N

umbe

r of a

ny c

onst

rain

ts o

r un

expe

cted

failu

res

whi

ch u

ser e

ncou

nter

du

ring

oper

atin

g co

ncur

rent

ly w

ith o

ther

so

ftwar

e T

= Ti

me

dura

tion

of c

oncu

rrent

ly o

pera

ting

othe

r sof

twar

e

0<=X

Th

e cl

oser

to

0 is

the

bette

r.

Rat

io

A= C

ount

T=

Tim

e X=

Cou

nt/

Tim

e

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

SQA

Ope

rato

r

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

6.4

Rep

lace

abili

ty m

etric

s

Exte

rnal

repl

acea

bilit

y m

etric

s M

etric

nam

e Pu

rpos

e of

the

met

rics

Met

hod

of a

pplic

atio

n M

easu

rem

ent,

form

ula

and

da

ta e

lem

ent c

ompu

tatio

ns

Inte

rpre

tatio

n of

mea

sure

d va

lue

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Con

tinue

d us

e of

dat

a C

an u

ser o

r m

aint

aine

r eas

ily

cont

inue

to u

se th

e sa

me

data

afte

r re

plac

ing

this

so

ftwar

e to

pre

viou

s on

e?

Is s

oftw

are

syst

em

mig

ratio

n go

ing

on

succ

essf

ully

?

Obs

erve

use

r’s o

r m

aint

aine

r’s b

ehav

iour

w

hen

user

is re

plac

ing

softw

are

to p

revi

ous

one.

X =

A / B

A

= nu

mbe

r of d

ata

whi

ch a

re u

sed

in o

ther

so

ftwar

e to

be

repl

aced

and

are

con

firm

ed

that

they

are

abl

e to

be

cont

inuo

usly

use

d B

= nu

mbe

r of d

ata

whi

ch a

re u

sed

in o

ther

so

ftwar

e to

be

repl

aced

and

pla

nned

to b

e co

ntin

uous

ly re

usab

le

0<=

X <=

1 Th

e la

rger

is

the

bette

r.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TE

This

met

ric c

an b

e ap

plie

d to

bot

h ca

ses

of re

plac

ing

an e

ntire

ly d

iffer

ent s

oftw

are

and

a di

ffere

nt v

ersi

on o

f the

sam

e so

ftwar

e se

ries

to p

revi

ous

one.

Fu

nctio

n in

clus

iven

ess

Can

use

r or

mai

ntai

ner e

asily

co

ntin

ue to

use

si

mila

r fun

ctio

ns a

fter

repl

acin

g th

is

softw

are

to p

revi

ous

one?

Is

sof

twar

e sy

stem

m

igra

tion

goin

g on

su

cces

sful

ly?

Obs

erve

use

r’s o

r m

aint

aine

r’s b

ehav

iour

w

hen

user

is re

plac

ing

softw

are

to p

revi

ous

one.

X =

A / B

A

= nu

mbe

r of f

unct

ions

whi

ch p

rodu

ce

sim

ilar r

esul

ts a

s pr

evio

usly

pro

duce

d an

d w

here

cha

nges

hav

e no

t be

en re

quire

d B

= nu

mbe

r of t

este

d fu

nctio

ns w

hich

are

si

mila

r to

func

tions

pro

vide

d by

ano

ther

so

ftwar

e to

be

repl

aced

0<=

X <=

1 Th

e la

rger

is

the

bette

r.

Abso

lute

A= C

ount

B= C

ount

X=

Cou

nt/

Cou

nt

Prob

lem

re

solu

tion

re

port

Ope

ratio

n re

port

5.3

Qua

lific

atio

n te

stin

g 5.

4 O

pera

tion

5.5

Mai

nte-

nanc

e

Dev

elop

er

Mai

ntai

ner

Ope

rato

r

FOO

TNO

TE

This

met

ric c

an b

e ap

plie

d to

bot

h ca

ses

of re

plac

ing

an e

ntire

ly d

iffer

ent s

oftw

are

and

a di

ffere

nt v

ersi

on o

f the

sam

e so

ftwar

e se

ries

to p

revi

ous

one.

Use

r sup

port

func

tiona

l co

nsis

tenc

y

How

con

sist

ent a

re

the

new

com

pone

nts

with

exi

stin

g us

er

inte

rface

?

Obs

erve

the

beha

viou

r of

the

user

and

ask

the

opin

ion.

X= 1

- A1

/ A

2

A= N

umbe

r of n

ew fu

nctio

ns w

hich

use

r fo

und

unac

cept

ably

inco

nsis

tent

with

the

user

’s e

xpec

tatio

n

B= N

umbe

r of n

ew fu

nctio

ns

0<=X

La

rger

is

bette

r.

Abso

lute

A1

= C

ount

A2

= C

ount

X=

C

ount

/ C

ount

Test

repo

rt O

pera

tion

repo

rt

5.3

Inte

grat

ion

5.3

Q

ualif

icat

ion

test

ing

5.4

Ope

ratio

n 6.

3 Q

ualit

y As

sura

nce

Use

r U

ser

inte

rface

de

sign

er

Mai

ntai

ner

Dev

elop

er

Test

er

SQA

FOO

TNO

TES

1 Th

e ca

se th

at a

diff

eren

t sof

twar

e is

intro

duce

d to

repl

ace

for a

pre

viou

s so

ftwar

e, a

new

diff

eren

t sof

twar

e ca

n be

iden

tifie

d as

a c

urre

nt v

ersi

on.

2 In

cas

e th

at th

e pa

ttern

of i

nter

actio

n is

cha

nged

to im

prov

e us

er in

terfa

ce in

a n

ew v

ersi

on,,

it is

sug

gest

ed to

obs

erve

use

r’s b

ehav

iour

and

to c

ount

the

num

ber o

f cas

es w

hich

the

user

fa

ils to

acc

ess

func

tions

cau

sed

by u

nacc

epta

ble

conf

orm

ity a

gain

st u

ser’s

exp

ecta

tion

deriv

ed fr

om p

revi

ous

vers

ion.

ISO/IEC TR 9126-2:2003(E)


Tabl

e 8.

6.5

Port

abili

ty c

ompl

ianc

e m

etric

s

Exte

rnal

por

tabi

lity

com

plia

nce

met

rics

Met

ric n

ame

Purp

ose

of th

e m

etric

s M

etho

d of

app

licat

ion

Mea

sure

men

t, fo

rmul

a an

d

data

ele

men

t com

puta

tions

In

terp

reta

tion

of m

easu

red

valu

e

Met

ric

scal

e ty

pe

Mea

sure

ty

pe

Inpu

t to

mea

sure

-m

ent

ISO

/IEC

12

207

SLC

P R

efer

ence

Targ

et

audi

ence

Port

abili

ty

com

plia

nce

How

com

plia

nt is

the

porta

bilit

y of

the

prod

uct t

o ap

plic

able

re

gula

tions

, st

anda

rds

and

conv

entio

ns ?

Cou

nt th

e nu

mbe

r of i

tem

s re

quiri

ng c

ompl

ianc

e th

at

have

bee

n m

et a

nd

com

pare

with

the

num

ber

of it

ems

requ

iring

co

mpl

ianc

e in

the

spec

ifica

tion.

X =

1- A

/ B

A=

Num

ber o

f por

tabi

lity

com

plia

nce

item

s sp

ecifi

ed th

at h

ave

not b

een

impl

emen

ted

durin

g te

stin

g B=

Tot

al n

umbe

r of p

orta

bilit

y co

mpl

ianc

e ite

ms

spec

ified

0<=

X <=

1 Th

e cl

oser

to

1.0

is th

e be

tter.

Abso

lute

A= C

ount

B= C

ount

X= C

ount

/C

ount

Prod

uct

desc

riptio

n (U

ser m

anua

l or

Sp

ecifi

catio

n)

of c

ompl

ianc

e an

d re

late

d st

anda

rds,

co

nven

tions

or

regu

latio

nsTe

st

spec

ifica

tion

and

repo

rt

5.3

Qua

lific

atio

n te

stin

g 6.

5 Va

lidat

ion

Supp

lier

Use

r

FOO

TNO

TE

It m

ay b

e us

eful

to c

olle

ct s

ever

al m

easu

red

valu

es a

long

tim

e, a

naly

se th

e tre

nd o

f inc

reas

ing

satis

fied

com

plia

nce

item

s, a

nd d

eter

min

e w

heth

er th

ey a

re fu

lly s

atis

fied.

ISO/IEC TR 9126-2:2003(E)


Annex A (informative)

Considerations When Using Metrics

A.1 Interpretation of measures

A.1.1 Potential differences between test and operational contexts of use

When planning the use of metrics or interpreting measures it is important to have a clear understanding of the intended context of use of the software, and any potential differences between the test and operational contexts of use. For example, the “time required to learn operation” measure is often different between skilled operators and unskilled operators in similar software systems. Examples of potential differences are given below.

a) Differences between testing environment and the operational environment

Are there any significant differences between the testing environment and the operational execution in user environment?

The following are examples:

• testing with higher / comparable / lower performance of CPU of operational computer;

• testing with higher / comparable / lower performance of operational network and communication;

• testing with higher / comparable / lower performance of operational operating system;

• testing with higher / comparable / lower performance of operational user interface.

b) Differences between testing execution and actual operational execution

Are there any significant differences between the testing execution and operational execution in user environment?


• coverage of functionality in test environment;

• test case sampling ratio;

• automated testing of real time transactions;

• stress loads;

• 24 hour 7 days a week (non stop) operation;

• appropriateness of data for testing of exceptions and errors;

• periodical processing;

• resource utilisation;

• levels of interruption;

• production pressures;

• distractions.

ISO/IEC TR 9126-2:2003(E)


c) User profile under observation

Are there any significant differences between test user profiles and operational user profiles?


• mix of type of users;

• user skill levels;

• specialist users or average users;

• limited user group or public users.

A.1.2 Issues affecting validity of results

The following issues may affect the validity of the data that is collected.

(a) procedures for collecting evaluation results:

• automatically with tools or facilities / manually collected / questionnaires or interviews;

(b) source of evaluation results

• developers' self reports / reviewers’ report / evaluator’s report;

(c) results data validation

• developers' self check / inspection by independent evaluators.

A.1.3 Balance of measurement resources

Is the balance of measures used at each stage appropriate for the evaluation purpose?

It is important to balance the effort used to apply an appropriate range of metrics for internal, external and quality in use measures.

A.1.4 Correctness of specification

Are there significant differences between the software specification and the real operational needs?

Measurements taken during software product evaluation at different stages are compared against product specifications. Therefore, it is very important to ensure by verification and validation that the product specifications used for evaluation reflect the actual and real needs in operation.

A.2 Validation of Metrics

A.2.1 Desirable Properties for Metrics

To obtain valid results from a quality evaluation, the metrics should have the properties stated below. If a metric does not have these properties, the metric description should explain the associated constraint on its validity and, as far as possible, how that situation can be handled.

a) Reliability (of metric): Reliability is associated with random error. A metric is free of random error if random variations do not affect the results of the metric.

ISO/IEC TR 9126-2:2003(E)


b) Repeatability (of metric): repeated use of the metric for the same product using the same evaluation specification (including the same environment), type of users, and environment by the same evaluators, should produce the same results within appropriate tolerances. The appropriate tolerances should include such things as fatigue, and learning effect.

c) Reproducibility (of metric): use of the metric for the same product using the same evaluation specification (including the same environment), type of users, and environment by different evaluators, should produce the same results within appropriate tolerances.

NOTE 1 It is recommended to use statistical analysis to measure the variability of the results.

d) Availability (of metric): The metric should clearly indicate the conditions (e.g. presence of specific attributes) which constrain its usage.

e) Indicativeness (of metric): Capability of the metric to identify parts or items of the software which should be improved, given the measured results compared to the expected ones.

NOTE 2 The selected or proposed metric should provide documented evidence of the availability of the metric for use, unlike those requiring project inspection only.

f) Correctness (of measure): The metric should have the following properties: 1) Objectivity (of measure): the metric results and its data input should be factual: i.e., not influenced by

the feelings or the opinions of the evaluator, test users, etc. (except for satisfaction or attractiveness metrics where user feelings and opinions are being measured).

2) Impartiality (of measure): the measurement should not be biased towards any particular result.

3) Sufficient precision (of measure): Precision is determined by the design of the metric, and particularly by the choice of the material definition used as the basis for the metric. The metric user will describe the precision and the sensitivity of the metric.

g) Meaningfulness (of measure): the measurement should produce meaningful results about the software behaviour or quality characteristics.

The metric should also be cost effective: that is, more costly metrics should provide higher value results.

A.2.2 Demonstrating the Validity of Metrics

The users of metrics should identify the methods for demonstrating the validity of metrics, as shown below:

(a) Correlation

The variation in the quality characteristics values (the measures of principal metrics in operational use) explained by the variation in the metric values, is given by the square of the linear coefficient.

An evaluator can predict quality characteristics without measuring them directly by using correlated metrics.

(b) Tracking

If a metric M is directly related to a quality characteristics value Q (the measures of principal metrics in operational use), for a given product or process, then a change value Q(T1) to Q(T2), would be accompanied by a change metric value from M(T1) to M(T2), in the same direction (for example, if Q increases, M increases).

An evaluator can detect movement of quality characteristics along a time period without measuring directly by using those metrics which have tracking ability.

ISO/IEC TR 9126-2:2003(E)


(c) Consistency

If quality characteristics values (the measures of principal metrics in operational use) Q1, Q2,..., Qn, corresponding to products or processes 1, 2,..., n, have the relationship Q1 > Q2 > ...> Qn, then the corresponding metric values would have the relationship M1 > M2 > ...> Mn.

An evaluator can notice exceptional and error prone components of software by using those metrics which have consistency ability.

(d) Predictability

If a metric is used at time T1 to predict a quality characteristic value Q (the measures of principal metrics in operational use) at T2, prediction error, which is {(predicted Q(T2) - actual Q(T2) ) / actual Q(T2)}, would be within allowed prediction error range.

An evaluator can predict the movement of quality characteristics in the future by using these metrics, which measure predictability.

(e) Discriminative

A metric would be able to discriminate between high and low quality software.

An evaluator can categorize software components and rate quality characteristics values by using those metrics which have discriminative ability.

A.3 Use of metrics for estimation (judgement) and prediction (forecast)

Estimation and prediction of the quality characteristics of the software product at the earlier stages are two of the most rewarding uses of metrics.

A.3.1 Quality characteristics prediction by current data

(a) Prediction by regression analysis When predicting the future value (measure) of the same characteristic (attribute) by using the current value (data) of it (the attribute), a regression analysis is useful based on a set of data that is observed in a sufficient period of time.

For example, the value of MTBF (Mean Time Between Failures) that is obtained during the testing stage (activities) can be used to estimate the MTBF in operation stage.

(b) Prediction by correlation analysis When predicting the future value (measure) of a characteristic (attribute) by using the current measured values of a different attribute, a correlation analysis is useful using a validated function which shows the correlation.

For example, the complexity of modules during coding stage may be used to predict time or effort required for program modification and test during maintenance process.

A.3.2 Current quality characteristics estimation on current facts

(a) Estimation by correlation analysis When estimating the current values of an attribute which are directly unmeasurable, or if there is any other measure that has strong correlation with the target measure, a correlation analysis is useful.

ISO/IEC TR 9126-2:2003(E)


For example, because the number of remaining faults in a software product is not measurable, it may be estimated by using the number and trend of detected faults.

Those metrics which are used for predicting the attributes that are not directly measurable should be estimated as explained below:

• Using models for predicting the attribute;

• Using formula for predicting the attribute;

• Using basis of experience for predicting the attribute;

• Using justification for predicting the attribute.

Those metrics which are used for predicting the attributes that are not directly measurable may be validated as explained below:

• Identify measures of attributes which are to be predicted;

• Identify the metrics which will be used for prediction;

• Perform a statistical analysis based validation;

• Document the results;

• Repeat the above periodically.

A.4 Detecting deviations and anomalies in quality problem prone components

The following quality control tools may be used to analyse deviations and anomalies in software product components:

(a) process charts (functional modules of software)

(b) Pareto analysis and diagrams

(c) histograms and scatter diagrams

(d) run diagrams, correlation diagrams and stratification

(e) Ishikawa (Fishbone) diagrams

(f) statistical process control (functional modules of software)

(g) check sheets

The above tools can be used to identify quality issues from data obtained by applying the metrics.

A.5 Displaying measurement results

(a) Displaying quality characteristics evaluation results The following graphical presentations are useful to display quality evaluation results for each of the quality characteristic and subcharacteristic.

Radar chart; Bar chart numbered histogram, multi-variates chart, Importance Performance Matrix, etc.

(b) Displaying measures There are useful graphical presentations such as Pareto chart, trend charts, histograms, correlation charts, etc.

ISO/IEC TR 9126-2:2003(E)


Annex B (informative)

Use of Quality in Use, External & Internal Metrics (Framework Example)

B.1 Introduction

This framework example is a high level description of how the ISO/IEC 9126 Quality model and related metrics may be used during the software development and implementation to achieve a quality product that meets user’s specified requirements. The concepts shown in this example may be implemented in various forms of customization to suit the individual, organization or project. The example uses the key life cycle processes from ISO/IEC 12207 as a reference to the traditional software development life cycle and quality evaluation process steps from ISO/IEC 14598-3 as a reference to the traditional Software Product Quality evaluation process. The concepts can be mapped onto other models of software life cycles if the user so wishes as long as the underlying concepts are understood.

B.2 Overview of Development and Quality Process

Table B.1 depicts an example model that links the Software Development life cycle process activities (activity 1 to activity 8) to their key deliverables and the relevant reference models for measuring quality of the deliverables (i.e., Quality in Use, External Quality, or Internal Quality).

Row 1 describes the software development life cycle process activities. (This may be customized to suit individual needs). Row 2 describes whether an actual measure or a prediction is possible for the category of measures (i.e., Quality in Use, External Quality, or Internal Quality). Row 3 describes the key deliverable that may be measured for Quality and Row 4 describes the metrics that may be applied on each deliverable at each process activity.

Table B.1 Quality Measurement Model Activity 1 Activity 2 Activity 3 Activity 4 Activity 5 Activity 6 Activity 7 Activity 8 Phase Requirement

analysis (Software and systems)

Architectural design (Software and systems)

Software detailed design

Software coding and testing

Software integration and software qualification testing

System integration and system qualification testing

Software installation

Software acceptance support

9126 series model reference

Required user quality, Required internal quality, Required external quality

Predicted quality in use, Predicted external quality, Measured internal quality

Predicted quality in use,Predicted external quality, Measured internal quality

Predicted quality in use,Measured external quality, Predicted external quality, Measured internal quality

Predicted quality in use,Measured external quality, Predicted external quality, Measured internal quality

Predicted quality in use, Measured external quality, Measured internal quality

Predicted quality in use,Measured external quality, Measured internal quality

Measured quality in use,Measured external quality, Measured internal quality

Key deliverables of activity

User quality requirements (specified), External quality requirements (specified), Internal quality requirements (specified)

Architecture design of Software / system

Software detailed design

Software code, Test results

Software product, Test results

Integrated system, Test results

Installed system

Delivered software product

Metrics used to measure

Internal metrics(External metrics may be applied to validate specifications)

Internal metrics Internal metrics

Internal metrics External metrics




Quality in use metrics Internal metrics External metrics

ISO/IEC TR 9126-2:2003(E)


B.3 Quality Approach Steps

B.3.1 General

Evaluation of the Quality during the development cycle is divided into the following steps. Step 1 has to be completed during the Requirement Analysis activity. Steps 2 to 5 have to be repeated during each process activity defined above.

B.3.2 Step #1 Quality requirements identification

For each of the Quality characteristics and subcharacteristics defined in the Quality model determine the User Needs weights using the two examples in Table B.2 for each category of the measurement. (Quality in Use, External and Internal Quality). Assigning relative weights will allow the evaluators to focus their efforts on the most important subcharacteristics.

Table B.2 User Needs Characteristics & Weights (a)

Quality in Use

CHARACTERISTIC WEIGHT (High/Medium/Low)

Effectiveness H Productivity H Safety L

Satisfaction M (b)

External & Internal Quality

CHARACTERISTIC SUBCHARACTERISTIC WEIGHT (High/Medium/Low)

Suitability H Accuracy H Interoperability L Security L

Functionality

Compliance M Maturity (hardware/software/data) L

Fault tolerance L Recoverability (data, process, technology) H

Reliability

Compliance H Understandability M Learnability L Operability H Attractiveness M

Usability

Compliance H Time behaviour H Resource utilization H

Efficiency

Compliance H Analyzability H Changeability M Stability L Testability M

Maintainability

Compliance H

ISO/IEC TR 9126-2:2003(E)


Table B.2 b) (continued)

Adaptability H Installability L Co-existence H Replaceability M

Portability

Compliance H

NOTE Weights can be expressed in the High/Medium/Low manner or using the ordinal type scale in the range 1-9 (e.g.: 1-3 = low, 4-6 = medium, 7-9 = high).

B.3.3 Step #2 Specification of the evaluation

This step is applied during every development process activity.

For each of the Quality subcharacteristics defined in the Quality model identify the metrics to be applied and the required levels to achieve the User Needs set in Step 1 and record as shown in the example in Table B.3.

Basic input and directions for the content formulation can be obtained from the example in Table B.1 that explains what can be measured at this stage of the development cycle.

NOTE It is possible, that some of the rows of the tables would be empty during the specific activities of the development cycle, because it would not be possible to measure all of the subcharacteristics early in the development process.

Table B.3 Quality measurement tables (a)

Quality in Use Measurement Category

CHARACTERISTIC METRICS REQUIRED LEVEL

ASSESSMENT ACTUAL RESULT

Effectiveness Productivity Safety

Satisfaction

(b)

External Quality Measurement Category

CHARACTERISTIC SUBCHARACTERISTIC METRICS REQUIRED LEVEL


Suitability Accuracy Interoperability Security

Functionality

Compliance Maturity (hardware/software/data)

Fault tolerance Recoverability (data, process, technology)

Reliability

Compliance

ISO/IEC TR 9126-2:2003(E)


Understandability Learnability Operability Attractiveness

Usability

Compliance Time behaviour Resource utilisation

Efficiency

Compliance Analyzability Changeability Stability Testability

Maintainability

Compliance Adaptability Instability Co-existence Replaceability

Portability

Compliance (c)

Internal Quality Measurement Category

CHARACTERISTIC SUBCHARACTERISTIC METRICS REQUIRED LEVEL


Suitability Accuracy Interoperability Security

Functionality

Compliance Maturity (hardware/software/data)

Fault tolerance Recoverability (data, process, technology)

Reliability

Compliance Understandability Learnability Operability Attractiveness

Usability

Compliance Time behaviour Resource utilisation

Efficiency

Compliance Analyzability Changeability Stability Testability

Maintainability

Compliance Adaptability Instability Co-existence Replaceability

Portability

Compliance

ISO/IEC TR 9126-2:2003(E)


B.3.4 Step #3 Design of the evaluation


Develop a measurement plan (similar to example in Table B.4) containing the deliverables that are used as input to the measurement process and the metrics to be applied.

Table B.4 Measurement plan

SUBCHARACTERISTIC DELIVERABLES TO BE EVALUATED

INTERNAL METRICS TO BE

APPLIED

EXTERNAL METRICS TO BE

APPLIED

QUALITY IN USE METRICS TO BE

APPLIED 1. Suitability 1.

2. 3.

1. 2. 3.

1. 2. 3.

(Not Applicable)

2. Satisfaction 1. 2. 3.

(Not Applicable) (Not Applicable) 1. 2. 3.

3. 4. 5. 6.

B.3.5 Step #4 Execution of the evaluation


Execute the evaluation plan and complete the column as shown in the examples in Table B.3. ISO/IEC 14598 series of standards should be used as a guidance for planning and executing the measurement process.

B.3.6 Step #5 Feedback to the organization


Once all measurements have been completed map the results into Table B.1 and document conclusions in the form of a report. Also identify specific areas where quality improvements are required for the product to meet the user needs.

ISO/IEC TR 9126-2:2003(E)


Annex C (informative)

Detailed explanation of metric scale types and measurement types

C.1 Metric Scale Types

One of the following measurement metric scale types should be identified for each measure, when a user of metrics has the result of a measurement and uses the measure for calculation or comparison. The average, ratio or difference values may have no meaning for some measures. Metric scale types are: Nominal scale, Ordinal scale, Intervals scale, Ratio scale, and Absolute scale. A scale should always be defined as M'=F(M), where F is the admissible function. Also the description of each measurement scale type contains a description of the admissible function (if M is a metric then M'=F(M) is also a metric).

(a) Nominal Scale

M'=F(M) where F is any one-to-one mapping.

This includes classification, for example, software fault types (data, control, other). An average has a meaning only if it is calculated with frequency of the same type. A ratio has a meaning only when it is calculated with frequency of each mapped type. Therefore, the ratio and average may be used to represent a difference in frequency of only the same type between early and later cases or two similar cases. Otherwise, they may be used to mutually compare the frequency of each other type respectively.

Examples: Town transport line identification number, Compiler error message identification number

Meaningful statements are Numbers of different categories only.

(b) Ordinal Scale

M'=F(M) where F is any monotonic increasing mapping that is, M(x)>=M(y) implies M'(x)>=M'(y).

This includes ordering, for example, software failure by severity (negligible, marginal, critical, catastrophic). An average has a meaning only if it is calculated with frequency of the same mapped order. A ratio has a meaning only when it is calculated with the frequency of each mapped order. Therefore, the ratio and the average may be used to represent a difference in frequency of only the same order between early and later cases or two similar cases. Otherwise, they may be used to compare mutually the frequency of each order.

Examples: School exam.result (excellent, good, acceptable, not acceptable),

Meaningful statements: Each will depend on its position in the order, for example the median.

(c) Interval Scale

M'=aM+b (a>0)

This includes ordered rating scales where the difference between two measures has an empirical meaning. However the ratio of two measures in an interval scale may not have the same empirical meaning.

Examples: Temperature (Celsius, Fahrenheit, Kelvin), difference between the actual computation time and the time predicted

Meaningful statements: An arithmetic average and anything that depends on an order

ISO/IEC TR 9126-2:2003(E)


(d) Ratio Scale

M'=aM (a>0)

This includes ordered rating scales, where the difference between two measures and also the proportion of two measures have the same empirical meaning. An average and a ratio have meaning respectively and they give actual meaning to the values.

Examples: Length, Weight, Time, Size, Count

Meaningful statements: Geometrical mean, Percentage

(e) Absolute Scale

M'=M they can be measured only in one way.

Any statement relating to measures is meaningful. For example the result of dividing one ratio scale type measure by another ratio scale type measure where the unit of measurement is the same is absolute. An absolute scale type measurement is in fact one without any unit.

Example: Number of lines of code with comments divided by the total lines of code

Meaningful statements: Everything

C.2 Measurement Types

C.2.0 General

In order to design a procedure for collecting data, interpreting fair meanings, and normalizing measures for comparison, a user of metrics should identify and take account of the measure type of measurement employed by a metric.

C.2.1 Size Measure Type

C.2.1.0 General

A measure of this type represents a particular size of software according to what it claims to measure within its definition.

NOTE Software may have many representations of size (like any entity can be measured in more than one dimension - mass, volume, surface area etc.).

Normalizing other measures with a size measure can give comparable values in terms of units of size. The size measures described below can be used for software quality measurement.

C.2.1.1 Functional Size Type

Functional size is an example of one type of size (one dimension) that software may have. Any one instance of software may have more than one functional size depending on, for example:

(a) the purpose for measuring the software size (It influences the scope of the software included in the measurement);

(b) the particular functional sizing method used (It will change the units and scale).

The definition of the concepts and process for applying a functional size measurement method (FSM Method) is provided by the standard ISO/IEC 14143-1.

ISO/IEC TR 9126-2:2003(E)


In order to use functional size for normalization it is necessary to ensure that the same functional sizing method is used and that the different software being compared have been measured for the same purpose and consequently have a comparable scope.

Although the following often claim that they represent functional sizes, it is not guaranteed they are equivalent to the functional size obtained from applying a FSM Method compliant with ISO/IEC 14143-1. However, they are widely used in software development:

1. number of spread sheets;

2. number of screens;

3. number of files or data sets which are processed;

4. number of itemized functional requirements described in user requirements specifications.

C.2.1.2 Program size type

In this clause, the term ‘programming’ represents the expressions that when executed result in actions, and the term ‘language’ represents the type of expression used.

1. Source program size

The programming language should be explained and it should be provided how the non executable statements, such as comment lines, are treated. The following measures are commonly used.

Non-comment source statements (NCSS) include executable statements and data declaration statements with logical source statements.

NOTE 1 New program size

A developer may use newly developed program size to represent development and maintenance work product size.

NOTE 2 Changed program size

A developer may use changed program size to represent size of software containing modified components.

NOTE 3 Computed program size

Example of computed program size formula is new lines of code + 0.2 x lines of code in modified components (NASA Goddard).

It may be necessary to distinguish a type of statements of source code into more detail as follows:

i. Statement Type

Logical Source Statement (LSS). The LSS measures the number of software instructions. The statements are irrespective of their relationship to lines and independent of the physical format in which they appear.

Physical Source Statement (PSS). The PSS measures the number of software source lines of code.

ii. Statement attribute

Executable statements;

Data declaration statements;

Compiler directive statements;

Comment source statements.

ISO/IEC TR 9126-2:2003(E)


iii. Origin

Modified source statements;

Added source statements;

Removed source statements;

♦ Newly Developed source statements: (= added source statements + modified source statements);

♦ Reused source statements: (= original - modified - removed source statements);

2. Program word count size

The measurement may be computed in the following manner using the Halstead's measure:

Program vocabulary = n1+n2; Observed program length = N1+N2, where:

• n1: Is the number of distinct operator words which are prepared and reserved by the program language in a program source code;

• n2: Is the number of distinct operand words which are defined by the programmer in a program source code;

• N1: Is the number of occurrences of distinct operators in a program source code;

• N2: Is the number of occurrences of distinct operands in a program source code.

3. Number of modules

The measurement is counting the number of independently executable objects such as modules of a program.

C.2.1.3 Utilized resource measure type

This type identifies resources utilized by the operation of the software being evaluated. Examples are:

(a) Amount of memory, for example, amount of disk or memory occupied temporally or permanently during the software execution;

(b) I/O load, for example, amount of traffic of communication data (meaningful for backup tools on a network);

(c) CPU load, for example, percentage of occupied CPU instruction sets per second (This measure type is meaningful for measuring CPU utilization and efficiency of process distribution in multi-thread software running on concurrent/parallel systems);

(d) Files and data records, for example, length in bytes of files or records;

(e) Documents, for example, number of document pages.

It may be important to take note of peak (maximal), minimum and average values, as well as periods of time and number of observations done.

C.2.1.4 Specified operating procedure step type

This type identifies static steps of procedures which are specified in a human-interface design specification or a user manual.

The measured value may differ depending on what kinds of description are used for measurement, such as a diagram or a text representing user operating procedures.

ISO/IEC TR 9126-2:2003(E)


C.2.2 Time measure type

C.2.2.0 General

The user of metrics of time measure type should record time periods, how many sites were examined and how many users took part in the measurements.

There are many ways in which time can be measured as a unit, as the following examples show.

(a) Real time unit

This is a physical time: i.e. second, minute, or hour. This unit is usually used for describing task processing time of real time software.

(b) Computer machinery time unit

This is computer processor's clock time: i.e. second, minute, or hour of CPU time.

(c) Official scheduled time unit

This includes working hours, calendar days, months or years.

(d) Component time unit

When there are multiple sites, component time identifies individual site and it is an accumulation of individual time of each site. This unit is usually used for describing component reliability, for example, component failure rate.

(e) System time unit

When there are multiple sites, system time does not identify individual sites but identifies all the sites running, as a whole in one system. This unit is usually used for describing system reliability, for example, system failure rate.

C.2.2.1 System operation time type

System operation time type provides a basis for measuring software availability. This is mainly used for reliability evaluation. It should be identified whether the software is under discontinuous operation or continuous operation. If the software operates discontinuously, it should be assured that the time measurement is done on the periods the software is active (this is obviously extended to continuous operation).

(a) Elapsed time

When the use of software is constant, for example in systems operating for the same length of time each week.

(b) Machine powered-on time

For real time, embedded or operating system software that is in full use the whole time the system is operational.

(c) Normalized machine time

As in “machine powered-on time”, but pooling data from several machines of different “powered-on-time” and applying a correction factor.

C.2.2.2 Execution time type

Execution time type is the time which is needed to execute software to complete a specified task. The distribution of several attempts should be analysed and mean, deviation or maximal values should be

ISO/IEC TR 9126-2:2003(E)


computed. The execution under the specific conditions, particularly overloaded condition, should be examined. Execution time type is mainly used for efficiency evaluation.

C.2.2.3 User time type

User time type is measured upon time periods spent by individual users on completing tasks by using operations of the software. Some examples are:

(a) Session time

Measured between start and end of a session. Useful, as example, for drawing behaviour of users of a home banking system. For an interactive program where idling time is of no interest or where interactive usability problems only are to be studied.

(b) Task time

Time spent by an individual user to accomplish a task by using operations of the software on each attempt. The start and end points of the measurement should be well defined.

(c) User time

Time spent by an individual user using the software from time started at a point in time. (Approximately, it is how many hours or days user uses the software from beginning).

C.2.2.4 Effort type

Effort type is the productive time associated with a specific project task.

(a) Individual effort

This is the productive time which is needed for the individual person who is a developer, maintainer, or operator to work to complete a specified task. Individual effort assumes only a certain number of productive hours per day.

(b) Task effort

Task effort is an accumulated value of all the individual project personnel: developer, maintainer, operator, user or others who worked to complete a specified task.

C.2.2.5 Time interval of events type

This measure type is the time interval between one event and the next one during an observation period. The frequency of an observation time period may be used in place of this measure. This is typically used for describing the time between failures occurring successively.

C.2.3 Count measure type

If attributes of documents of the software product are counted, they are static count types. If events or human actions are counted, they are kinetic count types.

C.2.3.1 Number of detected fault type

The measurement counts the detected faults during reviewing, testing, correcting, operating or maintaining. Severity levels may be used to categorize them to take into account the impact of the fault.

C.2.3.2 Program structural complexity number type

The measurement counts the program structural complexity. Examples are the number of distinct paths or the McCabe's cyclomatic number.

ISO/IEC TR 9126-2:2003(E)


C.2.3.3 Number of detected inconsistency type

This measure counts the detected inconsistent items which are prepared for the investigation.

(a) Number of failed conforming items

Examples:

• Conformance to specified items of requirements specifications;

• Conformance to rule, regulation, or standard;

• Conformance to protocols, data formats, media formats, character codes.

(b) Number of failed instances of user expectation

The measurement is to count satisfied/unsatisfied list items, which describe gaps between user's reasonable expectation and software product performance.

The measurement uses questionnaires to be answered by testers, customers, operators, or end users on what deficiencies were discovered.


• Function available or not;

• Function effectively operable or not;

• Function operable to user's specific intended use or not;

• Function is expected, needed or not needed.

C.2.3.4 Number of changes type

This type identifies software configuration items which are detected to have been changed. An example is the number of changed lines of source code.

C.2.3.5 Number of detected failures type

The measurement counts the detected number of failures during product development, testing, operating or maintenance. Severity levels may be used to categorize them to take into account the impact of the failure.

C.2.3.6 Number of attempts (trial) type

This measure counts the number of attempts at correcting the defect or fault. For example, during reviews, testing, and maintenance.

C.2.3.7 Stroke of human operating procedure type

This measure counts the number of strokes of user human action as kinetic steps of a procedure when a user is interactively operating the software. This measure quantifies the ergonomic usability as well as the effort to use. Therefore, this is used in usability measurement. Examples are number of strokes to perform a task, number of eye movements, etc.

C.2.3.8 Score type

This type identifies the score or the result of an arithmetic calculation. Score may include counting or calculation of weights checked on/off on checklists. Examples: Score of checklist; score of questionnaire; Delphi method; etc.

ISO/IEC TR 9126-2:2003(E)


Annex D (informative)

Term(s)

D.1 Definitions

Definitions are from ISO/IEC 14598-1 and ISO/IEC 9126-1 unless otherwise indicated.

D.1.1 Quality

External quality: The extent to which a product satisfies stated and implied needs when used under specified conditions.

Internal quality: The totality of attributes of a product that determine its ability to satisfy stated and implied needs when used under specified conditions.

NOTE 1 The term “attribute” is used (rather than the term “characteristic” used in 3.1.3) as the term “characteristic” is used in a more specific sense in ISO/IEC 9126 series.

Quality: The totality of characteristics of an entity that bear on its ability to satisfy stated and implied needs.

NOTE 2 In a contractual environment, or in a regulated environment, such as the nuclear safety field, needs are specified, whereas in other environments, implied needs should be identified and defined.

Quality in use: The capability of the software product to enable specified users to achieve specified goals with effectiveness, productivity, safety and satisfaction in specified contexts of use.

NOTE 3 Quality in use is the user’s view of the quality of an environment containing software, and is measured from the results of using the software in the environment, rather than properties of the software itself.

NOTE 4 The definition of quality in use in ISO/IEC 14598-1 does not currently include the new characteristic of “safety”.

Quality model: The set of characteristics and the relationships between them, which provide the basis for specifying quality requirements and evaluating quality.

D.1.2 Software and user

Software: All or part of the programs, procedures, rules, and associated documentation of an information processing system. (ISO/IEC 2382-1:1993)

NOTE 1 Software is an intellectual creation that is independent of the medium on which it is recorded.

Software product: The set of computer programs, procedures, and possibly associated documentation and data designated for delivery to a user. [ISO/IEC 12207]

NOTE 2 Products include intermediate products, and products intended for users such as developers and maintainers.

User: An individual that uses the software product to perform a specific function.

NOTE 3 Users may include operators, recipients of the results of the software, or developers or maintainers of software.

D.1.3 Measurement

Attribute: A measurable physical or abstract property of an entity.

ISO/IEC TR 9126-2:2003(E)


Direct measure: A measure of an attribute that does not depend upon a measure of any other attribute.

External measure: An indirect measure of a product derived from measures of the behaviour of the system of which it is a part.

NOTE 1 The system includes any associated hardware, software (either custom software or off-the-shelf software) and users.

NOTE 2 The number of faults found during testing is an external measure of the number of faults in the program because the number of faults are counted during the operation of a computer system running the program to identify the faults in the code.

NOTE 3 External measures can be used to evaluate quality attributes closer to the ultimate objectives of the design.

Indicator: A measure that can be used to estimate or predict another measure.

NOTE 4 The measure may be of the same or a different characteristic.

NOTE 5 Indicators may be used both to estimate software quality attributes and to estimate attributes of the production process. They are indirect measures of the attributes.

Indirect measure: A measure of an attribute that is derived from measures of one or more other attributes.

NOTE 6 An external measure of an attribute of a computing system (such as the response time to user input) is an indirect measure of attributes of the software as the measure will be influenced by attributes of the computing environment as well as attributes of the software.

Internal measure: A measure derived from the product itself, either direct or indirect; it is not derived from measures of the behaviour of the system of which it is a part.

NOTE 7 Lines of code, complexity, the number of faults found in a walk through and the Fog Index are all internal measures made on the product itself.

Measure (noun): The number or category assigned to an attribute of an entity by making a measurement.

Measure (verb): Make a measurement.

Measurement: The process of assigning a number or category to an entity to describe an attribute of that entity.

NOTE 8 “Category” is used to denote qualitative measures of attributes. For example, some important attributes of software products, e.g. the language of a source program (ADA, C, COBOL, etc.) are qualitative.

Metric: A measurement scale and the method used for measurement.

NOTE 9 Metrics can be internal or external.

Metrics includes methods for categorizing qualitative data.

ISO/IEC TR 9126-2:2003(E)

ICS 35.080 Price based on 86 pages

© ISO/IEC 2003 – All rights reserved