1 ©Serco Reproduction permitted provided the source is acknowledged Formalising usability as part of ISO standards Nigel Bevan [email protected]

1©Serco Reproduction permitted provided the source is acknowledged

Formalising usability as part of ISO standards

Nigel BevanNigel Bevan

[email protected]@usability.serco.com

www.usability.serco.comwww.usability.serco.comwww.usabilitynet.orgwww.usabilitynet.org

Serco Usability ServicesSerco Usability Services

UKUK


TopicsTopics

What is usability?What is usability?

Definitions of usability? ISO 9126-1 and ISO 9241-11Definitions of usability? ISO 9126-1 and ISO 9241-11

Specifying usability as quality in use: ISO 9126-4, the Specifying usability as quality in use: ISO 9126-4, the Common Industry Format and ISO 25030Common Industry Format and ISO 25030

Usability of everyday products: ISO 20282Usability of everyday products: ISO 20282

User interface usability: ISO 9241User interface usability: ISO 9241

User centred design process: ISO 13407User centred design process: ISO 13407

Usability capability: ISO TR 18529Usability capability: ISO TR 18529


What is usability?What is usability?

what do you think usability is ......?what do you think usability is ......?


IBMIBM

““IBM's focus on ease of use has led to technology IBM's focus on ease of use has led to technology breakthroughs and advances in design research”breakthroughs and advances in design research”

““For developers and manufacturers, the advantages of For developers and manufacturers, the advantages of creating usable products far outweigh the costs”creating usable products far outweigh the costs”

““Every dollar invested in ease of use returns $10 to Every dollar invested in ease of use returns $10 to $100”$100”

http://www.ibm.com/ibm/easy/http://www.ibm.com/ibm/easy/


MicrosoftMicrosoft

““The Usability Group is an integral part of the product The Usability Group is an integral part of the product design process at Microsoft”design process at Microsoft” Over 120 usability engineersOver 120 usability engineers Over 25 usability labsOver 25 usability labs

http://www.microsoft.com/usability/http://www.microsoft.com/usability/


Why is usability important?Why is usability important?

Focus on user and organisational needsFocus on user and organisational needs reduce development timesreduce development times less training and support and documentation is requiredless training and support and documentation is required

Improve productivityImprove productivity simpler interface, fewer user errorssimpler interface, fewer user errors

Improve the competitive edgeImprove the competitive edge increasing expectations for ease of useincreasing expectations for ease of use increasing usability of competitive productsincreasing usability of competitive products high profile of usability in advertisinghigh profile of usability in advertising

Improve the quality of lifeImprove the quality of life less stress, users are more satisfiedless stress, users are more satisfied lower staff turnoverlower staff turnover

Health and safety legislationHealth and safety legislation European Directive on Display Screen EquipmentEuropean Directive on Display Screen Equipment


Case study: Hewlett Packard Case study: Hewlett Packard

User-centred design methods were applied to redesign User-centred design methods were applied to redesign software used for identifying network problems:software used for identifying network problems:

OldOld NewNew

time to finish tasktime to finish task 9.4 min9.4 min 4.1 min4.1 min

problems identifiedproblems identified 16%16% 78%78%

average length of callaverage length of call 30 min30 min 10 min10 min

size of manualsize of manual 25 pages25 pages 4 pages4 pages

people needing the manualpeople needing the manual 53%53% 3%3%

user satisfaction ratinguser satisfaction rating 3.53.5 6.86.8

In addition to the benefits to customers, HP recovered their In addition to the benefits to customers, HP recovered their costs in 18 monthscosts in 18 months


ISO/IEC 9126 Software Product Quality ModelISO/IEC 9126 Software Product Quality Model

functionality reliability

usability efficiency

maintainability

portability

accuracysuitability

interoperabilitysecurity

understandabilitylearnabilityoperability

analysabilitychangeability

stabilitytestability

maturityfault tolerancerecoverability

availability

time behaviourresourceutilisation

adaptabilityinstallabilityco-existencereplaceability


ISO/IEC 9126-1 DefinitionsISO/IEC 9126-1 Definitions FunctionalityFunctionality

The capability of the software product to provide functions which meet stated The capability of the software product to provide functions which meet stated and implied needs when the software is used under specified conditions.and implied needs when the software is used under specified conditions.

ReliabilityReliability The capability of the software product to maintain a specified level of The capability of the software product to maintain a specified level of

performance when used under specified conditions performance when used under specified conditions

UsabilityUsability The capability of the software product to be understood, learned, used and The capability of the software product to be understood, learned, used and

attractive to the user, when used under specified conditions.attractive to the user, when used under specified conditions.

EfficiencyEfficiency The capability of the software product to provide appropriate performance, The capability of the software product to provide appropriate performance,

relative to the amount of resources used, under stated conditions.relative to the amount of resources used, under stated conditions.

MaintainabilityMaintainability The capability of the software product to be modified. Modifications may The capability of the software product to be modified. Modifications may

include corrections, improvements or adaptation of the software to changes in include corrections, improvements or adaptation of the software to changes in environment, and in requirements and functional specifications. environment, and in requirements and functional specifications.

PortabilityPortability The capability of the software product to be transferred from one environment to The capability of the software product to be transferred from one environment to

another.another.


Usability : The extent to which a product canbe used by specified users to achievespecified goals with effectiveness, efficiencyand satisfaction in a specified context ofuse.

Effectiveness : The accuracy andcompleteness with which users achievespecified goals.

Efficiency: The resources expended inrelation to the accuracy and completenesswith which users achieve goals.

Satisfaction : The comfort and acceptabilityof use.

ISO 9241-11 Guidance on UsabilityISO 9241-11 Guidance on Usability


ISO/IEC 9126-1 Software Product Quality ModelISO/IEC 9126-1 Software Product Quality Model

quality in use

functionality reliability

usability efficiency

maintainability portability

accuracysuitability

interoperabilitysecurity

understandabilitylearnabilityoperability

attractiveness

analysabilitychangeability

stabilitytestability

maturityfault tolerancerecoverability

availability

time behaviourresourceutilisation

adaptabilityinstallabilityco-existencereplaceability


Quality in UseQuality in Use

ISO/IEC 9126-1 (2001) Software Engineering - ISO/IEC 9126-1 (2001) Software Engineering - Product quality - Part 1: Quality modelProduct quality - Part 1: Quality model

quality in usequality in use

the capability of the software product to enable the capability of the software product to enable specified users to achieve specified goals with specified users to achieve specified goals with effectiveness, productivity, safety and satisfaction effectiveness, productivity, safety and satisfaction in specified contexts of usein specified contexts of use

Quality in use is the user’s view of the quality of a system Quality in use is the user’s view of the quality of a system containing software, and is measured in terms of the result of containing software, and is measured in terms of the result of using the software, rather than properties of the software using the software, rather than properties of the software itselfitself

Quality in use measures the combined effect of ease of use, Quality in use measures the combined effect of ease of use, functionality, efficiency and reliability.functionality, efficiency and reliability.


Approaches to achieving usabilityApproaches to achieving usability

process quality product quality quality in use

usability in context

development process

product effect of the product

user centred process

interface and interaction

ISO 13407 ISO 9241-11ISO 14598-1

ISO/IEC 9126-1ISO/IEC 9126-4

ISO 20282-2

organisational capability

life cycle processes

usability capability

ISOTR 18529 ISO 9241 parts 10, 12-17ISO/IEC 9126-2/3



1 Design for quality in useDesign for quality in use ISO 9241-11, ISO 9126-4: effectiveness, efficiency, (safety), ISO 9241-11, ISO 9126-4: effectiveness, efficiency, (safety),

satisfactionsatisfaction ISO 20282-2 Usability of everyday productsISO 20282-2 Usability of everyday products

2 Design the user interfaceDesign the user interface ISO 9241 parts 12 to 17 Ergonomic principlesISO 9241 parts 12 to 17 Ergonomic principles

3 Use user-centred methods in developmentUse user-centred methods in development ISO 13407 Human centred design processesISO 13407 Human centred design processes

4 Establish organisational processesEstablish organisational processes ISO TR 18529 ISO TR 18529 Human-centred lifecycle process descriptionsHuman-centred lifecycle process descriptions


requirements

User requirements

Functional requirements

Detailed specification

inspection

evaluationproduct

Deliver

Prototype

Code

What can be specified and measured?What can be specified and measured?

Test user function

Test user goal

Internalusability metricsISO/IEC 9126-3

requirements

Externalusability metricsISO/IEC 9126-2

Quality in usemetrics

ISO/IEC 9126-4


1. Design for quality in use1. Design for quality in use

Usability/quality in use measures:Usability/quality in use measures:

Effectiveness and efficiencyEffectiveness and efficiency MUSiC Performance Measurement MethodMUSiC Performance Measurement Method

http://www.usability.serco.com/publs.htmlhttp://www.usability.serco.com/publs.html

SatisfactionSatisfaction SUMI questionnaireSUMI questionnaire

www.ucc.ie/hfrg/questionnaires/sumiwww.ucc.ie/hfrg/questionnaires/sumi

Reporting usabilityReporting usability Common industry format for usability test resultsCommon industry format for usability test results includes intended context of use and actual context of includes intended context of use and actual context of

evaluationevaluation

www.nist.gov/iusrwww.nist.gov/iusr


ISO/IEC CD 25030: Software quality requirements ISO/IEC CD 25030: Software quality requirements and evaluation – Quality requirementsand evaluation – Quality requirements

Requirements can be categorised as: Requirements can be categorised as: functional requirementsfunctional requirements quality requirementsquality requirements other non-functional requirementsother non-functional requirements

Software quality requirements are categorised using the Software quality requirements are categorised using the ISO/IEC 9126-1 software quality model into ISO/IEC 9126-1 software quality model into requirements for: requirements for: quality in use (the effectiveness, productivity, safety and quality in use (the effectiveness, productivity, safety and

satisfaction with the software applied in its context of use)satisfaction with the software applied in its context of use) external quality (the behaviour of the finished software product)external quality (the behaviour of the finished software product) internal quality (static properties during development, such as internal quality (static properties during development, such as

adherence to coding standards, and meeting size and adherence to coding standards, and meeting size and complexity thresholds)complexity thresholds)


Requirements life cycle modelRequirements life cycle model

Definition

Measured quality

Measured quality

Stated requirements

Stated requirements

Stated requirements

Formalisation Evaluation

Measurable requirements



Internal quality

External quality

Quality in use

Quality requirements

Stake-holderneeds

Measured quality

Developed system


Other non-functional requirements


Quality requirements specificationQuality requirements specification

Define quality requirementsDefine quality requirements Identify stakeholdersIdentify stakeholders

end users, developers, producers, trainers, maintainers, disposers, acquirer, supplier organisations and regulatory bodies

Elicit requirements from stakeholdersElicit requirements from stakeholders Quality in use, External and Internal QualityQuality in use, External and Internal Quality

Analyse the set of requirementsAnalyse the set of requirements Resolve problemsResolve problems Confirm requirementsConfirm requirements Record requirementsRecord requirements

Formalise quality requirementsFormalise quality requirements Specify target values for measuresSpecify target values for measures Demonstrate traceabilityDemonstrate traceability Maintain requirementsMaintain requirements


Elicit requirements from stakeholdersElicit requirements from stakeholders

Include the needs, wants, desires, expectations and Include the needs, wants, desires, expectations and perceived constraints perceived constraints

Identify requirementsIdentify requirements Quality in useQuality in use ExternalExternal InternalInternal

Identify constraintsIdentify constraints Define scenarios of useDefine scenarios of use


Quality in useQuality in use

Extent to which a product used by specified users meets their needs to achieve specified goals with effectiveness, productivity and satisfaction in specified contexts of use

Effectiveness: accuracy and completenessEffectiveness: accuracy and completeness

Productivity: the resources expended by users in relation Productivity: the resources expended by users in relation to the effectiveness achievedto the effectiveness achieved


Why is quality in use important?Why is quality in use important?

It is about meeting business and user needsIt is about meeting business and user needs Effectiveness: success in achieving goalsEffectiveness: success in achieving goals Productivity: staff timeProductivity: staff time Satisfaction: willingness to use the systemSatisfaction: willingness to use the system


Quality in use: context of UseQuality in use: context of Use

The usability of a product is affected not only by The usability of a product is affected not only by the features of the product itself but also by its the features of the product itself but also by its Context of UseContext of Use

Context is the characteristics of:Context is the characteristics of: the users of the productthe users of the product the tasks they carry out the tasks they carry out the technical, organisational the technical, organisational

and physical environmentand physical environmentin which the product is in which the product is usedused

the date and time when the the date and time when the product is being usedproduct is being used


Example: A bank ATMExample: A bank ATM

Context description:Context description: The usersThe users

The user characteristicsThe user characteristics

The tasks users performThe tasks users perform

The technical environmentThe technical environment(hardware and software to(hardware and software tosupport system)support system)

The physical environmentThe physical environment

The social or organisational environmentThe social or organisational environment


New design features to meet contextual New design features to meet contextual needsneeds

Recess for wheelchair access.Recess for wheelchair access.

Speech output for visuallySpeech output for visuallyimpaired users.impaired users.

Customisation features forCustomisation features forrapid access.rapid access.

Finger print for identification.Finger print for identification.

Visor appears during sunny weather.Visor appears during sunny weather.

Buttons light during darkness.Buttons light during darkness.

Alarm button for security alert. Alarm button for security alert.


Efficiency: mean transaction timesEfficiency: mean transaction times

0

0.5

1

1.5

2

2.5

3

DepositingCash

WithdrawingCash

DepositingCheques

WithdrawingCheques

Minutes Old system

New system


User satisfaction: SUMI questionnaireUser satisfaction: SUMI questionnaire

0

10

20

30

40

50

60

Global Efficiency Affect Helpfulness Control Learnability

Old System

New System


The danger of not setting quality in use The danger of not setting quality in use requirementsrequirements

New software for issuing UK passportsNew software for issuing UK passports Installed in passport issuing officesInstalled in passport issuing offices Took operators twice as longTook operators twice as long Caused delays of up to 3 months in obtaining a passportCaused delays of up to 3 months in obtaining a passport Huge cost of additional clerical staffHuge cost of additional clerical staff


Example of quality in use requirementsExample of quality in use requirements

User performanceUser performance ““all data entry clerks will be able to complete the task with at all data entry clerks will be able to complete the task with at

least 95% accuracy in under 10 minutes”least 95% accuracy in under 10 minutes”

User satisfactionUser satisfaction ““the mean score on the SUMI scale will be greater than 50” More the mean score on the SUMI scale will be greater than 50” More

information on quality in use requirements information on quality in use requirements


Common Industry Format usability testsCommon Industry Format usability tests

NIST initiativeNIST initiative National Institute of Standards and TechnologyNational Institute of Standards and Technology

Objective: raise the profile of usability in procurementObjective: raise the profile of usability in procurement Suppliers provide standard test reports to purchasersSuppliers provide standard test reports to purchasers Suppliers include: Suppliers include:

IBM, Microsoft, HP, Sun, Kodak, Oracle, CompaqIBM, Microsoft, HP, Sun, Kodak, Oracle, Compaq Purchasers include: Purchasers include:

Boeing, Northwest Mutual Life, State Farm Insurance, FidelityBoeing, Northwest Mutual Life, State Farm Insurance, Fidelity Reports provided in confidenceReports provided in confidence Could permit comparisonsCould permit comparisons Agreed as US standard Agreed as US standard ANSI/NCITS 354ANSI/NCITS 354, to be submitted to ISO, to be submitted to ISO

http://www.nist.gov/iusrhttp://www.nist.gov/iusr


Report format - technicalReport format - technical

Test objectivesTest objectives ParticipantsParticipants

Total number and user groupsTotal number and user groups Key characteristics and capabilities expectedKey characteristics and capabilities expected How selected: with the key characteristics and capabilities?How selected: with the key characteristics and capabilities?

Context of evaluationContext of evaluation Task scenarios and task data usedTask scenarios and task data used Any performance criteria usedAny performance criteria used Physical and computer environmentPhysical and computer environment


Report format - resultsReport format - results

EffectivenessEffectiveness % completely and correctly achieving their goals% completely and correctly achieving their goals Separate data where assistance was givenSeparate data where assistance was given

EfficiencyEfficiency Mean time taken to complete the taskMean time taken to complete the task Standard deviation (and standard error)Standard deviation (and standard error)

SatisfactionSatisfaction Optional: own scales or standard questionnaireOptional: own scales or standard questionnaire


ISO 20282: Usability of everyday products ISO 20282: Usability of everyday products Part 1: Universal user profilePart 1: Universal user profile

Design should take account ofDesign should take account of Strength and Biomechanical AbilitiesStrength and Biomechanical Abilities HandednessHandedness Body Dimensions.Body Dimensions. Visual Abilities. Visual Abilities. Auditory AbilitiesAuditory Abilities Cognitive AbilitiesCognitive Abilities Language and literacyLanguage and literacy CultureCulture AgeAge GenderGender


ISO WD 20282-2: Test methodISO WD 20282-2: Test method

A summative test method for measuring usability based onA summative test method for measuring usability based on ISO 9241-11 (Guidance on usability)ISO 9241-11 (Guidance on usability) ISO 9126-4 (Quality in use metrics)ISO 9126-4 (Quality in use metrics) Common Industry Format for usability test reportsCommon Industry Format for usability test reports

Identify intended context of use and user groupsIdentify intended context of use and user groups

MeasuresMeasures Success rate (ease of operation)Success rate (ease of operation) Task time (efficiency)Task time (efficiency) Satisfaction scale (being developed)Satisfaction scale (being developed)





development process




ISO 13407 ISO 9241-11ISO 14598-1


ISO 20282-2






2. Design the user interface2. Design the user interfaceISO 9241: Ergonomic requirements for office work with visual display ISO 9241: Ergonomic requirements for office work with visual display terminalsterminals

Parts 10 to 17 give guidance on softwareParts 10 to 17 give guidance on software

1010 Dialogue principles Dialogue principles

1111 Guidance on usability Guidance on usability

1212 Presentation of information Presentation of information

1313 User guidance User guidance

1414 Menu dialogues Menu dialogues

1515 Command Language dialogues Command Language dialogues

1616 Direct Manipulation dialogues Direct Manipulation dialogues

1717 Form-filling dialogues Form-filling dialogues





development process




ISO 13407 ISO 9241-11ISO 14598-1


ISO 20282-2






3. ISO 13407: Human centred design process for 3. ISO 13407: Human centred design process for interactive systemsinteractive systems

1. Plan the human centred process

1. Plan the human centred process

2. Specify the context of use

2. Specify the context of use

4. Produce design solutions

4. Produce design solutions

3. Specify user and organisational

requirements

3. Specify user and organisational

requirements

5. Evaluate designs against

user requirements

5. Evaluate designs against

user requirements

Meets requirements


ISO 13407 ScopeISO 13407 Scope

Computer-based interactive systemsComputer-based interactive systems The detailed recommendations are applicable to computer The detailed recommendations are applicable to computer

hardware and softwarehardware and software

Can apply the same principles to the design of any Can apply the same principles to the design of any system for use by humanssystem for use by humans A penA pen A chairA chair A meeting roomA meeting room A conferenceA conference


TRUMP: Trial Usability Maturity ProcessTRUMP: Trial Usability Maturity Process

EU-funded trial application of user-centred design EU-funded trial application of user-centred design methods developed in previous research projects methods developed in previous research projects (INUSE and RESPECT)(INUSE and RESPECT)

Serco: apply the methodsSerco: apply the methods Lloyds Register: Usability Maturity AssessmentLloyds Register: Usability Maturity Assessment

Inland Revenue/EDS - IT for 60,000 staffInland Revenue/EDS - IT for 60,000 staff RAD methodologyRAD methodology

Israel Aircraft Industries - aerospace systemsIsrael Aircraft Industries - aerospace systems traditional methodologytraditional methodology


Typical approach to usabilityTypical approach to usability

feasibility requirements design implement release

Usability group user and task analysis

usability test

prototyping


TRUMP approach: “ubiquitous usability”TRUMP approach: “ubiquitous usability”

1. Stake-holder meeting

2. Context of use

3. Scenarios

4. Usability requirements

5. Evaluate existing system

6. Prototyping

7. Style guide

8. Evaluation

9. Usability testing

10. Collect feedback

feasibility requirements design implement release

System lifecycle

PlanProcess

SpecifyContext of

Use

DesignSolution

s

SpecifyRequirements

Evaluate against Requirements

www.usability.serco.com/trump/ucdmethods


requirements

User requirements



Function test

evaluationproduct

Deliver

Prototype

Code

Conventional design processConventional design processHow do we incorporate user centred design?How do we incorporate user centred design?


requirements

User requirements



evaluationproduct

2. Intended context of use

Context used for evaluation

6.Mock up

Use in context

Prototype

Code

6. Paper prototype

6. Paper specificatio

n

3. Intended scenarios of use

Scenarios used for evaluation

1. Plans

4. Quality in userequirements

4. External usability

requirements

4. Internal usability

requirements

5. Evaluate existing system

Function test

7. Style guide

Inspection7. Style guide


Methods: feasibilityMethods: feasibility

1. Stakeholder meeting 1. Stakeholder meeting A half-day meeting to identify and agree on the role of A half-day meeting to identify and agree on the role of usability, broadly identifying the intended context of use and usability, broadly identifying the intended context of use and usability goals, and how these relate to the business usability goals, and how these relate to the business objectives and success criteria for the system. objectives and success criteria for the system.

2. Context of use2. Context of useA half-day workshop to collect and agree detailed information A half-day workshop to collect and agree detailed information about the intended users, their tasks, and the technical and about the intended users, their tasks, and the technical and environmental constraints.environmental constraints.

3. Scenarios of use3. Scenarios of useA half day workshop to document examples of how users are A half day workshop to document examples of how users are expected carry out key tasks in a specified contexts, to expected carry out key tasks in a specified contexts, to provide an input to design and a basis for subsequent provide an input to design and a basis for subsequent usability testing. usability testing.


Methods: requirementsMethods: requirements

4. Evaluate an existing system4. Evaluate an existing systemEvaluate an earlier version or competitor system to identify Evaluate an earlier version or competitor system to identify usability problems and obtain measures of usability as an usability problems and obtain measures of usability as an input to usability requirements.input to usability requirements.

5. Usability requirements5. Usability requirementsA half-day workshop to establish requirements for A half-day workshop to establish requirements for effectiveness, efficiency and satisfaction for the user groups effectiveness, efficiency and satisfaction for the user groups and tasks identified in the context of use analysis and in the and tasks identified in the context of use analysis and in the scenarios.scenarios.


Methods: designMethods: design

6. Paper prototyping6. Paper prototypingEvaluation by users of quick low fidelity prototypes (using Evaluation by users of quick low fidelity prototypes (using paper or other materials) to clarify requirements and enable paper or other materials) to clarify requirements and enable draft interaction designs and screen designs to be rapidly draft interaction designs and screen designs to be rapidly simulated and tested.simulated and tested.

7. Style guide7. Style guide Identify, document and adhere to industry, corporate or Identify, document and adhere to industry, corporate or project conventions for screen and page design.project conventions for screen and page design.


Methods: evaluationMethods: evaluation

8. Evaluation of machine prototypes8. Evaluation of machine prototypes Informal usability testing with 3-5 representative users Informal usability testing with 3-5 representative users carrying out key tasks to provide rapid feedback on the carrying out key tasks to provide rapid feedback on the usability of prototypes. usability of prototypes.

9. Usability testing9. Usability testing Formal usability testing with 8 representatives of a user group Formal usability testing with 8 representatives of a user group carrying out key tasks to identify any remaining usability carrying out key tasks to identify any remaining usability problems and evaluate whether usability objectives have been problems and evaluate whether usability objectives have been achieved.achieved.

10. Collect feedback from users10. Collect feedback from usersCollect information from sources such as usability surveys, Collect information from sources such as usability surveys, help lines and support services to identify any problems that help lines and support services to identify any problems that should be fixed in future versions. should be fixed in future versions.


Cost benefits at IAICost benefits at IAI

Development cost/benefitsDevelopment cost/benefits A all the methods used (except Style Guide) resulted in savings in A all the methods used (except Style Guide) resulted in savings in

development costs of between $5K and $70K for each method, development costs of between $5K and $70K for each method, with a total saving of $330K. with a total saving of $330K.

The cost of using the methods was only $22K, giving a cost-The cost of using the methods was only $22K, giving a cost-benefit ration of benefit ration of 1:151:15. .

Sales cost/benefitsSales cost/benefits Increased sales because customers would be more likely to buy Increased sales because customers would be more likely to buy

the product were estimated to be $400Kthe product were estimated to be $400K Support cost/benefitsSupport cost/benefits

Reduced costs of training and support were estimated at £50K.Reduced costs of training and support were estimated at £50K. Overall cost benefitsOverall cost benefits

The overall costs of the maturity assessments and use of methods The overall costs of the maturity assessments and use of methods was $27K. The total estimated savings and increased sales is was $27K. The total estimated savings and increased sales is $780K, giving a cost-benefit ration of 1:29.$780K, giving a cost-benefit ration of 1:29.


Trial Application at IAI Bottom LineTrial Application at IAI Bottom Line

Positive Feedback from ParticipantsPositive Feedback from Participants A Definite Improvement in the Development Process A Definite Improvement in the Development Process Very Cost Effective and Low CostVery Cost Effective and Low Cost Mostly Intuitive however tailoring sometimes required Mostly Intuitive however tailoring sometimes required Expert Guidance needed in few techniquesExpert Guidance needed in few techniques LAHAV decided to incorporate TRUMP techniques in it’s LAHAV decided to incorporate TRUMP techniques in it’s

standard development processstandard development process


IAI ConclusionsIAI Conclusions

Low cost simple methods for improving systems usability Low cost simple methods for improving systems usability are hereare here

The methods can be easily integrated into any software The methods can be easily integrated into any software development processdevelopment process

The responsibility for quality in use should be transferred The responsibility for quality in use should be transferred to development organizationto development organization

The Methods and supporting means are available on the The Methods and supporting means are available on the WebWeb

No more Excuses No more Excuses





development process




ISO 13407 ISO 9241-11ISO 14598-1


ISO 20282-2






4. Establish usability capability 4. Establish usability capability ISO TR 18529: Human-centred lifecycle process descriptionsISO TR 18529: Human-centred lifecycle process descriptions

HCD.1 Ensure HCD content in system strategyHCD.1 Ensure HCD content in system strategy

HCD.2 Plan and manage the HCD processHCD.2 Plan and manage the HCD process

HCD.3 Specify the user and organisational requirementsHCD.3 Specify the user and organisational requirements

HCD.4 Understand and specify the context of useHCD.4 Understand and specify the context of use

HCD.5 Produce design solutionsHCD.5 Produce design solutions

HCD.6 Evaluate designs against requirementsHCD.6 Evaluate designs against requirements

HCD.7 Introduce and operate the systemHCD.7 Introduce and operate the system


Process improvement overviewProcess improvement overview

1.1. Identify needs for usability process improvement Identify needs for usability process improvement • Use the usability maturity model in ISO TR 18529 to assess the Use the usability maturity model in ISO TR 18529 to assess the

current capability of each organisationcurrent capability of each organisation

2.2. Make the identified improvements to the software Make the identified improvements to the software development processesdevelopment processes• Introduce simple user-based methods implementing ISO 13407Introduce simple user-based methods implementing ISO 13407

3.3. Reassess the usability capability the organisation Reassess the usability capability the organisation • Assess the extent of the improvementAssess the extent of the improvement

4.4. Identify the cost-benefits of the improvementsIdentify the cost-benefits of the improvements• Integrate the methods into the documented processesIntegrate the methods into the documented processes


Usability Maturity Model Usability Maturity Model HCD.2 Plan and manage the HCD processHCD.2 Plan and manage the HCD process

Consult stakeholders.Consult stakeholders. Identify and plan user involvement.Identify and plan user involvement. Select human-centred methods and techniques.Select human-centred methods and techniques. Ensure a human-centred approach within the project Ensure a human-centred approach within the project

team.team. Plan human-centred design process.Plan human-centred design process. Manage human-centred process.Manage human-centred process. Champion human-centred activities.Champion human-centred activities. Provide support for human-centred designProvide support for human-centred design


Usability Maturity Model Usability Maturity Model HCD.3 Specify the user and organisational HCD.3 Specify the user and organisational requirementsrequirements

Clarify and document the system goals.Clarify and document the system goals. Define stakeholders.Define stakeholders. Assess risk to stakeholders.Assess risk to stakeholders. Define the system.Define the system. Generate the user and organisational requirements.Generate the user and organisational requirements. Set usability objectives.Set usability objectives.


Usability Maturity ModelUsability Maturity Model HCD.4 Understand and specify the context of useHCD.4 Understand and specify the context of use

Identify and documentIdentify and document user’s tasks.user’s tasks. significant user attributes.significant user attributes. organisational environment.organisational environment. technical environment.technical environment. physical environment.physical environment.


Usability Maturity ModelUsability Maturity Model HCD.5 Produce design solutionsHCD.5 Produce design solutions

Allocate functionsAllocate functions Produce composite task modelProduce composite task model Produce system designProduce system design Use existing knowledge to develop design solutionsUse existing knowledge to develop design solutions Specify systemSpecify system Develop prototypesDevelop prototypes Provide user trainingProvide user training Provide user supportProvide user support


Usability Maturity ModelUsability Maturity Model HCD.6 Evaluate designs against HCD.6 Evaluate designs against requirementsrequirements

Specify and validate context of evaluation.Specify and validate context of evaluation. Evaluate early prototypes in order to define the Evaluate early prototypes in order to define the

requirements for the system.requirements for the system. Evaluate prototypes in order to improve the design.Evaluate prototypes in order to improve the design. Evaluate the system in order to check that the system Evaluate the system in order to check that the system

requirements have been met.requirements have been met. Evaluate the system in order to check that the Evaluate the system in order to check that the

required practice has been followed.required practice has been followed. Evaluate the system in use in order to ensure that it Evaluate the system in use in order to ensure that it

continues to meet organisational and user needs.continues to meet organisational and user needs.


United Kingdom’s tax collection departmentUnited Kingdom’s tax collection department

Employ over 60,000 staff in more than 600 local offices Employ over 60,000 staff in more than 600 local offices

Partnership with EDS for software development now Partnership with EDS for software development now running for more than 5 yearsrunning for more than 5 years

JAD: Joint Application DevelopmentJAD: Joint Application Development RAD: Rapid Application DevelopmentRAD: Rapid Application Development


Usability capability assessment Usability capability assessment

Formal CMM/SPICE style assessment by Lloyds RegisterFormal CMM/SPICE style assessment by Lloyds Register Interviewed 13 stakeholders in twelve 3-hour sessions Interviewed 13 stakeholders in twelve 3-hour sessions

over one week. over one week. A conventional software process assessment procedure based A conventional software process assessment procedure based

on SPICE (ISO 15504) was used. on SPICE (ISO 15504) was used. Produced a detailed profile and rich information about Produced a detailed profile and rich information about

where improvements would be beneficial. where improvements would be beneficial. At a feedback meeting on the last day, it provided the At a feedback meeting on the last day, it provided the

basis for an agreed set of improvement activitiesbasis for an agreed set of improvement activities A second assessment a year later showed significant A second assessment a year later showed significant

improvementsimprovements


Inland Revenue/EDS ConclusionsInland Revenue/EDS Conclusions

Significant improvement in all areas of usability maturitySignificant improvement in all areas of usability maturity IR/EDS now strong in the “core” areas of context, IR/EDS now strong in the “core” areas of context,

requirements, design and evaluationrequirements, design and evaluation Evaluation activity rated as outstandingEvaluation activity rated as outstanding The methods are very cost effective and will be The methods are very cost effective and will be

incorporated into the documented development processincorporated into the documented development process ““A step change in the awareness and practice of human A step change in the awareness and practice of human

centred design within IR/EDS”.centred design within IR/EDS”.


IAI LAHAV BackgroundIAI LAHAV Background

Expertise in Military Aircraft “Avionics Upgrade Expertise in Military Aircraft “Avionics Upgrade Programs”Programs”

Customers WorldwideCustomers Worldwide

User needs addressed by a group of PilotsUser needs addressed by a group of Pilots

Trial Application: PC-based Mission Planning systemTrial Application: PC-based Mission Planning system


Usability capability assessment workshopUsability capability assessment workshop

Process improvement workshopProcess improvement workshop One-day workshopOne-day workshop Review each process and activity in ISO TR 18529 in Review each process and activity in ISO TR 18529 in

detaildetail Score each process as: Score each process as:

Not, Partly, Largely or Fully performedNot, Partly, Largely or Fully performed Note whether the process is managed Note whether the process is managed

a documented part of normal proceduresa documented part of normal procedures Establish a target level for each process on the same Establish a target level for each process on the same

scalesscales A second assessment a year later showed significant A second assessment a year later showed significant

improvementsimprovements


Trial Application at IAI Bottom LineTrial Application at IAI Bottom Line

Positive Feedback from ParticipantsPositive Feedback from Participants

A Definite Improvement in the Development Process A Definite Improvement in the Development Process

Very Cost Effective and Low Cost methodsVery Cost Effective and Low Cost methods

LAHAV decided to incorporate TRUMP techniques in it’s LAHAV decided to incorporate TRUMP techniques in it’s standard development processstandard development process


Process improvement conclusionsProcess improvement conclusions

The usability maturity model was a valuable tool for The usability maturity model was a valuable tool for identifying needs for process improvementidentifying needs for process improvement Inland Revenue:summative assessment requiring three person Inland Revenue:summative assessment requiring three person

weeks effortweeks effort IAI: a simpler formative one-day assessmentIAI: a simpler formative one-day assessment

A very promising way to make major process A very promising way to make major process improvementsimprovements

www.usability.serco.com/trumpwww.usability.serco.com/trump


ISO 13407 ConclusionsISO 13407 Conclusions

ISO 13407 and ISO TR 18529 provide an internationally-agreed ISO 13407 and ISO TR 18529 provide an internationally-agreed description of good practice in user-centred designdescription of good practice in user-centred design

The content can be used to specify or evaluate:The content can be used to specify or evaluate: How usability should be incorporated into the development processHow usability should be incorporated into the development process Whether a project has applied user centred designWhether a project has applied user centred design Organisational capability in user-centred designOrganisational capability in user-centred design Basis for process improvementBasis for process improvement Appropriate content for education and training courses in user centred Appropriate content for education and training courses in user centred

designdesign Necessary areas of professional competence: professional accreditationNecessary areas of professional competence: professional accreditation


www.usabilitynet.orgwww.usabilitynet.org

Documents

1 ©Serco Reproduction permitted provided the source is acknowledged Formalising usability as part of ISO standards Nigel Bevan [email protected]