New ISO standards for usability, usability reports and usability measures slides

1

New ISO standards for usability, usability

reports and usability measures

HCII 2016

Nigel Bevan1, Jim Carter2, Jonathan Earthy3, Thomas Geis4, Susan Harker5

[email protected] Professional Usability [email protected] University of [email protected] Lloyd's Register EMEA

[email protected] ProContext Consulting [email protected] Loughborough University

© Nigel Bevan. Reproduction permitted provided the source is acknowledged.

2

Usability is the outcome of interaction:

Usability The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use

Effectiveness: The accuracy and completeness with which users achieve specified goals.

Efficiency: The resources expended in relation to the accuracy and completeness with which users achieve goals.

Satisfaction: The comfort and acceptability of use

ISO 9241-11 Guidance on Usability (1998)

3

ISO 9241-11 revised [feedback needed]

ISO9241-111998 ISODIS29241-112016Usability Extenttowhichaproductcanbe

usedbyspecifieduserstoachievespecifiedgoalswitheffectiveness,efficiencyandsatisfactioninaspecifiedcontextofuse

Extenttowhichasystem,productorservicecanbeusedbyspecifieduserstoachievespecifiedgoalswitheffectiveness,efficiencyandsatisfactioninaspecifiedcontextofuse

Effectiveness Accuracyandcompletenesswithwhichusersachievespecifiedgoals

Correctnessandcompletenesswithwhichusersachievedspecifiedgoals

Efficiency Resourcesexpendedinrelationtotheaccuracyandcompletenesswithwhichusersachievegoals

Resourcesusedinrelationtotheresultsachieved

Satisfaction Freedomfromdiscomfort,andpositiveattitudestowardstheuseoftheproduct.

Userexperienceresultingfromuseofasystem,productorservice

4

It is important to understand the user's

experience

Overall usability• All (potential) users• All (potential) goals

Specified usability• Specified user groups• Specified goals

Usability instance• Single user group• Single goal

Individual usability (UX)• Single user• Set of personal goals

5

Effectiveness is not just practical goals:

also personal goals

o Assigned goalsn By an organisationn Also typically in a usability test

o e.g. arrange travel to Toronto for HCII 2016

o Personal goalsn Interpretation of assigned goalsn Personal practical goals

o Have time for a holidayn Personal hedonic goal

o To experience a different airlineo To earn frequent flyer mileso To travel comfortably in business class

6

Effectiveness is not just positive outcomes:

also negative outcomes

Effectiveness

o Correctness is the extent to which actual outcomes match intended outcomes and there are no additional outcomes that lead to undesirable negative consequences.n Not just the lack of a correct outcome, but the potential negative consequencesn Some usability professionals believe that business consequences of usability

problems are outside their scope of responsibility

Exampleso Inputting data from a written form

n Some transcription errors are unimportant or will be subsequently identified and corrected

n Some could have negative consequences for the customer and/or the businesso Making a flight booking

n Some errors can cause inconvenience or major expense

7

Both objective and perceived success is

typically necessary

o Negative consequences if objective success is perceived as failuren You book a flight but don’t get a confirmation, so assume the booking

was not maden So you book again. Now you have two reservations!

o Perceiving objective failure as success can also have negative consequencesn You use a voting machine to make your vote. You think it was made,

but it is not counted because you did not press hard enough. n If you had realised it was not successful, you would have tried again.

8

But is “usability” the right term?

n In 9241-11, other concepts are special cases of usability:o Accessibility is effectiveness, efficiency and satisfaction for users with the

widest range of capabilitieso User experience is interpreted as user satisfactiono Negative consequences are part of effectiveness

n In ISO DIS 9241-220 (Processes for enabling, executing and assessing human-centred design within organizations) o Human-centred quality is the goal of human-centred design

n extent to which requirements relating to usability, accessibility, and user experience are achieved, and resulting negative consequences are minimized and positive outcomes are optimized

9

ISO DIS 9241-220 (replacing ISO TR 18529) [feedback needed]

ISO/IEC DIS 9241-220: Processes for enabling, executing and assessing human-centred design within organizationso Elaboration of ISO 9241-210 Human-centred design standardo Processes:

n Ensure enterprise focus on human-centred qualityn Enable human-centred design across projectsn Execute human-centred design within a project

o Plan and manage human-centred design for the projecto Context of use for each user groupo Establish the user and related stakeholder requirementso Design ergonomic solution that meets user requirementso User-centred evaluation

n Introduction, operation and end of life of a system

10

Identify the user needs (HCP.3.3.1)

Process purpose To provide the basis for deriving a comprehensive set of valid user requirements.

Process benefits The user needs relating to human-centred quality are identified.

Process outcomes a) The user needs are comprehensively described and documented.

There is sufficient information to derive user requirements b)that adequately satisfy the user needs.

Process activities (typical)

1. Analyse all data within the context of use description for user needs.

2. Consult relevant stakeholders to obtain a correct and complete set of user needs.

3. State each user need as an outcome to be achieved by the user.

4. Communicate the context of use and the identified needs to all relevant project stakeholders.

11

ISO 9241-220 applications

o ISO 9241-220 can be used for:n Implementing human-centred design as part of a systems development

processn Assessing an enterprise’s existing capability to carry out human-centred

design.n Improving the application of human-centred designn Providing a basis for education and trainingn Providing a basis for assessing personal competence

12

ISO/IEC 25066: Usability Evaluation Reports

Contents that should be included for different types of usability evaluation:o Inspection to identify usability defects and problems.o User observation

n Qualitative: Observing user behaviour to identify actual usability problems.

n Quantitative: Measuring user performance and responses to obtain data on effectiveness and efficiency.

o Obtaining subjective information from users including:n Qualitative: Problems, opinions and impressions given during or after a

usability evaluation.n Quantitative: Measures of user satisfaction or perception.

13

64 content items required or recommended in

the following categories

n Executive summary (if used)n Description of the object of evaluationn Purpose of the evaluationn Method

o Evaluators/participantso Tasks (if used in the evaluation)o Evaluation environment

n Procedureo Design of the evaluationo Data to be collected

n Resultso Data analysis

n Presentation of the resultsn Interpretation of results and recommendations

14

ISO/IEC 25022: Measurement of Quality in Use

Effectiveness Efficiency Satisfaction

Tasks completed Objectives achieved Errors in a task Tasks with errors Task error intensity

Task time Time efficiency Cost-effectiveness Productive time ratio Unnecessary actions Fatigue

Overall satisfaction Satisfaction with features Discretionary usage Feature utilisation Proportion of users complaining Proportion of user complaints

about a particular feature User trust User pleasure Physical comfort

15

Measures of negative consequences

Economic risk Health and safety risk Environmental risk

Return on investment (ROI) Time to achieve return on investment Business performance Benefits of IT Investment Service to customers Website visitors converted to

customers Revenue from each customer Errors with economic consequences

User health reporting frequency

User health and safety impact

Safety of people affected by use of the system

Environmental impact

16

Would you like to provide feedback on new

standards?

o UXPA provides liaison comments to ISO

[email protected]