1

Lecture 2Lecture 2

Concepts of usabilityConcepts of usabilityUsability testingUsability testing

How would you evaluate your How would you evaluate your knowledge levelsknowledge levels

1. My understanding of usability concept

Very bad Very goodOK

2

How would you evaluate your How would you evaluate your knowledge levelsknowledge levels

2. My knowledge on usability testing

Very bad Very goodOK

What is usability?What is usability?How to measure it?How to measure it?

3

Usability conceptUsability concept

ISO 9241:effectiveness (% of goal achieved)

+ efficiency (time to complete a task, or the error rate, or the amount of effort)

+ satisfaction (subjective rating scale)= Usability

What is context?What is context?

SystemSystemStakeholder analysis, role and benefitStakeholder analysis, role and benefitUserUser

Experience, knowledge and skillExperience, knowledge and skillPersonal attributesPersonal attributes

TaskTaskEnvironmentEnvironment

Technical, physical, organizationalTechnical, physical, organizational

4

RelationshipRelationship

Work environment

user

Computer system

software

Internal quality

External qualityQuality in use

Quality in useQuality in use

The capability of the software product The capability of the software product to enable specialed users to achieve to enable specialed users to achieve specialed goals with effectiveness, specialed goals with effectiveness, productivity, safety and satisfaction in productivity, safety and satisfaction in specialed contexts of use.specialed contexts of use.

ISO/IEC FDIS 9126ISO/IEC FDIS 9126--1 (2000)1 (2000)

5

Quality in useQuality in use

Effectiveness, productivity, safety, satisfaction

FunctionalityAccuracySuitability

Interoperabilitysecurity

ReliabilityMaturity

Fault toleranceRecoverability

availability

UsabilityUnderstandability

LearnabilityOperability

attractiveness

EfficiencyTime behaviour

Resourceutilisation

maintainabilityAnalysability

Changability, stability, testability

PortabilityAdaptability, installability

Co-existence, replaceability

SatisfactionSatisfaction

Satisfation: Satisfation: the level of comfortthe level of comfort that that the users feel when using a product the users feel when using a product and how acceptable the product is to and how acceptable the product is to users as a means of achieving their users as a means of achieving their goalsgoals

Subjective aspectSubjective aspectDifficult to measureDifficult to measureOften be strongly correlated with Often be strongly correlated with effectiveness and efficiencyeffectiveness and efficiency

6

Satisfaction levelsSatisfaction levels

FunctionalityFulfill the appropriate functionality

The context and the environment in which it will be used

UsabilityEfficiency, effectiveness, satisfaction

PleasurePeople’s emotion

Value, hope, taste, fear, etc

FunctionalityFulfill the appropriate functionality

The context and the environment in which it will be used

UsabilityEfficiency, effectiveness, satisfaction

PleasurePeople’s emotion

Value, hope, taste, fear, etc

Emotional feeling of pleasure Emotional feeling of pleasure

PropertiesPropertiesFeatures/functionalityFeatures/functionalityUsabilityUsabilityAestheticsAestheticsEffective performanceEffective performanceReliabilityReliabilityHighHigh--status productstatus productConvenienceConvenienceSizeSizeBrand nameBrand name

EmotionsEmotionsSecurity/comfort Security/comfort Confidence Confidence Pride Pride ExcitementExcitementSatisfactionSatisfactionEntertainmentEntertainmentFreedomFreedom

Sentiment/nostalgiaSentiment/nostalgia

7

Usability testingUsability testing

Usability testing determines whether the users can Usability testing determines whether the users can find and use the features in the amount of the time find and use the features in the amount of the time and effort they willing to expend searching.and effort they willing to expend searching.

Primary goal Primary goal –– improve the usabilityimprove the usabilityReal usersReal usersDo the real taskDo the real taskObservationObservationAnalysisAnalysis

Usability testingUsability testing

Usability is not: Usability is not: Quality assuranceQuality assuranceZero defectsZero defectsUtility of design featureUtility of design featureIntrinsic in productsIntrinsic in products

8

Making a listMaking a list

Usability testingUsability testing Quality assuranceQuality assurance

Lab. Test vs. field testsLab. Test vs. field tests

Lab testLab testWell designedWell designedNot choiceNot choiceaim is to test aim is to test whether a product whether a product meets certain meets certain predefined usability predefined usability criteriacriteria

Field testField testacceptability of a acceptability of a productproduct

however the benefits of having both types of test dataoutweigh the negative factors

9

Usability test methodsUsability test methods

Focus on Lab. testsFocus on Lab. tests

Usability testing lab.Usability testing lab.

10

Methods overviewMethods overview

Inspection methodInspection methodUser testingUser testing

11

Methods overviewMethods overview

Inspection methods: need to be carried out Inspection methods: need to be carried out by cognitive expert. They are good in by cognitive expert. They are good in finding problems finding problems

Heuristic evaluation: better predictorHeuristic evaluation: better predictorCognitive walkthrough: finding endCognitive walkthrough: finding end--user user problems towards easeproblems towards ease--of useof use

Actual user testing is still very importantActual user testing is still very importantUser testing and inspection methods do User testing and inspection methods do not have a high degree of overlapping not have a high degree of overlapping findingsfindings

Selecting methodsSelecting methods

Not to rely on any one method for Not to rely on any one method for determining the usability of product, determining the usability of product, but user a variety of methods.but user a variety of methods.

12

Learn about userLearn about user

Self definition (job, tasks, tools, and mental Self definition (job, tasks, tools, and mental models)models)individual (personal, physical and cultural individual (personal, physical and cultural characteristics, as well as motivation)characteristics, as well as motivation)stage of use (How they use products over time stage of use (How they use products over time and the choices they make about the levels of and the choices they make about the levels of expertise they want or need to achieve expertise they want or need to achieve

Characteristics of user populationsCharacteristics of user populationsNovicesNovicesAdvanced beginnersAdvanced beginnersCompetent performersCompetent performersExpert performersExpert performers

User and task analysisUser and task analysis

What usersWhat users’’ goals aregoals areWhat processes they use to achieve their What processes they use to achieve their goalsgoalsWhat characteristics shape the way they What characteristics shape the way they perform tasks and achieve goalsperform tasks and achieve goalsAffect from previous experienceAffect from previous experienceWhat is the most important to user or what What is the most important to user or what is most helpful to them in performing tasksis most helpful to them in performing tasksWhat impact the environment has on their What impact the environment has on their ability to perform tasks.ability to perform tasks.

13

UserUser’’s goals and taskss goals and tasks

UserUser’’s tasks are not necessary users tasks are not necessary user’’s s goalsgoalsIf the task interferes with the goal, users If the task interferes with the goal, users become frustrated.become frustrated.

the goal stackthe goal stackFalse goals (easily achieved by programmers)False goals (easily achieved by programmers)Corporate goals (companyCorporate goals (company’’s benefit)s benefit)Practical goals (the bridge between the companyPractical goals (the bridge between the company’’s s goals and the usergoals and the user’’s goals)s goals)Personal goals (true for everyone)Personal goals (true for everyone)

Site visitSite visit

Follow a user for a dayFollow a user for a dayQuestion user while they workQuestion user while they workTalking after taskTalking after taskThink aloudThink aloudCritical incident techniqueCritical incident techniqueScenarioScenarioCued recallCued recall

14

Constraints of site visitConstraints of site visit

Time constraintsTime constraintsBudget constraintsBudget constraintsPolitical and ethical constraintsPolitical and ethical constraints

After a site visitAfter a site visit

Report:Report:User analysisUser analysisTask analysisTask analysisEnvironmental analysisEnvironmental analysisMethodologyMethodologyRecommendationRecommendation

Personas Personas DailyDaily--use scenarios (often used)use scenarios (often used)NecessaryNecessary--use scenarios (infrequently used)use scenarios (infrequently used)

User learningUser learning

15

Iterative testing: Paper prototypeIterative testing: Paper prototype

goodgoodCheap, easy, can Cheap, easy, can do it oftendo it oftenTry new ideas and Try new ideas and conceptsconceptsTeam activitiesTeam activitiesEnhanced Enhanced communication communication among team among team membersmembersCommon baselineCommon baseline

BadBadComplex productsComplex productsCollect response Collect response timetimeinteraction qualityinteraction qualityTeam Team understandingunderstandingLonger time for Longer time for testingtesting

Purpose of usability testingPurpose of usability testing

Different stages of product Different stages of product development, The test is different.development, The test is different.

16

Planning for usability testingPlanning for usability testing

TeamTeamProduct Product GoalsGoalsMeasurementsMeasurementsUsersUsersTasksTasksResults analysisResults analysisTest planTest plan

LimitationsLimitations

TimeTimeMoneyMoneyTesting conditionsTesting conditionsWho wants to know the resultsWho wants to know the resultsWhat is best learned from usability What is best learned from usability testing?testing?

17

PlanningPlanning

Initial team meeting Planning meeting Results meeting

Pilot testwalkthroughRecruiting participants

Completion of test plan Approval of the test plan

Testing conditionTesting condition

Where?Where?Field?Field?Lab?Lab?Remote?Remote?

(this is depending on the product (this is depending on the product development stages and the location of development stages and the location of the users)the users)

SetSet--up for testingup for testing

18

What can we learn from usability What can we learn from usability test?test?

Usability test can not answer many Usability test can not answer many questions, such as color preference, questions, such as color preference, font size, use of particular graphics, font size, use of particular graphics, array array oof products available, since the f products available, since the answers to such questions cannot be answers to such questions cannot be obtain from 5 to 8 subjects.obtain from 5 to 8 subjects.

Preparing for usability testingPreparing for usability testing

Screening questionnaires:Screening questionnaires:The background information of users The background information of users towards the use of the product.towards the use of the product.These questions shall be sent to the These questions shall be sent to the users and have the answer back before users and have the answer back before the test.the test.

PrePre--test questionnairestest questionnairesThe questions answered before the test The questions answered before the test started.started.

19

Preparing for usability testingPreparing for usability testing

Creating scenariosCreating scenariosScenarios are description of typical Scenarios are description of typical tasks you want the participants to tasks you want the participants to perform.perform.Avoid creating steps or instructions, Avoid creating steps or instructions, focus on processfocus on processAvoid using product language, use Avoid using product language, use useruser’’s languages language

Objective measurementObjective measurement

Try to obtain if you canTry to obtain if you can

20

PostPost--task and posttask and post--test test questionnairesquestionnaires

Avoid bias responsesAvoid bias responsesNeutral questionsNeutral questionsCan be qualitative and quantitative Can be qualitative and quantitative responsesresponses

Roles of the team membersRoles of the team members

Based on their expertiseBased on their expertiseConsistentConsistentFacilitator: give user direct instructionFacilitator: give user direct instructionCamera/video operatorCamera/video operatorLogger: capture the measurementsLogger: capture the measurements

21

Important factorsImportant factors

Avoid biasAvoid biasMaintain consistencyMaintain consistency

Walkthrough Walkthrough

It is a chance to test the equipment, It is a chance to test the equipment, the scenarios, the questionnaires, the scenarios, the questionnaires, and the checklists for the team and the checklists for the team members.members.Team memberTeam member’’s practice s practice Intend to find the flaws in the planIntend to find the flaws in the plan

22

Pilot testPilot test

Test of the testTest of the testFind out practical problemsFind out practical problemsA few days earlier than real testA few days earlier than real test

Conducting the UConducting the U-- testtest

Team should avoid contacting with Team should avoid contacting with usersusersBriefing:Briefing:

Any misconception can be clarified.Any misconception can be clarified.Fill the forms.Fill the forms.Using videotape (?)Using videotape (?)UnbiasUnbias

People try to please the othersPeople try to please the others

23

Handling questionsHandling questions

If, user ask a question, donIf, user ask a question, don’’t answer it t answer it directlydirectly

““did I do that right?did I do that right?””““what do you think should happened?what do you think should happened?””or or ““tell me what makes you wonder tell me what makes you wonder whether you did it right?whether you did it right?””

Try to led the user shear his Try to led the user shear his experience with you.experience with you.

Intervene Intervene

Only when you see the user get lostOnly when you see the user get lostCareful! Careful!

Questions like: Questions like: ““what are you trying to what are you trying to do now?do now?”” or or ““what do you think the what do you think the system is doing now?system is doing now?””

HelpHelpNot to give the direction of how to solve Not to give the direction of how to solve the problem.the problem.Provide problem solving strategies.Provide problem solving strategies.

24

Collect dataCollect data

Category the dataCategory the dataIf two users have a problems, the If two users have a problems, the problem has to be taking careproblem has to be taking careIf only one single user has that If only one single user has that problem, depend how many users are problem, depend how many users are tested!tested!Find the explanation of the special Find the explanation of the special case.case.Conflict data? Believe what you see!Conflict data? Believe what you see!

AnalysisAnalysis

Determining the causes of problems.Determining the causes of problems.Make sure that one solution doesnMake sure that one solution doesn’’t t create some other problems.create some other problems.Scope and severity of problemsScope and severity of problems

25

Usability reportUsability report

Who is going to read the report?Who is going to read the report?

Standard form for industry Standard form for industry Usability report (1)Usability report (1)

Title PageTitle PageExecutive SummaryExecutive SummaryIntroductionIntroductionFull Product DescriptionFull Product DescriptionTest ObjectivesTest Objectives

26

Standard form for industry Standard form for industry Usability report (2)Usability report (2)

MethodMethodParticipantsParticipantsContext of Product Use in the TestContext of Product Use in the TestTasksTasksTest FacilityTest FacilityParticipantParticipant’’s Computing Environments Computing EnvironmentTest Administrator ToolsTest Administrator ToolsExperimental DesignExperimental Design

ProcedureProcedureParticipant General InstructionsParticipant General InstructionsParticipant Task InstructionsParticipant Task Instructions

Standard form for industry Standard form for industry Usability report (3)Usability report (3)

Usability MetricsUsability MetricsEffectivenessEffectivenessEfficiencyEfficiencySatisfactionSatisfaction

ResultsResultsData AnalysisData AnalysisData ScoringData ScoringData ReductionData ReductionStatistical AnalysisStatistical AnalysisPresentation of the ResultsPresentation of the ResultsPerformance ResultsPerformance ResultsSatisfaction ResultsSatisfaction Results

27

Standard form for industry Standard form for industry Usability report (4)Usability report (4)

AppendicesAppendicesReferencesReferences

ExerciseExercise

A short description of tasksA short description of tasksPlanning of usability test in detailPlanning of usability test in detailThe structure of usability reportThe structure of usability report

28

Usability test plan (1)Usability test plan (1)

What kind of knowledge they you need to What kind of knowledge they you need to have before you can carry out the usability have before you can carry out the usability tests of the product? Where and how can you tests of the product? Where and how can you get the knowledge? Do you need any skill get the knowledge? Do you need any skill training as well? You need to specify the training as well? You need to specify the discussion as detail as possible.discussion as detail as possible.What kind of team members do you like to What kind of team members do you like to have when you are asked to organize the have when you are asked to organize the usability tests?usability tests?Would you prefer to carry the test in the lab, Would you prefer to carry the test in the lab, or in the field, or both? Why?or in the field, or both? Why?

Usability test plan (2)Usability test plan (2)

What kind of preparations do you need to do What kind of preparations do you need to do before you can design the tests?before you can design the tests?What can be the best schedule of the plan? What can be the best schedule of the plan? Who shall be your tests subjects? How many of Who shall be your tests subjects? How many of them in each group? How can you find them?them in each group? How can you find them?How many tests you plan to carry out? How How many tests you plan to carry out? How would you design your testing scenarios would you design your testing scenarios (making as detail as possible of your testing (making as detail as possible of your testing scenario)? What kinds of design features and scenario)? What kinds of design features and functions you are planning to test?functions you are planning to test?

29

Usability test plan (3)Usability test plan (3)

Prepare your testing tools. What kind of tools Prepare your testing tools. What kind of tools you need? How shall they look like? What you you need? How shall they look like? What you plan to measures? Please make a detail design plan to measures? Please make a detail design of one lab test as detail as possible.of one lab test as detail as possible.Data analysis. Based on your plan of tests and Data analysis. Based on your plan of tests and your plan of measurements, can you make the your plan of measurements, can you make the analysis of the results? What kind of analysis of the results? What kind of conclusions you are expecting from the data conclusions you are expecting from the data analysis.analysis.Preparing you usability testing report. Making Preparing you usability testing report. Making the structure of how your usability testing report the structure of how your usability testing report shall be looked like.shall be looked like.

Exercise Exercise

After you finish the reportAfter you finish the reportRead the document about Read the document about ””7 common 7 common mistakesmistakes””A checklist of usability reportA checklist of usability report