Upload
sharyl-daniel
View
227
Download
0
Embed Size (px)
Citation preview
CS2003 Usability Engineering
Usability Evaluation
Dr Steve Love
Slide 2
Overview
• Continuing theme of usability evaluation techniques
• Think aloud protocol• Observation• Interviews• Diary studies• Laboratory setting• Field studies• Summary• References• Further reading
Think aloud protocol• The idea of using verbal protocols as a form of usability
evaluation was put forward by Ericsson and Simon (1985).
• Potentially provides information on what the user is actually thinking about when they are interacting with a service.
• Get feedback on users’ feelings at particular points in the interaction, especially if the system does something they didn’t expect or they ended up somewhere that they didn’t want to be.
• May find out that the navigation structure is not as clear-cut as you originally thought it was.
• The key to success in using this method is practice. You must give your participants practice in using the method before data collection begins.
• How many windows do you have in your house? Slide 3
Observation• In an observational study, the emphasis is on observing
the naturally occurring behaviour of people (with or without them knowing about it).
• In an HCI context, observational techniques have been used as part of the data collection process in the lab.
• For example, you can watch people’s reactions as they interact with a new smartphone interface, observing where they have difficulties or which parts of the interface they do not like.
• Two main types of observation:
• controlled observation: users are in a controlled lab environment (see example above)
• naturalistic observation : observing behaviour as it naturally occurs in its normal context of use (e.g. mobile phone use in public places). Slide 4
Observation techniques• Audiotape recording
• your own verbal record (recorded onto audiotape) of the behaviour of the people you are observing as it occurs.
• have to make sure that your recording was unobtrusive and did not distract users from carrying out the tasks that you had asked them to do.
• people may become uneasy about you looking at them while you are speaking into a small tape recorder at the same time.
• Videotape recording• could provide you with a very rich source of data• observing behaviour and reactions in real time• however, how would you feel seeing someone
pointing a video camera at you in a public place? Slide 5
Constructing an observational coding form• be clear about what it is you are interested in observing
(e.g. mobile phone use in public places.)• identify what you think are the dimensions or
categories of behaviour that you expect to observe.• test it out to see if it is covering the main behaviours
that you have highlighted• this process will also allow you to add new behaviour
categories that you had not previously thought • Another way to test the reliability of your coding form is
to get a few of your friends to use it (if you persuade them to help you) and record their observations and see how it compares with your own
• Obviously you would all have to observe the same social scene to do this, at the same time.
Slide 6
Interviews• Questioning an individual can be a quick way of getting information from them on usability issues, requirements gathering and a whole
host of mobile HCI research interests.
• Interviews can be used in the lab at the end of a formal testing session to help find out some more information on what participants thought about the experiment that they had just taken part in.
• It can also be used to investigate how participants’ interview answers compare to their behaviour which you will have observed during the experiment and also, the performance data that you may have logged during the experiment (e.g. timings for completing each task you asked them to undertake as part of the study).
• Interviews come in various formats and the following are probably the most commonly used in HCI to illicit information from
participants.Slide 7
Interview formats• Informal interviews• totally unstructured with the aim of the• interview being to try and collect as much data as
possible.• Semi-structured interviews• you have a set number of questions which are presented
in the same order• These types of questions could come at the end of a
Likert-type questionnaire• Structured interviews• all the questions are fixed and set beforehand• (e.g. how many text messages do you send per week?)• There are a set number of responses that respondents
can choose from (e.g., ‘please specify how many calls• you make per week using your mobile phone: 0–10, 11–
20, 21–30, 31–40, other ...) Slide 8
Diary studies• Ask participants to keep a personal record
of their thoughts and feelings about interacting with a particular piece of technology; for example, over a certain period of time
• This can provide a rich source of textual data for you to analyse.
• The key thing to remember here is that you have to give participants in your diary study specific instructions of what you want them to describe and write about
Slide 9
Exercise• Develop a set of guidelines for
participants taking part in a 4 week diary study looking into people’s use of smartphones
• What are the important aspects of user behaviour that you want participants to record in their diaries?
• Work in groups for the next 10 minutes
Slide 10
Lab settings• The traditional approach to usability
evaluation• You can control the environment in order
to test your specific hypotheses• Allows you to see how they react to
using• a new piece of mobile technology
without anything distracting them from the task in hand.
• Are there any problems with this approach?
Slide 11
Field studies• does not give you control over all your variables
and settings in the same way as a laboratory experiment
• it can be a very effective form of research method for mobile HCI research
• mobile devices and applications are used in dynamic social environments (e.g. in a café, on the train).
• it is important for researchers (and designers) to assess how context of use affects people’s performance and attitude towards using mobile devices and applications. Slide 12
Summary• There are a variety of usability
evaluation methods
• It is important you choose the method that will be the most appropriate for the aims of your usability study
• A triangulation approach is often the best way to proceed
Slide 13
Questions?
Slide 14
References and further reading
ReferenceEricsson, K.A. and Simon, H.A. (1985)
Protocol analysis: verbal reports as data. Cambridge MA: MIT Press.
Further readingFaulkner, X. (2000),Usability Engineering, Chapter 6:
Usability Evaluation
Palen,L. and Salzman, M (2002) voice-mail diary studies for naturalistic data capture under mobile conditions. Proceedings of CSCW 2002, New Orleans, November 16-29, pp87-95
Slide 15