Upload
brian-larson
View
34
Download
0
Embed Size (px)
DESCRIPTION
This sheet provides an overview and reading questions regarding think-aloud protocols.
Citation preview
© 2015 Brian N. Larson Topic: Think-‐aloud protocols Page 1
Topic: Think-‐aloud protocols This topic page includes introductory information, a list of readings, and questions to guide your reading and prepare you for class discussion; it may also include an individual or group assignment, which may or may not be graded.
Introduction Think-‐aloud protocols have been around for more than 40 years. They were intended as a way to gather information about the cognition of human agents dealing with ill-‐structured problems. Ill-‐structured problems contrast with well-‐structured problems; the latter type of problem is one where the beginning state, end states, and available moves are all well known or articulated up-‐front. Think of playing tic-‐tac-‐toe. Ill-‐structured problems lack some of that certainty; in fact, while doing an ill-‐structured problem, one might be uncertain about exactly what the problem is. In its most straight-‐forward form, the interviewer or facilitator asks the participant (sometimes called the “subject”) to work through a task and to talk all the while about what she is thinking while she is doing it. The participant is told to ignore the facilitator and not to ask the facilitator for assistance. The facilitator is present only to remind the participant to keep thinking aloud (e.g., by saying “Keep thinking aloud”). In theory, this should result in the participant sharing what is happening in her cognition from moment to moment without her “editing” it or attempting to shape it to make herself look good (addressing two concerns with qualitative interviewing). The major work on protocols is Ericsson and Simon (1993); that book is on reserve at the library and is also available electronically through the library. An earlier article by them (1980) discusses some of the main theoretical concerns with collecting “recollections,” which is what qualitative interviews (about which we have already read) do. That article is available on T-‐Square.
Readings for this topic • van den Haak, M., De Jong, M., & Jan Schellens, P. (2003). Retrospective vs. concurrent
think-‐aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology, 22(5), 339–351. http://doi.org/10.1080/0044929031000
• Optional: Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215–251.
Reading questions Read the following questions and comments and think about them as you read van den Haak et al.:
• The authors refer to “face validity.” Look this up at Wikipedia or elsewhere. Why do you think that protocol analysis has high face validity?
• They also refer to other kinds of validity, including predictive validity, congruent validity, and reliability. Think about what each of these terms means. Why do the authors and other usability folks care about these characteristics of their research methods?
• In what ways do the authors suggest that think-‐aloud protocols may be “reactive”? • Make sure you understand the difference between concurrent think-‐aloud (CTA) and
retrospective think-‐aloud (RTA). See p. 340 for definitions.
© 2015 Brian N. Larson Topic: Think-‐aloud protocols Page 2
• At p. 341, the authors discuss reasons that RTA might be better than CTA, and reasons why it might not be. Make sure you understand the widely accepted advantages and disadvantages of RTA. Note that the authors propose to test some of these views with their study.
• Note the organization of the authors’ presentation of their own study. After their lengthy introduction, they carefully describe their methods, present the results they obtained, and then discuss the implications of those results. This is what is sometimes called the IMRaD pattern: Introduction, Method, Results, and Discussion. You may want to consider it for your own projects in this class.
• On page 344, the instructions for the participant for the study in this article are quoted. Locate and identify them. You may want to use a similar instruction if you choose to use CTA for your projects in this class.
• Note how the authors organized the problems their participants had/found into categories: Layout problems, data entry problems, comprehensiveness problems, feedback problems. Think about your own projects in this class: What categories of problems might you anticipate users of the communications you are examining to have?
• When the authors (p. 349) say that CTA resulted in identifying significantly more problems “detected by means of observation only,” what do they mean? Contrast the fashion in which RTA proved more useful.
• The authors’ overall conclusion is a bit perplexing (p. 350): “the results of this study indicate that concurrent and retrospective think-‐aloud protocols can be regarded as equivalent, but clearly different evaluation methods. A strong, and new argument in favour of RTA protocols is that they may be less susceptible to the influence of task difficulty, both in terms of reactivity and in terms of completeness of the verbalisations.” What to they mean by “regarded as equivalent, but clearly different evaluation methods”? Can you rephrase that statement to make it sound less oxymoronic?
Classroom activities for this topic • Observe a think-‐aloud protocol in action.
Assignments for this topic • Post notes regarding the in-‐class think-‐aloud protocol to your group’s forum before the
next class after the TAP.
Works cited Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis : verbal reports as data (Rev. ed.). MIT Press. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215–251.