Keeping Your Moderating Skills Sharp

Keeping Your Moderating Skills Sharp Susan Mercer Senior Experience Researcher [email protected] @susanamercer

The Importance of Neutrality

• Clients want objective answers

• Asking biased questions lead to biased answers

• Participants want to please us • Providing types of feedback can subtly

communicate our expectations • They are more likely to answer future questions

that way

2

The Reality of Moderation

• Observed 17 practitioners • Many usability professionals get sloppy:

• Ask leading questions • Participant: “I would share this article with friends.” • Moderator: “By printing it?”

• Ask closed-ended questions • Moderator: “Would you print this?”

• Focus on known problems for “ammunition” • Use biasing response phrases “Good job”, “Yes,

that’s a problem.”

Source: Norgaard & Hornbaek, 2006 3

What does this mean?

• Current practices are not consistent • Moderating is taught through mentorship

• bad habits can easily be perpetuated

• We need to pay attention to our moderation and remain neutral to capture unbiased data.

4

5 Rules of Great Moderating(A Quick Review)

5 Rules of Great Moderating

1. Know your goals 2. Stay neutral 3. Be responsible 4. Be confident 5. Be genuine

6

1. Know Your Goals

• What are you trying to achieve? • Formative • Summative

• What topics are highest priority?

These will influence • What to focus on when time runs short • How you fine-tune your moderating style

7



8

2. Stay Neutral

• Be Quiet! • Don’t insinuate they gave a wrong answer

✘ “Why did you do that?” ✘ “What made you think to click that link?”

• Don’t put ideas into their heads ✘ “Does entering your SSN make you feel uncomfortable?” “What do you think about entering your SSN?”

• Use their words • If they call the dashboard a “chart area”, then call it a

“chart area” • Keep your tone naturally curious, not derisive

• “What did you expect here?”

9

2. Stay Neutral

• Don’t start with a verb ✘ Did you see that? ✘ Was that difficult? ✘ Is this frustrating?

• Start with What, When, Where, Why, How What would you change to make this easier? When in this process would you be ready to buy? Where did you expect to find that information? Why did you say this was easy? How does that error message make you feel?

10

2. Stay Neutral

• Use neutral, “continuing” response phrases • Acknowledge that they gave feedback • Don’t imply whether it is good or bad

✘ “Oh”, “Hmm”, “Interesting” - Assessing ✘ “OK”, “Yeah”, “That’s good” - Agreeing “Uh huh”, “Mhmm”, “Tell me more” – Continuing

• Other tips to staying neutral?

Source: Boren & Ramey, 2000 11



12

3. Be Responsible

You have responsibilities to:

• Participant • Company • Future Users • Study Integrity

13

3. Be Responsible

Situations happen during testing. You may need to juggle your responsibilities.

What are your priorities?

• Study Integrity • Company • Future Users • Participant 14

What to do?

If… • Participant overshares personal information • The participant rants about the website and

calls it “crap”

15



16

4. Be Confident

• Project confidence • If you make a mistake, keep going • Don’t over-apologize • Pause if you need to collect your thoughts • “Fake it until you make it”

17



18

5. Be Genuine

• Don’t play the notes on the page…

…play the music

• Don’t just read the questions from the page…

…have a conversation

19



6. What else?

20

Keeping sharp

Keep Improving

1. Identify your improvement areas • Watch your videos • Have colleagues give you feedback

22

Keep Improving

2. Learn from others • Watch others moderate • Be a participant • Listen to talk radio interviews

23

Keep Improving

3. Practice skills in everyday life • People watch (observe) in public • Be quiet and listen in everyday conversations

24

Your Thoughts?

25

Thank You! (Full presentation is on Slideshare. Follow me on twitter for link)

Susan Mercer Senior Experience Researcher [email protected] @susanamercer

26

References / Photo Credits

Boren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication, 43 (3), 261-278.

Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman. Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd. Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3),

215-251. Krahmer, E. and Ummelen, N. (2004) Thinking about thinking aloud: A comparison of two

verbal protocols for usability testing. IEEE Transactions on Professional Communication, 47 (2), 105-117.

Norgaard, M. and Hornbaek, K. (2006) What do usability evaluators do in practice? An exploratory study of think-aloud testing. DIS 2006, 209-218.

Slide 4: Samantha Louras Photography Slide 5: flickr: torgeaux Slide 14: flickr: mkorcusa, flickr: Calsidyrose, flickr: marioanima, flickr: breity, flickr: Qole Pejorian Slides 27, 29 and 30: flickr: Raphael Quinet, flickr: acme, flickr: betsyweber, flickr: KirkOls Slide 32: flickr: bberburb Slide 34: flickr: Tulane Public Relations Slide 35: flickr: apdk, flickr: marktristan Slide 38: flickr: Raphael Quinet Slide 39: flickr: l-i-n-k Slide 40: flickr: loppear

27

Technology

Keeping Your Moderating Skills Sharp