HCI460: Week 5 LectureHCI460: Week 5 LectureOctober 7, 2009October 7, 2009

2© April 21, 2023

A few more words about project 1b (Expert Evaluation Report)

Plan for next Wednesday

Test Plan: Common feedback themes

Moderating usability sessions

Observing sessions and note taking

Guest lecture:– Speaker: Ed Israelski, Human Factors Manager, Abbott– Title: “Usability Methods in Healthcare”

Time for questions, feedback on your Moderator’s Guide and Screener, group work

Outline

3© April 21, 2023

A Few More Words About Project 1b

Points (out of 15)

4

A Few More Words About Project 1b

© April 21, 2023

DLin-class

5© April 21, 2023

Everyone can resubmit their Project 1b.– No grade change– No deadline– Feedback will be provided verbally during office hours

We will look at how you have improved from the first report to the second and third.

Can I Resubmit?A Few More Words About Project 1b

6© April 21, 2023

Plan for Next Wednesday

7© April 21, 2023

Samsung TV remote control TRUEtrack glucose meter Policy declarations page for an auto insurance company Flip Video Mino HD Franklin Spelling Ace Tenori-On digital musical instrument (synthesizer) Livescribe Pulse Smartpen Corner Bakery printable pdf menu Slacker Radio application on the BlackBerry Storm Google Maps Brookstone AM/FM radio alarm clock

Formative Usability Testing of:Plan for Next Wednesday

8© April 21, 2023

Address:– 500 North Michigan Avenue– Floor 16– Suite 1610 – User Centric, Inc.

Directions:– http://www.usercentric.com/

directions_chicago.html

LocationPlan for Next Wednesday

9© April 21, 2023

SchedulePlan for Next Wednesday

TimeTest Room A

Observation Room ATest Room B

Observation Room B

5:45 – 6:15 Group 9 (TRUEtrack Glucose Meter)

Group 10 (Flip Video Mino HD)

Group 3 (Franklin Spelling Ace)

Group 6 (Tenori-On Synthesizer)

Group 5 (Livescribe Pulse Smartpen)

Group 8 (Slacker Radio Application on the

BlackBerry Storm)

6:15 – 6:45

6:45 – 7:15

7:30 – 8:00 Group 7 (Policy Declarations Page for an Auto

Insurance Company)

Group 11 (Google Maps)

Group 4 (Corner Bakery pdf Menu)

Group 1 (Samsung TV Remote Control)

Group 2 (Brookstone AM/FM Radio Alarm

Clock)

DL students – optional

8:00 – 8:30

8:30 – 9:00

10© April 21, 2023

Group 1: Dietz, Dulski, Dzienisowicz, Remington Group 2: Birdseye, Dash, Gamboa, Welense Group 3: Elliott, Petlick Group 4: Berberian, Devlin, Shah, Sienkiewicz Group 5: Albarracin, Ginez, Schwarz Group 6: Epps, Jones, Lund Group 7: Barbera, Schulte, Schwantes, Young Group 8: Canady, Doshi, Roberts Group 9: Garcia, Goldberg, Haines, Wickenkamp Group 10: Diemer, Komosa, Ranguelov, Young Group 11: Cheng, Freeman, Taylor

GroupsPlan for Next Wednesday

11© April 21, 2023

Groups 3, 5, 6, 8, 9 and 10 should arrive before 5:45pm.– If not all team members can arrive early, whoever can come in

early should bring the product you are testing.

Groups 1, 2, 4, 7 and 11 should arrive at 7:15pm.

Your early arrival will help us make sure that the lab setup works with your product.

Arrival TimesPlan for Next Wednesday

12© April 21, 2023

There will be only one test session per group.– You should conduct the rest of your sessions (min 5 total) on

your own.

Limit your session to 20-25 minutes.– Typical session lengths: 30, 45, 60, 90, 120 minutes but class

time is limited, so we will have shorter sessions.– You can conduct longer sessions on your own if you’d like.

Participants will be provided.

The moderator will be selected randomly from each group, so you all should be prepared to moderate.– Do a dry run before Wed session to make sure you can fit the

session into a 20-25-minute time slot.

Don’t forget to bring what you are testing.

Other Important DetailsPlan for Next Wednesday

13© April 21, 2023

You will receive feedback from:– Gavin or Aga– Your participant– Your observers (other groups observing your test session)

Feedback will be provided on:– The moderation– The procedure

Each observer will fill out simple feedback sheets:– 1 – 3 things that you think the moderator did well – 1 – 3 things that the moderator should improve on– 1 – 3 good things about the procedure– 1 – 3 ways in which the procedure could be improved

All feedback will be compiled and posted on COL by EOD 17th.

FeedbackPlan for Next Wednesday

14© April 21, 2023

Test Plan: Common Feedback Themes

15© April 21, 2023

Narrow down your scope.– Keep in mind session length– Too many objectives / tasks– Comparing products instead of testing a single product

• Good idea but not for formative testing with a few very short sessions.

ScopeTest Plans: Common Themes

16© April 21, 2023

Provide a business objective.– Not critical but very helpful– In real life, you should get business objectives from stakeholders– Why are we testing this product?

• To lower the return rate?• To decrease the volume of calls to the call center?

Business ObjectiveTest Plans: Common Themes

17© April 21, 2023

Objectives and tasks/questions must be in perfect alignment.– Objectives without tasks/questions can lead to disappointed

stakeholders.– Tasks/questions not addressing any objectives are a waste of

time. • Did you forget to list an important objective?

Objectives and TasksTest Plans: Common Themes

18© April 21, 2023

What are stimuli?– Stimuli = artifacts = stuff that will be tested– E.g., XYZ alarm clock, XYZ glucose meter, XYZ website

Use of quantitative measures (e.g., time on task, ratings)– Appropriate in summative testing– Why and how should we use quantitative measures in formative

testing?

Stimuli and MeasuresTest Plans: Common Themes

19© April 21, 2023

Moderating Usability Sessions

20© April 21, 2023

Who has moderated?– Explain insights/lessons learned

What makes a good moderator?

ExperienceModerating Usability Sessions

21© April 21, 2023

One basic tenet of usability testing is that it is virtually impossible to remain objective when usability testing your own product

Separate design and test teams

Reality is that some testing is better than no testing at all

Need for ObjectivityModerating Usability Sessions

22© April 21, 2023

Knowledgeable in UCD

Quick learner

Builds a rapport with participants / good communicator

Excellent memory

Good listener

Comfortable with ambiguity

Flexibility

Long attention span

Empathic “people person”

“Big picture” thinker

Characteristics of a Good Moderator (Rubin)

Moderating Usability Sessions

23© April 21, 2023

Reading the guide vs. just reading the guide

Biggest fear (study)

Biggest fear (formative session)

Characteristics of a Good Moderator (Lew/Bojko)

Moderating Usability Sessions

24© April 21, 2023

State task and participant does task– Discussions are limited

Think aloud protocol– Pros / cons

Retrospective review – Pros / cons

Depends on TechniqueModerating Usability Sessions

25© April 21, 2023

Leading…instead, enable

Being overly consumed with data collection

Acting too knowledgeable

Being rigidly adhering to moderator’s guide

Not relating well to every participant

Jumping to conclusions

Moderators Should Avoid… (Rubin)

Moderating Usability Sessions

26© April 21, 2023

When in doubt, read

The power of silence (simply pause)

What it is like to cut video clips

When trying to nail a point, avoid questions with yes/no, instead ask for elaboration

Manage time, but also take your time (yeah, I know)

Don’t jumping to conclusions, BUT be aware of trends– Especially when you are the single moderator

Moderators Should Remember… (Lew & Bojko)

Moderating Usability Sessions

27© April 21, 2023

How many sessions can be run in a day?– 60 min sessions?– 90 min sessions?– 120 min sessions?

Consider sample size of N=12– N=20– N=20, but with a between groups design (N=40)

Implications– Parallel sessions– Ramp up

Simultaneous Testing SessionsModerating Usability Sessions

28© April 21, 2023

Pros and cons

Notes handed back

Walk outside

Ear buds

IM

Do not get flustered– Okay to wait a moment

Communication During SessionModerating Usability Sessions

29© April 21, 2023

Confused

Bad– Define bad

Lying

Drunk

What if the Participant is…Moderating Usability Sessions

30© April 21, 2023

Somehow it looks worse after the luggage handlers touched it…– It is 7:45am. The first session starts at 9 am.

What if Your Mobile Usability Lab…Moderating Usability Sessions

31© April 21, 2023

Cancel study?– Cost / timeline

Ready the day before?– Technique

What if the Prototype is Not Ready?Moderating Usability Sessions

32© April 21, 2023

Observing Sessions and Note Taking

33© April 21, 2023

Two-person test teams vs. one-person test teams– Single session– One moderator and one note taker– Pros/Cons

Switching considerations– One and only one moderator?

Team Size Observing Sessions and Note Taking

34© April 21, 2023

Manage observation room– Redirect– Trends vs. outliers

• Note trends on a whiteboard– Assess politics– Achievable recommendations

Managing the Observation RoomObserving Sessions and Note Taking

35© April 21, 2023

Screener and Moderator’s Guide

36© April 21, 2023

Both the Screener and Moderator’s Guide are due on Friday 11:59 pm Central Time for in-class students (Tuesday for DL students).

Again, we will provide you with feedback.

Grade?– Project 2 will be graded after the final report is turned in.– As appendices, you should turn in:

• Test Plan • Moderator’s Guide improved based on our feedback• Screener

Questions:– How will the screener be used?– ….?

Questions About the Mod Guide & ScreenerScreener and Moderator’s Guide